prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How were the human judgements assembled?
Context 1: Introduction
The use of RNNs in the field of Statistical Machine Translation (SMT) has revolutionised the approaches to automated translation  As opposed to traditional shallow SMT models, which require a lot of memory to run, these neural translation models require only a small fraction of memory used, about 5% BIBREF0   Also, neural translation models are optimized such that every module is trained to jointly improve translation quality  With that being said, one of the main downsides of neural translation models is the heavy corpus requirement in order to ensure learning of deeper contexts  This is where the application of these encoder decoder architectures in translation to and/or from morphologically rich languages takes a severe hit 
For any language pair, the efficiency of an MT system depends on two major factors: the availability and size of parallel corpus used for training and the syntactic divergence between the two languages i e morphological richness, word order differences, grammatical structure etc  BIBREF0   The main differences between the languages stem from the fact that languages similar to English are predominantly fusional languages whereas many of the morphologically rich languages are agglutinative in nature  The nature of morphologically rich languages being structurally and semantically discordant from languages like English adds to the difficulty of SMT involving such languages 
In morphologically rich languages, any suffix can be added to any verb or noun to simply mean one specific thing about that particular word that the suffix commonly represents (agglutination)  This means that there exists a lot of inflectional forms of the same noun and verb base words, conveying similar notions  For example, in Tamil, there are at least 30,000 inflectional forms of any given verb and about 5,000 forms of inflectional forms for any noun  The merged words carry information about part of speech (POS) tags, tense, plurality and so forth that are important for analyzing text for Machine Translation (MT)  Not only are these hidden meanings not captured, the corresponding root words are trained as different units, thereby increasing the complexity of developing such MT systems BIBREF1  
To add to the complexities of being a morphologically rich language, there are several factors unique to Tamil that make translation very difficult  The availability of parallel corpus for Tamil is very scarce  Most of the other models in the field of English–Tamil MT have made use of their own translation corpora that were manually created for the purposes of research  Most of these corpora are not available online for use 
Another issue specific to Tamil is the addition of suffix characters included to the words in the language for smoothness in pronunciation  These characters are of so many different types; there is a unique suffix for each and every consonant in the language  These suffixes degrade performance of MT because the same words with different such pronounciation-based suffixes will be taken as different words in training 
Also to take into consideration is the existence of two different forms of the language being used  Traditionally defined Tamil and its pronunciations aren't acoustically pleasing to use  There's no linguistic flow between syllables and its usage in verbal communication is time consuming  Therefore, there exists two forms of the language, the written form, rigid in structure and syntax, and the spoken form, in which the flow and pace of the language is given priority over syntax and correctness of spelling  This divide leads to the corpus having 2 different versions of the language that increase the vocabulary even with the same words  This can be evidently seen in the corpus between the sentences used in the Bible, which is in traditional Tamil and sentences from movie subtitles, being in spoken Tamil format 
To account for such difficulties, a trade-off between domain specificity and size of the corpus is integral in building an English–Tamil neural MT system 
Corpus
The corpus selected for this experiment was a combination of different corpora from various domains  The major part of the corpus was made up by the EnTam v2 corpus BIBREF2   This corpus contained sentences taken from parallel news articles, English and Tamil bible corpus and movie subtitles  It also comprised of a tourism corpus that was obtained from TDIL (Technology Development for Indian Languages) and a corpus created from Tamil novels and short stories from AU-KBC, Anna university  The complete corpus consisted of 197,792 sentences  Fig  FIGREF20 shows the skinny shift and heatmap representations of the relativity between the sentences in terms of their sentence lengths 
An extra monolingual Tamil corpus, collated from various online sources was used for the word2vec embedding of the Tamil target language to enhance the richness of context of the word vectors  It was also used to create the language model for the phrase-based SMT model  This corpus contained 567,772 sentences and was self-collected by combining hundreds of ancient Tamil scriptures, novels and poems by accessing the websites of popular online ebook libraries in Python using the urllib package  Since the sources had Tamil text in different encodings, the encoding scheme was standardized to be UTF-8 for the entirety of the monolingual and parallel corpora using the chardet package  The corpora were cleaned for any stray special characters, unnecessary html tags and website URLs 
Word2Vec
The word embeddings of the source and target language sentences are used as initial vectors of the model to improve contextualization  The skip gram model of the word2vec algorithm optimizes the vectors by accounting for the average log probability of context words given a source word  DISPLAYFORM0
where k is the context window taken for the vectorization, INLINEFORM0 refers to the INLINEFORM1 word of the corpus and INLINEFORM2 is the size of the training corpus in terms of the number of words  Here, the probabily INLINEFORM3 is computed as a hierarchical softmax of the product of the transpose of the output vector of INLINEFORM4 and the input vector of INLINEFORM5 for each and every pair over the entire vocabulary  The processes of negative sampling and subsampling of frequent words that were used in the original model aren't used in this experiment BIBREF3  
For the process of creating semantically meaningful word embeddings, a monolingual corpus of 569,772 Tamil sentences was used  This gave the vectors more contextual richness due to the increased size of the corpus as opposed to using just the bilingual corpus' target side sentences BIBREF3  
In the experiment, the word2vec model was trained using a vector size of 100 to ensure that the bulk of the limited memory of the GPU will be used for the neural attention translation model  It has been shown that any size over that of 150 used for word vectorization gives similar results and that a size of 100 performs close to the model with 150-sized word vectors BIBREF7   A standard size of 5 was used as window size and the model was trained over 7 worker threads simultaneously  A batch size of 50 words was used for training  The negative sampling was set at 1 as it is the nature of morphologically rich languages to have a lot of important words that don't occur more than once in the corpus  The gensim word2vec toolkit was used to implement this word embedding process BIBREF8  
Neural Translation Model
The model used for translation is the one implemented by Bahdanau et al  Bahdanau2014  A bidirectional LSTM encoder first takes the source sentence and encodes it into a context vector which acts as input for the decoder  The decoder is attention-based where the hidden states of the decoder get as input the weighted sum of all the hidden layer outputs of the encoder alongwith the output of the previous hidden layer and the previously decoded word  This provides a contextual reference into the source language sentence BIBREF4  
Neural Machine Translation models directly compute the probability of the target language sentence given the source language sentence, word by word for every time step  The model with a basic decoder without the attention module computes the log probability of target sentence given source sentence as the sum of log probabilities of every word given every word before that  The attention-based model, on the other hand, calculates: DISPLAYFORM0
where INLINEFORM0 is the number of words in the target sentence, INLINEFORM1 is the target sentence, INLINEFORM2 is the source sentence, INLINEFORM3 is the fixed length output vector of the encoder and INLINEFORM4 is the weighted sum of all the hidden layer outputs of the encoder at every time step  Both the encoder's output context vector and the weighted sum (known as attention vector) help to improve the quality of translation by enabling selective source sentence lookup 
The decoder LSTM computes: DISPLAYFORM0
where the probability is computed as a function of the decoder's output in the previous time step INLINEFORM0 , the hidden layer vector of the decoder in the current timestep INLINEFORM1 and the context vector from the attention mechanism INLINEFORM2   The context vector INLINEFORM3 for time step INLINEFORM4 is computed as a weighted sum of the output of the entire sentence using a weight parameter INLINEFORM5 : DISPLAYFORM0
where INLINEFORM0 is the number of tokens in the source sentence, INLINEFORM1 refers to the value of the hidden layer of the encoder at time step INLINEFORM2 , and INLINEFORM3 is the alignment parameter  This parameter is calculated by means of a feed forward neural network to ensure that the alignment model is free from the difficulties of contextualization of long sentences into a single vector  The feed forward network is trained along with the neural translation model to jointly improve the performance of the translation  Mathematically, DISPLAYFORM0 DISPLAYFORM1
where INLINEFORM0 is the softmax output of the result of the feedforward network, INLINEFORM1 is the hidden state value of the decoder at timestep INLINEFORM2 and INLINEFORM3 is the encoder's hidden layer annotation at timestep INLINEFORM4   A concatenation of the forward and the reverse hidden layer parameters of the encoder is used at each step to compute the weights INLINEFORM5 for the attention mechanism  This is done to enable an overall context of the sentence, as opposed to a context of only all the previous words of the sentence for every word in consideration  Fig  FIGREF12 is the general architecture of the neural translation model without the Bidirectional LSTM encoder 
A global attention mechanism is preferred over local attention because the differences in the structures of the languages cannot be mapped efficiently to enable lookup into the right parts of the source sentence  Using local attention mechanism with a monotonic context lookup, where the region around INLINEFORM0 source word is looked up for the prediction of the INLINEFORM1 target word, is impractical because of the structural discordance between the English and Tamil sentences (see Figs  FIGREF37 and FIGREF44 )  The use of gaussian and other such distributions to facilitate local attention would also be inefficient because the existence of various forms of translations for the same source sentence involving morphological and structural variations that don't stay uniform through the entire corpus BIBREF5  
The No Peepholes (NP) variant of the LSTM cell, formulated in Greff et al  greff2015lstm is used in this experiment as it proved to give the best results amongst all the variants of an LSTM cell  It is specified by means of a gated mechanism designed to ensure that the vanishing gradient problem is prevented  LSTM maintains its hidden layer in two components, the cell vector INLINEFORM0 and the actual hidden layer output vector INLINEFORM1   The cell vector is ensured to never reach zero by means of a weighted sum of the previous layer's cell vector INLINEFORM2 regulated by the forget gate INLINEFORM3 and an activation of the weighted sum of the input INLINEFORM4 in the current timestep INLINEFORM5 and the previous timestep's hidden layer output vector INLINEFORM6   The combination is similarly regulated by the input gate INLINEFORM7   The hidden layer output is determined as an activation of the cell gate, regulated by the output gate INLINEFORM8   The interplay between these two vectors ( INLINEFORM9 and INLINEFORM10 ) at every timestep ensures that the problem of vanishing gradients doesn't occur  The three gates are also formed as a sigmoid of the weighted sum of the previous hidden layer output INLINEFORM11 and the input in the current timestep INLINEFORM12   The output generated out of the LSTM's hidden layer is specified as a weighted softmax over the hidden layer output INLINEFORM13   The learnable parameters of an LSTM cell are all the weights INLINEFORM14 and the biases INLINEFORM15   DISPLAYFORM0
The LSTM specified by equations 7 through 11 is the one used for the decoder of the model  The encoder uses a bidirectional RNN LSTM cell in which there are two hidden layer components INLINEFORM0 and INLINEFORM1 that contribute to the output INLINEFORM2 of each time step INLINEFORM3   Both the components have their own sets of LSTM equations in such a way that INLINEFORM4 for every timestep is computed from the first timestep till the INLINEFORM5 token is reached and INLINEFORM6 is computed from the INLINEFORM7 timestep backwards until the first token is reached  All the five vectors of the two components are all exactly the same as the LSTM equations specified with one variation in the computation of the result  DISPLAYFORM0
Morphological Segmentation
The morphological segmentation used is a semi-supervised extension to the generative probabilistic model of maximizing the probability of a INLINEFORM0 prefix,root,postfix INLINEFORM1 recursive split up of words based on an exhaustive combination of all possible morphemes  The details of this model are specified and extensively studied in Kohonen et al  kohonen2010semi  The model parameters INLINEFORM2 include the morph type count, morph token count of training data, the morph strings and their counts  The model is trained by maximizing the Maximum A Posteriori (MAP) probability using Bayes' rule: DISPLAYFORM0
where INLINEFORM0 refers to every word in the training lexicon  The prior INLINEFORM1 is estimated using the Minimum Description Length(MDL) principle  The likelihood INLINEFORM2 is estimated as: DISPLAYFORM0
where INLINEFORM0 refers to the intermediate analyses and INLINEFORM1 refers to the INLINEFORM2 morpheme of word INLINEFORM3  
An extension to the Viterbi algorithm is used for the decoding step based on exhaustive mapping of morphemes  To account for over-segmentation and under-segmentation issues associated with unsupervised morphological segmentation, extra parameters ( INLINEFORM0 ) and ( INLINEFORM1 ) are used with the cost function INLINEFORM2 DISPLAYFORM0
where INLINEFORM0 is the likelihood of the cost function, INLINEFORM1 describes the likelihood of contribution of the annotated dataset to the cost function and INLINEFORM2 is the likelihood of the labeled data  A decrease in the value of INLINEFORM3 will cause smaller segments and vice versa  INLINEFORM4 takes care of size discrepancies due to reduced availability of annotated corpus as compared to the training corpus BIBREF2 , BIBREF6  
The Python extension to the morphological segmentation tool morfessor 2 0 was used for this experiment to perform the segmentation  The annotation data for Tamil language collated and released by Anoop Kunchukkutan in the Indic NLP Library was used as the semi-supervised input to the model BIBREF9 , BIBREF6  
Experiment
The complexities of neural machine translation of morphologically rich languages were studied with respect to English to Tamil machine translation using the RNN LSTM Bi-directional encoder attention decoder architecture  To compare with a baseline system, a phrase based SMT system was implemented using the same corpus  The Factored SMT model with source-side preprocessing by Kumar et al  kumar2014improving was used as a reference for the translation between these language pairs  Also, an additional 569,772 monolingual Tamil sentences were used for the language model of the SMT system  The model used could be split up into various modules as expanded in Fig  FIGREF17  
Bucketing
The input source and target language sentences used for training were taken and divided into bucketed pairs of sentences of a fixed number of sizes  This relationship was determined by examining the distribution of words in the corpus primarily to minimize the number of PAD tokens in the sentence  The heat map of the number of words in the English–Tamil sentence pairs of the corpus revealed that the distribution is centered around the 10–20 words region  Therefore, more buckets in that region were applied as there would be enough number of examples in each of these bucket pairs for the model to learn about the sentences in each and every bucket  The exact scheme used for the RNNSearch models is specified by Fig  FIGREF21   The bucketing scheme for the RNNMorph model, involving morphs instead of words, was a simple shifted scheme of the one used in Fig  FIGREF21 , where every target sentence bucket count was increased uniformly by 5 
Model Details
Due to various computational constraints and lack of availability of comprehensive corpora, the vocabularies for English and Tamil languages for the RNNSearch model were restricted to 60,000 out of 67,768 and 150,000 out of 340,325 respectively  The vocabulary of the languages for the RNNMorph didn't have to be restricted and the actual number of words in the corpus i e  67,768 words for English and 41,906 words for Tamil could be accommodated into the training  Words not in the vocabulary from the test set input and output were replaced with the universal INLINEFORM0 UNK INLINEFORM1 token, symbolizing an unknown word  The LSTM hidden layer size, the training batch size, and the vocabulary sizes of the languages, together, acted as a bottleneck  The model was run on a 2GB NVIDIA GeForce GT 650M card with 384 cores and the memory allotment was constrained to the limits of the GPU  Therefore, after repeated experimentation, it was determined that with a batch size of 16, the maximum hidden layer size possible was 500, which was the size used  Attempts to reduce the batch size resulted in poor convergence, and so the parameters were set to center around the batch size of 16  The models used were of 4 layers of LSTM hidden units in the bidirectional encoder and attention decoder 
The model used a Stochastic Gradient Descent (SGD) optimization algorithm with a sampled softmax loss of 512 per sample to handle large vocabulary size of the target language BIBREF10   The model was trained with a learning rate 1 0 and a decay of rate 0 5 enforced manually  Gradient clipping based on the global norm of 5 0 was carried out to prevent gradients exploding and going to unrecoverable values tending towards infinity  The model described is the one used in the Tensorflow BIBREF11 seq2seq library 
Results and Discussion
The BLEU metric parameters (modified 1-gram, 2-gram, 3-gram and 4-gram precision values) and human evaluation metrics of adequacy, fluency and relative ranking values were used to evaluate the performance of the models 
BLEU Evaluation
The BLEU scores obtained using the various models used in the experiment are tabulated in Table TABREF25  
The BLEU metric computes the BLEU unigram, bigram, trigram and BLEU-4 modified precision values, each micro-averaged over the test set sentences BIBREF7   It was observed, as expected, that the performance of the phrase-based SMT model was inferior to that of the RNNSearch model  The baseline RNNSearch system was further refined by using word2vec vectors to embed semantic understanding, as observed with the slight increase in the BLEU scores  Fig  FIGREF26 plots the BLEU scores as a line graph for visualization of the improvement in performance  Also, the 4-gram BLEU scores for the various models were plotted as a bar graph in Fig  FIGREF26
Due to the agglutinative and morphologically rich nature of the target language i e  Tamil, the use of morphological segmentation to split the words into morphemes further improved the BLEU precision values in the RNNMorph model  One of the reasons for the large extent of increase in the BLEU score could be attributed to the overall increase in the number of word units per sentence  Since the BLEU score computes micro-average precision scores, an increase in both the numerator and denominator of the precision scores is apparent with an increase in the number of tokens due to morphological segmentation of the target language  Thus, the numeric extent of the increase of accuracy might not efficiently describe the improvement in performance of the translation 
Human Evaluation
To ensure that the increase in BLEU score correlated to actual increase in performance of translation, human evaluation metrics like adequacy, precision and ranking values (between RNNSearch and RNNMorph outputs) were estimated in Table TABREF30   A group of 50 native people who were well-versed in both English and Tamil languages acted as annotators for the evaluation  A collection of samples of about 100 sentences were taken from the test set results for comparison  This set included a randomized selection of the translation results to ensure the objectivity of evaluation  Fluency and adequacy results for the RNNMorph results are tabulated  Adequacy rating was calculated on a 5-point scale of how much of the meaning is conveyed by the translation (All, Most, Much, Little, None)  The fluency rating was calculated based on grammatical correctness on a 5-point scale of (Flawless, Good, Non-native, Disfluent, Incomprehensive)  For the comparison process, the RNNMorph and the RNNSearch + Word2Vec models’ sentence level translations were individually ranked between each other, permitting the two translations to have ties in the ranking  The intra-annotator values were computed for these metrics and the scores are shown in Table TABREF32 BIBREF12 , BIBREF13  
The human evaluation Kappa co-efficient results are calculated with respect to: DISPLAYFORM0
It was observed that the ranking Kappa co-efficient for intra-annotator ranking of the RNNMorph model was at 0 573, higher that the 0 410 of the RNNSearch+Word2Vec model, implying that the annotators found the RNNMorph model to produce better results when compared to the RNNSearch + Word2Vec model 
Model Parameters
The learning rate decay through the training process of the RNNMorph model is showcased in the graph in Fig  FIGREF34   This process was done manually where the learning rate was decayed after the end of specific epochs based on an observed stagnation in perplexity The RNNMorph model achieved saturation of perplexities much earlier through the epochs than the RNNSearch + Word2Vec model  This conforms to the expected outcome as the morphological segmentation has reduced the vocabulary size of the target language from 340,325 words to a mere 41,906 morphs 
The error function used was the sampled SoftMax loss to ensure a large target vocabulary could be accommodated BIBREF10   A zoomed inset graph (Fig  FIGREF35 ) has been used to visualize the values of the error function for the RNNSearch + Word2Vec and RNNMorph models with 4 hidden layers  It can be seen that the RNNMorph model is consistently better in terms of the perplexity values through the time steps 
Attention Vectors
In order to further demonstrate the quality of the RNNMorph model, the attention vectors of both the RNNSearch with Word2Vec embedding and RNNMorph models are compared for several good translations in Figs  FIGREF37 and FIGREF44   It is observed that the reduction in vocabulary size has improved the source sentence lookup by quite an extent  Each cell in the heatmap displays the magnitude of the attention layer weight INLINEFORM0 for the INLINEFORM1 Tamil word and the INLINEFORM2 English word in the respective sentences  The intensity of black corresponds to the magnitude of the cell INLINEFORM3   Also, the attention vectors of the RNNSearch model with Word2Vec embeddings tend to attend to INLINEFORM4 EOS INLINEFORM5 token in the middle of the sentence leading to incomplete translations  This could be due to the fact that only 44% of the Tamil vocabulary and 74% of the English vocabulary is taken for training in this model, as opposed to 100% of English and Tamil words in the RNNMorph model 
Target vocabulary size
A very large target vocabulary is an inadvertent consequence of the morphological richness of the Tamil language  This creates a potential restriction on the accuracy of the model as many inflectional forms of the same word are trained as independent units  One of the advantages of morphological segmentation of Tamil text is that the target vocabulary size decreased from 340,325 to a mere 41,906  This reduction helps improve the performance of the translation as the occurrence of unknown tokens was reduced compared to the RNNSearch model  This morphologically segmented vocabulary is divided into a collection of morphological roots and inflections as individual units 
Repetitions
Some of the translations of the RNNMorph model have repetitions of the same phrases (Fig  FIGREF53 ), whereas such repetitions occur much less frequently in the RNNSearch predictions  Such translations would make for good results if the repetitions weren't present and all parts of the sentence occur just once  These repetitions might be due to the increase in the general sequence length of the target sentences because of the morphological segmentation  While it is true the target vocabulary size has decreased due to morphological segmentation, the RNNMorph has more input units (morphs) per sentence, which makes it more demanding of the LSTM's memory units and the feed forward network of the attention model  Additionally, this behavior could also be attributed to the errors in the semi-supervised morphological segmentation due to the complexities of the Tamil language and the extent of the corpus 
Model Outputs
The translation outputs of the RNNSearch + Word2Vec and Morph2Vec models for the same input sentences from the test set demonstrate the effectiveness of using a morphological segmentation tool and how the morphemes have changed the sentence to be more grammatically sound  It is also observed (from Fig  FIGREF55 ) that most of the translation sentences of the Morph2Vec model have no INLINEFORM0 UNK INLINEFORM1 tokens  They exist in the predictions mostly only due to a word in the English test sentence not present in the source vocabulary 
Related Work
Professors CN Krishnan, Sobha et al developed a machine-aided-translation (MAT) system similar to the Anusaakara English Hindi MT system, using a small corpus and very few transfer rules, available at AU-KBC website BIBREF14   Balajapally et al  balajapally2006multilingual developed an example based machine translation (EBMT) system with 700000 sentences for English to INLINEFORM0 Tamil, Kannada, Hindi INLINEFORM1 transliterated text BIBREF15 , BIBREF16   Renganathan renganathan2002interactive developed a rule based MT system for English and Tamil using grammar rules for the language pair  Vetrivel et al  vetrivel2010english used HMMs to align and translate English and Tamil parallel sentences to build an SMT system  Irvine et al  irvine2013combining tried to combine parallel and similar corpora to improve the performance of English to Tamil SMT amongst other languages  Kasthuri et al  kasthuri2014rule used a rule based MT system using transfer lexicon and morphological analysis tools  Anglabharathi was developed at IIT Kanpur, a system translating English to a collection of Indian languages including Tamil using CFG like structures to create a pseudo target to convert to Indian languages BIBREF17 , BIBREF18   A variety of hybrid approaches have also been used for English–Tamil MT in combinations of rule based (transfer methods), interlingua representations BIBREF19 , BIBREF20 , BIBREF21   The use of Statistical Machine Translation took over the English–Tamil MT system research because of its desirable properties of language independence, better generalization features and a reduced requirement of linguistic expertise BIBREF1 , BIBREF22 , BIBREF23   Various enhancement techniques external to the MT system have also been proposed to improve the performance of translation using morphological pre and post processing techniques BIBREF24 , BIBREF25 , BIBREF26  
The use of RNN Encoder Decoder models in machine translation has shown good results in languages with similar grammatical structure  Deep MT systems have been performing better than the other shallow SMT models recently, with the availability of computational resources and hardware making it feasible to train such models  The first of these models came in 2014, with Cho et al SecondOneByCho  The model used was the RNN LSTM encoder decoder model with the context vector output of the encoder (run for every word in the sentence) is fed to every decoder unit along with the previous word output until INLINEFORM0 EOS INLINEFORM1 is reached  This model was used to score translation results of another MT system  Sutskever et al  sutskever2014sequence created a similar encoder decoder model with the decoder getting the context vector only for the first word of the target language sentence  After that, only the decoded target outputs act as inputs to the various time steps of the decoder  One major drawback of these models is the size of the context vector of the encoder being static in nature  The same sized vector was expected to to represent sentences of arbitrary length, which was impractical when it came to very long sentences 
The next breakthrough came from Bahdanau et al  Bahdanau2014 where variable length word vectors were used and instead of just the context vector, a weighted sum of the inputs is given for the decoder  This enabled selective lookup to the source sentence during decoding and is known as the attention mechanism BIBREF27   The attention mechanism was further analysed by Luong et al  luong2015effective where they made a distinction between global and local attention by means of AER scores of the attention vectors  A Gaussian distribution and a monotonic lookup were used to facilitate the corresponding local source sentence look-up 
Conclusion
Thus, it is seen that the use of morphological segmentation on a morphologically rich language before translation helps with the performance of the translation in multiple ways  Thus, machine translation involving morphologically rich languages should ideally be carried out only after morphological segmentation  If the translation has to be carried out between two morphologically rich languages, then both the languages' sentences should be individually segmented based on morphology  This is because while it is true that they are both morphologically rich languages, the schemes that the languages use for the process of agglutination might be different, in which case a mapping between the units would be difficult without the segmentation 
One drawback of morphological segmentation is the increase in complexity of the model due to an increase in the average sentence lengths  This cannot be avoided as it is essential to enable a correspondence between the sentences of the two languages when one of them is a simple fusional language  Even with the increase in the average sentence length, the attention models that have been developed to ensure correctness of translation of long sequences can be put to good use when involving morphologically rich languages  Another point to note here is that morphologically rich languages like Tamil generally have lesser number of words per sentence than languages like English due to the inherent property of agglutination 
Future Work
The model implemented in this paper only includes source-side morphological segmentation and does not include a target side morphological agglutination to give back the output in words rather than morphemes  In order to implement an end-to-end translation system for morphologically rich languages, a morphological generator is essential because the output units of the translation cannot be morphemes 
The same model implemented can be further enhanced by means of a better corpus that can generalize over more than just domain specific source sentences  Also, the use of a better GPU would result in a better allocation of the hidden layer sizes and the batch sizes thereby possibly increasing the scope and accuracy of learning of the translation model 
Although not directly related to Machine Translation, the novel encoder– decoder architecture proposed in by Rocktaschel et al  rocktaschel2015reasoning for Natural Language Inference (NLI) can be used for the same  Their model fuses inferences from each and every individual word, summarizing information at each step, thereby linking the hidden state of the encoder with that of the decoder by means of a weighted sum, trained for optimization 
Acknowledgements
I would like to thank Dr  M  Anand Kumar, Assistant Professor, Amrita Vishwa Vidyapeetham for his continuous support and guidance  I would also like to thank Dr  Arvindan, Professor, SSN College Of Engineering for his inputs and suggestions 
Context 2: Introduction
Cancer is one of the leading causes of death in the world, with over 80,000 deaths registered in Canada in 2017 (Canadian Cancer Statistics 2017)  A computer-aided system for cancer diagnosis usually involves a pathologist rendering a descriptive report after examining the tissue glass slides obtained from the biopsy of a patient  A pathology report contains specific analysis of cells and tissues, and other histopathological indicators that are crucial for diagnosing malignancies  An average sized laboratory may produces a large quantity of pathology reports annually (e g , in excess of 50,000), but these reports are written in mostly unstructured text and with no direct link to the tissue sample  Furthermore, the report for each patient is a personalized document and offers very high variability in terminology due to lack of standards and may even include misspellings and missing punctuation, clinical diagnoses interspersed with complex explanations, different terminology to label the same malignancy, and information about multiple carcinoma appearances included in a single report BIBREF0  
In Canada, each Provincial and Territorial Cancer Registry (PTCR) is responsible for collecting the data about cancer diseases and reporting them to Statistics Canada (StatCan)  Every year, Canadian Cancer Registry (CCR) uses the information sources of StatCan to compile an annual report on cancer and tumor diseases  Many countries have their own cancer registry programs  These programs rely on the acquisition of diagnostic, treatment, and outcome information through manual processing and interpretation from various unstructured sources (e g , pathology reports, autopsy/laboratory reports, medical billing summaries)  The manual classification of cancer pathology reports is a challenging, time-consuming task and requires extensive training BIBREF0  
With the continued growth in the number of cancer patients, and the increase in treatment complexity, cancer registries face a significant challenge in manually reviewing the large quantity of reports BIBREF1 , BIBREF0   In this situation, Natural Language Processing (NLP) systems can offer a unique opportunity to automatically encode the unstructured reports into structured data  Since, the registries already have access to the large quantity of historically labeled and encoded reports, a supervised machine learning approach of feature extraction and classification is a compelling direction for making their workflow more effective and streamlined  If successful, such a solution would enable processing reports in much lesser time allowing trained personnel to focus on their research and analysis  However, developing an automated solution with high accuracy and consistency across wide variety of reports is a challenging problem 
For cancer registries, an important piece of information in a pathology report is the associated ICD-O code which describes the patient's histological diagnosis, as described by the World Health Organization's (WHO) International Classification of Diseases for Oncology BIBREF2   Prediction of the primary diagnosis from a pathology report provides a valuable starting point for exploration of machine learning techniques for automated cancer surveillance  A major application for this purpose would be “auto-reporting” based on analysis of whole slide images, the digitization of the biopsy glass slides  Structured, summarized and categorized reports can be associated with the image content when searching in large archives  Such as system would be able to drastically increase the efficiency of diagnostic processes for the majority of cases where in spite of obvious primary diagnosis, still time and effort is required from the pathologists to write a descriptive report 
The primary objective of our study is to analyze the efficacy of existing machine learning approaches for the automated classification of pathology reports into different diagnosis categories  We demonstrate that TF-IDF feature vectors combined with linear SVM or XGBoost classifier can be an effective method for classification of the reports, achieving up to 83% accuracy  We also show that TF-IDF features are capable of identifying important keywords within a pathology report  Furthermore, we have created a new dataset consisting of 1,949 pathology reports across 37 primary diagnoses  Taken together, our exploratory experiments with a newly introduced dataset on pathology reports opens many new opportunities for researchers to develop a scalable and automatic information extraction from unstructured pathology reports 
Background
NLP approaches for information extraction within the biomedical research areas range from rule-based systems BIBREF3 , to domain-specific systems using feature-based classification BIBREF1 , to the recent deep networks for end-to-end feature extraction and classification BIBREF0   NLP has had varied degree of success with free-text pathology reports BIBREF4   Various studies have acknowledge the success of NLP in interpreting pathology reports, especially for classification tasks or extracting a single attribute from a report BIBREF4 , BIBREF5  
The Cancer Text Information Extraction System (caTIES) BIBREF6 is a framework developed in a caBIG project focuses on information extraction from pathology reports  Specifically, caTIES extracts information from surgical pathology reports (SPR) with good precision as well as recall 
Another system known as Open Registry BIBREF7 is capable of filtering the reports with disease codes containing cancer  In BIBREF8 , an approach called Automated Retrieval Console (ARC) is proposed which uses machine learning models to predict the degree of association of a given pathology or radiology with the cancer  The performance ranges from an F-measure of 0 75 for lung cancer to 0 94 for colorectal cancer  However, ARC uses domain-specific rules which hiders with the generalization of the approach to variety of pathology reports 
This research work is inspired by themes emerging in many of the above studies  Specifically, we are evaluating the task of predicting the primary diagnosis from the pathology report  Unlike previous approaches, the system does not rely on custom rule-based knowledge, domain specific features, balanced dataset with fewer number of classes 
Materials and Methods
We assembled a dataset of 1,949 cleaned pathology reports  Each report is associated with one of the 37 different primary diagnoses based on IDC-O codes  The reports are collected from four different body parts or primary sites from multiple patients  The distribution of reports across different primary diagnoses and primary sites is reported in tab:report-distribution  The dataset was developed in three steps as follows 
Collecting pathology reports: The total of 11,112 pathology reports were downloaded from NCI's Genomic Data Commons (GDC) dataset in PDF format BIBREF9   Out of all PDF files, 1,949 reports were selected across multiple patients from four specific primary sites—thymus, testis, lung, and kidney  The selection was primarily made based on the quality of PDF files 
Cleaning reports: The next step was to extract the text content from these reports  Due to the significant time expense of manually re-typing all the pathology reports, we developed a new strategy to prepare our dataset  We applied an Optical Character Recognition (OCR) software to convert the PDF reports to text files
Context 3: Introduction
Part-of-speech tagging is now a classic task in natural language processing, for which many systems have been developed or adapted for a large variety of languages  Its aim is to associate each “word” with a morphosyntactic tag, whose granularity can range from a simple morphosyntactic category, or part-of-speech (hereafter PoS), to finer categories enriched with morphological features (gender, number, case, tense, mood, etc ) 
The use of machine learning algorithms trained on manually annotated corpora has long become the standard way to develop PoS taggers  A large variety of algorithms have been used, such as (in approximative chronological order) bigram and trigram hidden Markov models BIBREF0 , BIBREF1 , BIBREF2 , decision trees BIBREF3 , BIBREF4 , maximum entropy Markov models (MEMMs) BIBREF5 and Conditional Random Fields (CRFs) BIBREF6 , BIBREF7   With such machine learning algorithms, it is possible to build PoS taggers for any language, provided adequate training data is available 
As a complement to annotated corpora, it has previously been shown that external lexicons are valuable sources of information, in particular morphosyntactic lexicons, which provide a large inventory of (word, PoS) pairs  Such lexical information can be used in the form of constraints at tagging time BIBREF8 , BIBREF9 or during the training process as additional features combined with standard features extracted from the training corpus BIBREF10 , BIBREF11 , BIBREF12  
In recent years, a different approach to modelling lexical information and integrating it into natural language processing systems has emerged, namely the use of vector representations for words or word sequences BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18   Such representations, which are generally extracted from large amounts of raw text, have proved very useful for numerous tasks including PoS tagging, in particular when used in recurrent neural networks (RNNs) and more specifically in mono- or bi-directional, word-level and/or character-level long short-term memory networks (LSTMs) BIBREF19 , BIBREF16 , BIBREF17 , BIBREF20  
Both approaches to representing lexical properties and to integrating them into a PoS tagger improve tagging results  Yet they rely on resources of different natures  The main advantage of word vectors is that they are built in an unsupervised way, only requiring large amounts of raw textual data  They also encode finer-grained information than usual morphosyntactic lexicons, most of which do not include any quantitative data, not even simple frequency information  Conversely, lexical resources often provide information about scarcely attested words, for which corpus-based approaches such as word vector representations are of limited relevance  Moreover, morphological or morphosyntactic lexicons already exist for a number of languages, including less-resourced langauges for which it might be difficult to obtain the large amounts of raw data necessary to extract word vector representations 
Our main goal is therefore to compare the respective impact of external lexicons and word vector representations on the accuracy of PoS models  This question has already been investigated for 6 languages by BIBREF18 using the state-of-the-art CRF-based tagging system MarMoT  The authors found that their best-performing word-vector-based PoS tagging models outperform their models that rely on morphosyntactic resources (lexicons or morphological analysers)  In this paper, we report on larger comparison, carried out in a larger multilingual setting and comparing different tagging models  Using different 16 datasets, we compare the performances of two feature-based models enriched with external lexicons and of two LSTM-based models enriched with word vector representations  A secondary goal of our work is to compare the relative improvements linked to the use of external lexical information in the two feature-based models, which use different models (MEMM vs  CRF) and feature sets 
More specifically, our starting point is the MElt system BIBREF12 , an MEMM tagging system  We first briefly describe this system and the way we adapted it by integrating our own set of corpus-based and lexical features  We then introduce the tagging models we have trained for 16 different languages using our adapted version of MElt  These models are trained on the Universal Dependencies (v1 2) corpus set BIBREF21 , complemented by morphosyntactic lexicons  We compare the accuracy of our models with the scores obtained by the CRF-based system MarMoT BIBREF22 , BIBREF18 , retrained on the same corpora and the same external morphosyntactic lexicons  We also compare our results to those obtained by the best bidirectional LSTM models described by BIBREF20 , which both make use of Polyglot word vector representations published by BIBREF23   We will show that an optimised enrichment of feature-based models with morphosyntactic lexicon results in significant accuracy gains  The macro-averaged accuracy of our enriched MElt models is above that of enriched MarMoT models and virtually identical to that of LSTMs enriched with word vector representations  More precisely, per-language results indicate that lexicons provide more useful information for languages with a high lexical variability (such as morphologically rich languages), whereas word vectors are more informative for languages with a lower lexical variability (such as English) 
MElt
MElt BIBREF12 is a tagging system based on maximum entropy Markov models (MEMM) BIBREF5 , a class of discriminative models that are suitable for sequence labelling BIBREF5   The basic set of features used by MElt is given in BIBREF12   It is a superset of the feature sets used by BIBREF5 and BIBREF24 and includes both local standard features (for example the current word itself and its prefixes and suffixes of length 1 to 4) and contextual standard features (for example the tag just assigned to the preceding word)  In particular, with respect to Ratnaparkhi's feature set, MElt's basic feature set lifts the restriction that local standard features used to analyse the internal composition of the current word should only apply to rare words 
One of the advantages of feature-based models such as MEMMs and CRFs is that complementary information can be easily added in the form of additional features  This was investigated for instance by BIBREF25 , whose best-performing model for PoS tagging dialogues was obtained with a version of MElt extended with dialogue-specific features  Yet the motivation of MElt's developers was first and foremost to investigate the best way to integrate lexical information extracted from large-scale morphosyntactic lexical resources into their models, on top of the training data BIBREF12   They showed that performances are better when this external lexical information is integrated in the form of additional lexical features than when the external lexicon is used as constraints at tagging time  These lexical features can also be divided into local lexical features (for example the list of possible tags known to the external lexicon for the current word) and contextual lexical features (for example the list of possible tags known to the external lexicon for surrounding words)  In particular, lexical contextual features provide a means to model the right context of the current word, made of words that have not yet been tagged by the system but for which the lexicon often provides a list of possible tags  Moreover, tagging accuracy for out-of-vocabulary (OOV) words is improved, as a result of the fact that words unknown to the training corpus might be known to the external lexicon 
Despite a few experiments published with MElt on languages other than French BIBREF12 , BIBREF40 , BIBREF41 , the original feature set used by MElt (standard and lexical features) was designed and tested mostly on this language, by building and evaluating tagging models on a variant of the French TreeBank  Since our goal was to carry out experiments in a multilingual setting, we have decided to design our own set of features, using the standard MElt features as a starting point  With respect to the original MElt feature set, we have added new ones, such as prefixes and suffixes of the following word, as well as a hybrid contextual feature obtained by concatenating the tag predicted for the preceding word and the tag(s) provided by the external lexicon for the following word 
In order to select the best performing feature set, we carried out a series of experiments using the multilingual dataset provided during the SPMRL parsing shared task BIBREF42   This included discarding useless or harmful features and selecting the maximal length of the prefixes and suffixes to be used as features, both for the current word and for the following word 
We incorporated in MElt the best performing feature set, described in Table TABREF1   All models discussed in this paper are based on this feature set 
Corpora
We carried out our experiments on the Universal Dependencies v1 2 treebanks BIBREF21 , hereafter UD1 2, from which morphosyntactically annotated corpora can be trivially extracted  All UD1 2 corpora use a common tag set, the 17 universal PoS tags, which is an extension of the tagset proposed by BIBREF43  
As our goal is to study the impact of lexical information for PoS tagging, we have restricted our experiments to UD1 2 corpora that cover languages for which we have morphosyntactic lexicons at our disposal, and for which BIBREF20 provide results  We considered UD1 2 corpora for the following 16 languages: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish  Although this language list contains only one non-Indo-European (Indonesian), four major Indo-European sub-families are represented (Germanic, Romance, Slavic, Indo-Iranian)

ground_truth: 50 human annotators ranked a random sample of 100 translations by Adequacy, Fluency and overall ranking on a 5-point scale.

adequacy, precision and ranking values

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What features are used?
Context 2:  Then, we manually inspected all generated text files to fix any grammar/spelling issues and irrelevant characters as an artefact produced by the OCR system 
Splitting into training-testing data: We split the cleaned reports into 70% and 30% for training and testing, respectively  This split resulted in 1,364 training, and 585 testing reports 
Pre-Processing of Reports
We pre-processed the reports by setting their text content to lowercase and filtering out any non-alphanumeric characters  We used NLTK library to remove stopping words, e g , `the', `an', `was', `if' and so on BIBREF10   We then analyzed the reports to find common bigrams, such as “lung parenchyma”, “microscopic examination”, “lymph node” etc  We joined the biagrams with a hyphen, converting them into a single word  We further removed the words that occur less than 2% in each of the diagnostic category  As well, we removed the words that occur more than 90% across all the categories  We stored each pre-processed report in a separate text file 
TF-IDF features
TF-IDF stands for Term Frequency-Inverse Document Frequency, and it is a useful weighting scheme in information retrieval and text mining  TF-IDF signifies the importance of a term in a document within a corpus  It is important to note that a document here refers to a pathology report, a corpus refers to the collection of reports, and a term refers to a single word in a report  The TF-IDF weight for a term INLINEFORM0 in a document INLINEFORM1 is given by DISPLAYFORM0
We performed the following steps to transform a pathology report into a feature vector:
Create a set of vocabulary containing all unique words from all the pre-processed training reports 
Create a zero vector INLINEFORM0 of the same length as the vocabulary 
For each word INLINEFORM0 in a report INLINEFORM1 , set the corresponding index in INLINEFORM2 to INLINEFORM3  
The resultant INLINEFORM0 is a feature vector for the report INLINEFORM1 and it is a highly sparse vector 
Keyword extraction and topic modelling
The keyword extraction involves identifying important words within reports that summarizes its content  In contrast, the topic modelling allows grouping these keywords using an intelligent scheme, enabling users to further focus on certain aspects of a document  All the words in a pathology report are sorted according to their TF-IDF weights  The top INLINEFORM0 sorted words constitute the top INLINEFORM1 keywords for the report  The INLINEFORM2 is empirically set to 50 within this research  The extracted keywords are further grouped into different topics by using latent Dirichlet allocation (LDA) BIBREF11   The keywords in a report are highlighted using the color theme based on their topics 
Evaluation metrics
Each model is evaluated using two standard NLP metrics—micro and macro averaged F-scores, the harmonic mean of related metrics precision and recall  For each diagnostic category INLINEFORM0 from a set of 37 different classes INLINEFORM1 , the number of true positives INLINEFORM2 , false positives INLINEFORM3 , and false negatives INLINEFORM4 , the micro F-score is defined as DISPLAYFORM0
whereas macro F-score is given by DISPLAYFORM0
In summary, micro-averaged metrics have class representation roughly proportional to their test set representation (same as accuracy for classification problem with a single label per data point), whereas macro-averaged metrics are averaged by class without weighting by class prevalence BIBREF12  
Experimental setting
In this study, we performed two different series of experiments: i) evaluating the performance of TF-IDF features and various machine learning classifiers on the task of predicting primary diagnosis from the text content of a given report, and ii) using TF-IDF and LDA techniques to highlight the important keywords within a report  For the first experiment series, training reports are pre-processed, then their TF-IDF features are extracted  The TF-IDF features and the training labels are used to train different classification models  These different classification models and their hyper-parameters are reported in tab:classifier  The performance of classifiers is measured quantitatively on the test dataset using the evaluation metrics discussed in the previous section  For the second experiment series, a random report is selected and its top 50 keywords are extracted using TF-IDF weights  These 50 keywords are highlighted using different colors based on their associated topic, which are extracted through LDA  A non-expert based qualitative inspection is performed on the extracted keywords and their corresponding topics 
Experiment Series 1
A classification model is trained to predict the primary diagnosis given the content of the cancer pathology report  The performance results on this task are reported in tab:results  We can observe that the XGBoost classifier outperformed all other models for both the micro F-score metric, with a score of 0 92, and the macro F-score metric, with a score of 0 31  This was an improvement of 7% for the micro F-score over the next best model, SVM-L, and a marginal improvement of 5% for macro F-score  It is interesting to note that SVM with linear kernels performs much better than SVM with RBF kernel, scoring 9% on the macro F-score and 12% more on the micro F-score  It is suspected that since words used in primary diagnosis itself occur in some reports, thus enabling the linear models to outperform complex models 
Experiment Series 2
fig:keywords shows the top 50 keywords highlighted using TF-IDF and LDA  The proposed approach has performed well in highlighting the important regions, for example the topic highlighted with a red color containing “presence range tumor necrosis” provides useful biomarker information to readers 
Conclusions
We proposed a simple yet efficient TF-IDF method to extract and corroborate useful keywords from pathology cancer reports  Encoding a pathology report for cancer and tumor surveillance is a laborious task, and sometimes it is subjected to human errors and variability in the interpretation  One of the most important aspects of encoding a pathology report involves extracting the primary diagnosis  This may be very useful for content-based image retrieval to combine with visual information  We used existing classification model and TF-IDF features to predict the primary diagnosis  We achieved up to 92% accuracy using XGBoost classifier  The prediction accuracy empowers the adoption of machine learning methods for automated information extraction from pathology reports 

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. which languages are explored?
Context 3:  Overall, the 16 languages considered in our experiments are typologically, morphologically and syntactically fairly diverse 
Lexicons
We generate our external lexicons using the set of source lexicons listed in Table TABREF3   Since external lexical information is exploited via features, there is no need for the external lexicons and the annotated corpora to use the same PoS inventory  Therefore, for each language, we simply extracted from the corresponding lexicon the PoS of each word based on its morphological tags, by removing all information provided except for its coarsest-level category  We also added entries for punctuations when the source lexicons did not contain any 
We also performed experiments in which we retained the full original tags provided by the lexicons, with all morphological features included  On average, results were slightly better than those presented in the paper, although not statistically significantly  Moreover, the granularity of tag inventories in the lexicons is diverse, which makes it difficult to draw general conclusions about results based on full tags  This is why we only report results based on (coarse) PoS extracted from the original lexicons 
Baseline models
In order to assess the respective contributions of external lexicons and word vector representations, we first compared the results of the three above-mentioned systems when trained without such additional lexical information  Table TABREF11 provides the results of MElt and MarMoT retrained on UD1 2 corpora, together with the results publised on the same corpora by BIBREF20 , using their best model not enhanced by external word vector representations —i e  the model they call INLINEFORM0 , which is a bidirectional LSTM that combines both word and character embeddings 
These results show that Plank et al 's (2016) bi-LSTM performs extremely well, surpassed by MarMoT on only 3 out of 16 datasets (Czech, French and Italian), and by MElt only once (Indonesian) 
Models enriched with external lexical information
Table TABREF13 provides the results of four systems enriched with lexical information  The feature-based systems MElt and MarMoT, respectively based on MEMMs and CRFs, are extended with the lexical information provided by our morphosyntactic lexicons  This extension takes the form of additional features, as described in Section SECREF2 for MElt  The results reported by BIBREF20 for their bidirectional LSTM when initialised with Polyglot embeddings trained on full wikipedias are also included, together with their new system FREQBIN, also initialised with Polyglot embeddings  FREQBIN trains bi-LSTMs to predict for each input word both a PoS and a label that represents its log frequency in the training data  As they word it, “the idea behind this model is to make the representation predictive for frequency, which encourages the model not to share representations between common and rare words, thus benefiting the handling of rare tokens ”
The results, which are also displayed in Figures FIGREF14 and FIGREF15 , show that all systems reach very similar results on average, although discrepancies can be observed from one dataset to another, on which we shall comment shortly  The best performing system in terms of macro-average is MElt (96 60%)  Both bi-LSTM systems reach the same score (96 58%), the difference with MElt's results being non significant, whereas MarMoT is only 0 14% behind (96 46%)  Given the better baseline scores of the neural approaches, these results show that the benefit of using external lexicons in the feature-based models MElt and MarMoT are much higher than those using Polyglot word vector representations as initialisations for bi-LSTMs 
Yet these very similar overall results reflect a different picture when focusing on OOV tagging accuracy  The best models for OOV tagging accuracy are, by far, FREQBIN models, which are beaten by MarMoT and by MElt only once each (on English and Danish respectively)  The comparison on OOV tagging between MElt and MarMoT shows that MElt performs better on average than MarMoT, despite the fact that MarMoT's baseline results were better than those reached by MElt  This shows that the information provided by external morphosyntactic lexicons is better exploited by MElt's lexical features than by those used by MarMoT  On the other hand, the comparison of both bi-LSTM-based approaches confirm that the FREQBIN models is better by over 10% absolute on OOV tagging accuracy (94 28% vs  83 59%), with 65% lower error rate 
One of the important differences between the lexical information provided by an external lexicon and word vectors built from raw corpora, apart from the very nature of the lexical information provided, is the coverage and accuracy of this lexical information on rare words  All words in a morphosyntactic lexicon are associated with information of a same granularity and quality, which is not the case with word representations such as provided by Polyglot  Models that take advantage of external lexicons should therefore perform comparatively better on datasets containing a higher proportion of rarer words, provided the lexicons' coverage is high  In order to confirm this intuition, we have used a lexical richness metric based on the type/token ratio  Since this ratio is well-known for being sensitive to corpus length, we normalised it by computing it over the 60,000 first tokens of each training set  When this normalised type/token ratio is plotted against the difference between the results of MElt and both bi-LSTM-based models, the expected correlation is clearly visible (see Figure FIGREF16 )  This explains why MElt obtains better results on the morphologically richer Slavic datasets (average normalised type/token ratio: 0 28, average accuracy difference: 0 32 compared to both bi-LSTM+Polyglot and FREQBIN+Polyglot) and, at the other end of the spectrum, significantly worse results on the English dataset (normalised type/token ratio: 0 15, average accuracy difference: -0 56 compared to bi-LSTM+Polyglot, -0 57 compared to FREQBIN+Polyglot) 
Conclusion
Two main conclusions can be drawn from our comparative results  First, feature-based tagging models adequately enriched with external morphosyntactic lexicons perform, on average, as well as bi-LSTMs enriched with word embeddings  Per-language results show that the best accuracy levels are reached by feature-based models, and in particular by our improved version of the MEMM-based system MElt, on datasets with high lexical variability (in short, for morphologically rich languages), whereas neural-based results perform better on datatsets with lower lexical variability (e g  for English) 
We have only compared the contribution of morphosyntactic lexicons to feature-based models (MEMMs, CRFs) and that of word vector representations to bi-LSTM-based models as reported by BIBREF20   As mentioned above, work on the contribution of word vector representations to feature-based approaches has been carried out by BIBREF18   However, the exploitation of existing morphosyntactic or morphological lexicons in neural models is a less studied question  Improvements over the state of the art might be achieved by integrating lexical information both from an external lexicon and from word vector representations into tagging models 
In that regard, further work will be required to understand which class of models perform the best  An option would be to integrate feature-based models such as a CRF with an LSTM-based layer, following recent proposals such as the one proposed by BIBREF45 for named entity recognition 

ground_truth: Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish and Swedish

Bulgarian, Croatian, Czech, Danish, English, French, German, Indonesian, Italian, Norwegian, Persian, Polish, Portuguese, Slovenian, Spanish , Swedish