prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what pretrained word embeddings were used?
Context 1: Introduction
Twitter, a micro-blogging and social networking site has emerged as a platform where people express themselves and react to events in real-time  It is estimated that nearly 500 million tweets are sent per day   Twitter data is particularly interesting because of its peculiar nature where people convey messages in short sentences using hashtags, emoticons, emojis etc  In addition, each tweet has meta data like location and language used by the sender  It's challenging to analyze this data because the tweets might not be grammatically correct and the users tend to use informal and slang words all the time  Hence, this poses an interesting problem for NLP researchers  Any advances in using this abundant and diverse data can help understand and analyze information about a person, an event, a product, an organization or a country as a whole  Many notable use cases of the twitter can be found here 
Along the similar lines, The Task 1 of WASSA-2017 BIBREF0 poses a problem of finding emotion intensity of four emotions namely anger, fear, joy, sadness from tweets  In this paper, we describe our approach and experiments to solve this problem  The rest of the paper is laid out as follows: Section 2 describes the system architecture, Section 3 reports results and inference from different experiments, while Section 4 points to ways that the problem can be further explored 
Preprocessing
The preprocessing step modifies the raw tweets before they are passed to feature extraction  Tweets are processed using tweetokenize tool  Twitter specific features are replaced as follows: username handles to USERNAME, phone numbers to PHONENUMBER, numbers to NUMBER, URLs to URL and times to TIME  A continuous sequence of emojis is broken into individual tokens  Finally, all tokens are converted to lowercase 
Feature Extraction
Many tasks related to sentiment or emotion analysis depend upon affect, opinion, sentiment, sense and emotion lexicons  These lexicons associate words to corresponding sentiment or emotion metrics  On the other hand, the semantic meaning of words, sentences, and documents are preserved and compactly represented using low dimensional vectors BIBREF1 instead of one hot encoding vectors which are sparse and high dimensional  Finally, there are traditional NLP features like word N-grams, character N-grams, Part-Of-Speech N-grams and word clusters which are known to perform well on various tasks 
Based on these observations, the feature extraction step is implemented as a union of different independent feature extractors (featurizers) in a light-weight and easy to use Python program EmoInt   It comprises of all features available in the baseline model BIBREF2 along with additional feature extractors and bi-gram support  Fourteen such feature extractors have been implemented which can be clubbed into 3 major categories:
[noitemsep]
Lexicon Features
Word Vectors
Syntax Features
Lexicon Features: AFINN BIBREF3 word list are manually rated for valence with an integer between -5 (Negative Sentiment) and +5 (Positive Sentiment)  Bing Liu BIBREF4 opinion lexicon extract opinion on customer reviews  +/-EffectWordNet BIBREF5 by MPQA group are sense level lexicons  The NRC Affect Intensity BIBREF6 lexicons provide real valued affect intensity  NRC Word-Emotion Association Lexicon BIBREF7 contains 8 sense level associations (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and 2 sentiment level associations (negative and positive)  Expanded NRC Word-Emotion Association Lexicon BIBREF8 expands the NRC word-emotion association lexicon for twitter specific language  NRC Hashtag Emotion Lexicon BIBREF9 contains emotion word associations computed on emotion labeled twitter corpus via Hashtags  NRC Hashtag Sentiment Lexicon and Sentiment140 Lexicon BIBREF10 contains sentiment word associations computed on twitter corpus via Hashtags and Emoticons  SentiWordNet BIBREF11 assigns to each synset of WordNet three sentiment scores: positivity, negativity, objectivity  Negation lexicons collections are used to count the total occurrence of negative words  In addition to these, SentiStrength BIBREF12 application which estimates the strength of positive and negative sentiment from tweets is also added 
Word Vectors: We focus primarily on the word vector representations (word embeddings) created specifically using the twitter dataset  GloVe BIBREF13 is an unsupervised learning algorithm for obtaining vector representations for words  200-dimensional GloVe embeddings trained on 2 Billion tweets are integrated  Edinburgh embeddings BIBREF14 are obtained by training skip-gram model on Edinburgh corpus BIBREF15   Since tweets are abundant with emojis, Emoji embeddings BIBREF16 which are learned from the emoji descriptions have been used  Embeddings for each tweet are obtained by summing up individual word vectors and then dividing by the number of tokens in the tweet 
Syntactic Features: Syntax specific features such as Word N-grams, Part-Of-Speech N-grams BIBREF17 , Brown Cluster N-grams BIBREF18 obtained using TweetNLP project have been integrated into the system 
The final feature vector is the concatenation of all the individual features  For example, we concatenate average word vectors, sum of NRC Affect Intensities, number of positive and negative Bing Liu lexicons, number of negation words and so on to get final feature vector  The scaling of final features is not required when used with gradient boosted trees  However, scaling steps like standard scaling (zero mean and unit normal) may be beneficial for neural networks as the optimizers work well when the data is centered around origin 
A total of fourteen different feature extractors have been implemented, all of which can be enabled or disabled individually to extract features from a given tweet 
Regression
The dev data set BIBREF19 in the competition was small hence, the train and dev sets were merged to perform 10-fold cross validation  On each fold, a model was trained and the predictions were collected on the remaining dataset  The predictions are averaged across all the folds to generalize the solution and prevent over-fitting  As described in Section SECREF6 , different combinations of feature extractors were used  After performing feature extraction, the data was then passed to various regressors Support Vector Regression, AdaBoost, RandomForestRegressor, and, BaggingRegressor of sklearn BIBREF20   Finally, the chosen top performing models had the least error on evaluation metrics namely Pearson's Correlation Coefficient and Spearman's rank-order correlation 
Parameter Optimization
In order to find the optimal parameter values for the EmoInt system, an extensive grid search was performed through the scikit-Learn framework over all subsets of the training set (shuffled), using stratified 10-fold cross validation and optimizing the Pearson's Correlation score  Best cross-validation results were obtained using AdaBoost meta regressor with base regressor as XGBoost BIBREF21 with 1000 estimators and 0 1 learning rate  Experiments and analysis of results are presented in the next section 
Experimental Results
As described in Section SECREF6 various syntax features were used namely, Part-of-Speech tags, brown clusters of TweetNLP project  However, these didn't perform well in cross validation  Hence, they were dropped from the final system  While performing grid-search as mentioned in Section SECREF14 , keeping all the lexicon based features same, choice of combination of emoji vector and word vectors are varied to minimize cross validation metric  Table TABREF16 describes the results for experiments conducted with different combinations of word vectors  Emoji embeddings BIBREF16 give better results than using plain GloVe and Edinburgh embeddings  Edinburgh embeddings outperform GloVe embeddings in Joy and Sadness category but lag behind in Anger and Fear category  The official submission comprised of the top-performing model for each emotion category  This system ranked 3 for the entire test dataset and 2 for the subset of the test data formed by taking every instance with a gold emotion intensity score greater than or equal to 0 5  Post competition, experiments were performed on ensembling diverse models for improving the accuracy  An ensemble obtained by averaging the results of the top 2 performing models outperforms all the individual models 
Feature Importance
The relative feature importance can be assessed by the relative depth of the feature used as a decision node in the tree  Features used at the top of the tree contribute to the final prediction decision of a larger fraction of the input samples  The expected fraction of the samples they contribute to can thus be used as an estimate of the relative importance of the features  By averaging the measure over several randomized trees, the variance of the estimate can be reduced and used as a measure of relative feature importance  In Figure FIGREF18 feature importance graphs are plotted for each emotion to infer which features are playing the major role in identifying emotional intensity in tweets  +/-EffectWordNet BIBREF5 , NRC Hashtag Sentiment Lexicon, Sentiment140 Lexicon BIBREF10 and NRC Hashtag Emotion Lexicon BIBREF9 are playing the most important role 
System Limitations
It is important to understand how the model performs in different scenarios  Table TABREF20 analyzes when the system performs the best and worst for each emotion  Since the features used are mostly lexicon based, the system has difficulties in capturing the overall sentiment and it leads to amplifying or vanishing intensity signals  For instance, in example 4 of fear louder and shaking lexicons imply fear but overall sentence doesn't imply fear  A similar pattern can be found in the 4 example of Anger and 3 example of Joy  The system has difficulties in understanding of sarcastic tweets, for instance, in the 3 tweet of Anger the user expressed anger but used lol which is used in a positive sense most of the times and hence the system did a bad job at predicting intensity  The system also fails in predicting sentences having deeper emotion and sentiment which humans can understand with a little context  For example, in sample 4 of sadness, the tweet refers to post travel blues which humans can understand  But with little context, it is difficult for the system to accurately estimate the intensity  The performance is poor with very short sentences as there are fewer indicators to provide a reasonable estimate 
Future Work & Conclusion
The paper studies the effectiveness of various affect lexicons word embeddings to estimate emotional intensity in tweets  A light-weight easy to use affect computing framework (EmoInt) to facilitate ease of experimenting with various lexicon features for text tasks is open-sourced  It provides plug and play access to various feature extractors and handy scripts for creating ensembles 
Few problems explained in the analysis section can be resolved with the help of sentence embeddings which take the context information into consideration  The features used in the system are generic enough to use them in other affective computing tasks on social media text, not just tweet data  Another interesting feature of lexicon-based systems is their good run-time performance during prediction, future work to benchmark the performance of the system can prove vital for deploying in a real-world setting 
Acknowledgement
We would like to thank the organizers of the WASSA-2017 Shared Task on Emotion Intensity, for providing the data, the guidelines and timely support 
Context 2: Introduction
Cancer is one of the leading causes of death in the world, with over 80,000 deaths registered in Canada in 2017 (Canadian Cancer Statistics 2017)  A computer-aided system for cancer diagnosis usually involves a pathologist rendering a descriptive report after examining the tissue glass slides obtained from the biopsy of a patient  A pathology report contains specific analysis of cells and tissues, and other histopathological indicators that are crucial for diagnosing malignancies  An average sized laboratory may produces a large quantity of pathology reports annually (e g , in excess of 50,000), but these reports are written in mostly unstructured text and with no direct link to the tissue sample  Furthermore, the report for each patient is a personalized document and offers very high variability in terminology due to lack of standards and may even include misspellings and missing punctuation, clinical diagnoses interspersed with complex explanations, different terminology to label the same malignancy, and information about multiple carcinoma appearances included in a single report BIBREF0  
In Canada, each Provincial and Territorial Cancer Registry (PTCR) is responsible for collecting the data about cancer diseases and reporting them to Statistics Canada (StatCan)  Every year, Canadian Cancer Registry (CCR) uses the information sources of StatCan to compile an annual report on cancer and tumor diseases  Many countries have their own cancer registry programs  These programs rely on the acquisition of diagnostic, treatment, and outcome information through manual processing and interpretation from various unstructured sources (e g , pathology reports, autopsy/laboratory reports, medical billing summaries)  The manual classification of cancer pathology reports is a challenging, time-consuming task and requires extensive training BIBREF0  
With the continued growth in the number of cancer patients, and the increase in treatment complexity, cancer registries face a significant challenge in manually reviewing the large quantity of reports BIBREF1 , BIBREF0   In this situation, Natural Language Processing (NLP) systems can offer a unique opportunity to automatically encode the unstructured reports into structured data  Since, the registries already have access to the large quantity of historically labeled and encoded reports, a supervised machine learning approach of feature extraction and classification is a compelling direction for making their workflow more effective and streamlined  If successful, such a solution would enable processing reports in much lesser time allowing trained personnel to focus on their research and analysis  However, developing an automated solution with high accuracy and consistency across wide variety of reports is a challenging problem 
For cancer registries, an important piece of information in a pathology report is the associated ICD-O code which describes the patient's histological diagnosis, as described by the World Health Organization's (WHO) International Classification of Diseases for Oncology BIBREF2   Prediction of the primary diagnosis from a pathology report provides a valuable starting point for exploration of machine learning techniques for automated cancer surveillance  A major application for this purpose would be “auto-reporting” based on analysis of whole slide images, the digitization of the biopsy glass slides  Structured, summarized and categorized reports can be associated with the image content when searching in large archives  Such as system would be able to drastically increase the efficiency of diagnostic processes for the majority of cases where in spite of obvious primary diagnosis, still time and effort is required from the pathologists to write a descriptive report 
The primary objective of our study is to analyze the efficacy of existing machine learning approaches for the automated classification of pathology reports into different diagnosis categories  We demonstrate that TF-IDF feature vectors combined with linear SVM or XGBoost classifier can be an effective method for classification of the reports, achieving up to 83% accuracy  We also show that TF-IDF features are capable of identifying important keywords within a pathology report  Furthermore, we have created a new dataset consisting of 1,949 pathology reports across 37 primary diagnoses  Taken together, our exploratory experiments with a newly introduced dataset on pathology reports opens many new opportunities for researchers to develop a scalable and automatic information extraction from unstructured pathology reports 
Background
NLP approaches for information extraction within the biomedical research areas range from rule-based systems BIBREF3 , to domain-specific systems using feature-based classification BIBREF1 , to the recent deep networks for end-to-end feature extraction and classification BIBREF0   NLP has had varied degree of success with free-text pathology reports BIBREF4   Various studies have acknowledge the success of NLP in interpreting pathology reports, especially for classification tasks or extracting a single attribute from a report BIBREF4 , BIBREF5  
The Cancer Text Information Extraction System (caTIES) BIBREF6 is a framework developed in a caBIG project focuses on information extraction from pathology reports  Specifically, caTIES extracts information from surgical pathology reports (SPR) with good precision as well as recall 
Another system known as Open Registry BIBREF7 is capable of filtering the reports with disease codes containing cancer  In BIBREF8 , an approach called Automated Retrieval Console (ARC) is proposed which uses machine learning models to predict the degree of association of a given pathology or radiology with the cancer  The performance ranges from an F-measure of 0 75 for lung cancer to 0 94 for colorectal cancer  However, ARC uses domain-specific rules which hiders with the generalization of the approach to variety of pathology reports 
This research work is inspired by themes emerging in many of the above studies  Specifically, we are evaluating the task of predicting the primary diagnosis from the pathology report  Unlike previous approaches, the system does not rely on custom rule-based knowledge, domain specific features, balanced dataset with fewer number of classes 
Materials and Methods
We assembled a dataset of 1,949 cleaned pathology reports  Each report is associated with one of the 37 different primary diagnoses based on IDC-O codes  The reports are collected from four different body parts or primary sites from multiple patients  The distribution of reports across different primary diagnoses and primary sites is reported in tab:report-distribution  The dataset was developed in three steps as follows 
Collecting pathology reports: The total of 11,112 pathology reports were downloaded from NCI's Genomic Data Commons (GDC) dataset in PDF format BIBREF9   Out of all PDF files, 1,949 reports were selected across multiple patients from four specific primary sites—thymus, testis, lung, and kidney  The selection was primarily made based on the quality of PDF files 
Cleaning reports: The next step was to extract the text content from these reports  Due to the significant time expense of manually re-typing all the pathology reports, we developed a new strategy to prepare our dataset  We applied an Optical Character Recognition (OCR) software to convert the PDF reports to text files  Then, we manually inspected all generated text files to fix any grammar/spelling issues and irrelevant characters as an artefact produced by the OCR system 
Splitting into training-testing data: We split the cleaned reports into 70% and 30% for training and testing, respectively  This split resulted in 1,364 training, and 585 testing reports 
Pre-Processing of Reports
We pre-processed the reports by setting their text content to lowercase and filtering out any non-alphanumeric characters  We used NLTK library to remove stopping words, e g , `the', `an', `was', `if' and so on BIBREF10   We then analyzed the reports to find common bigrams, such as “lung parenchyma”, “microscopic examination”, “lymph node” etc  We joined the biagrams with a hyphen, converting them into a single word  We further removed the words that occur less than 2% in each of the diagnostic category  As well, we removed the words that occur more than 90% across all the categories  We stored each pre-processed report in a separate text file 
TF-IDF features
TF-IDF stands for Term Frequency-Inverse Document Frequency, and it is a useful weighting scheme in information retrieval and text mining  TF-IDF signifies the importance of a term in a document within a corpus  It is important to note that a document here refers to a pathology report, a corpus refers to the collection of reports, and a term refers to a single word in a report  The TF-IDF weight for a term INLINEFORM0 in a document INLINEFORM1 is given by DISPLAYFORM0
We performed the following steps to transform a pathology report into a feature vector:
Create a set of vocabulary containing all unique words from all the pre-processed training reports 
Create a zero vector INLINEFORM0 of the same length as the vocabulary 
For each word INLINEFORM0 in a report INLINEFORM1 , set the corresponding index in INLINEFORM2 to INLINEFORM3  
The resultant INLINEFORM0 is a feature vector for the report INLINEFORM1 and it is a highly sparse vector 
Keyword extraction and topic modelling
The keyword extraction involves identifying important words within reports that summarizes its content  In contrast, the topic modelling allows grouping these keywords using an intelligent scheme, enabling users to further focus on certain aspects of a document  All the words in a pathology report are sorted according to their TF-IDF weights  The top INLINEFORM0 sorted words constitute the top INLINEFORM1 keywords for the report  The INLINEFORM2 is empirically set to 50 within this research  The extracted keywords are further grouped into different topics by using latent Dirichlet allocation (LDA) BIBREF11   The keywords in a report are highlighted using the color theme based on their topics 
Evaluation metrics
Each model is evaluated using two standard NLP metrics—micro and macro averaged F-scores, the harmonic mean of related metrics precision and recall  For each diagnostic category INLINEFORM0 from a set of 37 different classes INLINEFORM1 , the number of true positives INLINEFORM2 , false positives INLINEFORM3 , and false negatives INLINEFORM4 , the micro F-score is defined as DISPLAYFORM0
whereas macro F-score is given by DISPLAYFORM0
In summary, micro-averaged metrics have class representation roughly proportional to their test set representation (same as accuracy for classification problem with a single label per data point), whereas macro-averaged metrics are averaged by class without weighting by class prevalence BIBREF12  
Experimental setting
In this study, we performed two different series of experiments: i) evaluating the performance of TF-IDF features and various machine learning classifiers on the task of predicting primary diagnosis from the text content of a given report, and ii) using TF-IDF and LDA techniques to highlight the important keywords within a report  For the first experiment series, training reports are pre-processed, then their TF-IDF features are extracted  The TF-IDF features and the training labels are used to train different classification models  These different classification models and their hyper-parameters are reported in tab:classifier  The performance of classifiers is measured quantitatively on the test dataset using the evaluation metrics discussed in the previous section  For the second experiment series, a random report is selected and its top 50 keywords are extracted using TF-IDF weights  These 50 keywords are highlighted using different colors based on their associated topic, which are extracted through LDA  A non-expert based qualitative inspection is performed on the extracted keywords and their corresponding topics 
Experiment Series 1
A classification model is trained to predict the primary diagnosis given the content of the cancer pathology report  The performance results on this task are reported in tab:results  We can observe that the XGBoost classifier outperformed all other models for both the micro F-score metric, with a score of 0 92, and the macro F-score metric, with a score of 0 31  This was an improvement of 7% for the micro F-score over the next best model, SVM-L, and a marginal improvement of 5% for macro F-score  It is interesting to note that SVM with linear kernels performs much better than SVM with RBF kernel, scoring 9% on the macro F-score and 12% more on the micro F-score  It is suspected that since words used in primary diagnosis itself occur in some reports, thus enabling the linear models to outperform complex models 
Experiment Series 2
fig:keywords shows the top 50 keywords highlighted using TF-IDF and LDA  The proposed approach has performed well in highlighting the important regions, for example the topic highlighted with a red color containing “presence range tumor necrosis” provides useful biomarker information to readers 
Conclusions
We proposed a simple yet efficient TF-IDF method to extract and corroborate useful keywords from pathology cancer reports  Encoding a pathology report for cancer and tumor surveillance is a laborious task, and sometimes it is subjected to human errors and variability in the interpretation  One of the most important aspects of encoding a pathology report involves extracting the primary diagnosis  This may be very useful for content-based image retrieval to combine with visual information  We used existing classification model and TF-IDF features to predict the primary diagnosis  We achieved up to 92% accuracy using XGBoost classifier  The prediction accuracy empowers the adoption of machine learning methods for automated information extraction from pathology reports 
Context 3: Introduction
This paper describes our approach and results for Task 2 of the CoNLL–SIGMORPHON 2018 shared task on universal morphological reinflection BIBREF0   The task is to generate an inflected word form given its lemma and the context in which it occurs 
Morphological (re)inflection from context is of particular relevance to the field of computational linguistics: it is compelling to estimate how well a machine-learned system can capture the morphosyntactic properties of a word given its context, and map those properties to the correct surface form for a given lemma 
There are two tracks of Task 2 of CoNLL–SIGMORPHON 2018: in Track 1 the context is given in terms of word forms, lemmas and morphosyntactic descriptions (MSD); in Track 2 only word forms are available  See Table TABREF1 for an example  Task 2 is additionally split in three settings based on data size: high, medium and low, with high-resource datasets consisting of up to 70K instances per language, and low-resource datasets consisting of only about 1K instances 
The baseline provided by the shared task organisers is a seq2seq model with attention (similar to the winning system for reinflection in CoNLL–SIGMORPHON 2016, BIBREF1 ), which receives information about context through an embedding of the two words immediately adjacent to the target form  We use this baseline implementation as a starting point and achieve the best overall accuracy of 49 87 on Task 2 by introducing three augmentations to the provided baseline system: (1) We use an LSTM to encode the entire available context; (2) We employ a multi-task learning approach with the auxiliary objective of MSD prediction; and (3) We train the auxiliary component in a multilingual fashion, over sets of two to three languages 
In analysing the performance of our system, we found that encoding the full context improves performance considerably for all languages: 11 15 percentage points on average, although it also highly increases the variance in results  Multi-task learning, paired with multilingual training and subsequent monolingual finetuning, scored highest for five out of seven languages, improving accuracy by another 9 86% on average 
System Description
Our system is a modification of the provided CoNLL–SIGMORPHON 2018 baseline system, so we begin this section with a reiteration of the baseline system architecture, followed by a description of the three augmentations we introduce 
Baseline
The CoNLL–SIGMORPHON 2018 baseline is described as follows:
The system is an encoder-decoder on character sequences  It takes a lemma as input and generates a word form  The process is conditioned on the context of the lemma [   ] The baseline treats the lemma, word form and MSD of the previous and following word as context in track 1  In track 2, the baseline only considers the word forms of the previous and next word  [   ] The baseline system concatenates embeddings for context word forms, lemmas and MSDs into a context vector  The baseline then computes character embeddings for each character in the input lemma  Each of these is concatenated with a copy of the context vector  The resulting sequence of vectors is encoded using an LSTM encoder  Subsequently, an LSTM decoder generates the characters in the output word form using encoder states and an attention mechanism 
To that we add a few details regarding model size and training schedule:
the number of LSTM layers is one;
embedding size, LSTM layer size and attention layer size is 100;
models are trained for 20 epochs;
on every epoch, training data is subsampled at a rate of 0 3;
LSTM dropout is applied at a rate 0 3;
context word forms are randomly dropped at a rate of 0 1;
the Adam optimiser is used, with a default learning rate of 0 001; and
trained models are evaluated on the development data (the data for the shared task comes already split in train and dev sets) 
Our system
Here we compare and contrast our system to the baseline system  A diagram of our system is shown in Figure FIGREF4  
The idea behind this modification is to provide the encoder with access to all morpho-syntactic cues present in the sentence  In contrast to the baseline, which only encodes the immediately adjacent context of a target word, we encode the entire context  All context word forms, lemmas, and MSD tags (in Track 1) are embedded in their respective high-dimensional spaces as before, and their embeddings are concatenated  However, we now reduce the entire past context to a fixed-size vector by encoding it with a forward LSTM, and we similarly represent the future context by encoding it with a backwards LSTM 
We introduce an auxiliary objective that is meant to increase the morpho-syntactic awareness of the encoder and to regularise the learning process—the task is to predict the MSD tag of the target form  MSD tag predictions are conditioned on the context encoding, as described in UID15   Tags are generated with an LSTM one component at a time, e g  the tag PRO;NOM;SG;1 is predicted as a sequence of four components, INLINEFORM0 PRO, NOM, SG, 1 INLINEFORM1  
For every training instance, we backpropagate the sum of the main loss and the auxiliary loss without any weighting 
As MSD tags are only available in Track 1, this augmentation only applies to this track 
The parameters of the entire MSD (auxiliary-task) decoder are shared across languages 
Since a grouping of the languages based on language family would have left several languages in single-member groups (e g  Russian is the sole representative of the Slavic family), we experiment with random groupings of two to three languages  Multilingual training is performed by randomly alternating between languages for every new minibatch  We do not pass any information to the auxiliary decoder as to the source language of the signal it is receiving, as we assume abstract morpho-syntactic features are shared across languages 
After 20 epochs of multilingual training, we perform 5 epochs of monolingual finetuning for each language  For this phase, we reduce the learning rate to a tenth of the original learning rate, i e  0 0001, to ensure that the models are indeed being finetuned rather than retrained 
We keep all hyperparameters the same as in the baseline

ground_truth: Pretrained word embeddings  were not used

GloVe, Edinburgh embeddings BIBREF14, Emoji embeddings BIBREF16

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What features are used?
Context 4: Introduction
Question answering (QA) has been a blooming research field for the last decade  Selection-based QA implies a family of tasks that find answer contexts from large data given questions in natural language  Three tasks have been proposed for selection-based QA  Given a document, answer extraction BIBREF0 , BIBREF1 finds answer phrases whereas answer selection BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 and answer triggering BIBREF6 , BIBREF7 find answer sentences instead, although the presence of the answer context is not assumed within the provided document for answer triggering but it is for the other two tasks  Recently, various QA tasks that are not selection-based have been proposed BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 ; however, selection-based QA remains still important because of its practical value to real applications (e g , IBM Watson, MIT Start) 
Several datasets have been released for selection-based QA  wang:07a created the QASent dataset consisting of 277 questions, which has been widely used for benchmarking the answer selection task  feng:15a presented InsuranceQA comprising 16K+ questions on insurance contexts  yang:15a introduced WikiQA for answer selection and triggering  jurczyk:16 created SelQA for large real-scale answer triggering  rajpurkar2016squad presented SQuAD for answer extraction and selection as well as for reading comprehension  Finally, morales-EtAl:2016:EMNLP2016 provided InfoboxQA for answer selection 
These corpora make it possible to evaluate the robustness of statistical question answering learning  Although all of these corpora target on selection-based QA, they are designed for different purposes such that it is important to understand the nature of these corpora so a better use of them can be made  In this paper, we make both intrinsic and extrinsic analyses of four latest corpora based on Wikipedia, WikiQA, SelQA, SQuAD, and InfoboxQA  We first give a thorough intrinsic analysis regarding contextual similarities, question types, and answer categories (Section SECREF2 )  We then map questions in all corpora to the current version of English Wikipedia and benchmark another selection-based QA task, answer retrieval (Section SECREF3 )  Finally, we present an extrinsic analysis through a set of experiments cross-testing these corpora using a convolutional neural network architecture (Section SECREF4 ) 
Intrinsic Analysis
Four publicly available corpora are selected for our analysis  These corpora are based on Wikipedia, so more comparable than the others, and have already been used for the evaluation of several QA systems 
WikiQA BIBREF6 comprises questions selected from the Bing search queries, where user click data give the questions and their corresponding Wikipedia articles  The abstracts of these articles are then extracted to create answer candidates  The assumption is made that if many queries lead to the same article, it must contain the answer context; however, this assumption fails for some occasions, which makes this dataset more challenging  Since the existence of answer contexts is not guaranteed in this task, it is called answer triggering instead of answer selection 
SelQA BIBREF7 is a product of five annotation tasks through crowdsourcing  It consists of about 8K questions where a half of the questions are paraphrased from the other half, aiming to reduce contextual similarities between questions and answers  Each question is associated with a section in Wikipedia where the answer context is guaranteed, and also with five sections selected from the entire Wikipedia where the selection is made by the Lucene search engine  This second dataset does not assume the existence of the answer context, so can be used for the evaluation of answer triggering 
SQuAD BIBREF12 presents 107K+ crowdsourced questions on 536 Wikipedia articles, where the answer contexts are guaranteed to exist within the provided paragraph  It contains annotation of answer phrases as well as the pointers to the sentences including the answer phrases; thus, it can be used for both answer extraction and selection  This corpus also provides human accuracy on those questions, setting up a reasonable upper bound for machines  To avoid overfitting, the evaluation set is not publicly available although system outputs can be evaluated by their provided script 
InfoboxQA BIBREF13 gives 15K+ questions based on the infoboxes from 150 articles in Wikipedia  Each question is crowdsourced and associated with an infobox, where each line of the infobox is considered an answer candidate  This corpus emphasizes the gravity of infoboxes, which summary arguably the most commonly asked information about those articles  Although the nature of this corpus is different from the others, it can also be used to evaluate answer selection 
Analysis
All corpora provide datasets/splits for answer selection, whereas only (WikiQA, SQuAD) and (WikiQA, SelQA) provide datasets for answer extraction and answer triggering, respectively  SQuAD is much larger in size although questions in this corpus are often paraphrased multiple times  On the contrary, SQuAD's average candidates per question ( INLINEFORM0 ) is the smallest because SQuAD extracts answer candidates from paragraphs whereas the others extract them from sections or infoboxes that consist of bigger contexts  Although InfoboxQA is larger than WikiQA or SelQA, the number of token types ( INLINEFORM1 ) in InfoboxQA is smaller than those two, due to the repetitive nature of infoboxes 
All corpora show similar average answer candidate lengths ( INLINEFORM0 ), except for InfoboxQA where each line in the infobox is considered a candidate  SelQA and SQuAD show similar average question lengths ( INLINEFORM1 ) because of the similarity between their annotation schemes  It is not surprising that WikiQA's average question length is the smallest, considering their questions are taken from search queries  InfoboxQA's average question length is relatively small, due to the restricted information that can be asked from the infoboxes  InfoboxQA and WikiQA show the least question-answer word overlaps over questions and answers ( INLINEFORM2 and INLINEFORM3 in Table TABREF2 ), respectively  In terms of the F1-score for overlapping words ( INLINEFORM4 ), SQuAD gives the least portion of overlaps between question-answer pairs although WikiQA comes very close 
Fig

ground_truth: Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What architecture does the decoder have?
Context 3:  Training data is split 90:10 for training and validation  We train our models for 50 epochs, adding early stopping with a tolerance of five epochs of no improvement in the validation loss  We do not subsample from the training data 
We train models for 50 different random combinations of two to three languages in Track 1, and 50 monolingual models for each language in Track 2  Instead of picking the single model that performs best on the development set and thus risking to select a model that highly overfits that data, we use an ensemble of the five best models, and make the final prediction for a given target form with a majority vote over the five predictions 
Results and Discussion
Test results are listed in Table TABREF17   Our system outperforms the baseline for all settings and languages in Track 1 and for almost all in Track 2—only in the high resource setting is our system not definitively superior to the baseline 
Interestingly, our results in the low resource setting are often higher for Track 2 than for Track 1, even though contextual information is less explicit in the Track 2 data and the multilingual multi-tasking approach does not apply to this track  We interpret this finding as an indicator that a simpler model with fewer parameters works better in a setting of limited training data  Nevertheless, we focus on the low resource setting in the analysis below due to time limitations  As our Track 1 results are still substantially higher than the baseline results, we consider this analysis valid and insightful 
Ablation Study
We analyse the incremental effect of the different features in our system, focusing on the low-resource setting in Track 1 and using development data 
Encoding the entire context with an LSTM highly increases the variance of the observed results  So we trained fifty models for each language and each architecture  Figure FIGREF23 visualises the means and standard deviations over the trained models  In addition, we visualise the average accuracy for the five best models for each language and architecture, as these are the models we use in the final ensemble prediction  Below we refer to these numbers only 
The results indicate that encoding the full context with an LSTM highly enhances the performance of the model, by 11 15% on average  This observation explains the high results we obtain also for Track 2 
Adding the auxiliary objective of MSD prediction has a variable effect: for four languages (de, en, es, and sv) the effect is positive, while for the rest it is negative  We consider this to be an issue of insufficient data for the training of the auxiliary component in the low resource setting we are working with 
We indeed see results improving drastically with the introduction of multilingual training, with multilingual results being 7 96% higher than monolingual ones on average 
We studied the five best models for each language as emerging from the multilingual training (listed in Table TABREF27 ) and found no strong linguistic patterns  The en–sv pairing seems to yield good models for these languages, which could be explained in terms of their common language family and similar morphology  The other natural pairings, however, fr–es, and de–sv, are not so frequent among the best models for these pairs of languages 
Finally, monolingual finetuning improves accuracy across the board, as one would expect, by 2 72% on average 
The final observation to be made based on this breakdown of results is that the multi-tasking approach paired with multilingual training and subsequent monolingual finetuning outperforms the other architectures for five out of seven languages: de, en, fr, ru and sv  For the other two languages in the dataset, es and fi, the difference between this approach and the approach that emerged as best for them is less than 1%  The overall improvement of the multilingual multi-tasking approach over the baseline is 18 30% 
Error analysis
Here we study the errors produced by our system on the English test set to better understand the remaining shortcomings of the approach  A small portion of the wrong predictions point to an incorrect interpretation of the morpho-syntactic conditioning of the context, e g  the system predicted plan instead of plans in the context Our _ include raising private capital  The majority of wrong predictions, however, are nonsensical, like bomb for job, fify for fixing, and gnderrate for understand  This observation suggests that generally the system did not learn to copy the characters of lemma into inflected form, which is all it needs to do in a large number of cases  This issue could be alleviated with simple data augmentation techniques that encourage autoencoding BIBREF2  
MSD prediction
Figure FIGREF32 summarises the average MSD-prediction accuracy for the multi-tasking experiments discussed above  Accuracy here is generally higher than on the main task, with the multilingual finetuned setup for Spanish and the monolingual setup for French scoring best: 66 59% and 65 35%, respectively  This observation illustrates the added difficulty of generating the correct surface form even when the morphosyntactic description has been identified correctly 
We observe some correlation between these numbers and accuracy on the main task: for de, en, ru and sv, the brown, pink and blue bars here pattern in the same way as the corresponding INLINEFORM0 's in Figure FIGREF23   One notable exception to this pattern is fr where inflection gains a lot from multilingual training, while MSD prediction suffers greatly  Notice that the magnitude of change is not always the same, however, even when the general direction matches: for ru, for example, multilingual training benefits inflection much more than in benefits MSD prediction, even though the MSD decoder is the only component that is actually shared between languages  This observation illustrates the two-fold effect of multi-task training: an auxiliary task can either inform the main task through the parameters the two tasks share, or it can help the main task learning through its regularising effect 
Related Work
Our system is inspired by previous work on multi-task learning and multi-lingual learning, mainly building on two intuitions: (1) jointly learning related tasks tends to be beneficial BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 ; and (2) jointly learning related languages in an MTL-inspired framework tends to be beneficial BIBREF8 , BIBREF9 , BIBREF10   In the context of computational morphology, multi-lingual approaches have previously been employed for morphological reinflection BIBREF2 and for paradigm completion BIBREF11   In both of these cases, however, the available datasets covered more languages, 40 and 21, respectively, which allowed for linguistically-motivated language groupings and for parameter sharing directly on the level of characters  BIBREF10 explore parameter sharing between related languages for dependency parsing, and find that sharing is more beneficial in the case of closely related languages 
Conclusions
In this paper we described our system for the CoNLL–SIGMORPHON 2018 shared task on Universal Morphological Reinflection, Task 2, which achieved the best performance out of all systems submitted, an overall accuracy of 49 87  We showed in an ablation study that this is due to three core innovations, which extend a character-based encoder-decoder model: (1) a wide context window, encoding the entire available context; (2) multi-task learning with the auxiliary task of MSD prediction, which acts as a regulariser; (3) a multilingual approach, exploiting information across languages  In future work we aim to gain better understanding of the increase in variance of the results introduced by each of our modifications and the reasons for the varying effect of multi-task learning for different languages 
Acknowledgements
We gratefully acknowledge the support of the NVIDIA Corporation with the donation of the Titan Xp GPU used for this research 

ground_truth: LSTM

LSTM