prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What types of social media did they consider?
Context 1: Introduction
Explanations of happenings in one's life, causal explanations, are an important topic of study in social, psychological, economic, and behavioral sciences  For example, psychologists have analyzed people's causal explanatory style BIBREF0 and found strong negative relationships with depression, passivity, and hostility, as well as positive relationships with life satisfaction, quality of life, and length of life BIBREF1 , BIBREF2 , BIBREF0  
To help understand the significance of causal explanations, consider how they are applied to measuring optimism (and its converse, pessimism) BIBREF0   For example, in “My parser failed because I always have bugs ”, the emphasized text span is considered a causal explanation which indicates pessimistic personality – a negative event where the author believes the cause is pervasive  However, in “My parser failed because I barely worked on the code ”, the explanation would be considered a signal of optimistic personality – a negative event for which the cause is believed to be short-lived 
Language-based models which can detect causal explanations from everyday social media language can be used for more than automating optimism detection  Language-based assessments would enable other large-scale downstream tasks: tracking prevailing causal beliefs (e g , about climate change or autism), better extracting process knowledge from non-fiction (e g , gravity causes objects to move toward one another), or detecting attribution of blame or praise in product or service reviews (“I loved this restaurant because the fish was cooked to perfection”) 
In this paper, we introduce causal explanation analysis and its subtasks of detecting the presence of causality (causality prediction) and identifying explanatory phrases (causal explanation identification)  There are many challenges to achieving these task  First, the ungrammatical texts in social media incur poor syntactic parsing results which drastically affect the performance of discourse relation parsing pipelines   Many causal relations are implicit and do not contain any discourse markers (e g , `because')  Further, Explicit causal relations are also more difficult in social media due to the abundance of abbreviations and variations of discourse connectives (e g , `cuz' and `bcuz') 
Prevailing approaches for social media analyses, utilizing traditional linear models or bag of words models (e g , SVM trained with n-gram, part-of-speech (POS) tags, or lexicon-based features) alone do not seem appropriate for this task since they simply cannot segment the text into meaningful discourse units or discourse arguments such as clauses or sentences rather than random consecutive token sequences or specific word tokens  Even when the discourse units are clear, parsers may still fail to accurately identify discourse relations since the content of social media is quite different than that of newswire which is typically used for discourse parsing 
In order to overcome these difficulties of discourse relation parsing in social media, we simplify and minimize the use of syntactic parsing results and capture relations between discourse arguments, and investigate the use of a recursive neural network model (RNN)  Recent work has shown that RNNs are effective for utilizing discourse structures for their downstream tasks BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , but they have yet to be directly used for discourse relation prediction in social media  We evaluated our model by comparing it to off-the-shelf end-to-end discourse relation parsers and traditional models  We found that the SVM and random forest classifiers work better than the LSTM classifier for the causality detection, while the LSTM classifier outperforms other models for identifying causal explanation 
The contributions of this work include: (1) the proposal of models for both (a) causality prediction and (b) causal explanation identification, (2) the extensive evaluation of a variety of models from social media classification models and discourse relation parsers to RNN-based application models, demonstrating that feature-based models work best for causality prediction while RNNs are superior for the more difficult task of causal explanation identification, (3) performance analysis on architectural differences of the pipeline and the classifier structures, (4) exploration of the applications of causal explanation to downstream tasks, and (5) release of a novel, anonymized causality Facebook dataset along with our causality prediction and causal explanation identification models 
Related Work
Identifying causal explanations in documents can be viewed as discourse relation parsing  The Penn Discourse Treebank (PDTB) BIBREF7 has a `Cause' and `Pragmatic Cause' discourse type under a general `Contingency' class and Rhetorical Structure Theory (RST) BIBREF8 has a `Relations of Cause'  In most cases, the development of discourse parsers has taken place in-domain, where researchers have used the existing annotations of discourse arguments in newswire text (e g  Wall Street Journal) from the discourse treebank and focused on exploring different features and optimizing various types of models for predicting relations BIBREF9 , BIBREF10 , BIBREF11   In order to further develop automated systems, researchers have proposed end-to-end discourse relation parsers, building models which are trained and evaluated on the annotated PDTB and RST Discourse Treebank (RST DT)  These corpora consist of documents from Wall Street Journal (WSJ) which are much more well-organized and grammatical than social media texts BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15  
Only a few works have attempted to parse discourse relations for out-of-domain problems such as text categorizations on social media texts; Ji and Bhatia used models which are pretrained with RST DT for building discourse structures from movie reviews, and Son adapted the PDTB discourse relation parsing approach for capturing counterfactual conditionals from tweets BIBREF4 , BIBREF3 , BIBREF16   These works had substantial differences to what propose in this paper  First, Ji and Bhatia used a pretrained model (not fully optimal for some parts of the given task) in their pipeline; Ji's model performed worse than the baseline on the categorization of legislative bills, which is thought to be due to legislative discourse structures differing from those of the training set (WSJ corpus)  Bhatia also used a pretrained model finding that utilizing discourse relation features did not boost accuracy BIBREF4 , BIBREF3   Both Bhatia and Son used manual schemes which may limit the coverage of certain types of positive samples– Bhatia used a hand-crafted schema for weighting discourse structures for the neural network model and Son manually developed seven surface forms of counterfactual thinking for the rule-based system BIBREF4 , BIBREF16   We use social-media-specific features from pretrained models which are directly trained on tweets and we avoid any hand-crafted rules except for those included in the existing discourse argument extraction techniques 
The automated systems for discourse relation parsing involve multiple subtasks from segmenting the whole text into discourse arguments to classifying discourse relations between the arguments  Past research has found that different types of models and features yield varying performance for each subtask  Some have optimized models for discourse relation classification (i e  given a document indicating if the relation existing) without discourse argument parsing using models such as Naive-Bayes or SVMs, achieve relatively stronger accuracies but a simpler task than that associated with discourse arguments BIBREF10 , BIBREF11 , BIBREF9   Researchers who, instead, tried to build the end-to-end parsing pipelines considered a wider range of approaches including sequence models and RNNs BIBREF12 , BIBREF15 , BIBREF14 , BIBREF17   Particularly, when they tried to utilize the discourse structures for out-domain applications, they used RNN-based models and found that those models are advantageous for their downstream tasks BIBREF4 , BIBREF3  
In our case, for identifying causal explanations from social media using discourse structure, we build an RNN-based model for its structural effectiveness in this task (see details in section UID13 )  However, we also note that simpler models such as SVMs and logistic regression obtained the state-of-the-art performances for text categorization tasks in social media BIBREF18 , BIBREF19 , so we build relatively simple models with different properties for each stage of the full pipeline of our parser 
Methods
We build our model based on PDTB-style discourse relation parsing since PDTB has a relatively simpler text segmentation method; for explicit discourse relations, it finds the presence of discourse connectives within a document and extracts discourse arguments which parametrize the connective while for implicit relations, it considers all adjacent sentences as candidate discourse arguments 
Dataset
We created our own causal explanation dataset by collecting 3,268 random Facebook status update messages  Three well-trained annotators manually labeled whether or not each message contains the causal explanation and obtained 1,598 causality messages with substantial agreement ( $\kappa =0 61$ )  We used the majority vote for our gold standard  Then, on each causality message, annotators identified which text spans are causal explanations 
For each task, we used 80% of the dataset for training our model and 10% for tuning the hyperparameters of our models  Finally, we evaluated all of our models on the remaining 10% (Table 1 and Table 2 )  For causal explanation detection task, we extracted discourse arguments using our parser and selected discourse arguments which most cover the annotated causal explanation text span as our gold standard 
Model
We build two types of models  First, we develop feature-based models which utilize features of the successful models in social media analysis and causal relation discourse parsing  Then, we build a recursive neural network model which uses distributed representation of discourse arguments as this approach can even capture latent properties of causal relations which may exist between distant discourse arguments  We specifically selected bidirectional LSTM since the model with the discourse distributional structure built in this form outperformed the traditional models in similar NLP downstream tasks BIBREF3  
As the first step of our pipeline, we use Tweebo parser BIBREF20 to extract syntactic features from messages  Then, we demarcate sentences using punctuation (`,') tag and periods  Among those sentences, we find discourse connectives defined in PDTB annotation along with a Tweet POS tag for conjunction words which can also be a discourse marker  In order to decide whether these connectives are really discourse connectives (e g , I went home, but he stayed) as opposed to simple connections of two words (I like apple and banana) we see if verb phrases exist before and after the connective by using dependency parsing results  Although discourse connective disambiguation is a complicated task which can be much improved by syntactic features BIBREF21 , we try to minimize effects of syntactic parsing and simplify it since it is highly error-prone in social media  Finally, according to visual inspection, emojis (`E' tag) are crucial for discourse relation in social media so we take them as separate discourse arguments (e g ,in “My test result    :(” the sad feeling is caused by the test result, but it cannot be captured by plain word tokens) 
We trained a linear SVM, an rbf SVM, and a random forest with N-gram, charater N-gram, and tweet POS tags, sentiment tags, average word lengths and word counts from each message as they have a pivotal role in the models for many NLP downstream tasks in social media BIBREF19 , BIBREF18   In addition to these features, we also extracted First-Last, First3 features and Word Pairs from every adjacent pair of discourse arguments since these features were most helpful for causal relation prediction BIBREF9   First-Last, First3 features are first and last word and first three words of two discourse arguments of the relation, and Word Pairs are the cross product of words of those discourse arguments  These two features enable our model to capture interaction between two discourse arguments  BIBREF9 reported that these two features along with verbs, modality, context, and polarity (which can be captured by N-grams, sentiment tags and POS tags in our previous features) obtained the best performance for predicting Contingency class to which causality belongs 
We load the GLOVE word embedding BIBREF22 trained in Twitter for each token of extracted discourse arguments from messages  For the distributional representation of discourse arguments, we run a Word-level LSTM on the words' embeddings within each discourse argument and concatenate last hidden state vectors of forward LSTM ( $\overrightarrow{h}$ ) and backward LSTM ( $\overleftarrow{h}$ ) which is suggested by BIBREF3 ( $DA = [\overrightarrow{h};\overleftarrow{h}]$ )  Then, we feed the sequence of the vector representation of discourse arguments to the Discourse-argument-level LSTM (DA-level LSTM) to make a final prediction with log softmax function  With this structure, the model can learn the representation of interaction of tokens inside each discourse argument, then capture discourse relations across all of the discourse arguments in each message (Figure 2 )  In order to prevent the overfitting, we added a dropout layer between the Word-level LSTM and the DA-level LSTM layer 
We also explore subsets of the full RNN architecture, specifically with one of the two LSTM layers removed  In the first model variant, we directly input all word embeddings of a whole message to a BiLSTM layer and make prediction (Word LSTM) without the help of the distributional vector representations of discourse arguments  In the second model variant, we take the average of all word embeddings of each discourse argument ( $DA_k=\frac{1}{N_k} \sum _{i=1}^{N_k}W_{i}$ ), and use them as inputs to a BiLSTM layer (DA AVG LSTM) as the average vector of embeddings were quite effective for representing the whole sequence BIBREF3 , BIBREF5   As with the full architectures, for CP both of these variants ends with a many-to-one classification per message, while the CEI model ends with a sequence of classifications 
Experiment
We explored three types of models (RBF SVM, Linear SVM, and Random Forest Classifier) which have previously been shown empirically useful for the language analysis in social media  We filtered out low frequency Word Pairs features as they tend to be noisy and sparse BIBREF9   Then, we conducted univariate feature selection to restrict all remaining features to those showing at least a small relationship with the outcome  Specifically, we keep all features passing a family-wise error rate of $\alpha = 60$ with the given outcome  After comparing the performance of the optimized version of each model, we also conducted a feature ablation test on the best model in order to see how much each feature contributes to the causality prediction 
We used bidirectional LSTMs for causality classification and causal explanation identification since the discourse arguments for causal explanation can show up either before and after the effected events or results and we want our model to be optimized for both cases  However, there is a risk of overfitting due to the dataset which is relatively small for the high complexity of the model, so we added a dropout layer (p=0 3) between the Word-level LSTM and the DA-level LSTM 
For tuning our model, we explore the dimensionality of word vector and LSTM hidden state vectors of discourse arguments of 25, 50, 100, and 200 as pretrained GLOVE vectors were trained in this setting  For optimization, we used Stochastic Gradient Descent (SGD) and Adam BIBREF23 with learning rates 0 01 and 0 001 
We ignore missing word embeddings because our dataset is quite small for retraining new word embeddings  However, if embeddings are extracted as separate discourse arguments, we used the average of all vectors of all discourse arguments in that message  Average embeddings have performed well for representing text sequences in other tasks BIBREF5  
We first use state-of-the-art PDTB taggers for our baseline BIBREF13 , BIBREF12 for the evaluation of the causality prediction of our models ( BIBREF12 requires sentences extracted from the text as its input, so we used our parser to extract sentences from the message)  Then, we compare how models work for each task and disassembled them to inspect how each part of the models can affect their final prediction performances  We conducted McNemar's test to determine whether the performance differences are statistically significant at $p <  05$  
Results
We investigated various models for both causality detection and explanation identification  Based on their performances on the task, we analyzed the relationships between the types of models and the tasks, and scrutinized further for the best performing models  For performance analysis, we reported weighted F1 of classes 
Causality Prediction
In order to classify whether a message contains causal relation, we compared off-the-shelf PDTB parsers, linear SVM, RBF SVM, Random forest and LSTM classifiers  The off-the-shelf parsers achieved the lowest accuracies ( BIBREF12 and BIBREF13 in Table 3 )  This result can be expected since 1) these models were trained with news articles and 2) they are trained for all possible discourse relations in addition to causal relations (e g , contrast, condition, etc)  Among our suggested models, SVM and random forest classifier performed better than LSTM and, in the general trend, the more complex the models were, the worse they performed  This suggests that the models with more direct and simpler learning methods with features might classify the causality messages better than the ones more optimized for capturing distributional information or non-linear relationships of features 
Table 4 shows the results of a feature ablation test to see how each feature contributes to causality classification performance of the linear SVM classifier  POS tags caused the largest drop in F1  We suspect POS tags played a unique role because discourse connectives can have various surface forms (e g , because, cuz, bcuz, etc) but still the same POS tag `P'  Also POS tags can capture the occurrences of modal verbs, a feature previously found to be very useful for detecting similar discourse relations BIBREF9   N-gram features caused 0 022 F1 drop while sentiment tags did not affect the model when removed  Unlike the previous work where First-Last, First3 and Word pairs tended to gain a large F1 increase for multiclass discourse relation prediction, in our case, they did not affect the prediction performance compared to other feature types such as POS tags or N-grams 
Causal Explanation Identification
In this task, the model identifies causal explanations given the discourse arguments of the causality message  We explored over the same models as those we used for causality (sans the output layer), and found the almost opposite trend of performances (see Table 5 )  The Linear SVM obtained lowest F1 while the LSTM model made the best identification performance  As opposed to the simple binary classification of the causality messages, in order to detect causal explanation, it is more beneficial to consider the relation across discourse arguments of the whole message and implicit distributional representation due to the implicit causal relations between two distant arguments 
Architectural Variants
For causality prediction, we experimented with only word tokens in the whole message without help of Word-level LSTM layer (Word LSTM), and F1 dropped by 0 064 (CP in Table 6 )  Also, when we used the average of the sequence of word embeddings of each discourse argument as an input to the DA-level LSTM and it caused F1 drop of 0 073  This suggests that the information gained from both the interaction of words in and in between discourse arguments help when the model utilizes the distributional representation of the texts 
For causal explanation identification, in order to test how the LSTM classifier works without its capability of capturing the relations between discourse arguments, we removed DA-level LSTM layer and ran the LSTM directly on the word embedding sequence for each discourse argument for classifying whether the argument is causal explanation, and the model had 0 061 F1 drop (Word LSTM in CEI in Table 6 )  Also, when we ran DA-level LSTM on the average vectors of the word sequences of each discourse argument of messages, F1 decreased to 0 818  This follows the similar pattern observed from other types of models performances (i e , SVMs and Random Forest classifiers) that the models with higher complexity for capturing the interaction of discourse arguments tend to identify causal explanation with the higher accuracies 
For CEI task, we found that when the model ran on the sequence representation of discourse argument (DA AVG LSTM), its performance was higher than the plain sequence of word embeddings (Word LSTM)  Finally, in both subtasks, when the models ran on both Word-level and DA-Level (Full LSTM), they obtained the highest performance 
Complete Pipeline
Evaluations thus far zeroed-in on each subtask of causal explanation analysis (i e  CEI only focused on data already identified to contain causal explanations)  Here, we seek to evaluate the complete pipeline of CP and CEI, starting from all of test data (those or without causality) and evaluating the final accuracy of CEI predictions  This is intended to evaluate CEI performance under an applied setting where one does not already know whether a document has a causal explanation 
There are several approaches we could take to perform CEI starting from unannotated data  We could simply run CEI prediction by itself (CEI Only) or the pipeline of CP first and then only run CEI on documents predicted as causal (CP + CEI)  Further, the CEI model could be trained only on those documents annotated causal (as was done in the previous experiments) or on all training documents including many that are not causal 
Table 7 show results varying the pipeline and how CEI was trained  Though all setups performed decent ( $F1 > 0 81$ ) we see that the pipelined approach, first predicting causality (with the linear SVM) and then predicting causal explanations only for those with marked causal (CP + CEI $_{causal}$ ) yielded the strongest results  This also utilized the CEI model only trained on those annotated causal  Besides performance, an added benefit from this two step approach is that the CP step is less computational intensive of the CEI step and approximately 2/3 of documents will never need the CEI step applied 
We had an inevitable limitation on the size of our dataset, since there is no other causality dataset over social media and the annotation required an intensive iterative process  This might have limited performances of more complex models, but considering the processing time and the computation load, the combination of the linear model and the RNN-based model of our pipeline obtained both the high performance and efficiency for the practical applications to downstream tasks  In other words, it's possible the linear model will not perform as well if the training size is increased substantially  However, a linear model could still be used to do a first-pass, computationally efficient labeling, in order to shortlist social media posts for further labeling from an LSTM or more complex model 
Exploration
Here, we explore the use of causal explanation analysis for downstream tasks  First we look at the relationship between use of causal explanation and one's demographics: age and gender  Then, we consider their use in sentiment analysis for extracting the causes of polarity ratings  Research involving human subjects was approved by the University of Pennsylvania Institutional Review Board 
Conclusion
We developed a pipeline for causal explanation analysis over social media text, including both causality prediction and causal explanation identification  We examined a variety of model types and RNN architectures for each part of the pipeline, finding an SVM best for causality prediction and a hierarchy of BiLSTMs for causal explanation identification, suggesting the later task relies more heavily on sequential information  In fact, we found replacing either layer of the hierarchical LSTM architecture (the word-level or the DA-level) with a an equivalent “bag of features” approach resulted in reduced accuracy  Results of our whole pipeline of causal explanation analysis were found quite strong, achieving an $F1=0 868$ at identifying discourse arguments that are causal explanations 
Finally, we demonstrated use of our models in applications, finding associations between demographics and rate of mentioning causal explanations, as well as showing differences in the top words predictive of negative ratings in Yelp reviews  Utilization of discourse structure in social media analysis has been a largely untapped area of exploration, perhaps due to its perceived difficulty  We hope the strong results of causal explanation identification here leads to the integration of more syntax and deeper semantics into social media analyses and ultimately enables new applications beyond the current state of the art 
Acknowledgments
This work was supported, in part, by a grant from the Templeton Religion Trust (ID #TRT0048)  The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript  We also thank Laura Smith, Yiyi Chen, Greta Jawel and Vanessa Hernandez for their work in identifying causal explanations 
Context 2: Introduction
Nowadays deep learning techniques outperform the other conventional methods in most of the speech-related tasks  Training robust deep neural networks for each task depends on the availability of powerful processing GPUs, as well as standard and large scale datasets  In text-independent speaker verification, large-scale datasets are available, thanks to the NIST SRE evaluations and other data collection projects such as VoxCeleb BIBREF0 
In text-dependent speaker recognition, experiments with end-to-end architectures conducted on large proprietary databases have demonstrated their superiority over traditional approaches BIBREF1  Yet, contrary to text-independent speaker recognition, text-dependent speaker recognition lacks large-scale publicly available databases  The two most well-known datasets are probably RSR2015 BIBREF2 and RedDots BIBREF3  The former contains speech data collected from 300 individuals in a controlled manner, while the latter is used primarily for evaluation rather than training, due to its small number of speakers (only 64)  Motivated by this lack of large-scale dataset for text-dependent speaker verification, we chose to proceed with the collection of the DeepMine dataset, which we expect to become a standard benchmark for the task 
Apart from speaker recognition, large amounts of training data are required also for training automatic speech recognition (ASR) systems  Such datasets should not only be large in size, they should also be characterized by high variability with respect to speakers, age and dialects  While several datasets with these properties are available for languages like English, Mandarin, French, this is not the case for several other languages, such as Persian  To this end, we proceeded with collecting a large-scale dataset, suitable for building robust ASR models in Persian 
The main goal of the DeepMine project was to collect speech from at least a few thousand speakers, enabling research and development of deep learning methods  The project started at the beginning of 2017, and after designing the database and the developing Android and server applications, the data collection began in the middle of 2017  The project finished at the end of 2018 and the cleaned-up and final version of the database was released at the beginning of 2019  In BIBREF4, the running project and its data collection scenarios were described, alongside with some preliminary results and statistics  In this paper, we announce the final and cleaned-up version of the database, describe its different parts and provide various evaluation setups for each part  Finally, since the database was designed mainly for text-dependent speaker verification purposes, some baseline results are reported for this task on the official evaluation setups  Additional baseline results are also reported for Persian speech recognition  However, due to the space limitation in this paper, the baseline results are not reported for all the database parts and conditions  They will be defined and reported in the database technical documentation and in a future journal paper 
Data Collection
DeepMine is publicly available for everybody with a variety of licenses for different users  It was collected using crowdsourcing BIBREF4  The data collection was done using an Android application  Each respondent installed the application on his/her personal device and recorded several phrases in different sessions  The Android application did various checks on each utterance and if it passed all of them, the respondent was directed to the next phrase  For more information about data collection scenario, please refer to BIBREF4 
Data Collection ::: Post-Processing
In order to clean-up the database, the main post-processing step was to filter out problematic utterances  Possible problems include speaker word insertions (e g  repeating some part of a phrase), deletions, substitutions, and involuntary disfluencies  To detect these, we implemented an alignment stage, similar to the second alignment stage in the LibriSpeech project BIBREF5  In this method, a custom decoding graph was generated for each phrase  The decoding graph allows for word skipping and word insertion in the phrase 
For text-dependent and text-prompted parts of the database, such errors are not allowed  Hence, any utterances with errors were removed from the enrollment and test lists  For the speech recognition part, a sub-part of the utterance which is correctly aligned to the corresponding transcription is kept  After the cleaning step, around 190 thousand utterances with full transcription and 10 thousand with sub-part alignment have remained in the database 
Data Collection ::: Statistics
After processing the database and removing problematic respondents and utterances, 1969 respondents remained in the database, with 1149 of them being male and 820 female  297 of the respondents could not read English and have therefore read only the Persian prompts  About 13200 sessions were recorded by females and similarly, about 9500 sessions by males, i e  women are over-represented in terms of sessions, even though their number is 17% smaller than that of males  Other useful statistics related to the database are shown in Table TABREF4 
The last status of the database, as well as other related and useful information about its availability can be found on its website, together with a limited number of samples 
DeepMine Database Parts
The DeepMine database consists of three parts  The first one contains fixed common phrases to perform text-dependent speaker verification  The second part consists of random sequences of words useful for text-prompted speaker verification, and the last part includes phrases with word- and phoneme-level transcription, useful for text-independent speaker verification using a random phrase (similar to Part4 of RedDots)  This part can also serve for Persian ASR training  Each part is described in more details below  Table TABREF11 shows the number of unique phrases in each part of the database  For the English text-dependent part, the following phrases were selected from part1 of the RedDots database, hence the RedDots can be used as an additional training set for this part:
“My voice is my password ”
“OK Google ”
“Artificial intelligence is for real ”
“Actions speak louder than words ”
“There is no such thing as a free lunch ”
DeepMine Database Parts ::: Part1 - Text-dependent (TD)
This part contains a set of fixed phrases which are used to verify speakers in text-dependent mode  Each speaker utters 5 Persian phrases, and if the speaker can read English, 5 phrases selected from Part1 of the RedDots database are also recorded 
We have created three experimental setups with different numbers of speakers in the evaluation set  For each setup, speakers with more recording sessions are included in the evaluation set and the rest of the speakers are used for training in the background set (in the database, all background sets are basically training data)  The rows in Table TABREF13 corresponds to the different experimental setups and shows the numbers of speakers in each set  Note that, for English, we have filtered the (Persian native) speakers by the ability to read English  Therefore, there are fewer speakers in each set for English than for Persian  There is a small “dev” set in each setup which can be used for parameter tuning to prevent over-tuning on the evaluation set 
For each experimental setup, we have defined several official trial lists with different numbers of enrollment utterances per trial in order to investigate the effects of having different amounts of enrollment data  All trials in one trial list have the same number of enrollment utterances (3 to 6) and only one test utterance  All enrollment utterances in a trial are taken from different consecutive sessions and the test utterance is taken from yet another session  From all the setups and conditions, the 100-spk with 3-session enrollment (3-sess) is considered as the main evaluation condition  In Table TABREF14, the number of trials for Persian 3-sess are shown for the different types of trial in the text-dependent speaker verification (SV)  Note that for Imposter-Wrong (IW) trials (i e  imposter speaker pronouncing wrong phrase), we merely create one wrong trial for each Imposter-Correct (IC) trial to limit the huge number of possible trials for this case  So, the number of trials for IC and IW cases are the same 
DeepMine Database Parts ::: Part2 - Text-prompted (TP)
For this part, in each session, 3 random sequences of Persian month names are shown to the respondent in two modes: In the first mode, the sequence consists of all 12 months, which will be used for speaker enrollment  The second mode contains a sequence of 3 month names that will be used as a test utterance  In each 8 sessions received by a respondent from the server, there are 3 enrollment phrases of all 12 months (all in just one session), and $7 \times 3$ other test phrases, containing fewer words  For a respondent who can read English, 3 random sequences of English digits are also recorded in each session  In one of the sessions, these sequences contain all digits and the remaining ones contain only 4 digits 
Similar to the text-dependent case, three experimental setups with different number of speaker in the evaluation set are defined (corresponding to the rows in Table TABREF16)  However, different strategy is used for defining trials: Depending on the enrollment condition (1- to 3-sess), trials are enrolled on utterances of all words from 1 to 3 different sessions (i e  3 to 9 utterances)  Further, we consider two conditions for test utterances: seq test utterance with only 3 or 4 words and full test utterances with all words (i e  same words as in enrollment but in different order)  From all setups an all conditions, the 100-spk with 1-session enrolment (1-sess) is considered as the main evaluation condition for the text-prompted case  In Table TABREF16, the numbers of trials (sum for both seq and full conditions) for Persian 1-sess are shown for the different types of trials in the text-prompted SV  Again, we just create one IW trial for each IC trial 
DeepMine Database Parts ::: Part3 - Text-independent (TI)
In this part, 8 Persian phrases that have already been transcribed on the phone level are displayed to the respondent  These phrases are chosen mostly from news and Persian Wikipedia  If the respondent is unable to read English, instead of 5 fixed phrases and 3 random digit strings, 8 other Persian phrases are also prompted to the respondent to have exactly 24 phrases in each recording session 
This part can be useful at least for three potential applications  First, it can be used for text-independent speaker verification  The second application of this part (same as Part4 of RedDots) is text-prompted speaker verification using random text (instead of a random sequence of words)  Finally, the third application is large vocabulary speech recognition in Persian (explained in the next sub-section) 
Based on the recording sessions, we created two experimental setups for speaker verification  In the first one, respondents with at least 17 recording sessions are included to the evaluation set, respondents with 16 sessions to the development and the rest of respondents to the background set (can be used as training data)  In the second setup, respondents with at least 8 sessions are included to the evaluation set, respondents with 6 or 7 sessions to the development and the rest of respondents to the background set  Table TABREF18 shows numbers of speakers in each set of the database for text-independent SV case 
For text-independent SV, we have considered 4 scenarios for enrollment and 4 scenarios for test  The speaker can be enrolled using utterances from 1, 2 or 3 consecutive sessions (1sess to 3sess) or using 8 utterances from 8 different sessions  The test speech can be one utterance (1utt) for short duration scenario or all utterances in one session (1sess) for long duration case  In addition, test speech can be selected from 5 English phrases for cross-language testing (enrollment using Persian utterances and test using English utterances)

ground_truth: Facebook status update messages

Facebook status update messages

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what is the source of the data?
Context 2:  From all setups, 1sess-1utt and 1sess-1sess for 438-spk set are considered as the main evaluation setups for text-independent case  Table TABREF19 shows number of trials for these setups 
For text-prompted SV with random text, the same setup as text-independent case together with corresponding utterance transcriptions can be used 
DeepMine Database Parts ::: Part3 - Speech Recognition
As explained before, Part3 of the DeepMine database can be used for Persian read speech recognition  There are only a few databases for speech recognition in Persian BIBREF6, BIBREF7  Hence, this part can at least partly address this problem and enable robust speech recognition applications in Persian  Additionally, it can be used for speaker recognition applications, such as training deep neural networks (DNNs) for extracting bottleneck features BIBREF8, or for collecting sufficient statistics using DNNs for i-vector training 
We have randomly selected 50 speakers (25 for each gender) from the all speakers in the database which have net speech (without silence parts) between 25 minutes to 50 minutes as test speakers  For each speaker, the utterances in the first 5 sessions are included to (small) test-set and the other utterances of test speakers are considered as a large-test-set  The remaining utterances of the other speakers are included in the training set  The test-set, large-test-set and train-set contain 5 9, 28 5 and 450 hours of speech respectively 
There are about 8300 utterances in Part3 which contain only Persian full names (i e  first and family name pairs)  Each phrase consists of several full names and their phoneme transcriptions were extracted automatically using a trained Grapheme-to-Phoneme (G2P)  These utterances can be used to evaluate the performance of a systems for name recognition, which is usually more difficult than the normal speech recognition because of the lack of a reliable language model 
Experiments and Results
Due to the space limitation, we present results only for the Persian text-dependent speaker verification and speech recognition 
Experiments and Results ::: Speaker Verification Experiments
We conducted an experiment on text-dependent speaker verification part of the database, using the i-vector based method proposed in BIBREF9, BIBREF10 and applied it to the Persian portion of Part1  In this experiment, 20-dimensional MFCC features along with first and second derivatives are extracted from 16 kHz signals using HTK BIBREF11 with 25 ms Hamming windowed frames with 15 ms overlap 
The reported results are obtained with a 400-dimensional gender independent i-vector based system  The i-vectors are first length-normalized and are further normalized using phrase- and gender-dependent Regularized Within-Class Covariance Normalization (RWCCN) BIBREF10  Cosine distance is used to obtain speaker verification scores and phrase- and gender-dependent s-norm is used for normalizing the scores  For aligning speech frames to Gaussian components, monophone HMMs with 3 states and 8 Gaussian components in each state are used BIBREF10  We only model the phonemes which appear in the 5 Persian text-dependent phrases 
For speaker verification experiments, the results were reported in terms of Equal Error Rate (EER) and Normalized Detection Cost Function as defined for NIST SRE08 ($\mathrm {NDCF_{0 01}^{min}}$) and NIST SRE10 ($\mathrm {NDCF_{0 001}^{min}}$)  As shown in Table TABREF22, in text-dependent SV there are 4 types of trials: Target-Correct and Imposter-Correct refer to trials when the pass-phrase is uttered correctly by target and imposter speakers respectively, and in same manner, Target-Wrong and Imposter-Wrong refer to trials when speakers uttered a wrong pass-phrase  In this paper, only the correct trials (i e  Target-Correct as target trials vs Imposter-Correct as non-target trials) are considered for evaluating systems as it has been proved that these are the most challenging trials in text-dependent SV BIBREF8, BIBREF12 
Table TABREF23 shows the results of text-dependent experiments using Persian 100-spk and 3-sess setup  For filtering trials, the respondents' mobile brand and model were used in this experiment  In the table, the first two letters in the filter notation relate to the target trials and the second two letters (i e  right side of the colon) relate for non-target trials  For target trials, the first Y means the enrolment and test utterances were recorded using a device with the same brand by the target speaker  The second Y letter means both recordings were done using exactly the same device model  Similarly, the first Y for non-target trials means that the devices of target and imposter speakers are from the same brand (i e  manufacturer)  The second Y means that, in addition to the same brand, both devices have the same model  So, the most difficult target trials are “NN”, where the speaker has used different a device at the test time  In the same manner, the most difficult non-target trials which should be rejected by the system are “YY” where the imposter speaker has used the same device model as the target speaker (note that it does not mean physically the same device because each speaker participated in the project using a personal mobile device)  Hence, the similarity in the recording channel makes rejection more difficult 
The first row in Table TABREF23 shows the results for all trials  By comparing the results with the best published results on RSR2015 and RedDots BIBREF10, BIBREF8, BIBREF12, it is clear that the DeepMine database is more challenging than both RSR2015 and RedDots databases  For RSR2015, the same i-vector/HMM-based method with both RWCCN and s-norm has achieved EER less than 0 3% for both genders (Table VI in BIBREF10)  The conventional Relevance MAP adaptation with HMM alignment without applying any channel-compensation techniques (i e  without applying RWCCN and s-norm due to the lack of suitable training data) on RedDots Part1 for the male has achieved EER around 1 5% (Table XI in BIBREF10)  It is worth noting that EERs for DeepMine database without any channel-compensation techniques are 2 1 and 3 7% for males and females respectively 
One interesting advantage of the DeepMine database compared to both RSR2015 and RedDots is having several target speakers with more than one mobile device  This is allows us to analyse the effects of channel compensation methods  The second row in Table TABREF23 corresponds to the most difficult trials where the target trials come from mobile devices with different models while imposter trials come from the same device models  It is clear that severe degradation was caused by this kind of channel effects (i e  decreasing within-speaker similarities while increasing between-speaker similarities), especially for females 
The results in the third row show the condition when target speakers at the test time use exactly the same device that was used for enrollment  Comparing this row with the results in the first row proves how much improvement can be achieved when exactly the same device is used by the target speaker 
The results in the fourth row show the condition when imposter speakers also use the same device model at test time to fool the system  So, in this case, there is no device mismatch in all trials  By comparing the results with the third row, we can see how much degradation is caused if we only consider the non-target trials with the same device 
The fifth row shows similar results when the imposter speakers use device of the same brand as the target speaker but with a different model  Surprisingly, in this case, the degradation is negligible and it means that mobiles from a specific brand (manufacturer) have different recording channel properties 
The degraded female results in the sixth row as compared to the third row show the effect of using a different device model from the same brand for target trials  For males, the filters brings almost the same subsets of trials, which explains the very similar results in this case 
Looking at the first two and the last row of Table TABREF23, one can notice the significantly worse performance obtained for the female trials as compared to males  Note that these three rows include target trials where the devices used for enrollment do not necessarily match the devices used for recording test utterances  On the other hand, in rows 3 to 6, which exclude such mismatched trials, the performance for males and females is comparable  This suggest that the degraded results for females are caused by some problematic trials with device mismatch  The exact reason for this degradation is so far unclear and needs a further investigation 
In the last row of the table, the condition of the second row is relaxed: the target device should have different model possibly from the same brand and imposter device only needs to be from the same brand  In this case, as was expected, the performance degradation is smaller than in the second row 
Experiments and Results ::: Speech Recognition Experiments
In addition to speaker verification, we present several speech recognition experiments on Part3  The experiments were performed with the Kaldi toolkit BIBREF13  For training HMM-based MonoPhone model, only 20 thousands of shortest utterances are used and for other models the whole training data is used  The DNN based acoustic model is a time-delay DNN with low-rank factorized layers and skip connections without i-vector adaptation (a modified network from one of the best performing LibriSpeech recipes)  The network is shown in Table TABREF25: there are 16 F-TDNN layers, with dimension 1536 and linear bottleneck layers of dimension 256  The acoustic model is trained for 10 epochs using lattice-free maximum mutual information (LF-MMI) with cross-entropy regularization BIBREF14  Re-scoring is done using a pruned trigram language model and the size of the dictionary is around 90,000 words 
Table TABREF26 shows the results in terms of word error rate (WER) for different evaluated methods  As can be seen, the created database can be used to train well performing and practically usable Persian ASR models 
Conclusions
In this paper, we have described the final version of a large speech corpus, the DeepMine database  It has been collected using crowdsourcing and, according to the best of our knowledge, it is the largest public text-dependent and text-prompted speaker verification database in two languages: Persian and English  In addition, it is the largest text-independent speaker verification evaluation database, making it suitable to robustly evaluate state-of-the-art methods on different conditions  Alongside these appealing properties, it comes with phone-level transcription, making it suitable to train deep neural network models for Persian speech recognition 
We provided several evaluation protocols for each part of the database  The protocols allow researchers to investigate the performance of different methods in various scenarios and study the effects of channels, duration and phrase text on the performance  We also provide two test sets for speech recognition: One normal test set with a few minutes of speech for each speaker and one large test set with more (30 minutes on average) speech that can be used for any speaker adaptation method 
As baseline results, we reported the performance of an i-vector/HMM based method on Persian text-dependent part  Moreover, we conducted speech recognition experiments using conventional HMM-based methods, as well as state-of-the-art deep neural network based method using Kaldi toolkit with promising performance  Text-dependent results have shown that the DeepMine database is more challenging than RSR2015 and RedDots databases 
Acknowledgments
The data collection project was mainly supported by Sharif DeepMine company  The work on the paper was supported by Czech National Science Foundation (GACR) project "NEUREM3" No  19-26934X and the National Programme of Sustainability (NPU II) project "IT4Innovations excellence in science - LQ1602" 
Context 3: Introduction
This paper describes our approach and results for Task 2 of the CoNLL–SIGMORPHON 2018 shared task on universal morphological reinflection BIBREF0   The task is to generate an inflected word form given its lemma and the context in which it occurs 
Morphological (re)inflection from context is of particular relevance to the field of computational linguistics: it is compelling to estimate how well a machine-learned system can capture the morphosyntactic properties of a word given its context, and map those properties to the correct surface form for a given lemma 
There are two tracks of Task 2 of CoNLL–SIGMORPHON 2018: in Track 1 the context is given in terms of word forms, lemmas and morphosyntactic descriptions (MSD); in Track 2 only word forms are available  See Table TABREF1 for an example  Task 2 is additionally split in three settings based on data size: high, medium and low, with high-resource datasets consisting of up to 70K instances per language, and low-resource datasets consisting of only about 1K instances 
The baseline provided by the shared task organisers is a seq2seq model with attention (similar to the winning system for reinflection in CoNLL–SIGMORPHON 2016, BIBREF1 ), which receives information about context through an embedding of the two words immediately adjacent to the target form  We use this baseline implementation as a starting point and achieve the best overall accuracy of 49 87 on Task 2 by introducing three augmentations to the provided baseline system: (1) We use an LSTM to encode the entire available context; (2) We employ a multi-task learning approach with the auxiliary objective of MSD prediction; and (3) We train the auxiliary component in a multilingual fashion, over sets of two to three languages 
In analysing the performance of our system, we found that encoding the full context improves performance considerably for all languages: 11 15 percentage points on average, although it also highly increases the variance in results  Multi-task learning, paired with multilingual training and subsequent monolingual finetuning, scored highest for five out of seven languages, improving accuracy by another 9 86% on average 
System Description
Our system is a modification of the provided CoNLL–SIGMORPHON 2018 baseline system, so we begin this section with a reiteration of the baseline system architecture, followed by a description of the three augmentations we introduce 
Baseline
The CoNLL–SIGMORPHON 2018 baseline is described as follows:
The system is an encoder-decoder on character sequences  It takes a lemma as input and generates a word form  The process is conditioned on the context of the lemma [   ] The baseline treats the lemma, word form and MSD of the previous and following word as context in track 1  In track 2, the baseline only considers the word forms of the previous and next word  [   ] The baseline system concatenates embeddings for context word forms, lemmas and MSDs into a context vector  The baseline then computes character embeddings for each character in the input lemma  Each of these is concatenated with a copy of the context vector  The resulting sequence of vectors is encoded using an LSTM encoder  Subsequently, an LSTM decoder generates the characters in the output word form using encoder states and an attention mechanism 
To that we add a few details regarding model size and training schedule:
the number of LSTM layers is one;
embedding size, LSTM layer size and attention layer size is 100;
models are trained for 20 epochs;
on every epoch, training data is subsampled at a rate of 0 3;
LSTM dropout is applied at a rate 0 3;
context word forms are randomly dropped at a rate of 0 1;
the Adam optimiser is used, with a default learning rate of 0 001; and
trained models are evaluated on the development data (the data for the shared task comes already split in train and dev sets) 
Our system
Here we compare and contrast our system to the baseline system  A diagram of our system is shown in Figure FIGREF4  
The idea behind this modification is to provide the encoder with access to all morpho-syntactic cues present in the sentence  In contrast to the baseline, which only encodes the immediately adjacent context of a target word, we encode the entire context  All context word forms, lemmas, and MSD tags (in Track 1) are embedded in their respective high-dimensional spaces as before, and their embeddings are concatenated  However, we now reduce the entire past context to a fixed-size vector by encoding it with a forward LSTM, and we similarly represent the future context by encoding it with a backwards LSTM 
We introduce an auxiliary objective that is meant to increase the morpho-syntactic awareness of the encoder and to regularise the learning process—the task is to predict the MSD tag of the target form  MSD tag predictions are conditioned on the context encoding, as described in UID15   Tags are generated with an LSTM one component at a time, e g  the tag PRO;NOM;SG;1 is predicted as a sequence of four components, INLINEFORM0 PRO, NOM, SG, 1 INLINEFORM1  
For every training instance, we backpropagate the sum of the main loss and the auxiliary loss without any weighting 
As MSD tags are only available in Track 1, this augmentation only applies to this track 
The parameters of the entire MSD (auxiliary-task) decoder are shared across languages 
Since a grouping of the languages based on language family would have left several languages in single-member groups (e g  Russian is the sole representative of the Slavic family), we experiment with random groupings of two to three languages  Multilingual training is performed by randomly alternating between languages for every new minibatch  We do not pass any information to the auxiliary decoder as to the source language of the signal it is receiving, as we assume abstract morpho-syntactic features are shared across languages 
After 20 epochs of multilingual training, we perform 5 epochs of monolingual finetuning for each language  For this phase, we reduce the learning rate to a tenth of the original learning rate, i e  0 0001, to ensure that the models are indeed being finetuned rather than retrained 
We keep all hyperparameters the same as in the baseline  Training data is split 90:10 for training and validation  We train our models for 50 epochs, adding early stopping with a tolerance of five epochs of no improvement in the validation loss  We do not subsample from the training data 
We train models for 50 different random combinations of two to three languages in Track 1, and 50 monolingual models for each language in Track 2  Instead of picking the single model that performs best on the development set and thus risking to select a model that highly overfits that data, we use an ensemble of the five best models, and make the final prediction for a given target form with a majority vote over the five predictions 
Results and Discussion
Test results are listed in Table TABREF17   Our system outperforms the baseline for all settings and languages in Track 1 and for almost all in Track 2—only in the high resource setting is our system not definitively superior to the baseline 
Interestingly, our results in the low resource setting are often higher for Track 2 than for Track 1, even though contextual information is less explicit in the Track 2 data and the multilingual multi-tasking approach does not apply to this track  We interpret this finding as an indicator that a simpler model with fewer parameters works better in a setting of limited training data  Nevertheless, we focus on the low resource setting in the analysis below due to time limitations  As our Track 1 results are still substantially higher than the baseline results, we consider this analysis valid and insightful 
Ablation Study
We analyse the incremental effect of the different features in our system, focusing on the low-resource setting in Track 1 and using development data 
Encoding the entire context with an LSTM highly increases the variance of the observed results  So we trained fifty models for each language and each architecture  Figure FIGREF23 visualises the means and standard deviations over the trained models  In addition, we visualise the average accuracy for the five best models for each language and architecture, as these are the models we use in the final ensemble prediction  Below we refer to these numbers only 
The results indicate that encoding the full context with an LSTM highly enhances the performance of the model, by 11 15% on average  This observation explains the high results we obtain also for Track 2 
Adding the auxiliary objective of MSD prediction has a variable effect: for four languages (de, en, es, and sv) the effect is positive, while for the rest it is negative  We consider this to be an issue of insufficient data for the training of the auxiliary component in the low resource setting we are working with 
We indeed see results improving drastically with the introduction of multilingual training, with multilingual results being 7 96% higher than monolingual ones on average 
We studied the five best models for each language as emerging from the multilingual training (listed in Table TABREF27 ) and found no strong linguistic patterns  The en–sv pairing seems to yield good models for these languages, which could be explained in terms of their common language family and similar morphology  The other natural pairings, however, fr–es, and de–sv, are not so frequent among the best models for these pairs of languages 
Finally, monolingual finetuning improves accuracy across the board, as one would expect, by 2 72% on average 
The final observation to be made based on this breakdown of results is that the multi-tasking approach paired with multilingual training and subsequent monolingual finetuning outperforms the other architectures for five out of seven languages: de, en, fr, ru and sv  For the other two languages in the dataset, es and fi, the difference between this approach and the approach that emerged as best for them is less than 1%  The overall improvement of the multilingual multi-tasking approach over the baseline is 18 30% 
Error analysis
Here we study the errors produced by our system on the English test set to better understand the remaining shortcomings of the approach  A small portion of the wrong predictions point to an incorrect interpretation of the morpho-syntactic conditioning of the context, e g  the system predicted plan instead of plans in the context Our _ include raising private capital  The majority of wrong predictions, however, are nonsensical, like bomb for job, fify for fixing, and gnderrate for understand  This observation suggests that generally the system did not learn to copy the characters of lemma into inflected form, which is all it needs to do in a large number of cases  This issue could be alleviated with simple data augmentation techniques that encourage autoencoding BIBREF2  
MSD prediction
Figure FIGREF32 summarises the average MSD-prediction accuracy for the multi-tasking experiments discussed above  Accuracy here is generally higher than on the main task, with the multilingual finetuned setup for Spanish and the monolingual setup for French scoring best: 66 59% and 65 35%, respectively  This observation illustrates the added difficulty of generating the correct surface form even when the morphosyntactic description has been identified correctly 
We observe some correlation between these numbers and accuracy on the main task: for de, en, ru and sv, the brown, pink and blue bars here pattern in the same way as the corresponding INLINEFORM0 's in Figure FIGREF23   One notable exception to this pattern is fr where inflection gains a lot from multilingual training, while MSD prediction suffers greatly  Notice that the magnitude of change is not always the same, however, even when the general direction matches: for ru, for example, multilingual training benefits inflection much more than in benefits MSD prediction, even though the MSD decoder is the only component that is actually shared between languages  This observation illustrates the two-fold effect of multi-task training: an auxiliary task can either inform the main task through the parameters the two tasks share, or it can help the main task learning through its regularising effect 
Related Work
Our system is inspired by previous work on multi-task learning and multi-lingual learning, mainly building on two intuitions: (1) jointly learning related tasks tends to be beneficial BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 ; and (2) jointly learning related languages in an MTL-inspired framework tends to be beneficial BIBREF8 , BIBREF9 , BIBREF10   In the context of computational morphology, multi-lingual approaches have previously been employed for morphological reinflection BIBREF2 and for paradigm completion BIBREF11   In both of these cases, however, the available datasets covered more languages, 40 and 21, respectively, which allowed for linguistically-motivated language groupings and for parameter sharing directly on the level of characters  BIBREF10 explore parameter sharing between related languages for dependency parsing, and find that sharing is more beneficial in the case of closely related languages 
Conclusions
In this paper we described our system for the CoNLL–SIGMORPHON 2018 shared task on Universal Morphological Reinflection, Task 2, which achieved the best performance out of all systems submitted, an overall accuracy of 49 87  We showed in an ablation study that this is due to three core innovations, which extend a character-based encoder-decoder model: (1) a wide context window, encoding the entire available context; (2) multi-task learning with the auxiliary task of MSD prediction, which acts as a regulariser; (3) a multilingual approach, exploiting information across languages  In future work we aim to gain better understanding of the increase in variance of the results introduced by each of our modifications and the reasons for the varying effect of multi-task learning for different languages 
Acknowledgements
We gratefully acknowledge the support of the NVIDIA Corporation with the donation of the Titan Xp GPU used for this research 

ground_truth: Android application

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What architecture does the encoder have?
Context 4:  The learning rate is linearly warmed up from INLINEFORM4 to 1 for 16K steps and then annealed using a cosine learning rate schedule with a single phase to 0 0001 BIBREF20   We train on 32 Nvidia V100 SXM2 GPUs and use the NCCL2 library as well as the torch distributed package for inter-GPU communication  Training relies on 16-bit floating point operations BIBREF21 and it took six days for the bi-directional model and four days for the uni-directional model 
Sequence to sequence model
We use the transformer implementation of the fairseq toolkit BIBREF22   The WMT en-de and en-tr experiments are based on the Big Transformer sequence to sequence architecture with 6 blocks in the encoder and decoder  For abstractive summarization we use a base transformer model BIBREF16   We tune dropout values of between 0 1 and 0 4 on the validation set  Models are optimized with Adam BIBREF23 using INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 and we use the same learning rate schedule as BIBREF16 ; we perform 10K-200K depending on bitext size  All models use label smoothing with a uniform prior distribution over the vocabulary INLINEFORM3 BIBREF24 , BIBREF25   We run experiments on 8 GPUs and generate translations with a beam of size 5 
Machine translation
We first present a comparison of the various strategies in different simulated parallel corpus size settings  For each experiment, we tune the dropout applied to the language model representations, and we reduce the number of optimizer steps for smaller bitext setups as models converge faster; all other hyper-parameters are equal between setups  Our baseline is a Big Transformer model and we also consider a variant where we share token embeddings between the encoder and decoder (; Inan et al , 2016; Press & Wolf, 2016) 
Figure FIGREF10 shows results averaged over six test sets relative to the baseline which does not share source and target embeddings (Appendix SECREF6 shows a detailed breakdown)  performs very well with little labeled data but the gains erode to practically zero in large bitext settings  Pre-trained language model representations are most effective in low bitext setups  The best performing strategy is ELMo embeddings input to the encoder ()  This improves the baseline by 3 8 BLEU in the 160K bitext setting and it still improves the 5 2M setting by over 1 BLEU 
We further improve by sharing learned word representations in the decoder by tying input and output embeddings ()  This configuration performs even better than with a gain of 5 3 BLEU in the 160K setup  Sharing decoder embeddings is equally applicable to   Language model representations are much less effective in the decoder: improves the 160K bitext setup but yields no improvements thereafter and performs even worse  We conjecture that pre-trained representations give much easier wins in the encoder  Table TABREF14 shows additional results on newstest2018 
Pre-trained representations mostly impacts the training time of the sequence to sequence model (see Appendix SECREF7 ): slows throughput during training by about 5 3x and is even slower because of the need to backpropagate through the LM for fine-tuning (9 2x)  However, inference is only 12-14% slower than the baseline when adding pre-trained embeddings to the encoder (, )  This is because the LM computation can be paralelized for all input tokens  Inference is much slower when adding representations to the decoder because the LM needs to be invoked repeatedly  Our current implementation does not cache LM operations for the previous state and can be made much faster 
The baseline uses a BPE vocabulary estimated on the language model corpora (§ SECREF3 )  Appendix SECREF6 shows that this vocabulary actually leads to sligtly better performance than a joint BPE code learned on the bitext as is usual 
Next, we validate our findings on the WMT'18 English-Turkish task for which the bitext is truly limited (208K sentence-pairs)  We use the language model vocab for the the English side of the bitext and a BPE vocabulary learned on the Turkish side  Table TABREF15 shows that ELMo embeddings for the encoder improve English-Turkish translation 
Abstractive summarization
Following BIBREF11 , we experiment on the non-anonymized version of   When generating summaries, we follow standard practice of tuning the maximum output length and disallow repeating the same trigram BIBREF27 , BIBREF14   For this task we train language model representations on the combination of newscrawl and the training data  Table TABREF16 shows that pre-trained embeddings can significantly improve on top of a strong baseline transformer  We also compare to BIBREF26 who use a task-specific architecture compared to our generic sequence to sequence baseline  Pre-trained representations are complementary to their method 
Conclusion
We presented an analysis of different strategies to add pre-trained language model representations to sequence to sequence models for neural machine translation and abstractive document summarization  Adding pre-trained representations is very effective for the encoder network and while returns diminish when more labeled data becomes available, we still observe improvements when millions of examples are available  In future research we will investigate ways to improve the decoder with pre-trained representations 

ground_truth: LSTM

LSTM