prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what datasets did they use?
Context 1: Introduction
Accurate grapheme-to-phoneme conversion (g2p) is important for any application that depends on the sometimes inconsistent relationship between spoken and written language  Most prominently, this includes text-to-speech and automatic speech recognition  Most work on g2p has focused on a few languages for which extensive pronunciation data is available BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Most languages lack these resources  However, a low resource language's writing system is likely to be similar to the writing systems of languages that do have sufficient pronunciation data  Therefore g2p may be possible for low resource languages if this high resource data can be properly utilized 
We attempt to leverage high resource data by treating g2p as a multisource neural machine translation (NMT) problem  The source sequences for our system are words in the standard orthography in any language  The target sequences are the corresponding representation in the International Phonetic Alphabet (IPA)  Our results show that the parameters learned by the shared encoder–decoder are able to exploit the orthographic and phonemic similarities between the various languages in our data 
Low Resource g2p
Our approach is similar in goal to deri2016grapheme's model for adapting high resource g2p models for low resource languages  They trained weighted finite state transducer (wFST) models on a variety of high resource languages, then transferred those models to low resource languages, using a language distance metric to choose which high resource models to use and a phoneme distance metric to map the high resource language's phonemes to the low resource language's phoneme inventory  These distance metrics are computed based on data from Phoible BIBREF4 and URIEL BIBREF5  
Other low resource g2p systems have used a strategy of combining multiple models  schlippe2014combining trained several data-driven g2p systems on varying quantities of monolingual data and combined their outputs with a phoneme-level voting scheme  This led to improvements over the best-performing single system for small quantities of data in some languages  jyothilow trained recurrent neural networks for small data sets and found that a version of their system that combined the neural network output with the output of the wFST-based Phonetisaurus system BIBREF1 did better than either system alone 
A different approach came from kim2012universal, who used supervised learning with an undirected graphical model to induce the grapheme–phoneme mappings for languages written in the Latin alphabet  Given a short text in a language, the model predicts the language's orthographic rules  To create phonemic context features from the short text, the model naïvely maps graphemes to IPA symbols written with the same character, and uses the features of these symbols to learn an approximation of the phonotactic constraints of the language  In their experiments, these phonotactic features proved to be more valuable than geographical and genetic features drawn from WALS BIBREF6  
Multilingual Neural NLP
In recent years, neural networks have emerged as a common way to use data from several languages in a single system  Google's zero-shot neural machine translation system BIBREF7 shares an encoder and decoder across all language pairs  In order to facilitate this multi-way translation, they prepend an artificial token to the beginning of each source sentence at both training and translation time  The token identifies what language the sentence should be translated to  This approach has three benefits: it is far more efficient than building a separate model for each language pair; it allows for translation between languages that share no parallel data; and it improves results on low-resource languages by allowing them to implicitly share parameters with high-resource languages  Our g2p system is inspired by this approach, although it differs in that there is only one target “language”, IPA, and the artificial tokens identify the language of the source instead of the language of the target 
Other work has also made use of multilingually-trained neural networks  Phoneme-level polyglot language models BIBREF8 train a single model on multiple languages and additionally condition on externally constructed typological data about the language  ostling2017continuous used a similar approach, in which a character-level neural language model is trained on a massively multilingual corpus  A language embedding vector is concatenated to the input at each time step  The language embeddings their system learned correlate closely to the genetic relationships between languages  However, neither of these models was applied to g2p 
Grapheme-to-Phoneme
g2p is the problem of converting the orthographic representation of a word into a phonemic representation  A phoneme is an abstract unit of sound which may have different realizations in different contexts  For example, the English phoneme has two phonetic realizations (or allophones):
English speakers without linguistic training often struggle to perceive any difference between these sounds  Writing systems usually do not distinguish between allophones: and are both written as INLINEFORM0 p INLINEFORM1 in English  The sounds are written differently in languages where they contrast, such as Hindi and Eastern Armenian 
Most writing systems in use today are glottographic, meaning that their symbols encode solely phonological information  But despite being glottographic, in few writing systems do graphemes correspond one-to-one with phonemes  There are cases in which multiple graphemes represent a single phoneme, as in the word the in English:
There are cases in which a single grapheme represents multiple phonemes, such as syllabaries, in which each symbol represents a syllable 
In many languages, there are silent letters, as in the word hora in Spanish:
There are more complicated correspondences, such as the silent e in English that affects the pronunciation of the previous vowel, as seen in the pair of words cape and cap 
It is possible for an orthographic system to have any or all of the above phenomena while remaining unambiguous  However, some orthographic systems contain ambiguities  English is well-known for its spelling ambiguities  Abjads, used for Arabic and Hebrew, do not give full representation to vowels 
Consequently, g2p is harder than simply replacing each grapheme symbol with a corresponding phoneme symbol  It is the problem of replacing a grapheme sequence INLINEFORM0
with a phoneme sequence INLINEFORM0
where the sequences are not necessarily of the same length  Data-driven g2p is therefore the problem of finding the phoneme sequence that maximizes the likelihood of the grapheme sequence: INLINEFORM0
Data-driven approaches are especially useful for problems in which the rules that govern them are complex and difficult to engineer by hand  g2p for languages with ambiguous orthographies is such a problem  Multilingual g2p, in which the various languages have similar but different and possibly contradictory spelling rules, can be seen as an extreme case of that  Therefore, a data-driven sequence-to-sequence model is a natural choice 
Encoder–Decoder Models
In order to find the best phoneme sequence, we use a neural encoder–decoder model with attention BIBREF9   The model consists of two main parts: the encoder compresses each source grapheme sequence INLINEFORM0 into a fixed-length vector  The decoder, conditioned on this fixed-length vector, generates the output phoneme sequence INLINEFORM1  
The encoder and decoder are both implemented as recurrent neural networks, which have the advantage of being able to process sequences of arbitrary length and use long histories efficiently  They are trained jointly to minimize cross-entropy on the training data  We had our best results when using a bidirectional encoder, which consists of two separate encoders which process the input in forward and reverse directions  We used long short-term memory units BIBREF10 for both the encoder and decoder  For the attention mechanism, we used the general global attention architecture described by luong2015effective 
We implemented all models with OpenNMT BIBREF11   Our hyperparameters, which we determined by experimentation, are listed in Table TABREF8  
Training Multilingual Models
Presenting pronunciation data in several languages to the network might create problems because different languages have different pronunciation patterns  For example, the string `real' is pronounced differently in English, German, Spanish, and Portuguese  We solve this problem by prepending each grapheme sequence with an artificial token consisting of the language's ISO 639-3 code enclosed in angle brackets  The English word `real', for example, would be presented to the system as
INLINEFORM0 eng INLINEFORM1 r e a l
The artificial token is treated simply as an element of the grapheme sequence  This is similar to the approach taken by johnson2016google in their zero-shot NMT system  However, their source-side artificial tokens identify the target language, whereas ours identify the source language  An alternative approach, used by ostling2017continuous, would be to concatenate a language embedding to the input at each time step  They do not evaluate their approach on grapheme-to-phoneme conversion 
Data
In order to train a neural g2p system, one needs a large quantity of pronunciation data  A standard dataset for g2p is the Carnegie Mellon Pronouncing Dictionary BIBREF12   However, that is a monolingual English resource, so it is unsuitable for our multilingual task  Instead, we use the multilingual pronunciation corpus collected by deri2016grapheme for all experiments  This corpus consists of spelling–pronunciation pairs extracted from Wiktionary  It is already partitioned into training and test sets  Corpus statistics are presented in Table TABREF10  
In addition to the raw IPA transcriptions extracted from Wiktionary, the corpus provides an automatically cleaned version of transcriptions  Cleaning is a necessary step because web-scraped data is often noisy and may be transcribed at an inconsistent level of detail  The data cleaning used here attempts to make the transcriptions consistent with the phonemic inventories used in Phoible BIBREF4   When a transcription contains a phoneme that is not in its language's inventory in Phoible, that phoneme is replaced by the phoneme with the most similar articulatory features that is in the language's inventory  Sometimes this cleaning algorithm works well: in the German examples in Table TABREF11 , the raw German symbols and are both converted to   This is useful because the in Ansbach and the in Kaninchen are instances of the same phoneme, so their phonemic representations should use the same symbol  However, the cleaning algorithm can also have negative effects on the data quality  For example, the phoneme is not present in the Phoible inventory for German, but it is used in several German transcriptions in the corpus  The cleaning algorithm converts to in all German transcriptions, whereas would be a more reasonable guess  The cleaning algorithm also removes most suprasegmentals, even though these are often an important part of a language's phonology  Developing a more sophisticated procedure for cleaning pronunciation data is a direction for future work, but in this paper we use the corpus's provided cleaned transcriptions in order to ease comparison to previous results 
Experiments
We present experiments with two versions of our sequence-to-sequence model  LangID prepends each training, validation, and test sample with an artificial token identifying the language of the sample  NoLangID omits this token  LangID and NoLangID have identical structure otherwise  To translate the test corpus, we used a beam width of 100  Although this is an unusually wide beam and had negligible performance effects, it was necessary to compute our error metrics 
Evaluation
We use the following three evaluation metrics:
Phoneme Error Rate (PER) is the Levenshtein distance between the predicted phoneme sequences and the gold standard phoneme sequences, divided by the length of the gold standard phoneme sequences 
Word Error Rate (WER) is the percentage of words in which the predicted phoneme sequence does not exactly match the gold standard phoneme sequence 
Word Error Rate 100 (WER 100) is the percentage of words in the test set for which the correct guess is not in the first 100 guesses of the system 
In system evaluations, WER, WER 100, and PER numbers presented for multiple languages are averaged, weighting each language equally BIBREF13  
It would be interesting to compute error metrics that incorporate phoneme similarity, such as those proposed by hixon2011phonemic  PER weights all phoneme errors the same, even though some errors are more harmful than others: and are usually contrastive, whereas and almost never are  Such statistics would be especially interesting for evaluating a multilingual system, because different languages often map the same grapheme to phonemes that are only subtly different from each other  However, these statistics have not been widely reported for other g2p systems, so we omit them here 
Baseline
Results on LangID and NoLangID are compared to the system presented by deri2016grapheme, which is identified in our results as wFST  Their results can be divided into two parts:
High resource results, computed with wFSTs trained on a combination of Wiktionary pronunciation data and g2p rules extracted from Wikipedia IPA Help pages  They report high resource results for 85 languages 
Adapted results, where they apply various mapping strategies in order to adapt high resource models to other languages  The final adapted results they reported include most of the 85 languages with high resource results, as well as the various languages they were able to adapt them for, for a total of 229 languages  This test set omits 23 of the high resource languages that are written in unique scripts or for which language distance metrics could not be computed 
Training
We train the LangID and NoLangID versions of our model each on three subsets of the Wiktionary data:
LangID-High and NoLangID-High: Trained on data from the 85 languages for which BIBREF13 used non-adapted wFST models 
LangID-Adapted and NoLangID-Adapted: Trained on data from any of the 229 languages for which they built adapted models  Because many of these languages had no training data at all, the model is actually only trained on data in 157 languages  As is noted above, the Adapted set omits 23 languages which are in the High test set 
LangID-All and NoLangID-All: Trained on data in all 311 languages in the Wiktionary training corpus 
In order to ease comparison to Deri and Knight's system, we limited our use of the training corpus to 10,000 words per language  We set aside 10 percent of the data in each language for validation, so the maximum number of training words for any language is 9000 for our systems 
Adapted Results
On the 229 languages for which deri2016grapheme presented their final results, the LangID version of our system outperforms the baseline by a wide margin  The best performance came with the version of our model that was trained on data in all available languages, not just the languages it was tested on  Using a language ID token improves results considerably, but even NoLangID beats the baseline in WER and WER 100  Full results are presented in Table TABREF24  
High Resource Results
Having shown that our model exceeds the performance of the wFST-adaptation approach, we next compare it to the baseline models for just high resource languages  The wFST models here are purely monolingual – they do not use data adaptation because there is sufficient training data for each of them  Full results are presented in Table TABREF26   We omit models trained on the Adapted languages because they were not trained on high resource languages with unique writing systems, such as Georgian and Greek, and consequently performed very poorly on them 
In contrast to the larger-scale Adapted results, in the High Resource experiments none of the sequence-to-sequence approaches equal the performance of the wFST model in WER and PER, although LangID-High does come close  The LangID models do beat wFST in WER 100  A possible explanation is that a monolingual wFST model will never generate phonemes that are not part of the language's inventory  A multilingual model, on the other hand, could potentially generate phonemes from the inventories of any language it has been trained on 
Even if LangID-High does not present a more accurate result, it does present a more compact one: LangID-High is 15 4 MB, while the combined wFST high resource models are 197 5 MB 
Results on Unseen Languages
Finally, we report our models' results on unseen languages in Table TABREF28   The unseen languages are any that are present in the test corpus but absent from the training data  Deri and Knight did not report results specifically on these languages  Although the NoLangID models sometimes do better on WER 100, even here the LangID models have a slight advantage in WER and PER  This is somewhat surprising because the LangID models have not learned embeddings for the language ID tokens of unseen languages  Perhaps negative associations are also being learned, driving the model towards predicting more common pronunciations for unseen languages 
Language ID Tokens
Adding a language ID token always improves results in cases where an embedding has been learned for that token  The power of these embeddings is demonstrated by what happens when one feeds the same input word to the model with different language tokens, as is seen in Table TABREF30   Impressively, this even works when the source sequence is in the wrong script for the language, as is seen in the entry for Arabic 
Language Embeddings
Because these language ID tokens are so useful, it would be good if they could be effectively estimated for unseen languages  ostling2017continuous found that the language vectors their models learned correlated well to genetic relationships, so it would be interesting to see if the embeddings our source encoder learned for the language ID tokens showed anything similar  In a few cases they do (the languages closest to German in the vector space are Luxembourgish, Bavarian, and Yiddish, all close relatives)  However, for the most part the structure of these vectors is not interpretable  Therefore, it would be difficult to estimate the embedding for an unseen language, or to “borrow” the language ID token of a similar language  A more promising way forward is to find a model that uses an externally constructed typological representation of the language 
Phoneme Embeddings
In contrast to the language embeddings, the phoneme embeddings appear to show many regularities (see Table TABREF33 )  This is a sign that our multilingual model learns similar embeddings for phonemes that are written with the same grapheme in different languages  These phonemes tend to be phonetically similar to each other 
Perhaps the structure of the phoneme embedding space is what leads to our models' very good performance on WER 100  Even when the model's first predicted pronunciation is not correct, it tends to assign more probability mass to guesses that are more similar to the correct one  Applying some sort of filtering or reranking of the system output might therefore lead to better performance 
Future Work
Because the language ID token is so beneficial to performance, it would be very interesting to find ways to extend a similar benefit to unseen languages  One possible way to do so is with tokens that identify something other than the language, such as typological features about the language's phonemic inventory  This could enable better sharing of resources among languages  Such typological knowledge is readily available in databases like Phoible and WALS for a wide variety of languages  It would be interesting to explore if any of these features is a good predictor of a language's orthographic rules 
It would also be interesting to apply the artificial token approach to other problems besides multilingual g2p  One closely related application is monolingual English g2p  Some of the ambiguity of English spelling is due to the wide variety of loanwords in the language, many of which have unassimilated spellings  Knowing the origins of these loanwords could provide a useful hint for figuring out their pronunciations  The etymology of a word could be tagged in an analogous way to how language ID is tagged in multilingual g2p 
Context 2: Introduction
Recurrent neural networks (RNNs), including gated variants such as the long short-term memory (LSTM) BIBREF0 have become the standard model architecture for deep learning approaches to sequence modeling tasks  RNNs repeatedly apply a function with trainable parameters to a hidden state  Recurrent layers can also be stacked, increasing network depth, representational power and often accuracy  RNN applications in the natural language domain range from sentence classification BIBREF1 to word- and character-level language modeling BIBREF2   RNNs are also commonly the basic building block for more complex models for tasks such as machine translation BIBREF3 , BIBREF4 , BIBREF5 or question answering BIBREF6 , BIBREF7   Unfortunately standard RNNs, including LSTMs, are limited in their capability to handle tasks involving very long sequences, such as document classification or character-level machine translation, as the computation of features or states for different parts of the document cannot occur in parallel 
Convolutional neural networks (CNNs) BIBREF8 , though more popular on tasks involving image data, have also been applied to sequence encoding tasks BIBREF9   Such models apply time-invariant filter functions in parallel to windows along the input sequence  CNNs possess several advantages over recurrent models, including increased parallelism and better scaling to long sequences such as those often seen with character-level language data  Convolutional models for sequence processing have been more successful when combined with RNN layers in a hybrid architecture BIBREF10 , because traditional max- and average-pooling approaches to combining convolutional features across timesteps assume time invariance and hence cannot make full use of large-scale sequence order information 
We present quasi-recurrent neural networks for neural sequence modeling  QRNNs address both drawbacks of standard models: like CNNs, QRNNs allow for parallel computation across both timestep and minibatch dimensions, enabling high throughput and good scaling to long sequences  Like RNNs, QRNNs allow the output to depend on the overall order of elements in the sequence  We describe QRNN variants tailored to several natural language tasks, including document-level sentiment classification, language modeling, and character-level machine translation  These models outperform strong LSTM baselines on all three tasks while dramatically reducing computation time 
Model
Each layer of a quasi-recurrent neural network consists of two kinds of subcomponents, analogous to convolution and pooling layers in CNNs  The convolutional component, like convolutional layers in CNNs, allows fully parallel computation across both minibatches and spatial dimensions, in this case the sequence dimension  The pooling component, like pooling layers in CNNs, lacks trainable parameters and allows fully parallel computation across minibatch and feature dimensions 
Given an input sequence INLINEFORM0 of INLINEFORM1 INLINEFORM2 -dimensional vectors INLINEFORM3 , the convolutional subcomponent of a QRNN performs convolutions in the timestep dimension with a bank of INLINEFORM4 filters, producing a sequence INLINEFORM5 of INLINEFORM6 -dimensional candidate vectors INLINEFORM7   In order to be useful for tasks that include prediction of the next token, the filters must not allow the computation for any given timestep to access information from future timesteps  That is, with filters of width INLINEFORM8 , each INLINEFORM9 depends only on INLINEFORM10 through INLINEFORM11   This concept, known as a masked convolution BIBREF11 , is implemented by padding the input to the left by the convolution's filter size minus one 
We apply additional convolutions with separate filter banks to obtain sequences of vectors for the elementwise gates that are needed for the pooling function  While the candidate vectors are passed through a INLINEFORM0 nonlinearity, the gates use an elementwise sigmoid  If the pooling function requires a forget gate INLINEFORM1 and an output gate INLINEFORM2 at each timestep, the full set of computations in the convolutional component is then: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 , each in INLINEFORM3 , are the convolutional filter banks and INLINEFORM4 denotes a masked convolution along the timestep dimension  Note that if the filter width is 2, these equations reduce to the LSTM-like DISPLAYFORM0
Convolution filters of larger width effectively compute higher INLINEFORM0 -gram features at each timestep; thus larger widths are especially important for character-level tasks 
Suitable functions for the pooling subcomponent can be constructed from the familiar elementwise gates of the traditional LSTM cell  We seek a function controlled by gates that can mix states across timesteps, but which acts independently on each channel of the state vector  The simplest option, which BIBREF12 term “dynamic average pooling”, uses only a forget gate: DISPLAYFORM0
We term these three options f-pooling, fo-pooling, and ifo-pooling respectively; in each case we initialize INLINEFORM0 or INLINEFORM1 to zero  Although the recurrent parts of these functions must be calculated for each timestep in sequence, their simplicity and parallelism along feature dimensions means that, in practice, evaluating them over even long sequences requires a negligible amount of computation time 
A single QRNN layer thus performs an input-dependent pooling, followed by a gated linear combination of convolutional features  As with convolutional neural networks, two or more QRNN layers should be stacked to create a model with the capacity to approximate more complex functions 
Variants
Motivated by several common natural language tasks, and the long history of work on related architectures, we introduce several extensions to the stacked QRNN described above  Notably, many extensions to both recurrent and convolutional models can be applied directly to the QRNN as it combines elements of both model types 
Regularization An important extension to the stacked QRNN is a robust regularization scheme inspired by recent work in regularizing LSTMs 
The need for an effective regularization method for LSTMs, and dropout's relative lack of efficacy when applied to recurrent connections, led to the development of recurrent dropout schemes, including variational inference–based dropout BIBREF13 and zoneout BIBREF14   These schemes extend dropout to the recurrent setting by taking advantage of the repeating structure of recurrent networks, providing more powerful and less destructive regularization 
Variational inference–based dropout locks the dropout mask used for the recurrent connections across timesteps, so a single RNN pass uses a single stochastic subset of the recurrent weights  Zoneout stochastically chooses a new subset of channels to “zone out” at each timestep; for these channels the network copies states from one timestep to the next without modification 
As QRNNs lack recurrent weights, the variational inference approach does not apply  Thus we extended zoneout to the QRNN architecture by modifying the pooling function to keep the previous pooling state for a stochastic subset of channels  Conveniently, this is equivalent to stochastically setting a subset of the QRNN's INLINEFORM0 gate channels to 1, or applying dropout on INLINEFORM1 : DISPLAYFORM0
Thus the pooling function itself need not be modified at all  We note that when using an off-the-shelf dropout layer in this context, it is important to remove automatic rescaling functionality from the implementation if it is present  In many experiments, we also apply ordinary dropout between layers, including between word embeddings and the first QRNN layer 
Densely-Connected Layers We can also extend the QRNN architecture using techniques introduced for convolutional networks  For sequence classification tasks, we found it helpful to use skip-connections between every QRNN layer, a technique termed “dense convolution” by BIBREF15   Where traditional feed-forward or convolutional networks have connections only between subsequent layers, a “DenseNet” with INLINEFORM0 layers has feed-forward or convolutional connections between every pair of layers, for a total of INLINEFORM1   This can improve gradient flow and convergence properties, especially in deeper networks, although it requires a parameter count that is quadratic in the number of layers 
When applying this technique to the QRNN, we include connections between the input embeddings and every QRNN layer and between every pair of QRNN layers  This is equivalent to concatenating each QRNN layer's input to its output along the channel dimension before feeding the state into the next layer  The output of the last layer alone is then used as the overall encoding result 
Encoder–Decoder Models To demonstrate the generality of QRNNs, we extend the model architecture to sequence-to-sequence tasks, such as machine translation, by using a QRNN as encoder and a modified QRNN, enhanced with attention, as decoder  The motivation for modifying the decoder is that simply feeding the last encoder hidden state (the output of the encoder's pooling layer) into the decoder's recurrent pooling layer, analogously to conventional recurrent encoder–decoder architectures, would not allow the encoder state to affect the gate or update values that are provided to the decoder's pooling layer  This would substantially limit the representational power of the decoder 
Instead, the output of each decoder QRNN layer's convolution functions is supplemented at every timestep with the final encoder hidden state  This is accomplished by adding the result of the convolution for layer INLINEFORM0 (e g , INLINEFORM1 , in INLINEFORM2 ) with broadcasting to a linearly projected copy of layer INLINEFORM3 's last encoder state (e g , INLINEFORM4 , in INLINEFORM5 ): DISPLAYFORM0
where the tilde denotes that INLINEFORM0 is an encoder variable  Encoder–decoder models which operate on long sequences are made significantly more powerful with the addition of soft attention BIBREF3 , which removes the need for the entire input representation to fit into a fixed-length encoding vector  In our experiments, we computed an attentional sum of the encoder's last layer's hidden states  We used the dot products of these encoder hidden states with the decoder's last layer's un-gated hidden states, applying a INLINEFORM1 along the encoder timesteps, to weight the encoder states into an attentional sum INLINEFORM2 for each decoder timestep  This context, and the decoder state, are then fed into a linear layer followed by the output gate: DISPLAYFORM0
where INLINEFORM0 is the last layer 
While the first step of this attention procedure is quadratic in the sequence length, in practice it takes significantly less computation time than the model's linear and convolutional layers due to the simple and highly parallel dot-product scoring function 
Experiments
We evaluate the performance of the QRNN on three different natural language tasks: document-level sentiment classification, language modeling, and character-based neural machine translation  Our QRNN models outperform LSTM-based models of equal hidden size on all three tasks while dramatically improving computation speed  Experiments were implemented in Chainer BIBREF16  
Sentiment Classification
We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset BIBREF17   The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words BIBREF18   We compare only to other results that do not make use of additional unlabeled data (thus excluding e g , BIBREF19 ) 
Our best performance on a held-out development set was achieved using a four-layer densely-connected QRNN with 256 units per layer and word vectors initialized using 300-dimensional cased GloVe embeddings BIBREF20   Dropout of 0 3 was applied between layers, and we used INLINEFORM0 regularization of INLINEFORM1   Optimization was performed on minibatches of 24 examples using RMSprop BIBREF21 with learning rate of INLINEFORM2 , INLINEFORM3 , and INLINEFORM4  
Small batch sizes and long sequence lengths provide an ideal situation for demonstrating the QRNN's performance advantages over traditional recurrent architectures
Context 3: Introduction
Medical search engines are an essential component for many online medical applications, such as online diagnosis systems and medical document databases  A typical online diagnosis system, for instance, relies on a medical search engine  The search engine takes as input a user query that describes some symptoms and then outputs clinical concept entries that provide relevant information to assist in diagnosing the problem  One challenge medical search engines face is the segmentation of individual clinical entities  When a user query consists of multiple clinical entities, a search engine would often fail to recognize them as separate entities  For example, the user query “fever joint pain weight loss headache” contains four separate clinical entities: “fever”, “joint pain”, “weight loss”, and “headache”  But when the search engine does not recognize them as separate entities and proceeds to retrieve results for each word in the query, it may find "pain" in body locations other than "joint pain", or it may miss "headache" altogether, for example  Some search engines allow the users to enter a single clinical concept by selecting from an auto-completion pick list  But this could also result in retrieving inaccurate or partial results and lead to poor user experience 
We want to improve the medical search engine so that it can accurately retrieve all the relevant clinical concepts mentioned in a user query, where relevant clinical concepts are defined with respect to the terminologies the search engine uses  The problem of extracting clinical concept mentions from a user query can be seen as a variant of the Concept Extraction (CE) task in the frequently-cited NLP challenges in healthcare, such as 2010 i2b2/VA BIBREF0 and 2013 ShARe/CLEF Task 1 BIBREF1  Both CE tasks in 2010 i2b2/VA and 2013 ShARe/CLEF Task 1 ask the participants to design an algorithm to tag a set of predefined entities of interest in clinical notes  These entity tagging tasks are also known as clinical Named Entity Recognition (NER)  For example, the CE task in 2010 i2b2/VA defines three types of entities: “problem”, “treatment”, and “test”  The CE task in 2013 ShARe/CLEF defines various types of disorder such as “injury or poisoning”, "disease or syndrome”, etc  In addition to tagging, the CE task in 2013 ShARe/CLEF has an encoding component which requires selecting one and only one Concept Unique Identifier (CUI) from Systematized Nomenclature Of Medicine Clinical Terms (SNOMED-CT) for each disorder entity tagged  Our problem, similar to the CE task in 2013 ShARe/CLEF, also contains two sub-problems: tagging mentions of entities of interest (entity tagging), and selecting appropriate terms from a glossary to match the mentions (term matching)  However, several major differences exist  First, compared to clinical notes, the user queries are much shorter, less technical, and often less coherent  Second, instead of encoding, we are dealing with term matching where we rank a few best terms that match an entity, instead of selecting only one  This is because the users who type the queries may not have a clear idea about what they are looking for, or could be laymen who know little terminology, it may be more helpful to provide a set of likely results and let the users choose  Third, the types of entities are different  Each medical search engine may have its own types of entities to tag  There is also one minor difference in the tagging scheme between our problem and the CE task in 2013 ShARe/CLEF - We limit our scope to dealing with entities of consecutive words and not disjoint entities   We use only Beginning, Inside, Outside (BIO) tags  Given the differences listed above, we need to customize a framework consisting of an entity tagging and term matching component for our CE problem 
Related Work
An effective model that has been commonly used for NER problem is a Bi-directional LSTM with a Conditional Random Field (CRF) on the top layer (BiLSTM-CRF), which is described in the next section  Combining LSTM’s power of representing relations between words and CRF’s capability of accounting for tag sequence constraints, Huang et al  BIBREF2 proposed the BiLSTM-CRF model and used handcrafted word features as the input to the model  Lample et al  BIBREF3 used a combination of character-level and word-level word embeddings as the input to BiLSTM-CRF  Since then, similar models with variation in types of word embeddings have been used extensively for clinical CE tasks and produced state-of-the-art results BIBREF4, BIBREF5, BIBREF6, BIBREF7  Word embeddings have become the cornerstone of the neural models in NLP since the famous Word2vec BIBREF8 model demonstrated its power in word analogy tasks  One well-known example is that after training Word2vec on a large amount of news data, we can get word relations such as $vector(^{\prime }king^{\prime }) - vector(^{\prime }queen^{\prime }) + vector(^{\prime }woman^{\prime }) \approx vector(^{\prime }man^{\prime })$  More sophisticated word embedding technique emerged since Word2vec  It has been shown empirically that better quality in word embeddings leads to better performance in many downstream NLP including entity tagging BIBREF9, BIBREF10  Recently, contextualized word embeddings generated by deep learning models, such as ELMo BIBREF11, BERT BIBREF12, and Flair BIBREF13, have been shown to be more effective in various NLP tasks  In our project, we make use of a fine-tuned ELMo model and a fine-tuned Flair model in the medical domain  We experiment with the word embeddings from the two fine-tuned models as the input to the BiLSTM-CRF model separately and compare the results 
Tang et al  BIBREF14 provided straightforward algorithm for term matching  The algorithm starts with finding candidate terms that contain ALL the entity words, with term frequency - inverse document frequency (tf-idf) weighting  Then the candidates are ranked based on the pairwise cosine distance between the word embeddings of the candidates and the entity 
Framework
We adopt the tagging - encoding pipeline framework from the CE task in 2013 ShARe/CLEF  We first tag the clinical entities in the user query and then select relevant terms from a glossary in dermatology to match the entities 
Framework ::: Entity Tagging
We use the same BiLSTM-CRF model proposed by Huang et al  BIBREF2  An illustration of the architecture is shown in Figure FIGREF6   Given a sequence (or sentence) of n tokens, $r = (w_1, w_2,   , w_n)$, we use a fine-tuned ELMo model to generate contextual word embeddings for all the tokens in the sentence, where a token refers to a word or punctuation  We denote the ELMo embedding, $x$, for a token $w$ in the sentence $r$ by $x = ELMo(w|r)$  The notation and the procedure described here can be adopted for Flair embeddings or other embeddings  Now, given a sequence of tokens in ELMo embeddings, $X =(x_1, x_2,    , x_n)$, the BiLSTM layer generates a matrix of scores, $P(\theta )$ of size $n \times k$, where $k$ is the number of tag types, and $\theta $ is the parameters of the BiLSTM  To simplify notation, we will omit the $\theta $ and write $P$  Then, $P_{i,j}$ denotes the score of the token, $x_i$, being assigned to the $j$th tag  Since certain constraints may exist in the transition between tags, an "O" tag should not be followed by an "I" tag, for example, a transition matrix, $A$, of dimension $(k+2)\times (k+2)$, is initialized to model the constraints  The learnable parameters, $A_{i,j}$, represent the probability of the $j$th tag follows the $i$th tag in a sequence

ground_truth: the Carnegie Mellon Pronouncing Dictionary BIBREF12, the multilingual pronunciation corpus collected by deri2016grapheme , ranscriptions extracted from Wiktionary

multilingual pronunciation corpus collected by deri2016grapheme

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What sentiment classification dataset is used?
Context 2:  We observed a speedup of 3 2x on IMDb train time per epoch compared to the optimized LSTM implementation provided in NVIDIA's cuDNN library  For specific batch sizes and sequence lengths, a 16x speed gain is possible  Figure FIGREF15 provides extensive speed comparisons 
In Figure FIGREF12 , we visualize the hidden state vectors INLINEFORM0 of the final QRNN layer on part of an example from the IMDb dataset  Even without any post-processing, changes in the hidden state are visible and interpretable in regards to the input  This is a consequence of the elementwise nature of the recurrent pooling function, which delays direct interaction between different channels of the hidden state until the computation of the next QRNN layer 
Language Modeling
We replicate the language modeling experiment of BIBREF2 and BIBREF13 to benchmark the QRNN architecture for natural language sequence prediction  The experiment uses a standard preprocessed version of the Penn Treebank (PTB) by BIBREF25  
We implemented a gated QRNN model with medium hidden size: 2 layers with 640 units in each layer  Both QRNN layers use a convolutional filter width INLINEFORM0 of two timesteps  While the “medium” models used in other work BIBREF2 , BIBREF13 consist of 650 units in each layer, it was more computationally convenient to use a multiple of 32  As the Penn Treebank is a relatively small dataset, preventing overfitting is of considerable importance and a major focus of recent research  It is not obvious in advance which of the many RNN regularization schemes would perform well when applied to the QRNN  Our tests showed encouraging results from zoneout applied to the QRNN's recurrent pooling layer, implemented as described in Section SECREF5  
The experimental settings largely followed the “medium” setup of BIBREF2   Optimization was performed by stochastic gradient descent (SGD) without momentum  The learning rate was set at 1 for six epochs, then decayed by 0 95 for each subsequent epoch, for a total of 72 epochs  We additionally used INLINEFORM0 regularization of INLINEFORM1 and rescaled gradients with norm above 10  Zoneout was applied by performing dropout with ratio 0 1 on the forget gates of the QRNN, without rescaling the output of the dropout function  Batches consist of 20 examples, each 105 timesteps 
Comparing our results on the gated QRNN with zoneout to the results of LSTMs with both ordinary and variational dropout in Table TABREF14 , we see that the QRNN is highly competitive  The QRNN without zoneout strongly outperforms both our medium LSTM and the medium LSTM of BIBREF2 which do not use recurrent dropout and is even competitive with variational LSTMs  This may be due to the limited computational capacity that the QRNN's pooling layer has relative to the LSTM's recurrent weights, providing structural regularization over the recurrence 
Without zoneout, early stopping based upon validation loss was required as the QRNN would begin overfitting  By applying a small amount of zoneout ( INLINEFORM0 ), no early stopping is required and the QRNN achieves competitive levels of perplexity to the variational LSTM of BIBREF13 , which had variational inference based dropout of 0 2 applied recurrently  Their best performing variation also used Monte Carlo (MC) dropout averaging at test time of 1000 different masks, making it computationally more expensive to run 
When training on the PTB dataset with an NVIDIA K40 GPU, we found that the QRNN is substantially faster than a standard LSTM, even when comparing against the optimized cuDNN LSTM  In Figure FIGREF15 we provide a breakdown of the time taken for Chainer's default LSTM, the cuDNN LSTM, and QRNN to perform a full forward and backward pass on a single batch during training of the RNN LM on PTB  For both LSTM implementations, running time was dominated by the RNN computations, even with the highly optimized cuDNN implementation  For the QRNN implementation, however, the “RNN” layers are no longer the bottleneck  Indeed, there are diminishing returns from further optimization of the QRNN itself as the softmax and optimization overhead take equal or greater time  Note that the softmax, over a vocabulary size of only 10,000 words, is relatively small; for tasks with larger vocabularies, the softmax would likely dominate computation time 
It is also important to note that the cuDNN library's RNN primitives do not natively support any form of recurrent dropout  That is, running an LSTM that uses a state-of-the-art regularization scheme at cuDNN-like speeds would likely require an entirely custom kernel 
Character-level Neural Machine Translation
We evaluate the sequence-to-sequence QRNN architecture described in SECREF5 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation  This dataset consists of 209,772 sentence pairs of parallel training data from transcribed TED and TEDx presentations, with a mean sentence length of 103 characters for German and 93 for English  We remove training sentences with more than 300 characters in English or German, and use a unified vocabulary of 187 Unicode code points 
Our best performance on a development set (TED tst2013) was achieved using a four-layer encoder–decoder QRNN with 320 units per layer, no dropout or INLINEFORM0 regularization, and gradient rescaling to a maximum magnitude of 5  Inputs were supplied to the encoder reversed, while the encoder convolutions were not masked  The first encoder layer used convolutional filter width INLINEFORM1 , while the other encoder layers used INLINEFORM2   Optimization was performed for 10 epochs on minibatches of 16 examples using Adam BIBREF28 with INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , and INLINEFORM6   Decoding was performed using beam search with beam width 8 and length normalization INLINEFORM7   The modified log-probability ranking criterion is provided in the appendix 
Results using this architecture were compared to an equal-sized four-layer encoder–decoder LSTM with attention, applying dropout of 0 2  We again optimized using Adam; other hyperparameters were equal to their values for the QRNN and the same beam search procedure was applied  Table TABREF17 shows that the QRNN outperformed the character-level LSTM, almost matching the performance of a word-level attentional baseline 
Related Work
Exploring alternatives to traditional RNNs for sequence tasks is a major area of current research  Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by BIBREF12   While the motivation and constraints described in that work are different, BIBREF12 's concepts of “learnware” and “firmware” parallel our discussion of convolution-like and pooling-like subcomponents  As the use of a fully connected layer for recurrent connections violates the constraint of “strong typing”, all strongly-typed RNN architectures (including the T-RNN, T-GRU, and T-LSTM) are also quasi-recurrent  However, some QRNN models (including those with attention or skip-connections) are not “strongly typed”  In particular, a T-RNN differs from a QRNN as described in this paper with filter size 1 and f-pooling only in the absence of an activation function on INLINEFORM0   Similarly, T-GRUs and T-LSTMs differ from QRNNs with filter size 2 and fo- or ifo-pooling respectively in that they lack INLINEFORM1 on INLINEFORM2 and use INLINEFORM3 rather than sigmoid on INLINEFORM4  
The QRNN is also related to work in hybrid convolutional–recurrent models  BIBREF31 apply CNNs at the word level to generate INLINEFORM0 -gram features used by an LSTM for text classification  BIBREF32 also tackle text classification by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM  A similar approach was taken by BIBREF10 for character-level machine translation  Their model's encoder uses a convolutional layer followed by max-pooling to reduce sequence length, a four-layer highway network, and a bidirectional GRU  The parallelism of the convolutional, pooling, and highway layers allows training speed comparable to subword-level models without hard-coded text segmentation 
The QRNN encoder–decoder model shares the favorable parallelism and path-length properties exhibited by the ByteNet BIBREF33 , an architecture for character-level machine translation based on residual convolutions over binary trees  Their model was constructed to achieve three desired properties: parallelism, linear-time computational complexity, and short paths between any pair of words in order to better propagate gradient signals 
Conclusion
Intuitively, many aspects of the semantics of long sequences are context-invariant and can be computed in parallel (e g , convolutionally), but some aspects require long-distance context and must be computed recurrently  Many existing neural network architectures either fail to take advantage of the contextual information or fail to take advantage of the parallelism  QRNNs exploit both parallelism and context, exhibiting advantages from both convolutional and recurrent neural networks  QRNNs have better predictive accuracy than LSTM-based models of equal hidden size, even though they use fewer parameters and run substantially faster  Our experiments show that the speed and accuracy advantages remain consistent across tasks and at both word and character levels 
Extensions to both CNNs and RNNs are often directly applicable to the QRNN, while the model's hidden states are more interpretable than those of other recurrent architectures as its channels maintain their independence across timesteps  We believe that QRNNs can serve as a building block for long-sequence tasks that were previously impractical with traditional RNNs 
Beam search ranking criterion
The modified log-probability ranking criterion we used in beam search for translation experiments is: DISPLAYFORM0
where INLINEFORM0 is a length normalization parameter BIBREF34 , INLINEFORM1 is the INLINEFORM2 th output character, and INLINEFORM3 is a “target length” equal to the source sentence length plus five characters  This reduces at INLINEFORM4 to ordinary beam search with probabilities: DISPLAYFORM0
and at INLINEFORM0 to beam search with probabilities normalized by length (up to the target length): DISPLAYFORM0
Conveniently, this ranking criterion can be computed at intermediate beam-search timesteps, obviating the need to apply a separate reranking on complete hypotheses 
Context 3:  For example, if we index the tags by: 1:“B”, 2:“I”, and 3:“O”, then $A_{1,3}$ would be the probability that an “O” tag follows a “B” tag  A beginning transition and an end transition are inserted in $A$ and hence $A$ is of dimension $(k+2)\times (k+2)$ 
Given a sequence of tags, $Y=(y_1,y_2,   ,y_n)$, where each $y_i$, $1\le i \le n$, corresponds to an index of the tags, the score of the sequence is then given by
The probability of the sequence of tags is then calculated by a softmax,
where $\lbrace Y_x\rbrace $ denotes the set of all possible tag sequences  During training, the objective function is to maximize $\log (P(Y|X))$ by adjusting $A$ and $P$ 
Framework ::: Term Matching
The term matching algorithm of Tang et al  BIBREF14 is adopted with some major modifications  First, to identify candidate terms, we use a much looser string search algorithm where we stem the entity words with snowball stemmer and then find candidate terms which contain ANY non-stopword words in the entity  The stemming is mainly used to represent a word in its stem  For example, “legs” becomes “leg”, and “jammed” becomes “jam”  Thus, stemming can provide more tolerance when finding candidates  Similarly, finding candidates using the condition ANY (instead of ALL) also increases the tolerance  However, if the tagged entity contains stopwords such as “in”, “on”, etc , the pool of candidates will naturally grow very large and increase the computation cost for later part, because even completely irrelevant terms may contain stopwords  Therefore, we match based on non-stopwords only  To illustrate the points above, suppose a query is tagged with the entity
Ex  3 1 “severe burns on legs”,
and one relevant term is “leg burn”  After stemming, “burns” and “legs” in Ex UNKREF12 become “burn” and “leg”, respectively, allowing "leg burn" to be considered as a candidate  Although the word “severe” is not in the term “leg burn”, the term is still considered a candidate because we selected using ANY  The stopword “on” is ignored when finding candidate terms so that not every term that contains the word “on” is added to the candidate pool  When a candidate term, $C$, is found in this manner for the tagged entity, $E$, we calculate the semantic similarity score, $s$, between $C$ and $E$ in two steps  In the first step, calculate the maximum similarity score for each word in $C$ as shown in Figure FIGREF10  Given a word in the candidate term, $C_i$ ($1 \le i \le m$, $m$ is the number of words in the candidate term) and a word in the tagged entity, $E_j$ Their similarity score, $s_{ij}$ (shown as the element in the boxed matrix in Figure FIGREF10), is given by
where $ELMo(C_i|C)$ and $ELMo(E_j|E)$ are the ELMo embeddings for the word $C_i$ and $E_j$, respectively  The ELMo embeddings have the same dimension for all words when using the same fine-tuned ELMo model  Thus, we can use a distance function (e g , the cosine distance), denoted $d(\cdot )$ in equation DISPLAY_FORM13, to compute the semantic similarity between words  In step 2, we calculate the candidate-entity relevance score (similarity) using the formula
where $s_c$ is a score threshold, and $\mathbb {I} \lbrace max(\vec{S_i}) > s_c\rbrace $ is an indicator function that equals 1 if $max(\vec{S_i}) > s_c$ or equals 0 if not  In equation DISPLAY_FORM14 we define a metric that measures “information coverage” of the candidate terms with respect to a tagged entity  If the constituent words of a candidate term are relevant to the constituent words in the tagged entity, then the candidate term offers more information coverage  Intuitively, the more relevant words present in the candidate term, the more relevant the candidate is to the tagged entity  The purpose of the cutoff, $s_c$, is to screen the $(C_i,E_j)$ word pairs that are dissimilar, so that they do not contribute to information coverage  One can adjust the strictness of the entity - terminology matching by adjusting $s_c$  The higher we set $s_c$, the fewer candidate terms will be selected for a tagged entity  A normalization factor, $\frac{1}{m}$, is added to give preference to more concise candidate terms given the same amount of information coverage 
We need to create an extra stopword list to include words such as “configuration” and “color”, and exclude these words from the word count for a candidate term  This is because the terms associated with the description of color or configuration usually have the word “color” or “configuration” in them  On the other hand, a user query normally does not contain such words  For example, a tagged entity in a user query could be “round yellow patches”, for which the relevant terminologies include “round configuration” and “yellow color”  Since we applied a normalization factor, $\frac{1}{m}$, to the relevance score, the word “color” and “configuration” would lower the relevance score because they do not have a counterpart in the tagged entity  Therefore, we need to exclude them from word count  Once the process is complete, calculate $s(C,E)$ for all candidate terms and then we can apply a threshold on all $s(C,E)$ to ignore candidate terms with low information coverage  Finally, rank the terms by their $s(C,E)$ and return the ranked list as the results 
Experiments ::: Data
Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes  The synthesized user queries are generated using the aforementioned dermatology glossary  Tagged sentences are extracted from the clinical notes  Sentences with no clinical entity present are ignored  22,489 tagged sentences are extracted from the clinical notes  We will refer to these tagged sentences interchangeably as the i2b2 data  The sentences are shuffled and split into train/dev/test set with a ratio of 7:2:1  The synthesized user queries are composed by randomly selecting several clinical terms from the dermatology glossary and then combining them in no particular order  When combining the clinical terms, we attach the BIO tags to their constituent words  The synthesized user queries (13,697 in total) are then split into train/dev/test set with the same ratio  Next, each set in the i2b2 data and the corresponding set in the synthesized query data are combined to form a hybrid train/dev/test set, respectively  This way we ensure that in each hybrid train/dev/test set, the ratio between the i2b2 data and the synthesized query data is the same 
The reason for combining the two data is their drastic structural difference (See figure FIGREF16 for an example)  Previously, when trained on the i2b2 data only, the BiLSTM-CRF model was not able to segment clinical entities at the correct boundary

ground_truth: the IMDb movie review dataset BIBREF17

IMDb movie review

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. where did they obtain the annotated clinical notes from?
Context 3:  It would fail to recognize the user query in Figure FIGREF16(a) as four separate entities  On the other hand, if the model was trained solely on the synthesized user queries, we could imagine that it would fail miserably on any queries that resemble the sentence in Figure FIGREF16(b) because the model would have never seen an “O” tag in the training data  Therefore, it is necessary to use the hybrid training data containing both the i2b2 data and the synthesized user queries 
To make the hybrid training data, we need to unify the tags  Recall that in Section SECREF1 we point out that the tags are different for the different tasks and datasets  Since we use custom tags for dermatology glossary in our problem, we would need to convert the tags used in 2010 i2b2/VA  But this would be an infeasible job as we need experts to manually do that  An alternative is to avoid distinguishing the tag types and label all tags under the generic BIO tags 
Experiments ::: Setup
To show the effects of using the hybrid training data, we trained two models of the same architecture and hyperparameters  One model was trained on the hybrid data and will be referred to as hybrid NER model  The other model was trained on clinical notes only and will be referred to as i2b2 NER model  We evaluated the performance of the NER models by micro-F1 score on the test set of both the synthesized queries and the i2b2 data 
We used the BiLSTM-CRF implementation provided by the flair package BIBREF16  We set the hidden size value to be 256 in the LSTM structure and left everything else at default values for the SequenceTagger model on flair  For word embeddings, we used the ELMo embeddings fine-tuned on PubMed articles and flair embeddings BIBREF13 trained on $5\%$ of PubMed abstracts , respectively  We trained models for 10 epochs and experimented with different learning rate, mini batch size, and dropouts  We ran hyperparameter optimization tests to find the best combination  $S_c$ is set to be 0 6 in our experiment 
Experiments ::: Hyperparameter Tuning
We defined the following hyperparameter search space:
embeddings: [“ELMo on pubmed”, “stacked flair on pubmed”],
hidden_size: [128, 256],
learning_rate: [0 05, 0 1],
mini_batch_size: [32, 64, 128] 
The hyperparameter optimization was performed using Hyperopt   Three evaluations were run for each combination of hyperparameters  Each ran for 10 epochs  Then the results were averaged to give the performance for that particular combination of hyperparameters 
Experiments ::: Results
From the hyperparameter tuning we found that the best combination was
embeddings: “ELMo on pubmed”,
hidden_size: 256,
learning_rate: 0 05,
mini_batch_size: 32 
With the above hyperparameter setting, the hybrid NER model achieved a F1 score of $0 995$ on synthesized queries and $0 948$ on clinical notes while the i2b2 NER model achieved a F1 score of $0 441$ on synthesized queries and $0 927$ on clinical notes (See Table TABREF23) 
Since there was no ground truth available for the retrieved terms, we randomly picked a few samples to assess its performance  Some example outputs of our complete framework on real user queries are shown in Figure FIGREF24  For example, from the figure we see that the query "child fever double vision dizzy" was correctly tagged with four entities: "child", "fever", "double vision", and "dizzy"  A list of terms from our glossary was matched to each entity  In real world application, the lists of terms will be presented to the user as the retrieval results to their queries 
Discussion
In most real user queries we sampled, the entities were tagged at the correct boundary and the tagging was complete (such as the ones shown in Figure FIGREF24)  Only on a few user queries the tagging was controversial  For example, the query “Erythematous blanching round, oval patches on torso, extremities” was tagged as “Erythematous blanching” and “oval patches on torso”  The entity “extremities” was missing  The segmentation was also not correct  A more appropriate tagging would be “Erythematous blanching round, oval patches”, “torso”, and “extremities”  The tagging could be further improved by synthesizing more realistic user queries  Recall that the synthesized user queries were created by randomly combining terminologies from the dermatology glossary, which, while providing data that helped the model learn entity segmentation, did not reflect the co-occurrence information in real user queries  For example, there could be two clinical entities that often co-occur or never co-occur in a user query  But since the synthesized user queries we used combined terms randomly, the co-occurrence information was thus missing 
The final retrieval results of our framework were not evaluated quantitatively in terms of recall and precision, due the the lack of ground truth  When ground truth becomes available, we will be able to evaluate our framework more thoroughly  Recently, a fine-tuned BERT model in the medical domain called BioBERT BIBREF17 has attracted some attention in the medical NLP domain  We could experiment with BioBERT embeddings in the future  We could also include query expansion technique for term matching  When finding candidate terms for an entity, our first step was still based on string matching  Given that there might be multiple entities that could be matched to the same term, it could be hard to include all these entities in the glossary and hard to match terms to these entities 
Conclusion
In this project, we tackle the problem of extracting clinical concepts from user queries on medical search engines  By training a BiLSTM-CRF model on a hybrid data consisting of synthesized user queries and sentences from clinical note, we adopt a CE framework for clinical user queries with minimal effort spent on annotating user queries  We find that the hybrid data enables the NER model perform better on both tagging the user queries and the clinical note sentences  Furthermore, our framework is built on an easy-to-use deep learning NLP Python library, which lends it more prospective value to various online medical applications that employ medical search engines 
Acknowledgment
This paper results from a technical report of a project the authors have worked on with visualDx, a healthcare informatics company that provides web-based clinical decision support system  The authors would like to thank visualDx for providing them the opportunity to work on such an exciting project  In particular, the authors would like to thank Roy Robinson, the Vice President of Technology and Medical Informatics at visualDx, for providing the synthesized user queries, as well as preliminary feedback on the performance of our framework 

ground_truth: clinical notes from the CE task in 2010 i2b2/VA

clinical notes from the CE task in 2010 i2b2/VA