prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. where did they obtain the annotated clinical notes from?
Context 1: Introduction
Medical search engines are an essential component for many online medical applications, such as online diagnosis systems and medical document databases  A typical online diagnosis system, for instance, relies on a medical search engine  The search engine takes as input a user query that describes some symptoms and then outputs clinical concept entries that provide relevant information to assist in diagnosing the problem  One challenge medical search engines face is the segmentation of individual clinical entities  When a user query consists of multiple clinical entities, a search engine would often fail to recognize them as separate entities  For example, the user query “fever joint pain weight loss headache” contains four separate clinical entities: “fever”, “joint pain”, “weight loss”, and “headache”  But when the search engine does not recognize them as separate entities and proceeds to retrieve results for each word in the query, it may find "pain" in body locations other than "joint pain", or it may miss "headache" altogether, for example  Some search engines allow the users to enter a single clinical concept by selecting from an auto-completion pick list  But this could also result in retrieving inaccurate or partial results and lead to poor user experience 
We want to improve the medical search engine so that it can accurately retrieve all the relevant clinical concepts mentioned in a user query, where relevant clinical concepts are defined with respect to the terminologies the search engine uses  The problem of extracting clinical concept mentions from a user query can be seen as a variant of the Concept Extraction (CE) task in the frequently-cited NLP challenges in healthcare, such as 2010 i2b2/VA BIBREF0 and 2013 ShARe/CLEF Task 1 BIBREF1  Both CE tasks in 2010 i2b2/VA and 2013 ShARe/CLEF Task 1 ask the participants to design an algorithm to tag a set of predefined entities of interest in clinical notes  These entity tagging tasks are also known as clinical Named Entity Recognition (NER)  For example, the CE task in 2010 i2b2/VA defines three types of entities: “problem”, “treatment”, and “test”  The CE task in 2013 ShARe/CLEF defines various types of disorder such as “injury or poisoning”, "disease or syndrome”, etc  In addition to tagging, the CE task in 2013 ShARe/CLEF has an encoding component which requires selecting one and only one Concept Unique Identifier (CUI) from Systematized Nomenclature Of Medicine Clinical Terms (SNOMED-CT) for each disorder entity tagged  Our problem, similar to the CE task in 2013 ShARe/CLEF, also contains two sub-problems: tagging mentions of entities of interest (entity tagging), and selecting appropriate terms from a glossary to match the mentions (term matching)  However, several major differences exist  First, compared to clinical notes, the user queries are much shorter, less technical, and often less coherent  Second, instead of encoding, we are dealing with term matching where we rank a few best terms that match an entity, instead of selecting only one  This is because the users who type the queries may not have a clear idea about what they are looking for, or could be laymen who know little terminology, it may be more helpful to provide a set of likely results and let the users choose  Third, the types of entities are different  Each medical search engine may have its own types of entities to tag  There is also one minor difference in the tagging scheme between our problem and the CE task in 2013 ShARe/CLEF - We limit our scope to dealing with entities of consecutive words and not disjoint entities   We use only Beginning, Inside, Outside (BIO) tags  Given the differences listed above, we need to customize a framework consisting of an entity tagging and term matching component for our CE problem 
Related Work
An effective model that has been commonly used for NER problem is a Bi-directional LSTM with a Conditional Random Field (CRF) on the top layer (BiLSTM-CRF), which is described in the next section  Combining LSTM’s power of representing relations between words and CRF’s capability of accounting for tag sequence constraints, Huang et al  BIBREF2 proposed the BiLSTM-CRF model and used handcrafted word features as the input to the model  Lample et al  BIBREF3 used a combination of character-level and word-level word embeddings as the input to BiLSTM-CRF  Since then, similar models with variation in types of word embeddings have been used extensively for clinical CE tasks and produced state-of-the-art results BIBREF4, BIBREF5, BIBREF6, BIBREF7  Word embeddings have become the cornerstone of the neural models in NLP since the famous Word2vec BIBREF8 model demonstrated its power in word analogy tasks  One well-known example is that after training Word2vec on a large amount of news data, we can get word relations such as $vector(^{\prime }king^{\prime }) - vector(^{\prime }queen^{\prime }) + vector(^{\prime }woman^{\prime }) \approx vector(^{\prime }man^{\prime })$  More sophisticated word embedding technique emerged since Word2vec  It has been shown empirically that better quality in word embeddings leads to better performance in many downstream NLP including entity tagging BIBREF9, BIBREF10  Recently, contextualized word embeddings generated by deep learning models, such as ELMo BIBREF11, BERT BIBREF12, and Flair BIBREF13, have been shown to be more effective in various NLP tasks  In our project, we make use of a fine-tuned ELMo model and a fine-tuned Flair model in the medical domain  We experiment with the word embeddings from the two fine-tuned models as the input to the BiLSTM-CRF model separately and compare the results 
Tang et al  BIBREF14 provided straightforward algorithm for term matching  The algorithm starts with finding candidate terms that contain ALL the entity words, with term frequency - inverse document frequency (tf-idf) weighting  Then the candidates are ranked based on the pairwise cosine distance between the word embeddings of the candidates and the entity 
Framework
We adopt the tagging - encoding pipeline framework from the CE task in 2013 ShARe/CLEF  We first tag the clinical entities in the user query and then select relevant terms from a glossary in dermatology to match the entities 
Framework ::: Entity Tagging
We use the same BiLSTM-CRF model proposed by Huang et al  BIBREF2  An illustration of the architecture is shown in Figure FIGREF6   Given a sequence (or sentence) of n tokens, $r = (w_1, w_2,   , w_n)$, we use a fine-tuned ELMo model to generate contextual word embeddings for all the tokens in the sentence, where a token refers to a word or punctuation  We denote the ELMo embedding, $x$, for a token $w$ in the sentence $r$ by $x = ELMo(w|r)$  The notation and the procedure described here can be adopted for Flair embeddings or other embeddings  Now, given a sequence of tokens in ELMo embeddings, $X =(x_1, x_2,    , x_n)$, the BiLSTM layer generates a matrix of scores, $P(\theta )$ of size $n \times k$, where $k$ is the number of tag types, and $\theta $ is the parameters of the BiLSTM  To simplify notation, we will omit the $\theta $ and write $P$  Then, $P_{i,j}$ denotes the score of the token, $x_i$, being assigned to the $j$th tag  Since certain constraints may exist in the transition between tags, an "O" tag should not be followed by an "I" tag, for example, a transition matrix, $A$, of dimension $(k+2)\times (k+2)$, is initialized to model the constraints  The learnable parameters, $A_{i,j}$, represent the probability of the $j$th tag follows the $i$th tag in a sequence  For example, if we index the tags by: 1:“B”, 2:“I”, and 3:“O”, then $A_{1,3}$ would be the probability that an “O” tag follows a “B” tag  A beginning transition and an end transition are inserted in $A$ and hence $A$ is of dimension $(k+2)\times (k+2)$ 
Given a sequence of tags, $Y=(y_1,y_2,   ,y_n)$, where each $y_i$, $1\le i \le n$, corresponds to an index of the tags, the score of the sequence is then given by
The probability of the sequence of tags is then calculated by a softmax,
where $\lbrace Y_x\rbrace $ denotes the set of all possible tag sequences  During training, the objective function is to maximize $\log (P(Y|X))$ by adjusting $A$ and $P$ 
Framework ::: Term Matching
The term matching algorithm of Tang et al  BIBREF14 is adopted with some major modifications  First, to identify candidate terms, we use a much looser string search algorithm where we stem the entity words with snowball stemmer and then find candidate terms which contain ANY non-stopword words in the entity  The stemming is mainly used to represent a word in its stem  For example, “legs” becomes “leg”, and “jammed” becomes “jam”  Thus, stemming can provide more tolerance when finding candidates  Similarly, finding candidates using the condition ANY (instead of ALL) also increases the tolerance  However, if the tagged entity contains stopwords such as “in”, “on”, etc , the pool of candidates will naturally grow very large and increase the computation cost for later part, because even completely irrelevant terms may contain stopwords  Therefore, we match based on non-stopwords only  To illustrate the points above, suppose a query is tagged with the entity
Ex  3 1 “severe burns on legs”,
and one relevant term is “leg burn”  After stemming, “burns” and “legs” in Ex UNKREF12 become “burn” and “leg”, respectively, allowing "leg burn" to be considered as a candidate  Although the word “severe” is not in the term “leg burn”, the term is still considered a candidate because we selected using ANY  The stopword “on” is ignored when finding candidate terms so that not every term that contains the word “on” is added to the candidate pool  When a candidate term, $C$, is found in this manner for the tagged entity, $E$, we calculate the semantic similarity score, $s$, between $C$ and $E$ in two steps  In the first step, calculate the maximum similarity score for each word in $C$ as shown in Figure FIGREF10  Given a word in the candidate term, $C_i$ ($1 \le i \le m$, $m$ is the number of words in the candidate term) and a word in the tagged entity, $E_j$ Their similarity score, $s_{ij}$ (shown as the element in the boxed matrix in Figure FIGREF10), is given by
where $ELMo(C_i|C)$ and $ELMo(E_j|E)$ are the ELMo embeddings for the word $C_i$ and $E_j$, respectively  The ELMo embeddings have the same dimension for all words when using the same fine-tuned ELMo model  Thus, we can use a distance function (e g , the cosine distance), denoted $d(\cdot )$ in equation DISPLAY_FORM13, to compute the semantic similarity between words  In step 2, we calculate the candidate-entity relevance score (similarity) using the formula
where $s_c$ is a score threshold, and $\mathbb {I} \lbrace max(\vec{S_i}) > s_c\rbrace $ is an indicator function that equals 1 if $max(\vec{S_i}) > s_c$ or equals 0 if not  In equation DISPLAY_FORM14 we define a metric that measures “information coverage” of the candidate terms with respect to a tagged entity  If the constituent words of a candidate term are relevant to the constituent words in the tagged entity, then the candidate term offers more information coverage  Intuitively, the more relevant words present in the candidate term, the more relevant the candidate is to the tagged entity  The purpose of the cutoff, $s_c$, is to screen the $(C_i,E_j)$ word pairs that are dissimilar, so that they do not contribute to information coverage  One can adjust the strictness of the entity - terminology matching by adjusting $s_c$  The higher we set $s_c$, the fewer candidate terms will be selected for a tagged entity  A normalization factor, $\frac{1}{m}$, is added to give preference to more concise candidate terms given the same amount of information coverage 
We need to create an extra stopword list to include words such as “configuration” and “color”, and exclude these words from the word count for a candidate term  This is because the terms associated with the description of color or configuration usually have the word “color” or “configuration” in them  On the other hand, a user query normally does not contain such words  For example, a tagged entity in a user query could be “round yellow patches”, for which the relevant terminologies include “round configuration” and “yellow color”  Since we applied a normalization factor, $\frac{1}{m}$, to the relevance score, the word “color” and “configuration” would lower the relevance score because they do not have a counterpart in the tagged entity  Therefore, we need to exclude them from word count  Once the process is complete, calculate $s(C,E)$ for all candidate terms and then we can apply a threshold on all $s(C,E)$ to ignore candidate terms with low information coverage  Finally, rank the terms by their $s(C,E)$ and return the ranked list as the results 
Experiments ::: Data
Despite the greater similarity between our task and the 2013 ShARe/CLEF Task 1, we use the clinical notes from the CE task in 2010 i2b2/VA on account of 1) the data from 2010 i2b2/VA being easier to access and parse, 2) 2013 ShARe/CLEF containing disjoint entities and hence requiring more complicated tagging schemes  The synthesized user queries are generated using the aforementioned dermatology glossary  Tagged sentences are extracted from the clinical notes  Sentences with no clinical entity present are ignored  22,489 tagged sentences are extracted from the clinical notes  We will refer to these tagged sentences interchangeably as the i2b2 data  The sentences are shuffled and split into train/dev/test set with a ratio of 7:2:1  The synthesized user queries are composed by randomly selecting several clinical terms from the dermatology glossary and then combining them in no particular order  When combining the clinical terms, we attach the BIO tags to their constituent words  The synthesized user queries (13,697 in total) are then split into train/dev/test set with the same ratio  Next, each set in the i2b2 data and the corresponding set in the synthesized query data are combined to form a hybrid train/dev/test set, respectively  This way we ensure that in each hybrid train/dev/test set, the ratio between the i2b2 data and the synthesized query data is the same 
The reason for combining the two data is their drastic structural difference (See figure FIGREF16 for an example)  Previously, when trained on the i2b2 data only, the BiLSTM-CRF model was not able to segment clinical entities at the correct boundary  It would fail to recognize the user query in Figure FIGREF16(a) as four separate entities  On the other hand, if the model was trained solely on the synthesized user queries, we could imagine that it would fail miserably on any queries that resemble the sentence in Figure FIGREF16(b) because the model would have never seen an “O” tag in the training data  Therefore, it is necessary to use the hybrid training data containing both the i2b2 data and the synthesized user queries 
To make the hybrid training data, we need to unify the tags  Recall that in Section SECREF1 we point out that the tags are different for the different tasks and datasets  Since we use custom tags for dermatology glossary in our problem, we would need to convert the tags used in 2010 i2b2/VA  But this would be an infeasible job as we need experts to manually do that  An alternative is to avoid distinguishing the tag types and label all tags under the generic BIO tags 
Experiments ::: Setup
To show the effects of using the hybrid training data, we trained two models of the same architecture and hyperparameters  One model was trained on the hybrid data and will be referred to as hybrid NER model  The other model was trained on clinical notes only and will be referred to as i2b2 NER model  We evaluated the performance of the NER models by micro-F1 score on the test set of both the synthesized queries and the i2b2 data 
We used the BiLSTM-CRF implementation provided by the flair package BIBREF16  We set the hidden size value to be 256 in the LSTM structure and left everything else at default values for the SequenceTagger model on flair  For word embeddings, we used the ELMo embeddings fine-tuned on PubMed articles and flair embeddings BIBREF13 trained on $5\%$ of PubMed abstracts , respectively  We trained models for 10 epochs and experimented with different learning rate, mini batch size, and dropouts  We ran hyperparameter optimization tests to find the best combination  $S_c$ is set to be 0 6 in our experiment 
Experiments ::: Hyperparameter Tuning
We defined the following hyperparameter search space:
embeddings: [“ELMo on pubmed”, “stacked flair on pubmed”],
hidden_size: [128, 256],
learning_rate: [0 05, 0 1],
mini_batch_size: [32, 64, 128] 
The hyperparameter optimization was performed using Hyperopt   Three evaluations were run for each combination of hyperparameters  Each ran for 10 epochs  Then the results were averaged to give the performance for that particular combination of hyperparameters 
Experiments ::: Results
From the hyperparameter tuning we found that the best combination was
embeddings: “ELMo on pubmed”,
hidden_size: 256,
learning_rate: 0 05,
mini_batch_size: 32 
With the above hyperparameter setting, the hybrid NER model achieved a F1 score of $0 995$ on synthesized queries and $0 948$ on clinical notes while the i2b2 NER model achieved a F1 score of $0 441$ on synthesized queries and $0 927$ on clinical notes (See Table TABREF23) 
Since there was no ground truth available for the retrieved terms, we randomly picked a few samples to assess its performance  Some example outputs of our complete framework on real user queries are shown in Figure FIGREF24  For example, from the figure we see that the query "child fever double vision dizzy" was correctly tagged with four entities: "child", "fever", "double vision", and "dizzy"  A list of terms from our glossary was matched to each entity  In real world application, the lists of terms will be presented to the user as the retrieval results to their queries 
Discussion
In most real user queries we sampled, the entities were tagged at the correct boundary and the tagging was complete (such as the ones shown in Figure FIGREF24)  Only on a few user queries the tagging was controversial  For example, the query “Erythematous blanching round, oval patches on torso, extremities” was tagged as “Erythematous blanching” and “oval patches on torso”  The entity “extremities” was missing  The segmentation was also not correct  A more appropriate tagging would be “Erythematous blanching round, oval patches”, “torso”, and “extremities”  The tagging could be further improved by synthesizing more realistic user queries  Recall that the synthesized user queries were created by randomly combining terminologies from the dermatology glossary, which, while providing data that helped the model learn entity segmentation, did not reflect the co-occurrence information in real user queries  For example, there could be two clinical entities that often co-occur or never co-occur in a user query  But since the synthesized user queries we used combined terms randomly, the co-occurrence information was thus missing 
The final retrieval results of our framework were not evaluated quantitatively in terms of recall and precision, due the the lack of ground truth  When ground truth becomes available, we will be able to evaluate our framework more thoroughly  Recently, a fine-tuned BERT model in the medical domain called BioBERT BIBREF17 has attracted some attention in the medical NLP domain  We could experiment with BioBERT embeddings in the future  We could also include query expansion technique for term matching  When finding candidate terms for an entity, our first step was still based on string matching  Given that there might be multiple entities that could be matched to the same term, it could be hard to include all these entities in the glossary and hard to match terms to these entities 
Conclusion
In this project, we tackle the problem of extracting clinical concepts from user queries on medical search engines  By training a BiLSTM-CRF model on a hybrid data consisting of synthesized user queries and sentences from clinical note, we adopt a CE framework for clinical user queries with minimal effort spent on annotating user queries  We find that the hybrid data enables the NER model perform better on both tagging the user queries and the clinical note sentences  Furthermore, our framework is built on an easy-to-use deep learning NLP Python library, which lends it more prospective value to various online medical applications that employ medical search engines 
Acknowledgment
This paper results from a technical report of a project the authors have worked on with visualDx, a healthcare informatics company that provides web-based clinical decision support system  The authors would like to thank visualDx for providing them the opportunity to work on such an exciting project  In particular, the authors would like to thank Roy Robinson, the Vice President of Technology and Medical Informatics at visualDx, for providing the synthesized user queries, as well as preliminary feedback on the performance of our framework 
Context 2: Introduction
A hashtag is a keyphrase represented as a sequence of alphanumeric characters plus underscore, preceded by the # symbol  Hashtags play a central role in online communication by providing a tool to categorize the millions of posts generated daily on Twitter, Instagram, etc  They are useful in search, tracking content about a certain topic BIBREF0 , BIBREF1 , or discovering emerging trends BIBREF2  
Hashtags often carry very important information, such as emotion BIBREF3 , sentiment BIBREF4 , sarcasm BIBREF5 , and named entities BIBREF6 , BIBREF7   However, inferring the semantics of hashtags is non-trivial since many hashtags contain multiple tokens joined together, which frequently leads to multiple potential interpretations (e g , lion head vs  lionhead)  Table TABREF3 shows several examples of single- and multi-token hashtags  While most hashtags represent a mix of standard tokens, named entities and event names are prevalent and pose challenges to both human and automatic comprehension, as these are more likely to be rare tokens  Hashtags also tend to be shorter to allow fast typing, to attract attention or to satisfy length limitations imposed by some social media platforms  Thus, they tend to contain a large number of abbreviations or non-standard spelling variations (e g , #iloveu4eva) BIBREF8 , BIBREF9 , which hinders their understanding 
The goal of our study is to build efficient methods for automatically splitting a hashtag into a meaningful word sequence  Our contributions are:
Our new dataset includes segmentation for 12,594 unique hashtags and their associated tweets annotated in a multi-step process for higher quality than the previous dataset of 1,108 hashtags BIBREF10   We frame the segmentation task as a pairwise ranking problem, given a set of candidate segmentations  We build several neural architectures using this problem formulation which use corpus-based, linguistic and thesaurus based features  We further propose a multi-task learning approach which jointly learns segment ranking and single- vs  multi-token hashtag classification  The latter leads to an error reduction of 24 6% over the current state-of-the-art  Finally, we demonstrate the utility of our method by using hashtag segmentation in the downstream task of sentiment analysis  Feeding the automatically segmented hashtags to a state-of-the-art sentiment analysis method on the SemEval 2017 benchmark dataset results in a 2 6% increase in the official metric for the task 
Background and Preliminaries
Current approaches for hashtag segmentation can be broadly divided into three categories: (a) gazeteer and rule based BIBREF11 , BIBREF12 , BIBREF13 , (b) word boundary detection BIBREF14 , BIBREF15 , and (c) ranking with language model and other features BIBREF16 , BIBREF10 , BIBREF0 , BIBREF17 , BIBREF18   Hashtag segmentation approaches draw upon work on compound splitting for languages such as German or Finnish BIBREF19 and word segmentation BIBREF20 for languages with no spaces between words such as Chinese BIBREF21 , BIBREF22   Similar to our work, BIBREF10 BansalBV15 extract an initial set of candidate segmentations using a sliding window, then rerank them using a linear regression model trained on lexical, bigram and other corpus-based features  The current state-of-the-art approach BIBREF14 , BIBREF15 uses maximum entropy and CRF models with a combination of language model and hand-crafted features to predict if each character in the hashtag is the beginning of a new word 
Generating Candidate Segmentations  Microsoft Word Breaker BIBREF16 is, among the existing methods, a strong baseline for hashtag segmentation, as reported in BIBREF14 and BIBREF10   It employs a beam search algorithm to extract INLINEFORM0 best segmentations as ranked by the n-gram language model probability: INLINEFORM1
where INLINEFORM0 is the word sequence of segmentation INLINEFORM1 and INLINEFORM2 is the window size  More sophisticated ranking strategies, such as Binomial and word length distribution based ranking, did not lead to a further improvement in performance BIBREF16   The original Word Breaker was designed for segmenting URLs using language models trained on web data  In this paper, we reimplemented and tailored this approach to segmenting hashtags by using a language model specifically trained on Twitter data (implementation details in § SECREF26 )  The performance of this method itself is competitive with state-of-the-art methods (evaluation results in § SECREF46 )  Our proposed pairwise ranking method will effectively take the top INLINEFORM3 segmentations generated by this baseline as candidates for reranking 
However, in prior work, the ranking scores of each segmentation were calculated independently, ignoring the relative order among the top INLINEFORM0 candidate segmentations  To address this limitation, we utilize a pairwise ranking strategy for the first time for this task and propose neural architectures to model this 
Multi-task Pairwise Neural Ranking
We propose a multi-task pairwise neural ranking approach to better incorporate and distinguish the relative order between the candidate segmentations of a given hashtag  Our model adapts to address single- and multi-token hashtags differently via a multi-task learning strategy without requiring additional annotations  In this section, we describe the task setup and three variants of pairwise neural ranking models (Figure FIGREF11 ) 
Segmentation as Pairwise Ranking
The goal of hashtag segmentation is to divide a given hashtag INLINEFORM0 into a sequence of meaningful words INLINEFORM1   For a hashtag of INLINEFORM2 characters, there are a total of INLINEFORM3 possible segmentations but only one, or occasionally two, of them ( INLINEFORM4 ) are considered correct (Table TABREF9 ) 
We transform this task into a pairwise ranking problem: given INLINEFORM0 candidate segmentations { INLINEFORM1 }, we rank them by comparing each with the rest in a pairwise manner  More specifically, we train a model to predict a real number INLINEFORM2 for any two candidate segmentations INLINEFORM3 and INLINEFORM4 of hashtag INLINEFORM5 , which indicates INLINEFORM6 is a better segmentation than INLINEFORM7 if positive, and vice versa  To quantify the quality of a segmentation in training, we define a gold scoring function INLINEFORM8 based on the similarities with the ground-truth segmentation INLINEFORM9 : INLINEFORM10
We use the Levenshtein distance (minimum number of single-character edits) in this paper, although it is possible to use other similarity measurements as alternatives  We use the top INLINEFORM0 segmentations generated by Microsoft Word Breaker (§ SECREF2 ) as initial candidates 
Pairwise Neural Ranking Model
For an input candidate segmentation pair INLINEFORM0 , we concatenate their feature vectors INLINEFORM1 and INLINEFORM2 , and feed them into a feedforward network which emits a comparison score INLINEFORM3   The feature vector INLINEFORM4 or INLINEFORM5 consists of language model probabilities using Good-Turing BIBREF23 and modified Kneser-Ney smoothing BIBREF24 , BIBREF25 , lexical and linguistic features (more details in § SECREF23 )  For training, we use all the possible pairs INLINEFORM6 of the INLINEFORM7 candidates as the input and their gold scores INLINEFORM8 as the target  The training objective is to minimize the Mean Squared Error (MSE): DISPLAYFORM0
where INLINEFORM0 is the number of training examples 
To aggregate the pairwise comparisons, we follow a greedy algorithm proposed by BIBREF26 cohen1998learning and used for preference ranking BIBREF27   For each segmentation INLINEFORM0 in the candidate set INLINEFORM1 , we calculate a single score INLINEFORM2 , and find the segmentation INLINEFORM3 corresponding to the highest score  We repeat the same procedure after removing INLINEFORM4 from INLINEFORM5 , and continue until INLINEFORM6 reduces to an empty set  Figure FIGREF11 (a) shows the architecture of this model 
Margin Ranking (MR) Loss
As an alternative to the pairwise ranker (§ SECREF15 ), we propose a pairwise model which learns from candidate pairs INLINEFORM0 but ranks each individual candidate directly rather than relatively  We define a new scoring function INLINEFORM1 which assigns a higher score to the better candidate, i e , INLINEFORM2 , if INLINEFORM3 is a better candidate than INLINEFORM4 and vice-versa  Instead of concatenating the features vectors INLINEFORM5 and INLINEFORM6 , we feed them separately into two identical feedforward networks with shared parameters  During testing, we use only one of the networks to rank the candidates based on the INLINEFORM7 scores  For training, we add a ranking layer on top of the networks to measure the violations in the ranking order and minimize the Margin Ranking Loss (MR): DISPLAYFORM0
where INLINEFORM0 is the number of training samples  The architecture of this model is presented in Figure FIGREF11 (b) 
Adaptive Multi-task Learning
Both models in § SECREF15 and § SECREF17 treat all the hashtags uniformly  However, different features address different types of hashtags  By design, the linguistic features capture named entities and multi-word hashtags that exhibit word shape patterns, such as camel case  The ngram probabilities with Good-Turing smoothing gravitate towards multi-word segmentations with known words, as its estimate for unseen ngrams depends on the fraction of ngrams seen once which can be very low BIBREF28   The modified Kneser-Ney smoothing is more likely to favor segmentations that contain rare words, and single-word segmentations in particular  Please refer to § SECREF46 for a more detailed quantitative and qualitative analysis 
To leverage this intuition, we introduce a binary classification task to help the model differentiate single-word from multi-word hashtags  The binary classifier takes hashtag features INLINEFORM0 as the input and outputs INLINEFORM1 , which represents the probability of INLINEFORM2 being a multi-word hashtag  INLINEFORM3 is used as an adaptive gating value in our multi-task learning setup  The gold labels for this task are obtained at no extra cost by simply verifying whether the ground-truth segmentation has multiple words  We train the pairwise segmentation ranker and the binary single- vs  multi-token hashtag classifier jointly, by minimizing INLINEFORM4 for the pairwise ranker and the Binary Cross Entropy Error ( INLINEFORM5 ) for the classifier: DISPLAYFORM0
where INLINEFORM0 is the adaptive gating value, INLINEFORM1 indicates if INLINEFORM2 is actually a multi-word hashtag and INLINEFORM3 is the number of training examples  INLINEFORM4 and INLINEFORM5 are the weights for each loss  For our experiments, we apply equal weights 
More specifically, we divide the segmentation feature vector INLINEFORM0 into two subsets: (a) INLINEFORM1 with modified Kneser-Ney smoothing features, and (b) INLINEFORM2 with Good-Turing smoothing and linguistic features  For an input candidate segmentation pair INLINEFORM3 , we construct two pairwise vectors INLINEFORM4 and INLINEFORM5 by concatenation, then combine them based on the adaptive gating value INLINEFORM6 before feeding them into the feedforward network INLINEFORM7 for pairwise ranking: DISPLAYFORM0
We use summation with padding, as we find this simple ensemble method achieves similar performance in our experiments as the more complex multi-column networks BIBREF29   Figure FIGREF11 (c) shows the architecture of this model  An analogue multi-task formulation can also be used for the Margin Ranking loss as: DISPLAYFORM0
Features
We use a combination of corpus-based and linguistic features to rank the segmentations  For a candidate segmentation INLINEFORM0 , its feature vector INLINEFORM1 includes the number of words in the candidate, the length of each word, the proportion of words in an English dictionary or Urban Dictionary BIBREF30 , ngram counts from Google Web 1TB corpus BIBREF31 , and ngram probabilities from trigram language models trained on the Gigaword corpus BIBREF32 and 1 1 billion English tweets from 2010, respectively  We train two language models on each corpus: one with Good-Turing smoothing using SRILM BIBREF33 and the other with modified Kneser-Ney smoothing using KenLM BIBREF34   We also add boolean features, such as if the candidate is a named-entity present in the list of Wikipedia titles, and if the candidate segmentation INLINEFORM2 and its corresponding hashtag INLINEFORM3 satisfy certain word-shapes (more details in appendix SECREF61 ) 
Similarly, for hashtag INLINEFORM0 , we extract the feature vector INLINEFORM1 consisting of hashtag length, ngram count of the hashtag in Google 1TB corpus BIBREF31 , and boolean features indicating if the hashtag is in an English dictionary or Urban Dictionary, is a named-entity, is in camel case, ends with a number, and has all the letters as consonants  We also include features of the best-ranked candidate by the Word Breaker model 
Implementation Details
We use the PyTorch framework to implement our multi-task pairwise ranking model  The pairwise ranker consists of an input layer, three hidden layers with eight nodes in each layer and hyperbolic tangent ( INLINEFORM0 ) activation, and a single linear output node  The auxiliary classifier consists of an input layer, one hidden layer with eight nodes and one output node with sigmoid activation  We use the Adam algorithm BIBREF35 for optimization and apply a dropout of 0 5 to prevent overfitting  We set the learning rate to 0 01 and 0 05 for the pairwise ranker and auxiliary classifier respectively  For each experiment, we report results obtained after 100 epochs 
For the baseline model used to extract the INLINEFORM0 initial candidates, we reimplementated the Word Breaker BIBREF16 as described in § SECREF2 and adapted it to use a language model trained on 1 1 billion tweets with Good-Turing smoothing using SRILM BIBREF33 to give a better performance in segmenting hashtags (§ SECREF46 )  For all our experiments, we set INLINEFORM1  
Hashtag Segmentation Data
We use two datasets for experiments (Table TABREF29 ): (a) STAN INLINEFORM0 , created by BIBREF10 BansalBV15, which consists of 1,108 unique English hashtags from 1,268 randomly selected tweets in the Stanford Sentiment Analysis Dataset BIBREF36 along with their crowdsourced segmentations and our additional corrections; and (b) STAN INLINEFORM1 , our new expert curated dataset, which includes all 12,594 unique English hashtags and their associated tweets from the same Stanford dataset 
Experiments
In this section, we present experimental results that compare our proposed method with the other state-of-the-art approaches on hashtag segmentation datasets  The next section will show experiments of applying hashtag segmentation to the popular task of sentiment analysis 
Existing Methods
We compare our pairwise neural ranker with the following baseline and state-of-the-art approaches:
The original hashtag as a single token;
A rule-based segmenter, which employs a set of word-shape rules with an English dictionary BIBREF13 ;
A Viterbi model which uses word frequencies from a book corpus BIBREF0 ;
The specially developed GATE Hashtag Tokenizer from the open source toolkit, which combines dictionaries and gazetteers in a Viterbi-like algorithm BIBREF11 ;
A maximum entropy classifier (MaxEnt) trained on the STAN INLINEFORM0 training dataset  It predicts whether a space should be inserted at each position in the hashtag and is the current state-of-the-art BIBREF14 ;
Our reimplementation of the Word Breaker algorithm which uses beam search and a Twitter ngram language model BIBREF16 ;
A pairwise linear ranker which we implemented for comparison purposes with the same features as our neural model, but using perceptron as the underlying classifier BIBREF38 and minimizing the hinge loss between INLINEFORM0 and a scoring function similar to INLINEFORM1   It is trained on the STAN INLINEFORM2 dataset 
Evaluation Metrics
We evaluate the performance by the top INLINEFORM0 ( INLINEFORM1 ) accuracy (A@1, A@2), average token-level F INLINEFORM2 score (F INLINEFORM3 @1), and mean reciprocal rank (MRR)  In particular, the accuracy and MRR are calculated at the segmentation-level, which means that an output segmentation is considered correct if and only if it fully matches the human segmentation  Average token-level F INLINEFORM4 score accounts for partially correct segmentation in the multi-token hashtag cases 
Results
Tables TABREF32 and TABREF33 show the results on the STAN INLINEFORM0 and STAN INLINEFORM1 datasets, respectively  All of our pairwise neural rankers are trained on the 2,518 manually segmented hashtags in the training set of STAN INLINEFORM2 and perform favorably against other state-of-the-art approaches  Our best model (MSE+multitask) that utilizes different features adaptively via a multi-task learning procedure is shown to perform better than simply combining all the features together (MR and MSE)  We highlight the 24 6% error reduction on STAN INLINEFORM3 and 16 5% on STAN INLINEFORM4 of our approach over the previous SOTA BIBREF14 on the Multi-token hashtags, and the importance of having a separate evaluation of multi-word cases as it is trivial to obtain 100% accuracy for Single-token hashtags  While our hashtag segmentation model is achieving a very high accuracy@2, to be practically useful, it remains a challenge to get the top one predication exactly correct  Some hashtags are very difficult to interpret, e g , #BTVbrownSMB refers to the Social Media Breakfast (SMB) in Burlington, Vermont (BTV) 
The improved Word Breaker with our addition of a Twitter-specific language model is a very strong baseline, which echos the findings of the original Word Breaker paper BIBREF16 that having a large in-domain language model is extremely helpful for word segmentation tasks  It is worth noting that the other state-of-the-art system BIBREF14 also utilized a 4-gram language model trained on 476 million tweets from 2009 
Analysis and Discussion
To empirically illustrate the effectiveness of different features on different types of hashtags, we show the results for models using individual feature sets in pairwise ranking models (MSE) in Table TABREF45   Language models with modified Kneser-Ney smoothing perform best on single-token hashtags, while Good-Turing and Linguistic features work best on multi-token hashtags, confirming our intuition about their usefulness in a multi-task learning approach  Table TABREF47 shows a qualitative analysis with the first column ( INLINEFORM0 INLINEFORM1 INLINEFORM2 ) indicating which features lead to correct or wrong segmentations, their count in our data and illustrative examples with human segmentation 
As expected, longer hashtags with more than three tokens pose greater challenges and the segmentation-level accuracy of our best model (MSE+multitask) drops to 82 1%  For many error cases, our model predicts a close-to-correct segmentation, e g , #youbrownknowyoubrownupttoobrownearly, #iseebrownlondoniseebrownfrance, which is also reflected by the higher token-level F INLINEFORM0 scores across hashtags with different lengths (Figure FIGREF51 ) 
Since our approach heavily relies on building a Twitter language model, we experimented with its sizes and show the results in Figure FIGREF52   Our approach can perform well even with access to a smaller amount of tweets  The drop in F INLINEFORM0 score for our pairwise neural ranker is only 1 4% and 3 9% when using the language models trained on 10% and 1% of the total 1 1 billion tweets, respectively 
Language use in Twitter changes with time BIBREF9   Our pairwise ranker uses language models trained on the tweets from the year 2010  We tested our approach on a set of 500 random English hashtags posted in tweets from the year 2019 and show the results in Table TABREF55   With a segmentation-level accuracy of 94 6% and average token-level F INLINEFORM0 score of 95 6%, our approach performs favorably on 2019 hashtags 
Extrinsic Evaluation: Twitter Sentiment Analysis
We attempt to demonstrate the effectiveness of our hashtag segmentation system by studying its impact on the task of sentiment analysis in Twitter BIBREF39 , BIBREF40 , BIBREF41   We use our best model (MSE+multitask), under the name HashtagMaster, in the following experiments 
Experimental Setup
We compare the performance of the BiLSTM+Lex BIBREF42 sentiment analysis model under three configurations: (a) tweets with hashtags removed, (b) tweets with hashtags as single tokens excluding the # symbol, and (c) tweets with hashtags as segmented by our system, HashtagMaster  BiLSTM+Lex is a state-of-the-art open source system for predicting tweet-level sentiment BIBREF43   It learns a context-sensitive sentiment intensity score by leveraging a Twitter-based sentiment lexicon BIBREF44   We use the same settings as described by BIBREF42 teng-vo-zhang:2016:EMNLP2016 to train the model 
We use the dataset from the Sentiment Analysis in Twitter shared task (subtask A) at SemEval 2017 BIBREF41   Given a tweet, the goal is to predict whether it expresses POSITIVE, NEGATIVE or NEUTRAL sentiment  The training and development sets consist of 49,669 tweets and we use 40,000 for training and the rest for development  There are a total of 12,284 tweets containing 12,128 hashtags in the SemEval 2017 test set, and our hashtag segmenter ended up splitting 6,975 of those hashtags present in 3,384 tweets 
Results and Analysis
In Table TABREF59 , we report the results based on the 3,384 tweets where HashtagMaster predicted a split, as for the rest of tweets in the test set, the hashtag segmenter would neither improve nor worsen the sentiment prediction  Our hashtag segmenter successfully improved the sentiment analysis performance by 2% on average recall and F INLINEFORM0 comparing to having hashtags unsegmented  This improvement is seemingly small but decidedly important for tweets where sentiment-related information is embedded in multi-word hashtags and sentiment prediction would be incorrect based only on the text (see Table TABREF60 for examples)  In fact, 2,605 out of the 3,384 tweets have multi-word hashtags that contain words in the Twitter-based sentiment lexicon BIBREF44 and 125 tweets contain sentiment words only in the hashtags but not in the rest of the tweet  On the entire test set of 12,284 tweets, the increase in the average recall is 0 5% 
Other Related Work
Automatic hashtag segmentation can improve the performance of many applications besides sentiment analysis, such as text classification BIBREF13 , named entity linking BIBREF10 and modeling user interests for recommendations BIBREF45   It can also help in collecting data of higher volume and quality by providing a more nuanced interpretation of its content, as shown for emotion analysis BIBREF46 , sarcasm and irony detection BIBREF11 , BIBREF47   Better semantic analysis of hashtags can also potentially be applied to hashtag annotation BIBREF48 , to improve distant supervision labels in training classifiers for tasks such as sarcasm BIBREF5 , sentiment BIBREF4 , emotions BIBREF3 ; and, more generally, as labels for pre-training representations of words BIBREF49 , sentences BIBREF50 , and images BIBREF51  
Conclusion
We proposed a new pairwise neural ranking model for hashtag segmention and showed significant performance improvements over the state-of-the-art  We also constructed a larger and more curated dataset for analyzing and benchmarking hashtag segmentation methods  We demonstrated that hashtag segmentation helps with downstream tasks such as sentiment analysis  Although we focused on English hashtags, our pairwise ranking approach is language-independent and we intend to extend our toolkit to languages other than English as future work 
Acknowledgments
We thank Ohio Supercomputer Center BIBREF52 for computing resources and the NVIDIA for providing GPU hardware  We thank Alan Ritter, Quanze Chen, Wang Ling, Pravar Mahajan, and Dushyanta Dhyani for valuable discussions  We also thank the annotators: Sarah Flanagan, Kaushik Mani, and Aswathnarayan Radhakrishnan  This material is based in part on research sponsored by the NSF under grants IIS-1822754 and IIS-1755898, DARPA through the ARO under agreement number W911NF-17-C-0095, through a Figure-Eight (CrowdFlower) AI for Everyone Award and a Criteo Faculty Research Award to Wei Xu  The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of the U S  Government 
Word-shape rules
Our model uses the following word shape rules as boolean features  If the candidate segmentation INLINEFORM0 and its corresponding hashtag INLINEFORM1 satisfies a word shape rule, then the boolean feature is set to True 

ground_truth: clinical notes from the CE task in 2010 i2b2/VA

clinical notes from the CE task in 2010 i2b2/VA 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do the hashtag and SemEval datasets contain only English data?
Context 3: Introduction
Deep Learning approaches have achieved impressive results on various NLP tasks BIBREF0 , BIBREF1 , BIBREF2 and have become the de facto approach for any NLP task  However, these deep learning techniques have found to be less effective for low-resource languages when the available training data is very less BIBREF3   Recently, several approaches like Multi-task learning BIBREF4 , multilingual learning BIBREF5 , semi-supervised learning BIBREF2 , BIBREF6 and transfer learning BIBREF7 , BIBREF3 have been explored by the deep learning community to overcome data sparsity in low-resource languages  Transfer learning trains a model for a parent task and fine-tunes the learned parent model weights (features) for a related child task BIBREF7 , BIBREF8   This effectively reduces the requirement on training data for the child task as the model would have learned relevant features from the parent task data thereby, improving the performance on the child task 
Transfer learning has also been explored in the multilingual Neural Machine Translation BIBREF3 , BIBREF9 , BIBREF10   The goal is to improve the NMT performance on the source to target language pair (child task) using an assisting source language (assisting to target translation is the parent task)  Here, the parent model is trained on the assisting and target language parallel corpus and the trained weights are used to initialize the child model  The child model can now be fine-tuned on the source-target language pairs, if parallel corpus is available  The divergence between the source and the assisting language can adversely impact the benefits obtained from transfer learning  Multiple studies have shown that transfer learning works best when the languages are related BIBREF3 , BIBREF10 , BIBREF9   Several studies have tried to address lexical divergence between the source and the target languages BIBREF10 , BIBREF11 , BIBREF12   However, the effect of word order divergence and its mitigation has not been explored  In a practical setting, it is not uncommon to have source and assisting languages with different word order  For instance, it is possible to find parallel corpora between English and some Indian languages, but very little parallel corpora between Indian languages  Hence, it is natural to use English as an assisting language for inter-Indian language translation 
To see how word order divergence can be detrimental, let us consider the case of the standard RNN (Bi-LSTM) encoder-attention-decoder architecture BIBREF13   The encoder generates contextual representations (annotation vectors) for each source word, which are used by the attention network to match the source words to the current decoder state  The contextual representation is word-order dependent  Hence, if the assisting and the source languages do not have similar word order the generated contextual representations will not be consistent  The attention network (and hence the decoder) sees different contextual representations for similar words in parallel sentences across different languages  This makes it difficult to transfer knowledge learned from the assisting language to the source language 
We illustrate this by visualizing the contextual representations generated by the encoder of an English to Hindi NMT system for two versions of the English input: (a) original word order (SVO) (b) word order of the source language (SOV, for Bengali)  Figure FIGREF1 shows that the encoder representations obtained are very different  The attention network and the decoder now have to work with very different representations  Note that the plot below does not take into account further lexical and other divergences between source and assisting languages, since we demonstrated word order divergence with the same language on the source side 
To address this word order divergence, we propose to pre-order the assisting language sentences to match the word order of the source language  We consider an extremely resource constrained scenario, where we do not have any parallel corpus for the child task  We are limited to a bilingual dictionary for transfer information from the assisting to the source language  From our experiments, we show that there is a significant increase in the translation accuracy for the unseen source-target language pair 
Addressing Lexical Divergence
BIBREF3 explored transfer learning for NMT on low-resource languages  They studied the influence of language divergence between languages chosen for training the parent and child model, and showed that choosing similar languages for training the parent and child model leads to better improvements from transfer learning  A limitation of BIBREF3 approach is that they ignore the lexical similarity between languages and also the source language embeddings are randomly initialized  BIBREF10 , BIBREF11 , BIBREF12 take advantage of lexical similarity between languages in their work  BIBREF10 proposed to use Byte-Pair Encoding (BPE) to represent the sentences in both the parent and the child language to overcome the above limitation  They show using BPE benefits transfer learning especially when the involved languages are closely-related agglutinative languages  Similarly, BIBREF11 utilize lexical similarity between the source and assisting languages by training a character-level NMT system  BIBREF12 address lexical divergence by using bilingual embeddings and mixture of universal token embeddings  One of the languages' vocabulary, usually English vocabulary is considered as universal tokens and every word in the other languages is represented as a mixture of universal tokens  They show results on extremely low-resource languages 
Addressing Word Order Divergence
To the best of our knowledge, no work has addressed word order divergence in transfer learning for multilingual NMT  However, some work exists for other NLP tasks that could potentially address word order  For Named Entity Recognition (NER), BIBREF14 use a self-attention layer after the Bi-LSTM layer to address word-order divergence for Named Entity Recognition (NER) task  The approach does not show any significant improvements over multiple languages  A possible reason is that the divergence has to be addressed before/during construction of the contextual embeddings in the Bi-LSTM layer, and the subsequent self-attention layer does not address word-order divergence  BIBREF15 use adversarial training for cross-lingual question-question similarity ranking in community question answering  The adversarial training tries to force the encoder representations of similar sentences from different input languages to have similar representations 
Use of Pre-ordering
Pre-ordering the source language sentences to match the target language word order has been useful in addressing word-order divergence for Phrase-Based SMT BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19   Recently, BIBREF20 proposed a way to measure and reduce the divergence between the source and target languages based on morphological and syntactic properties, also termed as anisomorphism  They demonstrated that by reducing the anisomorphism between the source and target languages, consistent improvements in NMT performance were obtained  The NMT system used additional features like word forms, POS tags and dependency relations in addition to parallel corpora  On the other hand, BIBREF21 observed a drop in performance due to pre-ordering for NMT  Unlike BIBREF20 , the NMT system was trained on pre-ordered sentences and no additional features were provided to the system  Note that all these works address source-target divergence, not divergence between source languages in multilingual NMT 
Proposed Solution
Consider the task of translating from an extremely low-resource language (source) to a target language  The parallel corpus between the two languages if available may be too small to train a NMT model  Similar to existing works BIBREF3 , BIBREF10 , BIBREF12 , we use transfer learning to overcome data sparsity and train a NMT model between the source and the target languages  Specifically, the NMT model (parent model) is trained on the assisting language and target language pairs  We choose English as the assisting language in all our experiments  In our resource-scarce scenario, we have no parallel corpus for the child task  Hence, at test time, the source language sentence is translated using the parent model after performing a word-by-word translation into the assisting language 
Since the source language and the assisting language (English) have different word order, we hypothesize that it leads to inconsistencies in the contextual representations generated by the encoder for the two languages  In this paper, we propose to pre-order English sentences (assisting language sentences) to match the word-order of the source language and train the parent model on this pre-ordered corpus  In our experiments, we look at scenarios where the assisting language has SVO word order and the source language has SOV word order 
For instance, consider the English sentence Anurag will meet Thakur  One of the pre-ordering rule swaps the position of the noun phrase followed by a transitive verb with the transitive verb  The original and the resulting re-ordered parse tree will be as shown in the Table TABREF5   Applying this reordering rule to the above sentence Anurag will meet Thakur will yield the reordered sentence Anurag Thakur will meet  Additionally, the Table TABREF5 shows the parse trees for the above sentence with and without pre-ordering 
Pre-ordering should also be beneficial for other word order divergence scenarios (e g , SOV to SVO), but we leave verification of these additional scenarios for future work 
Experimental Setup
In this section, we describe the languages experimented with, datasets used, the network hyper-parameters used in our experiments 
Languages
We experimented with English INLINEFORM0 Hindi translation as the parent task  English is the assisting source language  Bengali, Gujarati, Marathi, Malayalam and Tamil are the primary source languages, and translation from these to Hindi constitute the child tasks  Hindi, Bengali, Gujarati and Marathi are Indo-Aryan languages, while Malayalam and Tamil are Dravidian languages  All these languages have a canonical SOV word order 
Datasets
For training English-Hindi NMT systems, we use the IITB English-Hindi parallel corpus BIBREF22 ( INLINEFORM0 sentences from the training set) and the ILCI English-Hindi parallel corpus ( INLINEFORM1 sentences)  The ILCI (Indian Language Corpora Initiative) multilingual parallel corpus BIBREF23 spans multiple Indian languages from the health and tourism domains  We use the 520-sentence dev-set of the IITB parallel corpus for validation  For each child task, we use INLINEFORM2 sentences from ILCI corpus as the test set 
Network
We use OpenNMT-Torch BIBREF24 to train the NMT system  We use the standard sequence-to-sequence architecture with attention BIBREF13   We use an encoder which contains two layers of bidirectional LSTMs with 500 neurons each  The decoder contains two LSTM layers with 500 neurons each  Input feeding approach BIBREF1 is used where the previous attention hidden state is fed as input to the decoder LSTM  We use a mini-batch of size 50 and use a dropout layer  We begin with an initial learning rate of INLINEFORM0 and decay the learning rate by a factor of INLINEFORM1 when the perplexity on validation set increases  The training is stopped when the learning rate falls below INLINEFORM2 or number of epochs=22  The English input is initialized with pre-trained embeddings trained using fastText BIBREF25  
English vocabulary consists of INLINEFORM0 tokens appearing at least 2 times in the English training corpus  For constructing the Hindi vocabulary we considered only those tokens appearing at least 5 times in the training split resulting in a vocabulary size of INLINEFORM1 tokens  For representing English and other source languages into a common space, we translate each word in the source language into English using a bilingual dictionary (Google Translate word translation in our case)  In an end-to-end solution, it would have been ideal to use bilingual embeddings or obtain word-by-word translations via bilingual embeddings BIBREF14   But, the quality of publicly available bilingual embeddings for English-Indian languages is very low for obtaining good-quality, bilingual representations BIBREF26 , BIBREF27   We also found that these embeddings were not useful for transfer learning 
We use the CFILT-preorder system for reordering English sentences to match the Indian language word order  It contains two re-ordering systems: (1) generic rules that apply to all Indian languages BIBREF17 , and (2) hindi-tuned rules which improve the generic rules by incorporating improvements found through an error analysis of English-Hindi reordering BIBREF28   These Hindi-tuned rules have been found to improve reordering for many English to Indian language pairs BIBREF29  
Results
In this section, we describe the results from our experiments on NMT task  We report the results on X-Hindi pair, where X is one of Bengali, Gujarati, Marathi, Tamil, and Malayalam  The results are presented in the Table TABREF6   We report BLEU scores and LeBLEU scores BIBREF30   We observe that both the pre-ordering configurations significantly improve the BLEU scores over the baseline scores  We observe larger gains when generic pre-ordering rules are used compared to the Hindi-tuned pre-ordering rules 
These results support our hypothesis that word-order divergence can limit the benefits of multilingual translation  Reducing the word order divergence can improve translation in extremely low-resource scenarios 
An analysis of the outputs revealed that pre-ordering significantly reducing the number of UNK tokens (placeholder for unknown words) in the test output (Table TABREF14 )  We hypothesize that due to word order divergence between English and Indian languages, the encoder representation generated is not consistent leading to decoder generating unknown words  However, the pre-ordered models generate better contextual representations leading to less number of unknown tokens and better translation which is also reflected in the BLEU scores 
Conclusion
In this paper, we show that handling word-order divergence between source and assisting languages is crucial for the success of multilingual NMT in an extremely low-resource setting  We show that pre-ordering the assisting language to match the word order of the source language significantly improves translation quality in an extremely low-resource setting  While the current work focused on Indian languages, we would like to validate the hypothesis on a more diverse set of languages 

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they match words before reordering them?
Context 4: 
Experiments ::: Metrics
For the MR extraction task, we measure the ROUGE-1 scores BIBREF14 for both the Dosage and Frequency extraction tasks  It should be noted that since Dosage is a single word token (after processing), both the reference and hypothesis are a single token, making its ROUGE-1 F1, Precision and Recall scores equal and be equal to percentage of times we find the correct dosage for the medications 
In our annotations, Frequency has conflicting tags (e g  {`Once a day', `twice a day'} and `daily'), hence metrics like Exact Match will be erroneous  To address this issue, we use the ROUGE scores to compare different models on the 10% test dataset and we use qualitative evaluation to measure the top-performing models on $\mathcal {D}_{test}$ 
Experiments ::: Model variations
We consider QA PGNet and Multi-decoder QA PGNet with lookup table embedding as baseline models and improve on the baselines with other variations described below 
Apart from learning-based baselines, we also create two naive baselines, one each for the Dosage and Frequency extraction tasks  For Dosage extraction, the baseline we consider is `Nearest Number', where we take the number nearest to the Medication Name as the prediction, and `none' if no number is mentioned or if the Medication Name is not detected in the input  For Frequency extraction, the baseline we consider is `Random Top-3' where we predict a random Frequency tag, from top-3 most frequent ones from our dataset - {`none', `daily', `twice a day'} 
Embedding: We developed different variations of our models with a simple lookup table embeddings learned from scratch and using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13 (trained and provided by the authors)  Refer to Table TABREF5 for the performance comparisons 
We derive embeddings from ELMo by learning a linear combination of its last three layer's hidden states (task-specific fine-tuning BIBREF11)  Similarly, for BERT-based embeddings, we take a linear combination of the hidden states from its last four layers, as this combination performs best without increasing the size of the embeddings BIBREF16  Since BERT and ClinicalBERT use word-piece vocabulary and computes sub-word embeddings, we compute word-level embedding by averaging the corresponding sub-word tokens  ELMo and BERT embeddings both have 1024 dimensions, ClinicalBERT have 768 as it is based on BERT base model, and the lookup table have 128 – higher dimension models leads to overfitting 
Pertaining Encoder: We trained the PGNet as a summarization task using the clinical summaries and used the trained model to initialize the encoders (and the embeddings) of the corresponding QA models  We use a vocab size of 4073 words, derived from the training dataset with a frequency threshold of 30 for the task  We trained the models using Adagrad optimizer with a learning rate of 0 015, normal gradient clipping set at 2 and trained for around 150000 iterations (stopped using validation dataset)  On the summarization task PGNet obtained ROUGE-1 F1 scores of 41 42 with ELMo and 39 15 with BERT embeddings  We compare the effects of pretraining the model in Table: TABREF5, models with `pretrained encoder' had their encoders and embeddings pretrained with the summarization task 
Results and Discussion ::: Difference in networks and approaches
Embeddings: On Dosage extraction, in general, ELMo obtains better performance than BERT, refer to Table TABREF5  This could be because we concatenated the numbers with a hyphen, and as ELMo uses character-level tokens it can learn the tagging better than BERT, a similar observation is also noted in BIBREF17  On the other hand, on Frequency extraction, without pretraining, ELMo's performance is lagging by a big margin of $\sim $8 5 ROUGE-1 F1 compared to BERT-based embeddings 
Although in cases without encoder pretraining, ClinicalBERT performed the best in the Frequency extraction task (by a small margin), in general, it does not perform as well as BERT  This could also be a reflection of the fact that the language and style of writing used in clinical notes is very different from the way doctors converse with patients and the embedding dimension difference  Lookup table embedding performed decently in the frequency extraction task but lags behind in the Dosage extraction task 
From the metrics and qualitative inspection, we find that the Frequency extraction is an easier task than the Dosage extraction  This is because, in the conversations, frequency information usually occurs in isolation and near the medications, but a medication's dosage can occur 1) near other medication's dosages, 2) with previous dosages (when a dosage for a medication is revised), and 3) after a large number of words from the medication 
Other Variations: Considering various models' performance (without pretraining) and the resource constraint, we choose ELMo and BERT embeddings to analyze the effects of pretraining the encoder  When the network's encoder (and embedding) is pretrained with the summarization task, we 1) see a small decrease in the average number of iterations required for training, 2) improvement in individual performances of all models for both the sub-tasks, and 3) get best performance metrics across all variations, refer to Table TABREF5  Both in terms of performance and the training speed, there is no clear winner between shared and multi-decoder approaches  Medication tagging and data augmentation increase the best-performing model's ROUGE-1 F1 score by $\sim $1 5 for the Dosage extraction task 
We also measure the performance of Multitask Question Answering Network (MQAN) BIBREF18 a QA model trained by the authors on the Decathlon multitask challenge  Since MQAN was not trained to produce the output sequence in our MR tags, it would not be fair to compute ROUGE scores  Instead, we randomly sample the MQAN's predictions from the 10% test dataset and qualitatively evaluate it  From the evaluations, we find that MQAN can not distinguish between frequency and dosage, and mixed the answers  MQAN correctly predicted the dosage for 29 73% and frequency for 24 24% percent of the medications compared to 84 12% and 76 34% for the encoder pretrained BERT QA PGNet model trained on our dataset  This could be because of the difference in the training dataset, domain and the tasks in the Decathlon challenge compared to ours 
Almost all our models perform better than the naive baselines and the ones using lookup table embeddings, and our best performing models outperform them significantly  Among all the variations, the best performing models are ELMo with Multi-decoder (Dosage extraction) and BERT with shared-decoder QA PGNet architecture with pretrained encoder (Frequency extraction)  We choose these two models for our subsequent analysis 
Results and Discussion ::: Breakdown of Performance
We categorize the 10% test dataset into different categories based on the complexity and type of the data and analyze the breakdown of the system's performance in Table TABREF11  We breakdown the Frequency extraction into two categories, 1) None: ground truth Frequency tag is `none', and 2) NN (Not None): ground truth Frequency tag is not `none'  Similarly, the Dosage extraction into 5 categories, 1) None: ground truth dosage tag is `none', 2) MM (Multiple Medicine): input segment has more than one Medication mentioned, 3) MN (Multiple Numbers): input segment has more than one number present, and 4) NBM (Number between correct Dosage and Medicine) : between the Medication Name and the correct Dosage in the input segment there are other numbers present  Note that the categories of the Dosage extraction task are not exhaustive, and one tag can belong to multiple categories 
From the performance breakdown of Dosage extraction task, we see that 1) the models predict `none' better than other categories, i e , the models are correctly able to identify when a medication's dosage is absent, 2) there is performance dip in hard cases (MM, MN, and NBM), 3) the models are able to figure out the correct dosage (decently) for a medication even when there are multiple numbers/dosage present, and 4) the model struggles the most in the NBM category  The models' low performance in NBM could be because we have a comparatively lower number of examples to train in this category  The Frequency extraction task performs equally well when the tag is `none` or not  In most categories, we see an increase in performance when using pretrained encoders 
Results and Discussion ::: Training Dataset Size
We vary the number of MR tags used to train the model and analyze the model's performance when training the networks, using publicly available contextual embeddings, compared to using pretrained embeddings and encoder (pretrained on the summarization task)  Out of the 5,016 files in the 80% train dataset only 2,476 have atleast one MR tag  Therefore, out of the 2476 files, we randomly choose 100, 500, and 1000 files and trained the best performing model variations to observe the performance differences, refer to Figure FIGREF12  For all these experiments we used the same vocabulary size (456), the same hyper/training parameters, and the same 10% test split of 627 files 
As expected, we see that the encoder pretrained models have higher performance on all the different training data sizes, i e , they achieve higher performance on a lower number of data points, refer to Figure FIGREF12  The difference, as expected, shrinks as the training data size increases 
Results and Discussion ::: Evaluating on ASR transcripts
To test the performance of our models on real-world conditions, we use commercially available ASR services (Google and IBM) to transcribe the $\mathcal {D}_{test}$ files and measure the performance of our models without assuming any annotations (except when calculating the metrics)  It should be noted that this is not the case in our previous evaluations using `10% test' dataset where we use the segmentation information  For ground truth annotations on ASR transcripts, we aligned the MR tags from human written transcripts to the ASR transcript using their grounded timing information  Additionally, since ASR is prone to errors, during the alignment, if a medication from an MR tag is not recognized correctly in the ASR transcript, we remove the corresponding MR tag 
In our evaluations, we use Google Cloud Speech-to-Text (G-STT) and IBM Watson Speech to Text (IBM-STT) as these were among the top-performing ASR APIs on medical speech BIBREF19 and were readily available to us  We used G-STT, with the `video model' with punctuation settings  Unlike our human written transcripts, the transcript provided by G-STT is not verbatim and does not have disfluencies  IBM-STT, on the other hand, does not give punctuation so we used the speaker changes to add end-of-sentence punctuation 
In our $\mathcal {D}_{test}$ dataset, on initial study we see a Word Error Rate of $\sim $50% for the ASR APIs and this number is not accurate because, 1) of the de-identification, 2) disfluencies (verbatim) difference between the human written and ASR transcript, and 3) minor alignment differences between the audio and the ground truth transcript 
During this evaluation, we followed the same preprocessing methods we used during training  Then, we auto segment the transcript into small contiguous segments similar to the grounded sentences in the annotations for tags extraction  To segment the transcript, we follow a simple procedure  First, we detected all the medications in a transcript using RxNorm BIBREF20 via string matching  For all the detected medications, we selected $2 \le x \le 5$ nearby sentences as the input to our model  We increased $x$ iteratively until we encountered a quantity entity – detected using spaCy's entity recognizer, and we set $x$ as 2 if we did not detect any entities in the range 
We show the model's performance on ASR transcripts and human written transcripts with automatic segmentation, and human written transcripts with human (defined) segmentation, in Table TABREF18  The number of recognized medications in IBM-STT is only 95 compared to 725 (human written), we mainly consider the models' performance on G-STT's transcripts (343) 
On the Medications that were recognized correctly, the models can perform decently on ASR transcripts in comparison to human transcripts (within 5 points ROUGE-1 F1 for both tasks, refer to Table TABREF18)  This shows that the models are robust to ASR variations discussed above  The lower performance compared to human transcripts is mainly due to incorrect recognition of Dosage and other medications in the same segments (changing the meaning of the text)  By comparing the performance of the model on the human written transcripts with human (defined) segmentation and the same with auto segmentation, we see a 10 point drop in Dosage and 6 point drop in Frequency extraction tasks  This points out the need for more sophisticated segmentation algorithms 
With G-STT, our best model obtained ROUGE-1 F1 of 71 75 (which equals to percentage of times dosage is correct in this case) for Dosage extraction and 40 13 for Frequency extraction tasks  To measure the percentage of times the correct frequency was extracted by the model, we qualitatively compared the extracted and predicted frequency  We find that for 73 58% of the medications the model can find the correct frequency from the transcripts 
Conclusion
In this paper, we explore the Medication Regimen (MR) extraction task of extracting dosage and frequency for the medications mentioned in a doctor-patient conversation transcript  We explore different variations of abstractive QA models and new architecture in the intersection of QA and IE frameworks and provide a comparative performance analysis of the methods along with other techniques like pretraining to improve the overall performance  Finally, we demonstrate the performance of our best-performing models by automatically extracting MR tags from spontaneous doctor-patient conversations (using commercially available ASR)  Our best model can correctly extract the dosage for 71 75% (interpretation of ROUGE-1 score) and frequency for 73 58% (on qualitative evaluation) of the medications discussed in the transcripts generated using Google Speech-To-Text  In summary, we demonstrate that research on NLP can be translated into real-world clinical settings to realize its benefits for both doctors and patients 
Using ASR transcripts in our training process to improve our performance on both the tasks and extending the medication regimen extraction network to extract other important medical information can be interesting lines of future work 
Acknowledgements
We thank: University of Pittsburgh Medical Center (UPMC), and Abridge AI Inc  for providing access to the de-identified data corpus; Dr  Shivdev Rao, a faculty member and practicing cardiologist in UPMC's Heart and Vascular Institute and Prof  Florian Metze, Associate Research Professor, Carnegie Mellon University for helpful discussions; Ben Schloss, Steven Coleman, and Deborah Osakue for data business development and annotation management 

ground_truth: Unanswerable

CFILT-preorder system