prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was their performance on emotion detection?
Context 1: Introduction
This work is licenced under a Creative Commons Attribution 4 0 International Licence  Licence details: http://creativecommons org/licenses/by/4 0/
In the spirit of the brevity of social media's messages and reactions, people have got used to express feelings minimally and symbolically, as with hashtags on Twitter and Instagram  On Facebook, people tend to be more wordy, but posts normally receive more simple “likes” than longer comments  Since February 2016, Facebook users can express specific emotions in response to a post thanks to the newly introduced reaction feature (see Section SECREF2 ), so that now a post can be wordlessly marked with an expression of say “joy" or “surprise" rather than a generic “like” 
It has been observed that this new feature helps Facebook to know much more about their users and exploit this information for targeted advertising BIBREF0 , but interest in people's opinions and how they feel isn't limited to commercial reasons, as it invests social monitoring, too, including health care and education BIBREF1   However, emotions and opinions are not always expressed this explicitly, so that there is high interest in developing systems towards their automatic detection  Creating manually annotated datasets large enough to train supervised models is not only costly, but also—especially in the case of opinions and emotions—difficult, due to the intrinsic subjectivity of the task BIBREF2 , BIBREF3   Therefore, research has focused on unsupervised methods enriched with information derived from lexica, which are manually created BIBREF3 , BIBREF4   Since go2009twitter have shown that happy and sad emoticons can be successfully used as signals for sentiment labels, distant supervision, i e  using some reasonably safe signals as proxies for automatically labelling training data BIBREF5 , has been used also for emotion recognition, for example exploiting both emoticons and Twitter hashtags BIBREF6 , but mainly towards creating emotion lexica  mohammad2015using use hashtags, experimenting also with highly fine-grained emotion sets (up to almost 600 emotion labels), to create the large Hashtag Emotion Lexicon  Emoticons are used as proxies also by hallsmarmulti, who use distributed vector representations to find which words are interchangeable with emoticons but also which emoticons are used in a similar context 
We take advantage of distant supervision by using Facebook reactions as proxies for emotion labels, which to the best of our knowledge hasn't been done yet, and we train a set of Support Vector Machine models for emotion recognition  Our models, differently from existing ones, exploit information which is acquired entirely automatically, and achieve competitive or even state-of-the-art results for some of the emotion labels on existing, standard evaluation datasets  For explanatory purposes, related work is discussed further and more in detail when we describe the benchmarks for evaluation (Section SECREF3 ) and when we compare our models to existing ones (Section SECREF5 )  We also explore and discuss how choosing different sets of Facebook pages as training data provides an intrinsic domain-adaptation method 
Facebook reactions as labels
For years, on Facebook people could leave comments to posts, and also “like” them, by using a thumbs-up feature to explicitly express a generic, rather underspecified, approval  A “like” could thus mean “I like what you said", but also “I like that you bring up such topic (though I find the content of the article you linked annoying)" 
In February 2016, after a short trial, Facebook made a more explicit reaction feature available world-wide  Rather than allowing for the underspecified “like” as the only wordless response to a post, a set of six more specific reactions was introduced, as shown in Figure FIGREF1 : Like, Love, Haha, Wow, Sad and Angry  We use such reactions as proxies for emotion labels associated to posts 
We collected Facebook posts and their corresponding reactions from public pages using the Facebook API, which we accessed via the Facebook-sdk python library  We chose different pages (and therefore domains and stances), aiming at a balanced and varied dataset, but we did so mainly based on intuition (see Section SECREF4 ) and with an eye to the nature of the datasets available for evaluation (see Section SECREF5 )  The choice of which pages to select posts from is far from trivial, and we believe this is actually an interesting aspect of our approach, as by using different Facebook pages one can intrinsically tackle the domain-adaptation problem (See Section SECREF6 for further discussion on this)  The final collection of Facebook pages for the experiments described in this paper is as follows: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney 
Note that thankful was only available during specific time spans related to certain events, as Mother's Day in May 2016 
For each page, we downloaded the latest 1000 posts, or the maximum available if there are fewer, from February 2016, retrieving the counts of reactions for each post  The output is a JSON file containing a list of dictionaries with a timestamp, the post and a reaction vector with frequency values, which indicate how many users used that reaction in response to the post (Figure FIGREF3 )  The resulting emotion vectors must then be turned into an emotion label 
In the context of this experiment, we made the simple decision of associating to each post the emotion with the highest count, ignoring like as it is the default and most generic reaction people tend to use  Therefore, for example, to the first post in Figure FIGREF3 , we would associate the label sad, as it has the highest score (284) among the meaningful emotions we consider, though it also has non-zero scores for other emotions  At this stage, we didn't perform any other entropy-based selection of posts, to be investigated in future work 
Emotion datasets
Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset  In order to compare our performance to state-of-the-art results, we have used them as well  In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7   A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation 
Affective Text dataset
Task 14 at SemEval 2007 BIBREF7 was concerned with the classification of emotions and valence in news headlines  The headlines where collected from several news websites including Google news, The New York Times, BBC News and CNN  The used emotion labels were Anger, Disgust, Fear, Joy, Sadness, Surprise, in line with the six basic emotions of Ekman's standard model BIBREF8   Valence was to be determined as positive or negative  Classification of emotion and valence were treated as separate tasks  Emotion labels were not considered as mututally exclusive, and each emotion was assigned a score from 0 to 100  Training/developing data amounted to 250 annotated headlines (Affective development), while systems were evaluated on another 1000 (Affective test)  Evaluation was done using two different methods: a fine-grained evaluation using Pearson's r to measure the correlation between the system scores and the gold standard; and a coarse-grained method where each emotion score was converted to a binary label, and precision, recall, and f-score were computed to assess performance  As it is done in most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , we also treat this as a classification problem (coarse-grained)  This dataset has been extensively used for the evaluation of various unsupervised methods BIBREF2 , but also for testing different supervised learning techniques and feature portability BIBREF10  
Fairy Tales dataset
This is a dataset collected by alm2008affect, where about 1,000 sentences from fairy tales (by B  Potter, H C  Andersen and Grimm) were annotated with the same six emotions of the Affective Text dataset, though with different names: Angry, Disgusted, Fearful, Happy, Sad, and Surprised  In most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , only sentences where all annotators agreed are used, and the labels angry and disgusted are merged  We adopt the same choices 
ISEAR
The ISEAR (International Survey on Emotion Antecedents and Reactions BIBREF11 , BIBREF12 ) is a dataset created in the context of a psychology project of the 1990s, by collecting questionnaires answered by people with different cultural backgrounds  The main aim of this project was to gather insights in cross-cultural aspects of emotional reactions  Student respondents, both psychologists and non-psychologists, were asked to report situations in which they had experienced all of seven major emotions (joy, fear, anger, sadness, disgust, shame and guilt)  In each case, the questions covered the way they had appraised a given situation and how they reacted  The final dataset contains reports by approximately 3000 respondents from all over the world, for a total of 7665 sentences labelled with an emotion, making this the largest dataset out of the three we use 
Overview of datasets and emotions
We summarise datasets and emotion distribution from two viewpoints  First, because there are different sets of emotions labels in the datasets and Facebook data, we need to provide a mapping and derive a subset of emotions that we are going to use for the experiments  This is shown in Table TABREF8 , where in the “Mapped” column we report the final emotions we use in this paper: anger, joy, sadness, surprise  All labels in each dataset are mapped to these final emotions, which are therefore the labels we use for training and testing our models 
Second, the distribution of the emotions for each dataset is different, as can be seen in Figure FIGREF9  
In Figure FIGREF9 we also provide the distribution of the emotions anger, joy, sadness, surprise per Facebook page, in terms of number of posts (recall that we assign to a post the label corresponding to the majority emotion associated to it, see Section SECREF2 )  We can observe that for example pages about news tend to have more sadness and anger posts, while pages about cooking and tv-shows have a high percentage of joy posts  We will use this information to find the best set of pages for a given target domain (see Section SECREF5 ) 
Model
There are two main decisions to be taken in developing our model: (i) which Facebook pages to select as training data, and (ii) which features to use to train the model, which we discuss below  Specifically, we first set on a subset of pages and then experiment with features  Further exploration of the interaction between choice of pages and choice of features is left to future work, and partly discussed in Section SECREF6   For development, we use a small portion of the Affective data set described in Section SECREF4 , that is the portion that had been released as development set for SemEval's 2007 Task 14 BIBREF7 , which contains 250 annotated sentences (Affective development, Section SECREF4 )  All results reported in this section are on this dataset  The test set of Task 14 as well as the other two datasets described in Section SECREF3 will be used to evaluate the final models (Section SECREF4 ) 
Selecting Facebook pages
Although page selection is a crucial ingredient of this approach, which we believe calls for further and deeper, dedicated investigation, for the experiments described here we took a rather simple approach  First, we selected the pages that would provide training data based on intuition and availability, then chose different combinations according to results of a basic model run on development data, and eventually tested feature combinations, still on the development set 
For the sake of simplicity and transparency, we first trained an SVM with a simple bag-of-words model and default parameters as per the Scikit-learn implementation BIBREF13 on different combinations of pages  Based on results of the attempted combinations as well as on the distribution of emotions in the development dataset (Figure FIGREF9 ), we selected a best model (B-M), namely the combined set of Time, The Guardian and Disney, which yields the highest results on development data  Time and The Guardian perform well on most emotions but Disney helps to boost the performance for the Joy class 
Features
In selecting appropriate features, we mainly relied on previous work and intuition  We experimented with different combinations, and all tests were still done on Affective development, using the pages for the best model (B-M) described above as training data  Results are in Table TABREF20   Future work will further explore the simultaneous selection of features and page combinations 
We use a set of basic text-based features to capture the emotion class  These include a tf-idf bag-of-words feature, word (2-3) and character (2-5) ngrams, and features related to the presence of negation words, and to the usage of punctuation 
This feature is used in all unsupervised models as a source of information, and we mainly include it to assess its contribution, but eventually do not use it in our final model 
We used the NRC10 Lexicon because it performed best in the experiments by BIBREF10 , which is built around the emotions anger, anticipation, disgust, fear, joy, sadness, and surprise, and the valence values positive and negative  For each word in the lexicon, a boolean value indicating presence or absence is associated to each emotion  For a whole sentence, a global score per emotion can be obtained by summing the vectors for all content words of that sentence included in the lexicon, and used as feature 
As additional feature, we also included Word Embeddings, namely distributed representations of words in a vector space, which have been exceptionally successful in boosting performance in a plethora of NLP tasks  We use three different embeddings:
Google embeddings: pre-trained embeddings trained on Google News and obtained with the skip-gram architecture described in BIBREF14   This model contains 300-dimensional vectors for 3 million words and phrases 
Facebook embeddings: embeddings that we trained on our scraped Facebook pages for a total of 20,000 sentences  Using the gensim library BIBREF15 , we trained the embeddings with the following parameters: window size of 5, learning rate of 0 01 and dimensionality of 100  We filtered out words with frequency lower than 2 occurrences 
Retrofitted embeddings: Retrofitting BIBREF16 has been shown as a simple but efficient way of informing trained embeddings with additional information derived from some lexical resource, rather than including it directly at the training stage, as it's done for example to create sense-aware BIBREF17 or sentiment-aware BIBREF18 embeddings  In this work, we retrofit general embeddings to include information about emotions, so that emotion-similar words can get closer in space  Both the Google as well as our Facebook embeddings were retrofitted with lexical information obtained from the NRC10 Lexicon mentioned above, which provides emotion-similarity for each token  Note that differently from the previous two types of embeddings, the retrofitted ones do rely on handcrafted information in the form of a lexical resource 
Results on development set
We report precision, recall, and f-score on the development set  The average f-score is reported as micro-average, to better account for the skewed distribution of the classes as well as in accordance to what is usually reported for this task BIBREF19  
From Table TABREF20 we draw three main observations  First, a simple tf-idf bag-of-word mode works already very well, to the point that the other textual and lexicon-based features don't seem to contribute to the overall f-score (0 368), although there is a rather substantial variation of scores per class  Second, Google embeddings perform a lot better than Facebook embeddings, and this is likely due to the size of the corpus used for training  Retrofitting doesn't seem to help at all for the Google embeddings, but it does boost the Facebook embeddings, leading to think that with little data, more accurate task-related information is helping, but corpus size matters most  Third, in combination with embeddings, all features work better than just using tf-idf, but removing the Lexicon feature, which is the only one based on hand-crafted resources, yields even better results  Then our best model (B-M) on development data relies entirely on automatically obtained information, both in terms of training data as well as features 
Results
In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3  
Our B-M model relies on subsets of Facebook pages for training, which were chosen according to their performance on the development set as well as on the observation of emotions distribution on different pages and in the different datasets, as described in Section SECREF4   The feature set we use is our best on the development set, namely all the features plus Google-based embeddings, but excluding the lexicon  This makes our approach completely independent of any manual annotation or handcrafted resource  Our model's performance is compared to the following systems, for which results are reported in the referred literature  Please note that no other existing model was re-implemented, and results are those reported in the respective papers 
Discussion, conclusions and future work
We have explored the potential of using Facebook reactions in a distant supervised setting to perform emotion classification  The evaluation on standard benchmarks shows that models trained as such, especially when enhanced with continuous vector representations, can achieve competitive results without relying on any handcrafted resource  An interesting aspect of our approach is the view to domain adaptation via the selection of Facebook pages to be used as training data 
We believe that this approach has a lot of potential, and we see the following directions for improvement  Feature-wise, we want to train emotion-aware embeddings, in the vein of work by tang:14, and iacobacci2015sensembed  Retrofitting FB-embeddings trained on a larger corpus might also be successful, but would rely on an external lexicon 
The largest room for yielding not only better results but also interesting insights on extensions of this approach lies in the choice of training instances, both in terms of Facebook pages to get posts from, as well as in which posts to select from the given pages  For the latter, one could for example only select posts that have a certain length, ignore posts that are only quotes or captions to images, or expand posts by including content from linked html pages, which might provide larger and better contexts BIBREF23   Additionally, and most importantly, one could use an entropy-based measure to select only posts that have a strong emotion rather than just considering the majority emotion as training label  For the former, namely the choice of Facebook pages, which we believe deserves the most investigation, one could explore several avenues, especially in relation to stance-based issues BIBREF24   In our dataset, for example, a post about Chile beating Colombia in a football match during the Copa America had very contradictory reactions, depending on which side readers would cheer for  Similarly, the very same political event, for example, would get very different reactions from readers if it was posted on Fox News or The Late Night Show, as the target audience is likely to feel very differently about the same issue  This also brings up theoretical issues related more generally to the definition of the emotion detection task, as it's strongly dependent on personal traits of the audience  Also, in this work, pages initially selected on availability and intuition were further grouped into sets to make training data according to performance on development data, and label distribution  Another criterion to be exploited would be vocabulary overlap between the pages and the datasets 
Lastly, we could develop single models for each emotion, treating the problem as a multi-label task  This would even better reflect the ambiguity and subjectivity intrinsic to assigning emotions to text, where content could be at same time joyful or sad, depending on the reader 
Acknowledgements
In addition to the anonymous reviewers, we want to thank Lucia Passaro and Barbara Plank for insightful discussions, and for providing comments on draft versions of this paper 
Context 2: Introduction
To appear in Proceedings of International Workshop on Health Intelligence (W3PHIAI) of the 34th AAAI Conference on Artificial Intelligence, 2020 
Physician burnout is a growing concern, estimated to be experienced by at least 35% of physicians in the developing world and 50% in the United States BIBREF0  BIBREF1 found that for every hour physicians provide direct clinical facetime to patients, nearly two additional hours are spent on EHR (Electronic Health Records) and administrative or desk work  As per the study conducted by Massachusetts General Physicians Organization (MPGO) BIBREF2 and as reported by BIBREF3, the average time spent on administrative tasks increased from 23 7% in 2014 to 27 9% in 2017  Both the surveys found that time spent on administrative tasks was positively associated with higher likelihood of burnout  Top reasons under administrative burden include working on the ambulatory EHR, handling medication reconciliation (sometimes done by aids), medication renewals, and medical billing and coding  The majority of these reasons revolve around documentation of information exchanged between doctors and patients during the clinical encounters  Automatically extracting such clinical information BIBREF4, BIBREF5 can not only help alleviate the documentation burden on the physician, but also allow them to dedicate more time directly with patients 
Among all the clinical information extraction tasks, Medication Regimen (Medication, dosage, and frequency) extraction is particularly interesting due to its ability to help doctors with medication orders cum renewals, medication reconciliation, potentially verifying the reconciliations for errors, and, other medication-centered EHR documentation tasks  In addition, the same information when provided to patients can help them with better recall of doctor instructions which might aid in compliance with the care plan  This is particularly important given that patients forget or wrongly recollect 40-80% BIBREF6 of what is discussed in the clinic, and accessing EHR data has its own challenges 
Spontaneous clinical conversations happening between a doctor and a patient, have several distinguishing characteristics from a normal monologue or prepared speech: it involves multiple speakers with overlapping dialogues, covers a variety of speech patterns, and the vocabulary can range from colloquial to complex domain-specific language  With recent advancements in Conversational Speech Recognition BIBREF7 rendering the systems less prone to errors, the subsequent challenge of understanding and extracting relevant information from the conversations is receiving increasing research focus BIBREF4, BIBREF8 
In this paper, we focus on local information extraction in transcribed clinical conversations  Specifically, we extract dosage (e g  5mg) and frequency (e g  once a day) for the medications (e g  aspirin) from these transcripts, collectively referred to as Medication Regimen (MR) extraction  The information extraction is local as we extract the information from a segment of the transcript and not the entire transcript since doing the latter is difficult owing to the long meandering nature of the conversations often with multiple medication regimens and care plans being discussed 
The challenges associated with the Medication Regimen (MR) extraction task include understanding the spontaneous dialog with clinical vocabulary and understanding the relationship between different entities as the discussion can contain multiple medications and dosages (e g  doctor revising a dosage or reviewing all the current medications) 
We frame this problem as a Question Answering (QA) task by generating questions using templates  We base the QA model on pointer-generator networks BIBREF9 augmented with Co-Attentions BIBREF10  In addition, we develop models combining QA and Information Extraction frameworks using multi-decoder (one each for dosage and frequency) architecture 
Lack of availability of a large volume of data is a typical challenge in healthcare  A conversation corpus by itself is a rare commodity in the healthcare data space because of the cost and difficulty in handing (because of data privacy concerns)  Moreover, transcribing and labeling the conversations is a costly process as it requires domain-specific medical annotation expertise  To address data shortage and improve the model performance, we investigate different high-performance contextual embeddings (ELMO BIBREF11, BERT BIBREF12 and ClinicalBERT BIBREF13), and pretrain the models on a clinical summarization task  We further investigate the effects of training data size on our models 
On the MR extraction task, ELMo with encoder multi-decoder architecture and BERT with encoder-decoder with encoders pretrained on the summarization task perform the best  The best-performing models improve our baseline's dosage and frequency extractions ROUGE-1 F1 scores from 54 28 and 37 13 to 89 57 and 45 94, respectively 
Using our models, we present the first fully automated system to extract MR tags from spontaneous doctor-patient conversations  We evaluate the system (using our best performing models) on the transcripts generated from Automatic Speech Recognition (ASR) APIs offered by Google and IBM  In Google ASR's transcripts, our best model obtained ROUGE-1 F1 of 71 75 for Dosage extraction (which in this specific case equals to the percentage of times dosage is correct, refer Metrics Section for more details) and 40 13 for Frequency extraction tasks  On qualitative evaluation, we find that for 73 58% of the medications the model can find the correct frequency  These results demonstrate that the research on NLP can be used effectively in a real clinical setting to benefit both doctors and patients
Data
Our dataset consists of a total of 6,693 real doctor-patient conversations recorded in a clinical setting using distant microphones of varying quality  The recordings have an average duration of 9min 28s and have a verbatim transcript of 1,500 words on average (written by the experts)  Both the audio and the transcript are de-identified (by removing the identifying information) with digital zeros and [de-identified] tags, respectively  The sentences in the transcript are grounded to the audio with the timestamps of its first and last word 
The transcript of the conversations are annotated with summaries and Medication Regimen tags (MR tags), both grounded using the timestamps of the sentences from the transcript deemed relevant by the expert annotators, refer to Table TABREF1  The transcript for a typical conversation can be quite long, and not easy for many of the high performing deep learning models to act on  Moreover, the medical information about a concept/condition/entity can change during the conversation after a significant time gap  For example, dosage of a medication can be different when discussing current medication the patient is on vs when they are prescribed a different dosage  Hence, we have annotations, that are grounded to a short segment of the transcript 
The summaries (#words - $\mu = 9 7; \sigma = 10 1$) are medically relevant and local  The MR tags are also local and are of the form {Medication Name, Dosage, Frequency}  If dosage ($\mu = 2 0; \sigma = 0$) or frequency ($\mu = 2 1; \sigma = 1 07$) information for a medication is not present in a grounded sentence, the corresponding field in the MR tag will be marked as `none' 
In the MR tags, Medication Name and Dosage (usually a quantity followed by its units) can be relatively easily extracted from the transcript except for the units of the dosage which is sometimes inferred  In contrast, due to high degree of linguistic variation with which Frequency is often expressed, extracting Frequency requires an additional inference step  For example, `take one in the morning and at noon' from the transcript is tagged as `twice a day' in the frequency tag, likewise `take it before sleeping' is tagged as `at night time' 
Out of overall 6,693 files, we set aside a random sample of 423 files (denoted as $\mathcal {D}_{test}$) for final evaluation  The remaining 6,270 files are used for training with 80% train (5016), 10% validation (627), and 10% test (627) split  Overall, the 6,270 files contains 156,186 summaries and 32,000 MR tags out of which 8,654 MR tags contain values for at least one of the Dosage or Frequency, which we used for training to avoid overfitting (the remaining MR tags have both Dosage and Frequency as `none')  Note that we have two test datasets: `10% test' - used to evaluate all the models, and $\mathcal {D}_{test}$ - used to measure the performance of best performing models on ASR transcripts 
Approach
We frame the Medication Regimen extraction problem as a Question Answering (QA) task, which forms the basis for our first approach  It can also be considered as a specific inference or relation extract task, since we extract specific information about an entity (Medication Name), hence our second approach is at the intersection of Question Answering (QA) and Information Extraction (IE) domains  Both the approaches involve using a contiguous segment of the transcript and the Medication Name as input, to find/infer the medication's Dosage and Frequency  When testing the approaches mimicking real-world conditions, we extract Medication Name from the transcript separately using ontology, refer to SECREF19 
In the first approach, we frame the MR task as a QA task and generate questions using the template: “What is the $ <$dosage/frequency$>$ for $<$Medication Name$>$"  Here, we use an abstractive QA model based on pointer-generator networks BIBREF9 augmented with coattention encoder BIBREF10 (QA-PGNet) 
In the second approach, we frame the problem as a conditioned IE task, where the information extracted depends on an entity (Medication Name)  Here, we use a multi-decoder pointer-generator network augmented with coattention encoder (Multi-decoder QA-PGNet)  Instead of using templates to generate questions and using a single decoder to extract different types of information as in the QA approach (which might lead to performance degradation), here we consider separate decoders for extracting specific types of information about an entity $E$ (Medication Name) 
Approach ::: Pointer-generator Network (PGNet)
The network is a sequence-to-sequence attention model that can both copy a word from the input $I$ containing $P$ word tokens or generate a word from its vocabulary $vocab$, to produce the output sequence 
First, the tokens of the $I$ are converted to embeddings and are fed one-by-one to the encoder, a single bi-LSTM layer, which encodes the tokens in $I$ into a sequence of hidden states - $H=encoder(I)$, where $ H=[h_1   h_P]$ 
For each decoder time step $t$, in a loop, we compute, 1) attention $a_t$ (using the last decoder state $s_{t-1}$), over the input tokens $I$, and 2) the decoder state $s_t$ using $a_t$  Then, at each time step, using both $a_t$ and $s_t$ we can find the probability $P_t(w)$, of producing a word $w$ (from both $vocab$ and $I$)  For convenience, we denote the attention and the decoder as $decoder_{pg}(H)=P(w)$, where $P(w)=[P_1(w)   P_T(w)]$  The output can then be decoded from $P(w)$, which is decoded until it produces an `end of output token' or the number of steps reach the maximum allowed limit 
Approach ::: QA PGNet
We first encode both the question - $H_Q = encoder(Q)$, and the input - $H_I = encoder(I)$, separately using encoders (with shared weights)  Then, to condition $I$ on $Q$ (and vice versa), we use the coattention encoder BIBREF10 which attends to both the $I$ and $Q$ simultaneously to generates the coattention context - $C_D = coatt(H_I, H_Q)$  Finally, using the pointer-generator decoder we find the probability distribution of the output sequence - $P(w) = decoder_{pg}([H_I; C_D])$, which is then decoded to generate the answer sequence 
Approach ::: Multi-decoder (MD) QA PGNet
After encoding the inputs into $H_I$ and $H_E$, for extracting $K$ types of information about an entity in an IE fashion, we use the following multi-decoder (MD) setup:
Predictions for each of the $K$ decoders are then decoded using $P^k(w)$ 
All the networks discussed above are trained using a negative log-likelihood loss for the target word at each time step and summed over all the decoder time steps 
Experiments
We initialized MR extraction models' vocabulary from the training dataset after removing words with a frequency lower than 30 in the dataset, resulting in 456 words  Our vocabulary is small because of the size of the dataset, hence we rely on the model's ability to copy words to produce the output effectively  In all our model variations, the embedding and the network's hidden dimension are set to be equal  The networks were trained with a learning rate of 0 0015, dropout of 0 5 on the embedding layer, normal gradient clipping set at 2, batch size of 8, and optimized with Adagrad BIBREF15 and the training was stopped using the $10\%$ validation dataset 
Experiments ::: Data Processing
We did the following basic preprocessing to our data, 1) added `none' to the beginning of the input utterance, so the network could point to it when there was no relevant information in the input, 2) filtered outliers with a large number of grounded transcript sentences ($>$150 words), and 3) converted all text to lower case 
To improve performance, we 1) standardized all numbers (both digits and words) to words concatenated with a hyphen (e g  110 -$>$ one-hundred-ten), in both input and output, 2) removed units from Dosage as sometimes the units were not explicitly mentioned in the transcript segment but were written by the annotators using domain knowledge, 3) prepended all medication mentions with `rx-' tag, as this helps model's performance when multiple medications are discussed in a segment (in both input and output), and 4) when a transcript segment has multiple medications or dosages being discussed we randomly shuffle them (in both input and output) and create a new data point, to increases the number of training data points  Randomly shuffling the entities increases the number of training MR tags from 8,654 to 11,521  Based on the data statistics after data processing, we fixed the maximum encoder steps to 100, dosage decoder steps to 1, and frequency decoder steps to 3 (for both the QA and Multi-decoder QA models) 
Experiments ::: Metrics
For the MR extraction task, we measure the ROUGE-1 scores BIBREF14 for both the Dosage and Frequency extraction tasks  It should be noted that since Dosage is a single word token (after processing), both the reference and hypothesis are a single token, making its ROUGE-1 F1, Precision and Recall scores equal and be equal to percentage of times we find the correct dosage for the medications 
In our annotations, Frequency has conflicting tags (e g  {`Once a day', `twice a day'} and `daily'), hence metrics like Exact Match will be erroneous  To address this issue, we use the ROUGE scores to compare different models on the 10% test dataset and we use qualitative evaluation to measure the top-performing models on $\mathcal {D}_{test}$ 
Experiments ::: Model variations
We consider QA PGNet and Multi-decoder QA PGNet with lookup table embedding as baseline models and improve on the baselines with other variations described below 
Apart from learning-based baselines, we also create two naive baselines, one each for the Dosage and Frequency extraction tasks  For Dosage extraction, the baseline we consider is `Nearest Number', where we take the number nearest to the Medication Name as the prediction, and `none' if no number is mentioned or if the Medication Name is not detected in the input  For Frequency extraction, the baseline we consider is `Random Top-3' where we predict a random Frequency tag, from top-3 most frequent ones from our dataset - {`none', `daily', `twice a day'} 
Embedding: We developed different variations of our models with a simple lookup table embeddings learned from scratch and using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13 (trained and provided by the authors)  Refer to Table TABREF5 for the performance comparisons 
We derive embeddings from ELMo by learning a linear combination of its last three layer's hidden states (task-specific fine-tuning BIBREF11)  Similarly, for BERT-based embeddings, we take a linear combination of the hidden states from its last four layers, as this combination performs best without increasing the size of the embeddings BIBREF16  Since BERT and ClinicalBERT use word-piece vocabulary and computes sub-word embeddings, we compute word-level embedding by averaging the corresponding sub-word tokens  ELMo and BERT embeddings both have 1024 dimensions, ClinicalBERT have 768 as it is based on BERT base model, and the lookup table have 128 – higher dimension models leads to overfitting 
Pertaining Encoder: We trained the PGNet as a summarization task using the clinical summaries and used the trained model to initialize the encoders (and the embeddings) of the corresponding QA models  We use a vocab size of 4073 words, derived from the training dataset with a frequency threshold of 30 for the task  We trained the models using Adagrad optimizer with a learning rate of 0 015, normal gradient clipping set at 2 and trained for around 150000 iterations (stopped using validation dataset)  On the summarization task PGNet obtained ROUGE-1 F1 scores of 41 42 with ELMo and 39 15 with BERT embeddings  We compare the effects of pretraining the model in Table: TABREF5, models with `pretrained encoder' had their encoders and embeddings pretrained with the summarization task 
Results and Discussion ::: Difference in networks and approaches
Embeddings: On Dosage extraction, in general, ELMo obtains better performance than BERT, refer to Table TABREF5  This could be because we concatenated the numbers with a hyphen, and as ELMo uses character-level tokens it can learn the tagging better than BERT, a similar observation is also noted in BIBREF17  On the other hand, on Frequency extraction, without pretraining, ELMo's performance is lagging by a big margin of $\sim $8 5 ROUGE-1 F1 compared to BERT-based embeddings 
Although in cases without encoder pretraining, ClinicalBERT performed the best in the Frequency extraction task (by a small margin), in general, it does not perform as well as BERT  This could also be a reflection of the fact that the language and style of writing used in clinical notes is very different from the way doctors converse with patients and the embedding dimension difference  Lookup table embedding performed decently in the frequency extraction task but lags behind in the Dosage extraction task 
From the metrics and qualitative inspection, we find that the Frequency extraction is an easier task than the Dosage extraction  This is because, in the conversations, frequency information usually occurs in isolation and near the medications, but a medication's dosage can occur 1) near other medication's dosages, 2) with previous dosages (when a dosage for a medication is revised), and 3) after a large number of words from the medication 
Other Variations: Considering various models' performance (without pretraining) and the resource constraint, we choose ELMo and BERT embeddings to analyze the effects of pretraining the encoder  When the network's encoder (and embedding) is pretrained with the summarization task, we 1) see a small decrease in the average number of iterations required for training, 2) improvement in individual performances of all models for both the sub-tasks, and 3) get best performance metrics across all variations, refer to Table TABREF5  Both in terms of performance and the training speed, there is no clear winner between shared and multi-decoder approaches  Medication tagging and data augmentation increase the best-performing model's ROUGE-1 F1 score by $\sim $1 5 for the Dosage extraction task 
We also measure the performance of Multitask Question Answering Network (MQAN) BIBREF18 a QA model trained by the authors on the Decathlon multitask challenge  Since MQAN was not trained to produce the output sequence in our MR tags, it would not be fair to compute ROUGE scores  Instead, we randomly sample the MQAN's predictions from the 10% test dataset and qualitatively evaluate it  From the evaluations, we find that MQAN can not distinguish between frequency and dosage, and mixed the answers  MQAN correctly predicted the dosage for 29 73% and frequency for 24 24% percent of the medications compared to 84 12% and 76 34% for the encoder pretrained BERT QA PGNet model trained on our dataset  This could be because of the difference in the training dataset, domain and the tasks in the Decathlon challenge compared to ours 
Almost all our models perform better than the naive baselines and the ones using lookup table embeddings, and our best performing models outperform them significantly  Among all the variations, the best performing models are ELMo with Multi-decoder (Dosage extraction) and BERT with shared-decoder QA PGNet architecture with pretrained encoder (Frequency extraction)  We choose these two models for our subsequent analysis 
Results and Discussion ::: Breakdown of Performance
We categorize the 10% test dataset into different categories based on the complexity and type of the data and analyze the breakdown of the system's performance in Table TABREF11  We breakdown the Frequency extraction into two categories, 1) None: ground truth Frequency tag is `none', and 2) NN (Not None): ground truth Frequency tag is not `none'  Similarly, the Dosage extraction into 5 categories, 1) None: ground truth dosage tag is `none', 2) MM (Multiple Medicine): input segment has more than one Medication mentioned, 3) MN (Multiple Numbers): input segment has more than one number present, and 4) NBM (Number between correct Dosage and Medicine) : between the Medication Name and the correct Dosage in the input segment there are other numbers present  Note that the categories of the Dosage extraction task are not exhaustive, and one tag can belong to multiple categories 
From the performance breakdown of Dosage extraction task, we see that 1) the models predict `none' better than other categories, i e , the models are correctly able to identify when a medication's dosage is absent, 2) there is performance dip in hard cases (MM, MN, and NBM), 3) the models are able to figure out the correct dosage (decently) for a medication even when there are multiple numbers/dosage present, and 4) the model struggles the most in the NBM category  The models' low performance in NBM could be because we have a comparatively lower number of examples to train in this category  The Frequency extraction task performs equally well when the tag is `none` or not  In most categories, we see an increase in performance when using pretrained encoders 
Results and Discussion ::: Training Dataset Size
We vary the number of MR tags used to train the model and analyze the model's performance when training the networks, using publicly available contextual embeddings, compared to using pretrained embeddings and encoder (pretrained on the summarization task)  Out of the 5,016 files in the 80% train dataset only 2,476 have atleast one MR tag  Therefore, out of the 2476 files, we randomly choose 100, 500, and 1000 files and trained the best performing model variations to observe the performance differences, refer to Figure FIGREF12  For all these experiments we used the same vocabulary size (456), the same hyper/training parameters, and the same 10% test split of 627 files 
As expected, we see that the encoder pretrained models have higher performance on all the different training data sizes, i e , they achieve higher performance on a lower number of data points, refer to Figure FIGREF12  The difference, as expected, shrinks as the training data size increases 
Results and Discussion ::: Evaluating on ASR transcripts
To test the performance of our models on real-world conditions, we use commercially available ASR services (Google and IBM) to transcribe the $\mathcal {D}_{test}$ files and measure the performance of our models without assuming any annotations (except when calculating the metrics)  It should be noted that this is not the case in our previous evaluations using `10% test' dataset where we use the segmentation information  For ground truth annotations on ASR transcripts, we aligned the MR tags from human written transcripts to the ASR transcript using their grounded timing information  Additionally, since ASR is prone to errors, during the alignment, if a medication from an MR tag is not recognized correctly in the ASR transcript, we remove the corresponding MR tag 
In our evaluations, we use Google Cloud Speech-to-Text (G-STT) and IBM Watson Speech to Text (IBM-STT) as these were among the top-performing ASR APIs on medical speech BIBREF19 and were readily available to us  We used G-STT, with the `video model' with punctuation settings  Unlike our human written transcripts, the transcript provided by G-STT is not verbatim and does not have disfluencies  IBM-STT, on the other hand, does not give punctuation so we used the speaker changes to add end-of-sentence punctuation 
In our $\mathcal {D}_{test}$ dataset, on initial study we see a Word Error Rate of $\sim $50% for the ASR APIs and this number is not accurate because, 1) of the de-identification, 2) disfluencies (verbatim) difference between the human written and ASR transcript, and 3) minor alignment differences between the audio and the ground truth transcript 
During this evaluation, we followed the same preprocessing methods we used during training  Then, we auto segment the transcript into small contiguous segments similar to the grounded sentences in the annotations for tags extraction  To segment the transcript, we follow a simple procedure  First, we detected all the medications in a transcript using RxNorm BIBREF20 via string matching  For all the detected medications, we selected $2 \le x \le 5$ nearby sentences as the input to our model  We increased $x$ iteratively until we encountered a quantity entity – detected using spaCy's entity recognizer, and we set $x$ as 2 if we did not detect any entities in the range 
We show the model's performance on ASR transcripts and human written transcripts with automatic segmentation, and human written transcripts with human (defined) segmentation, in Table TABREF18  The number of recognized medications in IBM-STT is only 95 compared to 725 (human written), we mainly consider the models' performance on G-STT's transcripts (343) 
On the Medications that were recognized correctly, the models can perform decently on ASR transcripts in comparison to human transcripts (within 5 points ROUGE-1 F1 for both tasks, refer to Table TABREF18)  This shows that the models are robust to ASR variations discussed above  The lower performance compared to human transcripts is mainly due to incorrect recognition of Dosage and other medications in the same segments (changing the meaning of the text)  By comparing the performance of the model on the human written transcripts with human (defined) segmentation and the same with auto segmentation, we see a 10 point drop in Dosage and 6 point drop in Frequency extraction tasks  This points out the need for more sophisticated segmentation algorithms 
With G-STT, our best model obtained ROUGE-1 F1 of 71 75 (which equals to percentage of times dosage is correct in this case) for Dosage extraction and 40 13 for Frequency extraction tasks  To measure the percentage of times the correct frequency was extracted by the model, we qualitatively compared the extracted and predicted frequency  We find that for 73 58% of the medications the model can find the correct frequency from the transcripts 
Conclusion
In this paper, we explore the Medication Regimen (MR) extraction task of extracting dosage and frequency for the medications mentioned in a doctor-patient conversation transcript  We explore different variations of abstractive QA models and new architecture in the intersection of QA and IE frameworks and provide a comparative performance analysis of the methods along with other techniques like pretraining to improve the overall performance  Finally, we demonstrate the performance of our best-performing models by automatically extracting MR tags from spontaneous doctor-patient conversations (using commercially available ASR)  Our best model can correctly extract the dosage for 71 75% (interpretation of ROUGE-1 score) and frequency for 73 58% (on qualitative evaluation) of the medications discussed in the transcripts generated using Google Speech-To-Text  In summary, we demonstrate that research on NLP can be translated into real-world clinical settings to realize its benefits for both doctors and patients 
Using ASR transcripts in our training process to improve our performance on both the tasks and extending the medication regimen extraction network to extract other important medical information can be interesting lines of future work 
Acknowledgements
We thank: University of Pittsburgh Medical Center (UPMC), and Abridge AI Inc  for providing access to the de-identified data corpus; Dr  Shivdev Rao, a faculty member and practicing cardiologist in UPMC's Heart and Vascular Institute and Prof  Florian Metze, Associate Research Professor, Carnegie Mellon University for helpful discussions; Ben Schloss, Steven Coleman, and Deborah Osakue for data business development and annotation management 
Context 3: Introduction
Text simplification aims to reduce the lexical and structural complexity of a text, while still retaining the semantic meaning, which can help children, non-native speakers, and people with cognitive disabilities, to understand text better  One of the methods of automatic text simplification can be generally divided into three categories: lexical simplification (LS) BIBREF0 , BIBREF1 , rule-based BIBREF2 , and machine translation (MT) BIBREF3 , BIBREF4   LS is mainly used to simplify text by substituting infrequent and difficult words with frequent and easier words  However, there are several challenges for the LS approach: a great number of transformation rules are required for reasonable coverage and should be applied based on the specific context; third, the syntax and semantic meaning of the sentence is hard to retain  Rule-based approaches use hand-crafted rules for lexical and syntactic simplification, for example, substituting difficult words in a predefined vocabulary  However, such approaches need a lot of human-involvement to manually define these rules, and it is impossible to give all possible simplification rules  MT-based approach has attracted great attention in the last several years, which addresses text simplification as a monolingual machine translation problem translating from 'ordinary' and 'simplified' sentences 
In recent years, neural Machine Translation (NMT) is a newly-proposed deep learning approach and achieves very impressive results BIBREF5 , BIBREF6 , BIBREF7   Unlike the traditional phrased-based machine translation system which operates on small components separately, NMT system is being trained end-to-end, without the need to have external decoders, language models or phrase tables  Therefore, the existing architectures in NMT are used for text simplification BIBREF8 , BIBREF4   However, most recent work using NMT is limited to the training data that are scarce and expensive to build  Language models trained on simplified corpora have played a central role in statistical text simplification BIBREF9 , BIBREF10   One main reason is the amount of available simplified corpora typically far exceeds the amount of parallel data  The performance of models can be typically improved when trained on more data  Therefore, we expect simplified corpora to be especially helpful for NMT models 
In contrast to previous work, which uses the existing NMT models, we explore strategy to include simplified training corpora in the training process without changing the neural network architecture  We first propose to pair simplified training sentences with synthetic ordinary sentences during training, and treat this synthetic data as additional training data  We obtain synthetic ordinary sentences through back-translation, i e  an automatic translation of the simplified sentence into the ordinary sentence BIBREF11   Then, we mix the synthetic data into the original (simplified-ordinary) data to train NMT model  Experimental results on two publicly available datasets show that we can improve the text simplification quality of NMT models by mixing simplified sentences into the training set over NMT model only using the original training data 
Related Work
Automatic TS is a complicated natural language processing (NLP) task, which consists of lexical and syntactic simplification levels BIBREF12   It has attracted much attention recently as it could make texts more accessible to wider audiences, and used as a pre-processing step, improve performances of various NLP tasks and systems BIBREF13 , BIBREF14 , BIBREF15   Usually, hand-crafted, supervised, and unsupervised methods based on resources like English Wikipedia and Simple English Wikipedia (EW-SEW) BIBREF10 are utilized for extracting simplification rules  It is very easy to mix up the automatic TS task and the automatic summarization task BIBREF3 , BIBREF16 , BIBREF6   TS is different from text summarization as the focus of text summarization is to reduce the length and redundant content 
At the lexical level, lexical simplification systems often substitute difficult words using more common words, which only require a large corpus of regular text to obtain word embeddings to get words similar to the complex word BIBREF1 , BIBREF9   Biran et al  BIBREF0 adopted an unsupervised method for learning pairs of complex and simpler synonyms from a corpus consisting of Wikipedia and Simple Wikipedia  At the sentence level, a sentence simplification model was proposed by tree transformation based on statistical machine translation (SMT) BIBREF3   Woodsend and Lapata BIBREF17 presented a data-driven model based on a quasi-synchronous grammar, a formalism that can naturally capture structural mismatches and complex rewrite operations  Wubben et al  BIBREF18 proposed a phrase-based machine translation (PBMT) model that is trained on ordinary-simplified sentence pairs  Xu et al  BIBREF19 proposed a syntax-based machine translation model using simplification-specific objective functions and features to encourage simpler output 
Compared with SMT, neural machine translation (NMT) has shown to produce state-of-the-art results BIBREF5 , BIBREF7   The central approach of NMT is an encoder-decoder architecture implemented by recurrent neural networks, which can represent the input sequence as a vector, and then decode that vector into an output sequence  Therefore, NMT models were used for text simplification task, and achieved good results BIBREF8 , BIBREF4 , BIBREF20   The main limitation of the aforementioned NMT models for text simplification depended on the parallel ordinary-simplified sentence pairs  Because ordinary-simplified sentence pairs are expensive and time-consuming to build, the available largest data is EW-SEW that only have 296,402 sentence pairs  The dataset is insufficiency for NMT model if we want to NMT model can obtain the best parameters  Considering simplified data plays an important role in boosting fluency for phrase-based text simplification, and we investigate the use of simplified data for text simplification

ground_truth: Answer with content missing: (Table 3) Best author's model B-M average micro f-score is 0.409, 0.459, 0.411 on Affective, Fairy Tales and ISEAR datasets respectively. 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is the data de-identified?
Context 3:  We are the first to show that we can effectively adapt neural translation models for text simplifiation with simplified corpora 
Simplified Corpora
We collected a simplified dataset from Simple English Wikipedia that are freely available, which has been previously used for many text simplification methods BIBREF0 , BIBREF10 , BIBREF3   The simple English Wikipedia is pretty easy to understand than normal English Wikipedia  We downloaded all articles from Simple English Wikipedia  For these articles, we removed stubs, navigation pages and any article that consisted of a single sentence  We then split them into sentences with the Stanford CorNLP BIBREF21 , and deleted these sentences whose number of words are smaller than 10 or large than 40  After removing repeated sentences, we chose 600K sentences as the simplified data with 11 6M words, and the size of vocabulary is 82K 
Text Simplification using Neural Machine Translation
Our work is built on attention-based NMT BIBREF5 as an encoder-decoder network with recurrent neural networks (RNN), which simultaneously conducts dynamic alignment and generation of the target simplified sentence 
The encoder uses a bidirectional RNN that consists of forward and backward RNN  Given a source sentence INLINEFORM0 , the forward RNN and backward RNN calculate forward hidden states INLINEFORM1 and backward hidden states INLINEFORM2 , respectively  The annotation vector INLINEFORM3 is obtained by concatenating INLINEFORM4 and INLINEFORM5  
The decoder is a RNN that predicts a target simplificated sentence with Gated Recurrent Unit (GRU) BIBREF22   Given the previously generated target (simplified) sentence INLINEFORM0 , the probability of next target word INLINEFORM1 is DISPLAYFORM0
where INLINEFORM0 is a non-linear function, INLINEFORM1 is the embedding of INLINEFORM2 , and INLINEFORM3 is a decoding state for time step INLINEFORM4  
State INLINEFORM0 is calculated by DISPLAYFORM0
where INLINEFORM0 is the activation function GRU 
The INLINEFORM0 is the context vector computed as a weighted annotation INLINEFORM1 , computed by DISPLAYFORM0
where the weight INLINEFORM0 is computed by DISPLAYFORM0 DISPLAYFORM1
where INLINEFORM0 , INLINEFORM1 and INLINEFORM2 are weight matrices  The training objective is to maximize the likelihood of the training data  Beam search is employed for decoding 
Synthetic Simplified Sentences
We train an auxiliary system using NMT model from the simplified sentence to the ordinary sentence, which is first trained on the available parallel data  For leveraging simplified sentences to improve the quality of NMT model for text simplification, we propose to adapt the back-translation approach proposed by Sennrich et al  BIBREF11 to our scenario  More concretely, Given one sentence in simplified sentences, we use the simplified-ordinary system in translate mode with greedy decoding to translate it to the ordinary sentences, which is denoted as back-translation  This way, we obtain a synthetic parallel simplified-ordinary sentences  Both the synthetic sentences and the available parallel data are used as training data for the original NMT system 
Evaluation
We evaluate the performance of text simplification using neural machine translation on available parallel sentences and additional simplified sentences 
Dataset  We use two simplification datasets (WikiSmall and WikiLarge)  WikiSmall consists of ordinary and simplified sentences from the ordinary and simple English Wikipedias, which has been used as benchmark for evaluating text simplification BIBREF17 , BIBREF18 , BIBREF8   The training set has 89,042 sentence pairs, and the test set has 100 pairs  WikiLarge is also from Wikipedia corpus whose training set contains 296,402 sentence pairs BIBREF19 , BIBREF20   WikiLarge includes 8 (reference) simplifications for 2,359 sentences split into 2,000 for development and 359 for testing 
Metrics  Three metrics in text simplification are chosen in this paper  BLEU BIBREF5 is one traditional machine translation metric to assess the degree to which translated simplifications differed from reference simplifications  FKGL measures the readability of the output BIBREF23   A small FKGL represents simpler output  SARI is a recent text-simplification metric by comparing the output against the source and reference simplifications BIBREF20  
We evaluate the output of all systems using human evaluation  The metric is denoted as Simplicity BIBREF8   The three non-native fluent English speakers are shown reference sentences and output sentences  They are asked whether the output sentence is much simpler (+2), somewhat simpler (+1), equally (0), somewhat more difficult (-1), and much more difficult (-2) than the reference sentence 
Methods  We use OpenNMT BIBREF24 as the implementation of the NMT system for all experiments BIBREF5 

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what language does this paper focus on?
Context 3:  We generally follow the default settings and training procedure described by Klein et al (2017)  We replace out-of-vocabulary words with a special UNK symbol  At prediction time, we replace UNK words with the highest probability score from the attention layer  OpenNMT system used on parallel data is the baseline system  To obtain a synthetic parallel training set, we back-translate a random sample of 100K sentences from the collected simplified corpora  OpenNMT used on parallel data and synthetic data is our model  The benchmarks are run on a Intel(R) Core(TM) i7-5930K CPU@3 50GHz, 32GB Mem, trained on 1 GPU GeForce GTX 1080 (Pascal) with CUDA v  8 0 
We choose three statistical text simplification systems  PBMT-R is a phrase-based method with a reranking post-processing step BIBREF18   Hybrid performs sentence splitting and deletion operations based on discourse representation structures, and then simplifies sentences with PBMT-R BIBREF25   SBMT-SARI BIBREF19 is syntax-based translation model using PPDB paraphrase database BIBREF26 and modifies tuning function (using SARI)  We choose two neural text simplification systems  NMT is a basic attention-based encoder-decoder model which uses OpenNMT framework to train with two LSTM layers, hidden states of size 500 and 500 hidden units, SGD optimizer, and a dropout rate of 0 3 BIBREF8   Dress is an encoder-decoder model coupled with a deep reinforcement learning framework, and the parameters are chosen according to the original paper BIBREF20   For the experiments with synthetic parallel data, we back-translate a random sample of 60 000 sentences from the collected simplified sentences into ordinary sentences  Our model is trained on synthetic data and the available parallel data, denoted as NMT+synthetic 
Results  Table 1 shows the results of all models on WikiLarge dataset  We can see that our method (NMT+synthetic) can obtain higher BLEU, lower FKGL and high SARI compared with other models, except Dress on FKGL and SBMT-SARI on SARI  It verified that including synthetic data during training is very effective, and yields an improvement over our baseline NMF by 2 11 BLEU, 1 7 FKGL and 1 07 SARI  We also substantially outperform Dress, who previously reported SOTA result  The results of our human evaluation using Simplicity are also presented in Table 1  NMT on synthetic data is significantly better than PBMT-R, Dress, and SBMT-SARI on Simplicity  It indicates that our method with simplified data is effective at creating simpler output 
Results on WikiSmall dataset are shown in Table 2  We see substantial improvements (6 37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences  Compared with statistical machine translation models (PBMT-R, Hybrid, SBMT-SARI), our method (NMT+synthetic) still have better results, but slightly worse FKGL and SARI  Similar to the results in WikiLarge, the results of our human evaluation using Simplicity outperforms the other models  In conclusion, Our method produces better results comparing with the baselines, which demonstrates the effectiveness of adding simplified training data 
Conclusion
In this paper, we propose one simple method to use simplified corpora during training of NMT systems, with no changes to the network architecture  In the experiments on two datasets, we achieve substantial gains in all tasks, and new SOTA results, via back-translation of simplified sentences into the ordinary sentences, and treating this synthetic data as additional training data  Because we do not change the neural network architecture to integrate simplified corpora, our method can be easily applied to other Neural Text Simplification (NTS) systems  We expect that the effectiveness of our method not only varies with the quality of the NTS system used for back-translation, but also depends on the amount of available parallel and simplified corpora  In the paper, we have only utilized data from Wikipedia for simplified sentences  In the future, many other text sources are available and the impact of not only size, but also of domain should be investigated 

ground_truth: English

Simple English