prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What type of evaluation is proposed for this task?
Context 1: Introduction
Multi-document summarization (MDS), the transformation of a set of documents into a short text containing their most important aspects, is a long-studied problem in NLP  Generated summaries have been shown to support humans dealing with large document collections in information seeking tasks BIBREF0 , BIBREF1 , BIBREF2   However, when exploring a set of documents manually, humans rarely write a fully-formulated summary for themselves  Instead, user studies BIBREF3 , BIBREF4 show that they note down important keywords and phrases, try to identify relationships between them and organize them accordingly  Therefore, we believe that the study of summarization with similarly structured outputs is an important extension of the traditional task 
A representation that is more in line with observed user behavior is a concept map BIBREF5 , a labeled graph showing concepts as nodes and relationships between them as edges (Figure FIGREF2 )  Introduced in 1972 as a teaching tool BIBREF6 , concept maps have found many applications in education BIBREF7 , BIBREF8 , for writing assistance BIBREF9 or to structure information repositories BIBREF10 , BIBREF11   For summarization, concept maps make it possible to represent a summary concisely and clearly reveal relationships  Moreover, we see a second interesting use case that goes beyond the capabilities of textual summaries: When concepts and relations are linked to corresponding locations in the documents they have been extracted from, the graph can be used to navigate in a document collection, similar to a table of contents  An implementation of this idea has been recently described by BIBREF12  
The corresponding task that we propose is concept-map-based MDS, the summarization of a document cluster in the form of a concept map  In order to develop and evaluate methods for the task, gold-standard corpora are necessary, but no suitable corpus is available  The manual creation of such a dataset is very time-consuming, as the annotation includes many subtasks  In particular, an annotator would need to manually identify all concepts in the documents, while only a few of them will eventually end up in the summary 
To overcome these issues, we present a corpus creation method that effectively combines automatic preprocessing, scalable crowdsourcing and high-quality expert annotations  Using it, we can avoid the high effort for single annotators, allowing us to scale to document clusters that are 15 times larger than in traditional summarization corpora  We created a new corpus of 30 topics, each with around 40 source documents on educational topics and a summarizing concept map that is the consensus of many crowdworkers (see Figure FIGREF3 ) 
As a crucial step of the corpus creation, we developed a new crowdsourcing scheme called low-context importance annotation  In contrast to traditional approaches, it allows us to determine important elements in a document cluster without requiring annotators to read all documents, making it feasible to crowdsource the task and overcome quality issues observed in previous work BIBREF13   We show that the approach creates reliable data for our focused summarization scenario and, when tested on traditional summarization corpora, creates annotations that are similar to those obtained by earlier efforts 
To summarize, we make the following contributions: (1) We propose a novel task, concept-map-based MDS (§ SECREF2 ), (2) present a new crowdsourcing scheme to create reference summaries (§ SECREF4 ), (3) publish a new dataset for the proposed task (§ SECREF5 ) and (4) provide an evaluation protocol and baseline (§ SECREF7 )  We make these resources publicly available under a permissive license 
Task
Concept-map-based MDS is defined as follows: Given a set of related documents, create a concept map that represents its most important content, satisfies a specified size limit and is connected 
We define a concept map as a labeled graph showing concepts as nodes and relationships between them as edges  Labels are arbitrary sequences of tokens taken from the documents, making the summarization task extractive  A concept can be an entity, abstract idea, event or activity, designated by its unique label  Good maps should be propositionally coherent, meaning that every relation together with the two connected concepts form a meaningful proposition 
The task is complex, consisting of several interdependent subtasks  One has to extract appropriate labels for concepts and relations and recognize different expressions that refer to the same concept across multiple documents  Further, one has to select the most important concepts and relations for the summary and finally organize them in a graph satisfying the connectedness and size constraints 
Related Work
Some attempts have been made to automatically construct concept maps from text, working with either single documents BIBREF14 , BIBREF9 , BIBREF15 , BIBREF16 or document clusters BIBREF17 , BIBREF18 , BIBREF19   These approaches extract concept and relation labels from syntactic structures and connect them to build a concept map  However, common task definitions and comparable evaluations are missing  In addition, only a few of them, namely Villalon 2012 and Valerio 2006, define summarization as their goal and try to compress the input to a substantially smaller size  Our newly proposed task and the created large-cluster dataset fill these gaps as they emphasize the summarization aspect of the task 
For the subtask of selecting summary-worthy concepts and relations, techniques developed for traditional summarization BIBREF20 and keyphrase extraction BIBREF21 are related and applicable  Approaches that build graphs of propositions to create a summary BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 seem to be particularly related, however, there is one important difference: While they use graphs as an intermediate representation from which a textual summary is then generated, the goal of the proposed task is to create a graph that is directly interpretable and useful for a user  In contrast, these intermediate graphs, e g  AMR, are hardly useful for a typical, non-linguist user 
For traditional summarization, the most well-known datasets emerged out of the DUC and TAC competitions  They provide clusters of news articles with gold-standard summaries  Extending these efforts, several more specialized corpora have been created: With regard to size, Nakano 2010 present a corpus of summaries for large-scale collections of web pages  Recently, corpora with more heterogeneous documents have been suggested, e g  BIBREF26 and BIBREF27   The corpus we present combines these aspects, as it has large clusters of heterogeneous documents, and provides a necessary benchmark to evaluate the proposed task 
For concept map generation, one corpus with human-created summary concept maps for student essays has been created BIBREF28   In contrast to our corpus, it only deals with single documents, requires a two orders of magnitude smaller amount of compression of the input and is not publicly available  
Other types of information representation that also model concepts and their relationships are knowledge bases, such as Freebase BIBREF29 , and ontologies  However, they both differ in important aspects: Whereas concept maps follow an open label paradigm and are meant to be interpretable by humans, knowledge bases and ontologies are usually more strictly typed and made to be machine-readable  Moreover, approaches to automatically construct them from text typically try to extract as much information as possible, while we want to summarize a document 
Low-Context Importance Annotation
Lloret 2013 describe several experiments to crowdsource reference summaries  Workers are asked to read 10 documents and then select 10 summary sentences from them for a reward of $0 05  They discovered several challenges, including poor work quality and the subjectiveness of the annotation task, indicating that crowdsourcing is not useful for this purpose 
To overcome these issues, we introduce a new task design, low-context importance annotation, to determine summary-worthy parts of documents  Compared to Lloret et al 's approach, it is more in line with crowdsourcing best practices, as the tasks are simple, intuitive and small BIBREF30 and workers receive reasonable payment BIBREF31   Most importantly, it is also much more efficient and scalable, as it does not require workers to read all documents in a cluster 
Task Design
We break down the task of importance annotation to the level of single propositions  The goal of our crowdsourcing scheme is to obtain a score for each proposition indicating its importance in a document cluster, such that a ranking according to the score would reveal what is most important and should be included in a summary  In contrast to other work, we do not show the documents to the workers at all, but provide only a description of the document cluster's topic along with the propositions  This ensures that tasks are small, simple and can be done quickly (see Figure FIGREF4 ) 
In preliminary tests, we found that this design, despite the minimal context, works reasonably on our focused clusters on common educational topics  For instance, consider Figure FIGREF4 : One can easily say that P1 is more important than P2 without reading the documents 
We distinguish two task variants:
Instead of enforcing binary importance decisions, we use a 5-point Likert-scale to allow more fine-grained annotations  The obtained labels are translated into scores (5  1) and the average of all scores for a proposition is used as an estimate for its importance  This follows the idea that while single workers might find the task subjective, the consensus of multiple workers, represented in the average score, tends to be less subjective due to the “wisdom of the crowd”  We randomly group five propositions into a task 
As an alternative, we use a second task design based on pairwise comparisons  Comparisons are known to be easier to make and more consistent BIBREF32 , but also more expensive, as the number of pairs grows quadratically with the number of objects  To reduce the cost, we group five propositions into a task and ask workers to order them by importance per drag-and-drop  From the results, we derive pairwise comparisons and use TrueSkill BIBREF35 , a powerful Bayesian rank induction model BIBREF34 , to obtain importance estimates for each proposition 
Pilot Study
To verify the proposed approach, we conducted a pilot study on Amazon Mechanical Turk using data from TAC2008 BIBREF36   We collected importance estimates for 474 propositions extracted from the first three clusters using both task designs  Each Likert-scale task was assigned to 5 different workers and awarded $0 06  For comparison tasks, we also collected 5 labels each, paid $0 05 and sampled around 7% of all possible pairs  We submitted them in batches of 100 pairs and selected pairs for subsequent batches based on the confidence of the TrueSkill model 
Following the observations of Lloret 2013, we established several measures for quality control  First, we restricted our tasks to workers from the US with an approval rate of at least 95%  Second, we identified low quality workers by measuring the correlation of each worker's Likert-scores with the average of the other four scores  The worst workers (at most 5% of all labels) were removed 
In addition, we included trap sentences, similar as in BIBREF13 , in around 80 of the tasks  In contrast to Lloret et al 's findings, both an obvious trap sentence (This sentence is not important) and a less obvious but unimportant one (Barack Obama graduated from Harvard Law) were consistently labeled as unimportant (1 08 and 1 14), indicating that the workers did the task properly 
For Likert-scale tasks, we follow Snow 2008 and calculate agreement as the average Pearson correlation of a worker's Likert-score with the average score of the remaining workers  This measure is less strict than exact label agreement and can account for close labels and high- or low-scoring workers  We observe a correlation of 0 81, indicating substantial agreement  For comparisons, the majority agreement is 0 73  To further examine the reliability of the collected data, we followed the approach of Kiritchenko 2016 and simply repeated the crowdsourcing for one of the three topics  Between the importance estimates calculated from the first and second run, we found a Pearson correlation of 0 82 (Spearman 0 78) for Likert-scale tasks and 0 69 (Spearman 0 66) for comparison tasks  This shows that the approach, despite the subjectiveness of the task, allows us to collect reliable annotations 
In addition to the reliability studies, we extrinsically evaluated the annotations in the task of summary evaluation  For each of the 58 peer summaries in TAC2008, we calculated a score as the sum of the importance estimates of the propositions it contains  Table TABREF13 shows how these peer scores, averaged over the three topics, correlate with the manual responsiveness scores assigned during TAC in comparison to ROUGE-2 and Pyramid scores  The results demonstrate that with both task designs, we obtain importance annotations that are similarly useful for summary evaluation as pyramid annotations or gold-standard summaries (used for ROUGE) 
Based on the pilot study, we conclude that the proposed crowdsourcing scheme allows us to obtain proper importance annotations for propositions  As workers are not required to read all documents, the annotation is much more efficient and scalable as with traditional methods 
Corpus Creation
This section presents the corpus construction process, as outlined in Figure FIGREF16 , combining automatic preprocessing, scalable crowdsourcing and high-quality expert annotations to be able to scale to the size of our document clusters  For every topic, we spent about $150 on crowdsourcing and 1 5h of expert annotations, while just a single annotator would already need over 8 hours (at 200 words per minute) to read all documents of a topic 
Source Data
As a starting point, we used the DIP corpus BIBREF37 , a collection of 49 clusters of 100 web pages on educational topics (e g  bullying, homeschooling, drugs) with a short description of each topic  It was created from a large web crawl using state-of-the-art information retrieval  We selected 30 of the topics for which we created the necessary concept map annotations 
Proposition Extraction
As concept maps consist of propositions expressing the relation between concepts (see Figure FIGREF2 ), we need to impose such a structure upon the plain text in the document clusters  This could be done by manually annotating spans representing concepts and relations, however, the size of our clusters makes this a huge effort: 2288 sentences per topic (69k in total) need to be processed  Therefore, we resort to an automatic approach 
The Open Information Extraction paradigm BIBREF38 offers a representation very similar to the desired one  For instance, from
Students with bad credit history should not lose hope and apply for federal loans with the FAFSA 
Open IE systems extract tuples of two arguments and a relation phrase representing propositions:
(s  with bad credit history, should not lose, hope)
(s  with bad credit history, apply for, federal loans with the FAFSA)
While the relation phrase is similar to a relation in a concept map, many arguments in these tuples represent useful concepts  We used Open IE 4, a state-of-the-art system BIBREF39 to process all sentences  After removing duplicates, we obtained 4137 tuples per topic 
Since we want to create a gold-standard corpus, we have to ensure that we produce high-quality data  We therefore made use of the confidence assigned to every extracted tuple to filter out low quality ones  To ensure that we do not filter too aggressively (and miss important aspects in the final summary), we manually annotated 500 tuples sampled from all topics for correctness  On the first 250 of them, we tuned the filter threshold to 0 5, which keeps 98 7% of the correct extractions in the unseen second half  After filtering, a topic had on average 2850 propositions (85k in total) 
Proposition Filtering
Despite the similarity of the Open IE paradigm, not every extracted tuple is a suitable proposition for a concept map  To reduce the effort in the subsequent steps, we therefore want to filter out unsuitable ones  A tuple is suitable if it (1) is a correct extraction, (2) is meaningful without any context and (3) has arguments that represent proper concepts  We created a guideline explaining when to label a tuple as suitable for a concept map and performed a small annotation study  Three annotators independently labeled 500 randomly sampled tuples  The agreement was 82% ( INLINEFORM0 )  We found tuples to be unsuitable mostly because they had unresolvable pronouns, conflicting with (2), or arguments that were full clauses or propositions, conflicting with (3), while (1) was mostly taken care of by the confidence filtering in § SECREF21  
Due to the high number of tuples we decided to automate the filtering step  We trained a linear SVM on the majority voted annotations  As features, we used the extraction confidence, length of arguments and relations as well as part-of-speech tags, among others  To ensure that the automatic classification does not remove suitable propositions, we tuned the classifier to avoid false negatives  In particular, we introduced class weights, improving precision on the negative class at the cost of a higher fraction of positive classifications  Additionally, we manually verified a certain number of the most uncertain negative classifications to further improve performance  When 20% of the classifications are manually verified and corrected, we found that our model trained on 350 labeled instances achieves 93% precision on negative classifications on the unseen 150 instances  We found this to be a reasonable trade-off of automation and data quality and applied the model to the full dataset 
The classifier filtered out 43% of the propositions, leaving 1622 per topic  We manually examined the 17k least confident negative classifications and corrected 955 of them  We also corrected positive classifications for certain types of tuples for which we knew the classifier to be imprecise  Finally, each topic was left with an average of 1554 propositions (47k in total) 
Importance Annotation
Given the propositions identified in the previous step, we now applied our crowdsourcing scheme as described in § SECREF4 to determine their importance  To cope with the large number of propositions, we combine the two task designs: First, we collect Likert-scores from 5 workers for each proposition, clean the data and calculate average scores  Then, using only the top 100 propositions according to these scores, we crowdsource 10% of all possible pairwise comparisons among them  Using TrueSkill, we obtain a fine-grained ranking of the 100 most important propositions 
For Likert-scores, the average agreement over all topics is 0 80, while the majority agreement for comparisons is 0 78  We repeated the data collection for three randomly selected topics and found the Pearson correlation between both runs to be 0 73 (Spearman 0 73) for Likert-scores and 0 72 (Spearman 0 71) for comparisons  These figures show that the crowdsourcing approach works on this dataset as reliably as on the TAC documents 
In total, we uploaded 53k scoring and 12k comparison tasks to Mechanical Turk, spending $4425 45 including fees  From the fine-grained ranking of the 100 most important propositions, we select the top 50 per topic to construct a summary concept map in the subsequent steps 
Proposition Revision
Having a manageable number of propositions, an annotator then applied a few straightforward transformations that correct common errors of the Open IE system  First, we break down propositions with conjunctions in either of the arguments into separate propositions per conjunct, which the Open IE system sometimes fails to do  And second, we correct span errors that might occur in the argument or relation phrases, especially when sentences were not properly segmented  As a result, we have a set of high quality propositions for our concept map, consisting of, due to the first transformation, 56 1 propositions per topic on average 
Concept Map Construction
In this final step, we connect the set of important propositions to form a graph  For instance, given the following two propositions
(student, may borrow, Stafford Loan)
(the student, does not have, a credit history)
one can easily see, although the first arguments differ slightly, that both labels describe the concept student, allowing us to build a concept map with the concepts student, Stafford Loan and credit history  The annotation task thus involves deciding which of the available propositions to include in the map, which of their concepts to merge and, when merging, which of the available labels to use  As these decisions highly depend upon each other and require context, we decided to use expert annotators rather than crowdsource the subtasks 
Annotators were given the topic description and the most important, ranked propositions  Using a simple annotation tool providing a visualization of the graph, they could connect the propositions step by step  They were instructed to reach a size of 25 concepts, the recommended maximum size for a concept map BIBREF6   Further, they should prefer more important propositions and ensure connectedness  When connecting two propositions, they were asked to keep the concept label that was appropriate for both propositions  To support the annotators, the tool used ADW BIBREF40 , a state-of-the-art approach for semantic similarity, to suggest possible connections  The annotation was carried out by graduate students with a background in NLP after receiving an introduction into the guidelines and tool and annotating a first example 
If an annotator was not able to connect 25 concepts, she was allowed to create up to three synthetic relations with freely defined labels, making the maps slightly abstractive  On average, the constructed maps have 0 77 synthetic relations, mostly connecting concepts whose relation is too obvious to be explicitly stated in text (e g  between Montessori teacher and Montessori education) 
To assess the reliability of this annotation step, we had the first three maps created by two annotators  We casted the task of selecting propositions to be included in the map as a binary decision task and observed an agreement of 84% ( INLINEFORM0 )  Second, we modeled the decision which concepts to join as a binary decision on all pairs of common concepts, observing an agreement of 95% ( INLINEFORM1 )  And finally, we compared which concept labels the annotators decided to include in the final map, observing 85% ( INLINEFORM2 ) agreement  Hence, the annotation shows substantial agreement BIBREF41  
Corpus Analysis
In this section, we describe our newly created corpus, which, in addition to having summaries in the form of concept maps, differs from traditional summarization corpora in several aspects 
Document Clusters
The corpus consists of document clusters for 30 different topics  Each of them contains around 40 documents with on average 2413 tokens, which leads to an average cluster size of 97,880 token  With these characteristics, the document clusters are 15 times larger than typical DUC clusters of ten documents and five times larger than the 25-document-clusters (Table TABREF26 )  In addition, the documents are also more variable in terms of length, as the (length-adjusted) standard deviation is twice as high as in the other corpora  With these properties, the corpus represents an interesting challenge towards real-world application scenarios, in which users typically have to deal with much more than ten documents 
Because we used a large web crawl as the source for our corpus, it contains documents from a variety of genres  To further analyze this property, we categorized a sample of 50 documents from the corpus  Among them, we found professionally written articles and blog posts (28%), educational material for parents and kids (26%), personal blog posts (16%), forum discussions and comments (12%), commented link collections (12%) and scientific articles (6%) 
In addition to the variety of genres, the documents also differ in terms of language use  To capture this property, we follow Zopf 2016 and compute, for every topic, the average Jensen-Shannon divergence between the word distribution of one document and the word distribution in the remaining documents  The higher this value is, the more the language differs between documents  We found the average divergence over all topics to be 0 3490, whereas it is 0 3019 in DUC 2004 and 0 3188 in TAC 2008A 
Concept Maps
As Table TABREF33 shows, each of the 30 reference concept maps has exactly 25 concepts and between 24 and 28 relations  Labels for both concepts and relations consist on average of 3 2 tokens, whereas the latter are a bit shorter in characters 
To obtain a better picture of what kind of text spans have been used as labels, we automatically tagged them with their part-of-speech and determined their head with a dependency parser  Concept labels tend to be headed by nouns (82%) or verbs (15%), while they also contain adjectives, prepositions and determiners  Relation labels, on the other hand, are almost always headed by a verb (94%) and contain prepositions, nouns and particles in addition  These distributions are very similar to those reported by Villalon 2010 for their (single-document) concept map corpus 
Analyzing the graph structure of the maps, we found that all of them are connected  They have on average 7 2 central concepts with more than one relation, while the remaining ones occur in only one proposition  We found that achieving a higher number of connections would mean compromising importance, i e  including less important propositions, and decided against it 
Baseline Experiments
In this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus 
Conclusion
In this work, we presented low-context importance annotation, a novel crowdsourcing scheme that we used to create a new benchmark corpus for concept-map-based MDS  The corpus has large-scale document clusters of heterogeneous web documents, posing a challenging summarization task  Together with the corpus, we provide implementations of a baseline method and evaluation scripts and hope that our efforts facilitate future research on this variant of summarization 
Acknowledgments
We would like to thank Teresa Botschen, Andreas Hanselowski and Markus Zopf for their help with the annotation work and Christian Meyer for his valuable feedback  This work has been supported by the German Research Foundation as part of the Research Training Group “Adaptive Preparation of Information from Heterogeneous Sources” (AIPHES) under grant No  GRK 1994/1 
Context 3: Introduction
To appear in Proceedings of International Workshop on Health Intelligence (W3PHIAI) of the 34th AAAI Conference on Artificial Intelligence, 2020 
Physician burnout is a growing concern, estimated to be experienced by at least 35% of physicians in the developing world and 50% in the United States BIBREF0  BIBREF1 found that for every hour physicians provide direct clinical facetime to patients, nearly two additional hours are spent on EHR (Electronic Health Records) and administrative or desk work  As per the study conducted by Massachusetts General Physicians Organization (MPGO) BIBREF2 and as reported by BIBREF3, the average time spent on administrative tasks increased from 23 7% in 2014 to 27 9% in 2017  Both the surveys found that time spent on administrative tasks was positively associated with higher likelihood of burnout  Top reasons under administrative burden include working on the ambulatory EHR, handling medication reconciliation (sometimes done by aids), medication renewals, and medical billing and coding  The majority of these reasons revolve around documentation of information exchanged between doctors and patients during the clinical encounters  Automatically extracting such clinical information BIBREF4, BIBREF5 can not only help alleviate the documentation burden on the physician, but also allow them to dedicate more time directly with patients 
Among all the clinical information extraction tasks, Medication Regimen (Medication, dosage, and frequency) extraction is particularly interesting due to its ability to help doctors with medication orders cum renewals, medication reconciliation, potentially verifying the reconciliations for errors, and, other medication-centered EHR documentation tasks  In addition, the same information when provided to patients can help them with better recall of doctor instructions which might aid in compliance with the care plan  This is particularly important given that patients forget or wrongly recollect 40-80% BIBREF6 of what is discussed in the clinic, and accessing EHR data has its own challenges 
Spontaneous clinical conversations happening between a doctor and a patient, have several distinguishing characteristics from a normal monologue or prepared speech: it involves multiple speakers with overlapping dialogues, covers a variety of speech patterns, and the vocabulary can range from colloquial to complex domain-specific language  With recent advancements in Conversational Speech Recognition BIBREF7 rendering the systems less prone to errors, the subsequent challenge of understanding and extracting relevant information from the conversations is receiving increasing research focus BIBREF4, BIBREF8 
In this paper, we focus on local information extraction in transcribed clinical conversations  Specifically, we extract dosage (e g  5mg) and frequency (e g  once a day) for the medications (e g  aspirin) from these transcripts, collectively referred to as Medication Regimen (MR) extraction  The information extraction is local as we extract the information from a segment of the transcript and not the entire transcript since doing the latter is difficult owing to the long meandering nature of the conversations often with multiple medication regimens and care plans being discussed 
The challenges associated with the Medication Regimen (MR) extraction task include understanding the spontaneous dialog with clinical vocabulary and understanding the relationship between different entities as the discussion can contain multiple medications and dosages (e g  doctor revising a dosage or reviewing all the current medications) 
We frame this problem as a Question Answering (QA) task by generating questions using templates  We base the QA model on pointer-generator networks BIBREF9 augmented with Co-Attentions BIBREF10  In addition, we develop models combining QA and Information Extraction frameworks using multi-decoder (one each for dosage and frequency) architecture 
Lack of availability of a large volume of data is a typical challenge in healthcare  A conversation corpus by itself is a rare commodity in the healthcare data space because of the cost and difficulty in handing (because of data privacy concerns)  Moreover, transcribing and labeling the conversations is a costly process as it requires domain-specific medical annotation expertise  To address data shortage and improve the model performance, we investigate different high-performance contextual embeddings (ELMO BIBREF11, BERT BIBREF12 and ClinicalBERT BIBREF13), and pretrain the models on a clinical summarization task  We further investigate the effects of training data size on our models 
On the MR extraction task, ELMo with encoder multi-decoder architecture and BERT with encoder-decoder with encoders pretrained on the summarization task perform the best  The best-performing models improve our baseline's dosage and frequency extractions ROUGE-1 F1 scores from 54 28 and 37 13 to 89 57 and 45 94, respectively 
Using our models, we present the first fully automated system to extract MR tags from spontaneous doctor-patient conversations  We evaluate the system (using our best performing models) on the transcripts generated from Automatic Speech Recognition (ASR) APIs offered by Google and IBM  In Google ASR's transcripts, our best model obtained ROUGE-1 F1 of 71 75 for Dosage extraction (which in this specific case equals to the percentage of times dosage is correct, refer Metrics Section for more details) and 40 13 for Frequency extraction tasks  On qualitative evaluation, we find that for 73 58% of the medications the model can find the correct frequency  These results demonstrate that the research on NLP can be used effectively in a real clinical setting to benefit both doctors and patients
Data
Our dataset consists of a total of 6,693 real doctor-patient conversations recorded in a clinical setting using distant microphones of varying quality  The recordings have an average duration of 9min 28s and have a verbatim transcript of 1,500 words on average (written by the experts)  Both the audio and the transcript are de-identified (by removing the identifying information) with digital zeros and [de-identified] tags, respectively  The sentences in the transcript are grounded to the audio with the timestamps of its first and last word 
The transcript of the conversations are annotated with summaries and Medication Regimen tags (MR tags), both grounded using the timestamps of the sentences from the transcript deemed relevant by the expert annotators, refer to Table TABREF1  The transcript for a typical conversation can be quite long, and not easy for many of the high performing deep learning models to act on  Moreover, the medical information about a concept/condition/entity can change during the conversation after a significant time gap  For example, dosage of a medication can be different when discussing current medication the patient is on vs when they are prescribed a different dosage  Hence, we have annotations, that are grounded to a short segment of the transcript 
The summaries (#words - $\mu = 9 7; \sigma = 10 1$) are medically relevant and local  The MR tags are also local and are of the form {Medication Name, Dosage, Frequency}  If dosage ($\mu = 2 0; \sigma = 0$) or frequency ($\mu = 2 1; \sigma = 1 07$) information for a medication is not present in a grounded sentence, the corresponding field in the MR tag will be marked as `none' 
In the MR tags, Medication Name and Dosage (usually a quantity followed by its units) can be relatively easily extracted from the transcript except for the units of the dosage which is sometimes inferred  In contrast, due to high degree of linguistic variation with which Frequency is often expressed, extracting Frequency requires an additional inference step  For example, `take one in the morning and at noon' from the transcript is tagged as `twice a day' in the frequency tag, likewise `take it before sleeping' is tagged as `at night time' 
Out of overall 6,693 files, we set aside a random sample of 423 files (denoted as $\mathcal {D}_{test}$) for final evaluation  The remaining 6,270 files are used for training with 80% train (5016), 10% validation (627), and 10% test (627) split  Overall, the 6,270 files contains 156,186 summaries and 32,000 MR tags out of which 8,654 MR tags contain values for at least one of the Dosage or Frequency, which we used for training to avoid overfitting (the remaining MR tags have both Dosage and Frequency as `none')  Note that we have two test datasets: `10% test' - used to evaluate all the models, and $\mathcal {D}_{test}$ - used to measure the performance of best performing models on ASR transcripts 
Approach
We frame the Medication Regimen extraction problem as a Question Answering (QA) task, which forms the basis for our first approach

ground_truth: Answer with content missing: (Evaluation Metrics section) Precision, Recall, F1-scores, Strict match, METEOR, ROUGE-2

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What models are used in the experiment?
Context 2: Introduction
Offensive content has become pervasive in social media and a reason of concern for government organizations, online communities, and social media platforms  One of the most common strategies to tackle the problem is to train systems capable of recognizing offensive content, which then can be deleted or set aside for human moderation  In the last few years, there have been several studies published on the application of computational methods to deal with this problem  Most prior work focuses on a different aspect of offensive language such as abusive language BIBREF0 , BIBREF1 , (cyber-)aggression BIBREF2 , (cyber-)bullying BIBREF3 , BIBREF4 , toxic comments INLINEFORM0 , hate speech BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , and offensive language BIBREF11   Prior work has focused on these aspects of offensive language in Twitter BIBREF3 , BIBREF7 , BIBREF8 , BIBREF11 , Wikipedia comments, and Facebook posts BIBREF2  
Recently, Waseem et  al  ( BIBREF12 ) acknowledged the similarities among prior work and discussed the need for a typology that differentiates between whether the (abusive) language is directed towards a specific individual or entity or towards a generalized group and whether the abusive content is explicit or implicit  Wiegand et al  ( BIBREF11 ) followed this trend as well on German tweets  In their evaluation, they have a task to detect offensive vs not offensive tweets and a second task for distinguishing between the offensive tweets as profanity, insult, or abuse  However, no prior work has explored the target of the offensive language, which is important in many scenarios, e g , when studying hate speech with respect to a specific target 
Therefore, we expand on these ideas by proposing a a hierarchical three-level annotation model that encompasses:
Using this annotation model, we create a new large publicly available dataset of English tweets  The key contributions of this paper are as follows:
Related Work
Different abusive and offense language identification sub-tasks have been explored in the past few years including aggression identification, bullying detection, hate speech, toxic comments, and offensive language 
Aggression identification: The TRAC shared task on Aggression Identification BIBREF2 provided participants with a dataset containing 15,000 annotated Facebook posts and comments in English and Hindi for training and validation  For testing, two different sets, one from Facebook and one from Twitter were provided  Systems were trained to discriminate between three classes: non-aggressive, covertly aggressive, and overtly aggressive 
Bullying detection: Several studies have been published on bullying detection  One of them is the one by xu2012learning which apply sentiment analysis to detect bullying in tweets  xu2012learning use topic models to to identify relevant topics in bullying  Another related study is the one by dadvar2013improving which use user-related features such as the frequency of profanity in previous messages to improve bullying detection 
Hate speech identification: It is perhaps the most widespread abusive language detection sub-task  There have been several studies published on this sub-task such as kwok2013locate and djuric2015hate who build a binary classifier to distinguish between `clean' comments and comments containing hate speech and profanity  More recently, Davidson et al  davidson2017automated presented the hate speech detection dataset containing over 24,000 English tweets labeled as non offensive, hate speech, and profanity 
Offensive language: The GermEval BIBREF11 shared task focused on Offensive language identification in German tweets  A dataset of over 8,500 annotated tweets was provided for a course-grained binary classification task in which systems were trained to discriminate between offensive and non-offensive tweets and a second task where the organizers broke down the offensive class into three classes: profanity, insult, and abuse 
Toxic comments: The Toxic Comment Classification Challenge was an open competition at Kaggle which provided participants with comments from Wikipedia labeled in six classes: toxic, severe toxic, obscene, threat, insult, identity hate 
While each of these sub-tasks tackle a particular type of abuse or offense, they share similar properties and the hierarchical annotation model proposed in this paper aims to capture this  Considering that, for example, an insult targeted at an individual is commonly known as cyberbulling and that insults targeted at a group are known as hate speech, we pose that OLID's hierarchical annotation model makes it a useful resource for various offensive language identification sub-tasks 
Hierarchically Modelling Offensive Content
In the OLID dataset, we use a hierarchical annotation model split into three levels to distinguish between whether language is offensive or not (A), and type (B) and target (C) of the offensive language  Each level is described in more detail in the following subsections and examples are shown in Table TABREF10  
Level A: Offensive language Detection
Level A discriminates between offensive (OFF) and non-offensive (NOT) tweets 
Not Offensive (NOT): Posts that do not contain offense or profanity;
Offensive (OFF): We label a post as offensive if it contains any form of non-acceptable language (profanity) or a targeted offense, which can be veiled or direct  This category includes insults, threats, and posts containing profane language or swear words 
Level B: Categorization of Offensive Language
Level B categorizes the type of offense and two labels are used: targeted (TIN) and untargeted (INT) insults and threats 
Targeted Insult (TIN): Posts which contain an insult/threat to an individual, group, or others (see next layer);
Untargeted (UNT): Posts containing non-targeted profanity and swearing  Posts with general profanity are not targeted, but they contain non-acceptable language 
Level C: Offensive Language Target Identification
Level C categorizes the targets of insults and threats as individual (IND), group (GRP), and other (OTH) 
Individual (IND): Posts targeting an individual  It can be a a famous person, a named individual or an unnamed participant in the conversation  Insults and threats targeted at individuals are often defined as cyberbulling 
Group (GRP): The target of these offensive posts is a group of people considered as a unity due to the same ethnicity, gender or sexual orientation, political affiliation, religious belief, or other common characteristic  Many of the insults and threats targeted at a group correspond to what is commonly understood as hate speech 
Other (OTH): The target of these offensive posts does not belong to any of the previous two categories (e g  an organization, a situation, an event, or an issue) 
Data Collection
The data included in OLID has been collected from Twitter  We retrieved the data using the Twitter API by searching for keywords and constructions that are often included in offensive messages, such as `she is' or `to:BreitBartNews'  We carried out a first round of trial annotation of 300 instances with six experts  The goal of the trial annotation was to 1) evaluate the proposed tagset; 2) evaluate the data retrieval method; and 3) create a gold standard with instances that could be used as test questions in the training and test setting annotation which was carried out using crowdsourcing  The breakdown of keywords and their offensive content in the trial data of 300 tweets is shown in Table TABREF14   We included a left (@NewYorker) and far-right (@BreitBartNews) news accounts because there tends to be political offense in the comments  One of the best offensive keywords was tweets that were flagged as not being safe by the Twitter `safe' filter (the `-' indicates `not safe')  The vast majority of content on Twitter is not offensive so we tried different strategies to keep a reasonable number of tweets in the offensive class amounting to around 30% of the dataset including excluding some keywords that were not high in offensive content such as `they are` and `to:NewYorker`  Although `he is' is lower in offensive content we kept it as a keyword to avoid gender bias  In addition to the keywords in the trial set, we searched for more political keywords which tend to be higher in offensive content, and sampled our dataset such that 50% of the the tweets come from political keywords and 50% come from non-political keywords  In addition to the keywords `gun control', and `to:BreitbartNews', political keywords used to collect these tweets are `MAGA', `antifa', `conservative' and `liberal'  We computed Fliess' INLINEFORM0 on the trial set for the five annotators on 21 of the tweets  INLINEFORM1 is  83 for Layer A (OFF vs NOT) indicating high agreement  As to normalization and anonymization, no user metadata or Twitter IDs have been stored, and URLs and Twitter mentions have been substituted to placeholders  We follow prior work in related areas (burnap2015cyber,davidson2017automated) and annotate our data using crowdsourcing using the platform Figure Eight  We ensure data quality by: 1) we only received annotations from individuals who were experienced in the platform; and 2) we used test questions to discard annotations of individuals who did not reach a certain threshold  Each instance in the dataset was annotated by multiple annotators and inter-annotator agreement has been calculated  We first acquired two annotations for each instance  In case of 100% agreement, we considered these as acceptable annotations, and in case of disagreement, we requested more annotations until the agreement was above 66%  After the crowdsourcing annotation, we used expert adjudication to guarantee the quality of the annotation  The breakdown of the data into training and testing for the labels from each level is shown in Table TABREF15  
Experiments and Evaluation
We assess our dataset using traditional and deep learning methods  Our simplest model is a linear SVM trained on word unigrams  SVMs have produced state-of-the-art results for many text classification tasks BIBREF13   We also train a bidirectional Long Short-Term-Memory (BiLSTM) model, which we adapted from the sentiment analysis system of sentimentSystem,rasooli2018cross and altered to predict offensive labels instead  It consists of (1) an input embedding layer, (2) a bidirectional LSTM layer, (3) an average pooling layer of input features  The concatenation of the LSTM's and average pool layer is passed through a dense layer and the output is passed through a softmax function  We set two input channels for the input embedding layers: pre-trained FastText embeddings BIBREF14 , as well as updatable embeddings learned by the model during training  Finally, we also apply a Convolutional Neural Network (CNN) model based on the architecture of BIBREF15 , using the same multi-channel inputs as the above BiLSTM 
Our models are trained on the training data, and evaluated by predicting the labels for the held-out test set  The distribution is described in Table TABREF15   We evaluate and compare the models using the macro-averaged F1-score as the label distribution is highly imbalanced  Per-class Precision (P), Recall (R), and F1-score (F1), also with other averaged metrics are also reported  The models are compared against baselines of predicting all labels as the majority or minority classes 
Offensive Language Detection
The performance on discriminating between offensive (OFF) and non-offensive (NOT) posts is reported in Table TABREF18   We can see that all systems perform significantly better than chance, with the neural models being substantially better than the SVM  The CNN outperforms the RNN model, achieving a macro-F1 score of 0 80 
Categorization of Offensive Language
In this experiment, the two systems were trained to discriminate between insults and threats (TIN) and untargeted (UNT) offenses, which generally refer to profanity  The results are shown in Table TABREF19  
The CNN system achieved higher performance in this experiment compared to the BiLSTM, with a macro-F1 score of 0 69  All systems performed better at identifying target and threats (TIN) than untargeted offenses (UNT) 
Offensive Language Target Identification
The results of the offensive target identification experiment are reported in Table TABREF20   Here the systems were trained to distinguish between three targets: a group (GRP), an individual (IND), or others (OTH)  All three models achieved similar results far surpassing the random baselines, with a slight performance edge for the neural models 
The performance of all systems for the OTH class is 0  This poor performances can be explained by two main factors  First, unlike the two other classes, OTH is a heterogeneous collection of targets  It includes offensive tweets targeted at organizations, situations, events, etc  making it more challenging for systems to learn discriminative properties of this class  Second, this class contains fewer training instances than the other two  There are only 395 instances in OTH, and 1,075 in GRP, and 2,407 in IND 
Conclusion and Future Work
This paper presents OLID, a new dataset with annotation of type and target of offensive language  OLID is the official dataset of the shared task SemEval 2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) BIBREF16   In OffensEval, each annotation level in OLID is an independent sub-task  The dataset contains 14,100 tweets and is released freely to the research community  To the best of our knowledge, this is the first dataset to contain annotation of type and target of offenses in social media and it opens several new avenues for research in this area  We present baseline experiments using SVMs and neural networks to identify the offensive tweets, discriminate between insults, threats, and profanity, and finally to identify the target of the offensive messages  The results show that this is a challenging task  A CNN-based sentence classifier achieved the best results in all three sub-tasks 
In future work, we would like to make a cross-corpus comparison of OLID and datasets annotated for similar tasks such as aggression identification BIBREF2 and hate speech detection BIBREF8   This comparison is, however, far from trivial as the annotation of OLID is different 
Acknowledgments
The research presented in this paper was partially supported by an ERAS fellowship awarded by the University of Wolverhampton 
Context 3:  It can also be considered as a specific inference or relation extract task, since we extract specific information about an entity (Medication Name), hence our second approach is at the intersection of Question Answering (QA) and Information Extraction (IE) domains  Both the approaches involve using a contiguous segment of the transcript and the Medication Name as input, to find/infer the medication's Dosage and Frequency  When testing the approaches mimicking real-world conditions, we extract Medication Name from the transcript separately using ontology, refer to SECREF19 
In the first approach, we frame the MR task as a QA task and generate questions using the template: “What is the $ <$dosage/frequency$>$ for $<$Medication Name$>$"  Here, we use an abstractive QA model based on pointer-generator networks BIBREF9 augmented with coattention encoder BIBREF10 (QA-PGNet) 
In the second approach, we frame the problem as a conditioned IE task, where the information extracted depends on an entity (Medication Name)  Here, we use a multi-decoder pointer-generator network augmented with coattention encoder (Multi-decoder QA-PGNet)  Instead of using templates to generate questions and using a single decoder to extract different types of information as in the QA approach (which might lead to performance degradation), here we consider separate decoders for extracting specific types of information about an entity $E$ (Medication Name) 
Approach ::: Pointer-generator Network (PGNet)
The network is a sequence-to-sequence attention model that can both copy a word from the input $I$ containing $P$ word tokens or generate a word from its vocabulary $vocab$, to produce the output sequence 
First, the tokens of the $I$ are converted to embeddings and are fed one-by-one to the encoder, a single bi-LSTM layer, which encodes the tokens in $I$ into a sequence of hidden states - $H=encoder(I)$, where $ H=[h_1   h_P]$ 
For each decoder time step $t$, in a loop, we compute, 1) attention $a_t$ (using the last decoder state $s_{t-1}$), over the input tokens $I$, and 2) the decoder state $s_t$ using $a_t$  Then, at each time step, using both $a_t$ and $s_t$ we can find the probability $P_t(w)$, of producing a word $w$ (from both $vocab$ and $I$)  For convenience, we denote the attention and the decoder as $decoder_{pg}(H)=P(w)$, where $P(w)=[P_1(w)   P_T(w)]$  The output can then be decoded from $P(w)$, which is decoded until it produces an `end of output token' or the number of steps reach the maximum allowed limit 
Approach ::: QA PGNet
We first encode both the question - $H_Q = encoder(Q)$, and the input - $H_I = encoder(I)$, separately using encoders (with shared weights)  Then, to condition $I$ on $Q$ (and vice versa), we use the coattention encoder BIBREF10 which attends to both the $I$ and $Q$ simultaneously to generates the coattention context - $C_D = coatt(H_I, H_Q)$  Finally, using the pointer-generator decoder we find the probability distribution of the output sequence - $P(w) = decoder_{pg}([H_I; C_D])$, which is then decoded to generate the answer sequence 
Approach ::: Multi-decoder (MD) QA PGNet
After encoding the inputs into $H_I$ and $H_E$, for extracting $K$ types of information about an entity in an IE fashion, we use the following multi-decoder (MD) setup:
Predictions for each of the $K$ decoders are then decoded using $P^k(w)$ 
All the networks discussed above are trained using a negative log-likelihood loss for the target word at each time step and summed over all the decoder time steps 
Experiments
We initialized MR extraction models' vocabulary from the training dataset after removing words with a frequency lower than 30 in the dataset, resulting in 456 words  Our vocabulary is small because of the size of the dataset, hence we rely on the model's ability to copy words to produce the output effectively  In all our model variations, the embedding and the network's hidden dimension are set to be equal  The networks were trained with a learning rate of 0 0015, dropout of 0 5 on the embedding layer, normal gradient clipping set at 2, batch size of 8, and optimized with Adagrad BIBREF15 and the training was stopped using the $10\%$ validation dataset 
Experiments ::: Data Processing
We did the following basic preprocessing to our data, 1) added `none' to the beginning of the input utterance, so the network could point to it when there was no relevant information in the input, 2) filtered outliers with a large number of grounded transcript sentences ($>$150 words), and 3) converted all text to lower case 
To improve performance, we 1) standardized all numbers (both digits and words) to words concatenated with a hyphen (e g  110 -$>$ one-hundred-ten), in both input and output, 2) removed units from Dosage as sometimes the units were not explicitly mentioned in the transcript segment but were written by the annotators using domain knowledge, 3) prepended all medication mentions with `rx-' tag, as this helps model's performance when multiple medications are discussed in a segment (in both input and output), and 4) when a transcript segment has multiple medications or dosages being discussed we randomly shuffle them (in both input and output) and create a new data point, to increases the number of training data points  Randomly shuffling the entities increases the number of training MR tags from 8,654 to 11,521  Based on the data statistics after data processing, we fixed the maximum encoder steps to 100, dosage decoder steps to 1, and frequency decoder steps to 3 (for both the QA and Multi-decoder QA models) 
Experiments ::: Metrics
For the MR extraction task, we measure the ROUGE-1 scores BIBREF14 for both the Dosage and Frequency extraction tasks  It should be noted that since Dosage is a single word token (after processing), both the reference and hypothesis are a single token, making its ROUGE-1 F1, Precision and Recall scores equal and be equal to percentage of times we find the correct dosage for the medications 
In our annotations, Frequency has conflicting tags (e g  {`Once a day', `twice a day'} and `daily'), hence metrics like Exact Match will be erroneous  To address this issue, we use the ROUGE scores to compare different models on the 10% test dataset and we use qualitative evaluation to measure the top-performing models on $\mathcal {D}_{test}$ 
Experiments ::: Model variations
We consider QA PGNet and Multi-decoder QA PGNet with lookup table embedding as baseline models and improve on the baselines with other variations described below 
Apart from learning-based baselines, we also create two naive baselines, one each for the Dosage and Frequency extraction tasks  For Dosage extraction, the baseline we consider is `Nearest Number', where we take the number nearest to the Medication Name as the prediction, and `none' if no number is mentioned or if the Medication Name is not detected in the input  For Frequency extraction, the baseline we consider is `Random Top-3' where we predict a random Frequency tag, from top-3 most frequent ones from our dataset - {`none', `daily', `twice a day'} 
Embedding: We developed different variations of our models with a simple lookup table embeddings learned from scratch and using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13 (trained and provided by the authors)  Refer to Table TABREF5 for the performance comparisons 
We derive embeddings from ELMo by learning a linear combination of its last three layer's hidden states (task-specific fine-tuning BIBREF11)  Similarly, for BERT-based embeddings, we take a linear combination of the hidden states from its last four layers, as this combination performs best without increasing the size of the embeddings BIBREF16  Since BERT and ClinicalBERT use word-piece vocabulary and computes sub-word embeddings, we compute word-level embedding by averaging the corresponding sub-word tokens  ELMo and BERT embeddings both have 1024 dimensions, ClinicalBERT have 768 as it is based on BERT base model, and the lookup table have 128 – higher dimension models leads to overfitting 
Pertaining Encoder: We trained the PGNet as a summarization task using the clinical summaries and used the trained model to initialize the encoders (and the embeddings) of the corresponding QA models  We use a vocab size of 4073 words, derived from the training dataset with a frequency threshold of 30 for the task  We trained the models using Adagrad optimizer with a learning rate of 0 015, normal gradient clipping set at 2 and trained for around 150000 iterations (stopped using validation dataset)  On the summarization task PGNet obtained ROUGE-1 F1 scores of 41 42 with ELMo and 39 15 with BERT embeddings  We compare the effects of pretraining the model in Table: TABREF5, models with `pretrained encoder' had their encoders and embeddings pretrained with the summarization task 
Results and Discussion ::: Difference in networks and approaches
Embeddings: On Dosage extraction, in general, ELMo obtains better performance than BERT, refer to Table TABREF5  This could be because we concatenated the numbers with a hyphen, and as ELMo uses character-level tokens it can learn the tagging better than BERT, a similar observation is also noted in BIBREF17  On the other hand, on Frequency extraction, without pretraining, ELMo's performance is lagging by a big margin of $\sim $8 5 ROUGE-1 F1 compared to BERT-based embeddings 
Although in cases without encoder pretraining, ClinicalBERT performed the best in the Frequency extraction task (by a small margin), in general, it does not perform as well as BERT  This could also be a reflection of the fact that the language and style of writing used in clinical notes is very different from the way doctors converse with patients and the embedding dimension difference  Lookup table embedding performed decently in the frequency extraction task but lags behind in the Dosage extraction task 
From the metrics and qualitative inspection, we find that the Frequency extraction is an easier task than the Dosage extraction  This is because, in the conversations, frequency information usually occurs in isolation and near the medications, but a medication's dosage can occur 1) near other medication's dosages, 2) with previous dosages (when a dosage for a medication is revised), and 3) after a large number of words from the medication 
Other Variations: Considering various models' performance (without pretraining) and the resource constraint, we choose ELMo and BERT embeddings to analyze the effects of pretraining the encoder  When the network's encoder (and embedding) is pretrained with the summarization task, we 1) see a small decrease in the average number of iterations required for training, 2) improvement in individual performances of all models for both the sub-tasks, and 3) get best performance metrics across all variations, refer to Table TABREF5  Both in terms of performance and the training speed, there is no clear winner between shared and multi-decoder approaches

ground_truth: linear SVM, bidirectional Long Short-Term-Memory (BiLSTM), Convolutional Neural Network (CNN)

linear SVM, bidirectional Long Short-Term-Memory (BiLSTM), Convolutional Neural Network (CNN)

linear SVM trained on word unigrams,  bidirectional Long Short-Term-Memory (BiLSTM),  Convolutional Neural Network (CNN) 

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is the data de-identified?
Context 3:  Medication tagging and data augmentation increase the best-performing model's ROUGE-1 F1 score by $\sim $1 5 for the Dosage extraction task 
We also measure the performance of Multitask Question Answering Network (MQAN) BIBREF18 a QA model trained by the authors on the Decathlon multitask challenge  Since MQAN was not trained to produce the output sequence in our MR tags, it would not be fair to compute ROUGE scores  Instead, we randomly sample the MQAN's predictions from the 10% test dataset and qualitatively evaluate it  From the evaluations, we find that MQAN can not distinguish between frequency and dosage, and mixed the answers  MQAN correctly predicted the dosage for 29 73% and frequency for 24 24% percent of the medications compared to 84 12% and 76 34% for the encoder pretrained BERT QA PGNet model trained on our dataset  This could be because of the difference in the training dataset, domain and the tasks in the Decathlon challenge compared to ours 
Almost all our models perform better than the naive baselines and the ones using lookup table embeddings, and our best performing models outperform them significantly  Among all the variations, the best performing models are ELMo with Multi-decoder (Dosage extraction) and BERT with shared-decoder QA PGNet architecture with pretrained encoder (Frequency extraction)  We choose these two models for our subsequent analysis 
Results and Discussion ::: Breakdown of Performance
We categorize the 10% test dataset into different categories based on the complexity and type of the data and analyze the breakdown of the system's performance in Table TABREF11  We breakdown the Frequency extraction into two categories, 1) None: ground truth Frequency tag is `none', and 2) NN (Not None): ground truth Frequency tag is not `none'  Similarly, the Dosage extraction into 5 categories, 1) None: ground truth dosage tag is `none', 2) MM (Multiple Medicine): input segment has more than one Medication mentioned, 3) MN (Multiple Numbers): input segment has more than one number present, and 4) NBM (Number between correct Dosage and Medicine) : between the Medication Name and the correct Dosage in the input segment there are other numbers present  Note that the categories of the Dosage extraction task are not exhaustive, and one tag can belong to multiple categories 
From the performance breakdown of Dosage extraction task, we see that 1) the models predict `none' better than other categories, i e , the models are correctly able to identify when a medication's dosage is absent, 2) there is performance dip in hard cases (MM, MN, and NBM), 3) the models are able to figure out the correct dosage (decently) for a medication even when there are multiple numbers/dosage present, and 4) the model struggles the most in the NBM category  The models' low performance in NBM could be because we have a comparatively lower number of examples to train in this category  The Frequency extraction task performs equally well when the tag is `none` or not  In most categories, we see an increase in performance when using pretrained encoders 
Results and Discussion ::: Training Dataset Size
We vary the number of MR tags used to train the model and analyze the model's performance when training the networks, using publicly available contextual embeddings, compared to using pretrained embeddings and encoder (pretrained on the summarization task)  Out of the 5,016 files in the 80% train dataset only 2,476 have atleast one MR tag  Therefore, out of the 2476 files, we randomly choose 100, 500, and 1000 files and trained the best performing model variations to observe the performance differences, refer to Figure FIGREF12  For all these experiments we used the same vocabulary size (456), the same hyper/training parameters, and the same 10% test split of 627 files 
As expected, we see that the encoder pretrained models have higher performance on all the different training data sizes, i e , they achieve higher performance on a lower number of data points, refer to Figure FIGREF12  The difference, as expected, shrinks as the training data size increases 
Results and Discussion ::: Evaluating on ASR transcripts
To test the performance of our models on real-world conditions, we use commercially available ASR services (Google and IBM) to transcribe the $\mathcal {D}_{test}$ files and measure the performance of our models without assuming any annotations (except when calculating the metrics)  It should be noted that this is not the case in our previous evaluations using `10% test' dataset where we use the segmentation information  For ground truth annotations on ASR transcripts, we aligned the MR tags from human written transcripts to the ASR transcript using their grounded timing information  Additionally, since ASR is prone to errors, during the alignment, if a medication from an MR tag is not recognized correctly in the ASR transcript, we remove the corresponding MR tag 
In our evaluations, we use Google Cloud Speech-to-Text (G-STT) and IBM Watson Speech to Text (IBM-STT) as these were among the top-performing ASR APIs on medical speech BIBREF19 and were readily available to us  We used G-STT, with the `video model' with punctuation settings  Unlike our human written transcripts, the transcript provided by G-STT is not verbatim and does not have disfluencies  IBM-STT, on the other hand, does not give punctuation so we used the speaker changes to add end-of-sentence punctuation 
In our $\mathcal {D}_{test}$ dataset, on initial study we see a Word Error Rate of $\sim $50% for the ASR APIs and this number is not accurate because, 1) of the de-identification, 2) disfluencies (verbatim) difference between the human written and ASR transcript, and 3) minor alignment differences between the audio and the ground truth transcript 
During this evaluation, we followed the same preprocessing methods we used during training  Then, we auto segment the transcript into small contiguous segments similar to the grounded sentences in the annotations for tags extraction  To segment the transcript, we follow a simple procedure  First, we detected all the medications in a transcript using RxNorm BIBREF20 via string matching  For all the detected medications, we selected $2 \le x \le 5$ nearby sentences as the input to our model  We increased $x$ iteratively until we encountered a quantity entity – detected using spaCy's entity recognizer, and we set $x$ as 2 if we did not detect any entities in the range 
We show the model's performance on ASR transcripts and human written transcripts with automatic segmentation, and human written transcripts with human (defined) segmentation, in Table TABREF18  The number of recognized medications in IBM-STT is only 95 compared to 725 (human written), we mainly consider the models' performance on G-STT's transcripts (343) 
On the Medications that were recognized correctly, the models can perform decently on ASR transcripts in comparison to human transcripts (within 5 points ROUGE-1 F1 for both tasks, refer to Table TABREF18)  This shows that the models are robust to ASR variations discussed above  The lower performance compared to human transcripts is mainly due to incorrect recognition of Dosage and other medications in the same segments (changing the meaning of the text)  By comparing the performance of the model on the human written transcripts with human (defined) segmentation and the same with auto segmentation, we see a 10 point drop in Dosage and 6 point drop in Frequency extraction tasks  This points out the need for more sophisticated segmentation algorithms 
With G-STT, our best model obtained ROUGE-1 F1 of 71 75 (which equals to percentage of times dosage is correct in this case) for Dosage extraction and 40 13 for Frequency extraction tasks  To measure the percentage of times the correct frequency was extracted by the model, we qualitatively compared the extracted and predicted frequency  We find that for 73 58% of the medications the model can find the correct frequency from the transcripts 
Conclusion
In this paper, we explore the Medication Regimen (MR) extraction task of extracting dosage and frequency for the medications mentioned in a doctor-patient conversation transcript  We explore different variations of abstractive QA models and new architecture in the intersection of QA and IE frameworks and provide a comparative performance analysis of the methods along with other techniques like pretraining to improve the overall performance  Finally, we demonstrate the performance of our best-performing models by automatically extracting MR tags from spontaneous doctor-patient conversations (using commercially available ASR)  Our best model can correctly extract the dosage for 71 75% (interpretation of ROUGE-1 score) and frequency for 73 58% (on qualitative evaluation) of the medications discussed in the transcripts generated using Google Speech-To-Text  In summary, we demonstrate that research on NLP can be translated into real-world clinical settings to realize its benefits for both doctors and patients 
Using ASR transcripts in our training process to improve our performance on both the tasks and extending the medication regimen extraction network to extract other important medical information can be interesting lines of future work 
Acknowledgements
We thank: University of Pittsburgh Medical Center (UPMC), and Abridge AI Inc  for providing access to the de-identified data corpus; Dr  Shivdev Rao, a faculty member and practicing cardiologist in UPMC's Heart and Vascular Institute and Prof  Florian Metze, Associate Research Professor, Carnegie Mellon University for helpful discussions; Ben Schloss, Steven Coleman, and Deborah Osakue for data business development and annotation management 

ground_truth: Yes

Yes