prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Are the images from a specific domain?
Context 1: Introduction
Automatically generating text to describe the content of images, also known as image captioning, is a multimodal task of considerable interest in both the computer vision and the NLP communities  Image captioning can be framed as a translation task from an image to a descriptive natural language statement  Many existing captioning models BIBREF0, BIBREF1, BIBREF2, BIBREF3 follow the typical encoder-decoder framework where a convolutional network is used to condense images into visual feature representations, combined with a recurrent network for language generation  While these models demonstrate promising results, quantifying image captioning performance remains a challenging problem, in a similar way to other generative tasks BIBREF4, BIBREF5 
Evaluating candidate captions for human preference is slow and laborious  To alleviate this problem, many automatic evaluation metrics have been proposed, such as BLEU BIBREF6, METEOR BIBREF7, ROUGE BIBREF8 and CIDEr BIBREF9  These n-gram-based metrics evaluate captioning performance based on surface similarity between a candidate caption and reference statements  A more recent evaluation metric for image captioning is SPICE BIBREF10, which takes into account semantic propositional content of generated captions by scoring a caption based upon a graph-based semantic representation transformed from reference captions 
The rationale behind these evaluation metrics is that human reference captions serve as an approximate target and comparing model outputs to this target is a proxy for how well a system performs  Thus, a candidate caption is not directly evaluated with respect to image content, but compared to a set of human statements about that image 
However, in image captioning, visual scenes with multiple objects and relations correspond to a diversity of valid descriptions  Consider the example image and captions from the ShapeWorld framework BIBREF11 shown in Figure FIGREF1  The first three captions are true statements about the image and express relevant ideas, but describe different objects, attributes and spatial relationships, while the fourth caption is wrong despite referring to the same objects as in the third caption  This casts doubt on the sufficiency of using a set of reference captions to approximate the content of an image  We argue that, while existing metrics have undeniably been useful for real-world captioning evaluation, their focus on approximate surface comparison limits deeper insights into the learning process and eventual behavior of captioning models 
To address this problem, we propose a set of principled evaluation criteria which evaluate image captioning models for grammaticality, truthfulness and diversity (GTD)  These criteria correspond to necessary requirements for image captioning systems: (a) that the output is grammatical, (b) that the output statement is true with respect to the image, and (c) that outputs are diverse and mirror the variability of training captions 
Practical evaluation of GTD is currently only possible on synthetic data  We construct a range of datasets designed for image captioning evaluation  We call this diagnostic evaluation benchmark ShapeWorldICE (ShapeWorld for Image Captioning Evaluation)  We illustrate the evaluation of specific image captioning models on ShapeWorldICE  We empirically demonstrate that the existing metrics BLEU and SPICE do not capture true caption-image agreement in all scenarios, while the GTD framework allows a fine-grained investigation of how well existing models cope with varied visual situations and linguistic constructions 
We believe that as a supplementary evaluation method to real-world metrics, the GTD framework provides evaluation insights that are sufficiently interesting to motivate future work 
Related work ::: Existing evaluation of image captioning
As a natural language generation task, image captioning frequently uses evaluation metrics such as BLEU BIBREF6, METEOR BIBREF7, ROUGE BIBREF8 and CIDEr BIBREF9  These metrics use n-gram similarity between the candidate caption and reference captions to approximate the correlation between a candidate caption and the associated ground truth  SPICE BIBREF10 is a more recent metric specifically designed for image captioning  For SPICE, both the candidate caption and reference captions are parsed to scene graphs, and the agreement between tuples extracted from these scene graphs is examined  SPICE more closely relates to our truthfulness evaluation than the other metrics, but it still uses overlap comparison to reference captions as a proxy to ground truth  In contrast, our truthfulness metric directly evaluates a candidate caption against a model of the actual visual content 
Many researchers have pointed out problems with existing reference-based metrics including low correlations with human judgment BIBREF12, BIBREF10, BIBREF13 and strong baselines using nearest-neighbor methods BIBREF14 or relying solely on object detection BIBREF15  Fundamental concerns have been raised with respect to BLEU, including variability in parameterization and precise score calculation leading to significantly different results BIBREF16  Its validity as a metric for tasks other than machine translation has been questioned BIBREF17, particularly for tasks for which the output content is not narrowly constrained, like dialogue BIBREF18 
Some recent work focuses on increasing the diversity of generated captions, for which various measures are proposed  Devlin et al  BIBREF19 explored the concept of caption diversity by evaluating performance on compositionally novel images  van Miltenburg et al BIBREF20 framed image captioning as a word recall task and proposed several metrics, predominantly focusing on diversity at the word level  However, this direction is still relatively new and lacks standardized benchmarks and metrics 
Related work ::: Synthetic datasets
Recently, many synthetic datasets have been proposed as diagnostic tools for deep learning models, such as CLEVR BIBREF21 for visual question answering (VQA), the bAbI tasks BIBREF22 for text understanding and reasoning, and ShapeWorld BIBREF11 for visually grounded language understanding  The primary motivation is to reduce complexity which is considered irrelevant to the evaluation focus, to enable better control over the data, and to provide more detailed insights into strengths and limitations of existing models 
In this work, we develop the evaluation datasets within the ShapeWorld framework  ShapeWorld is a controlled data generation framework consisting of abstract colored shapes (see Figure FIGREF1 for an example)  We use ShapeWorld to generate training and evaluation data for two major reasons  ShapeWorld supports customized data generation according to user specification, which enables a variety of model inspections in terms of language construction, visual complexity and reasoning ability  Another benefit is that each training and test instance generated in ShapeWorld is returned as a triplet of $<$image, caption, world model$>$  The world model stores information about the underlying microworld used to generate an image and a descriptive caption, internally represented as a list of entities with their attributes, such as shape, color, position  During data generation, ShapeWorld randomly samples a world model from a set of available entities and attributes  The generated world model is then used to realize a corresponding instance consisting of image and caption  The world model gives the actual semantic information contained in an image, which allows evaluation of caption truthfulness 
GTD Evaluation Framework
In the following we introduce GTD in more detail, consider it as an evaluation protocol covering necessary aspects of the multifaceted captioning task, rather than a specific metric 
GTD Evaluation Framework ::: Grammaticality
An essential criterion for an image captioning model is that the captions generated are grammatically well-formed  Fully accurate assessment of grammaticality in a general context is itself a difficult task, but becomes more feasible in a very constrained context like our diagnostic language data  We take parseability with the English Resource Grammar BIBREF23 as a surrogate for grammaticality, meaning that a sentence is considered grammatically well-formed if we obtain a parse using the ERG 
The ERG is a broad-coverage grammar based on the head-driven phrase structure grammar (HPSG) framework  It is linguistically precise: sentences only parse if they are valid according to its hand-built rules  It is designed to be general-purpose: verified coverage is around 80% for Wikipedia, and over 90% for corpora with shorter sentences and more limited vocabulary (for details see BIBREF24 flickinger2011accuracy)  Since the ShapeWorld training data – the only language source for models to learn from – is generated using the same grammar, the ERG has $\sim $100% coverage of grammaticality in the model output space 
GTD Evaluation Framework ::: Truthfulness
The second aspect we investigate is truthfulness, that is, whether a candidate caption is compatible with the content of the image it is supposed to describe  We evaluate caption truthfulness on the basis of a linguistically-motivated approach using formal semantics  We convert the output of the ERG parse for a grammatical caption to a Dependency Minimal Recursion Semantics (DMRS) graph using the pydmrs tool BIBREF25  Each converted DMRS is a logical semantic graph representation corresponding to the caption  We construct a logical proposition from the DMRS graph, and evaluate it against the actual world model of the corresponding image  A caption can be said to agree with an image only if the proposition evaluates as true on the basis of the world model  By examining the logical agreement between a caption representation and a world model, we can check whether the semantics of this caption agrees with the visual content which the world model represents  Thus we do not rely on a set of captions as a surrogate for the content of an image, but instead leverage the fact that we have the ground truth, thus enabling the evaluation of true image-caption agreement 
GTD Evaluation Framework ::: Diversity
While grammaticality and truthfulness are essential requirements for image captions, these criteria alone can easily be “gamed” by specializing on a small set of generic statements which are true most of the time  In the context of abstract shapes, such captions include examples like “There is a shape” or “At least zero shapes are blue” (which is technically true even if there is no blue shape)  This motivates the third fundamental requirement of captioning output to be diverse 
As ShapeWorldICE exploits a limited size of open-class words, we emphasize the diversity in ShapeWorldICE at the sentence level rather than the word level  Since the ground-truth reference captions in ShapeWorld are randomly sampled, we take the sampled captions accompanying the test images as a proxy for optimal caption diversity, and compare it with the empirical output diversity of the evaluated model on these test images  Practically, we look at language constructions used and compute the corresponding diversity score as the ratio of observed number versus optimal number:
Language constructions here correspond to reduced caption representations which only record whether an object is described by shape (e g , “square”), color (e g , “red shape”) or color-shape combination (e g , “red square”)  So the statement “A square is red” and “A circle is blue” are considered the same, while “A shape is red” is different 
Experimental Setup ::: Datasets
We develop a variety of ShapeWorldICE datasets, with a similar idea to the “skill tasks” in the bAbI framework BIBREF22  Table TABREF4 gives an overview for different ShapeWorldICE datasets we use in this paper  We consider three different types of captioning tasks, each of which focuses on a distinct aspect of reasoning abilities  Existential descriptions examine whether a certain object is present in an image  Spatial descriptions identify spatial relationships among visual objects  Quantification descriptions involve count-based and ratio-based statements, with an explicit focus on inspecting models for their counting ability  We develop two variants for each type of dataset to enable different levels of visual complexity or specific aspects of the same reasoning type  All the training and test captions sampled in this work are in English 
Each dataset variant consists of around 200k training instances and 4,096 validation instances, plus 4,096 test instances  Each training instance consists of an image and a reference caption  At test time, only the test images are available to the evaluated models  Underlying world models are kept from the models and are used for later GTD evaluation  For each test instance, we sample ten reference captions of the same distribution as the training captions to enable the comparison of our proposed metrics to BLEU and SPICE  We fine-tune our model hyperparameters based on the performance on the validation set  All reported results are measured on the test split with the parameters yielding the best validation performance 
Experimental Setup ::: Models
We experiment with two image captioning models: the Show&Tell model BIBREF0 and the LRCN1u model BIBREF1  Both models follow the basic encoder-decoder architecture design that uses a CNN encoder to condense the visual information into an image embedding, which in turn conditions an LSTM decoder to generate a natural language caption  The main difference between the two models is the way they condition the decoder  The Show&Tell model feeds the image embedding as the “predecessor word embedding” to the first produced word, while the LRCN1u model concatenates the image features with the embedded previous word as the input to the sequence model at each time step 
We follow the common practice in image captioning to use a CNN component pretrained on object detection and fine-tune its parameters on the image captioning task  The encoder and decoder components are jointly optimized with respect to the standard cross-entropy sequence loss on the respective ShapeWorldICE dataset  For all our experiments, we train models end-to-end for a fixed number of 100k iterations with a batch size of 64  We use Adam optimization BIBREF26 with a learning rate of 0 001  Word embeddings are randomly initialized and jointly trained during the training 
Results
We train and evaluate the Show&Tell and LRCN1u models on the ShapeWorldICE datasets  Here we discuss in detail the diagnostic results of these experiments  During training, we periodically record model output on the test images, to be able to analyze the development of our evaluation metrics throughout the process  We also compute BLEU-4 scores and SPICE scores of generated captions for comparison, using 10 reference captions per test image 
LRCN1u exhibits clearly superior performance in terms of truthfulness  We start off by comparing performance of the Show&Tell model and the LRCN1u model, see Figure FIGREF8  While both models learn to produce grammatical sentences early on, it can be seen that LRCN1u is clearly superior in terms of truthfulness, achieving 100% halfway through training, whereas Show&Tell only slowly reaches around 90% by the end of 100k iterations  This indicates that incorporating visual features at every generation step is beneficial for producing true captions  The diversity ratios of captions generated by two models both increase substantially as the training progresses, with LRCN1u exhibiting a slightly greater caption diversity at the end of training 
We observed similar results on other ShapeWorldICE datasets that we experimented with, validating the superiority of LRCN1u over Show&Tell on ShapeWorldICE  Consequently, we decided to focus on the LRCN1u architecture in subsequent evaluations, where we report detailed results with respect to the GTD framework on a variety of datasets 
Correlation between the BLEU/SPICE scores and the ground truth  From the learning curves shown in Figure FIGREF9, we find low or no correlation between the BLEU/SPICE scores and caption truthfulness 
On Existential-OneShape, the BLEU curve follows the trend of the truthfulness curve in general, indicating that BLEU is able to capture caption truthfulness well in this simple scenario  However, while BLEU reports equivalent model performance on Existential-MultiShapes and Spatial-MultiShapes, the truthfulness metric demonstrates very different results  The BLEU score for generated Existential-MultiShapes captions increases rapidly at the beginning of training and then plateaus despite the continuous increase in truthfulness ratio  Captions generated on Spatial-MultiShapes attain a relatively high BLEU score from an early stage of training, but exhibit low agreement ($<$0 6 truthfulness ratio) with ground-truth visual scenes  In the case of Spatial-MultiShapes, spatial descriptors for two objects are chosen from a fixed set (“above”, “below”, “to the left of” and “to the right of”)  It is very likely for a generated spatial descriptor to match one of the descriptors mentioned in reference captions  In this particular case, the model is apt to infer a caption which has high n-gram overlaps with reference captions, resulting in a relatively high BLEU score  Thus an increased BLEU score does not necessarily indicate improved performance 
While the truthfulness and BLEU scores in Figure FIGREF9 both increase rapidly early on and then stay stable at a high rate after training for 20k iterations, the SPICE curve instead shows a downward trend in the later stage of training  We examined the output SPICE score for each test instance  SPICE reports a precision score of 1 0 for most test instances after 20k iterations, which is consistent with the truthfulness and BLEU scores  However, SPICE forms the reference scene graph as the union of the scene graphs extracted from individual reference captions, thus introducing redundancies  SPICE uses the F1 score of scene graph matching between the candidate and reference and hence is lowered by imperfect recall 
Comparing SPICE curves for three datasets shown in Figure FIGREF9-FIGREF9, they suggest an increase in task complexity, but they do not reflect the successively closing gap of caption truthfulness scores between two Existential datasets, or the substantial difference in caption truthfulness between captions on Existential-MultiShapes and Spatial-MultiShapes 
In the remainder of the paper we discuss in detail the diagnostic results of the LRCN1u model demonstrated by the GTD evaluation framework 
Perfect grammaticality for all caption types  As shown in Figure FIGREF15, generated captions for all types of ShapeWorldICE datasets attain quasi-perfect grammaticality scores in fewer than 5,000 iterations, suggesting that the model quickly learns to generate grammatically well-formed sentences 
Failure to learn complex spatial relationships  While CNNs can produce rich visual representations that can be used for a variety of vision tasks BIBREF27, it remains an open question whether these condensed visual representations are rich enough for multimodal tasks that require higher-level abilities of scene understanding and visual reasoning  From Figure FIGREF16, we can see that while the model performs rather well on Existential datasets, it exhibits a worse performance on Spatial data  The caption agreement ratio in the simple Spatial-TwoShapes scenario is relatively high, but drops significantly on Spatial-MultiShapes, demonstrating the deficiencies of the model in learning spatial relationships from complex visual scenes 
The counting task is non-trivial  Counting has long been considered to be a challenging task in multimodal reasoning BIBREF28, BIBREF29  To explore how well the LRCN1u model copes with counting tasks, we generated two Quantification datasets  The Quant-Count captions describe the number of objects with certain attributes that appear in an image (e g  “Exactly four shapes are crosses”), while the Quant-Ratio captions describe the ratio of certain objects (e g  “A third of the shapes are blue squares”) 
From Figure FIGREF16, we notice that the LRCN1u model performs poorly on these datasets in terms of truthfulness, reflected in the 0 50 and 0 46 scores achieved by the model on the Quant-Count and Quant-Ratio tasks respectively  The learning curve for Quant-Ratio exhibits a more gradual rise as the training progresses, suggesting a greater complexity for the ratio-based task 
Caption diversity benefits from varied language constructions in the training data  The diversity ratios of generated captions for different ShapeWorldICE datasets are illustrated in Figure FIGREF17  We can see that the diversity of inferred captions is largely sensitive to the caption variability in the dataset itself  For simple datasets (such as Existential-OneShape) where language constructions in the training set are less diverse, the output captions tend to have uniform sentence structures  The high diversity ratios of generated Spatial and Quantification captions suggest that caption diversity benefits from heterogeneous language constructions in complex datasets 
Discussions and Conclusions
Evaluation metrics are required as a proxy for performance in real applications  As such, they should, as far as possible, allow measurement of fundamental aspects of the performance of models on tasks  In this work, we propose the GTD evaluation framework as a supplement to standard image captioning evaluation which explicitly focuses on grammaticality, truthfulness and diversity  We developed the ShapeWorldICE evaluation suite to allow in-depth and fine-grained inspection of model behaviors  We have empirically verified that GTD captures different aspects of performance to existing metrics by evaluating image captioning models on the ShapeWorldICE suite  We hope that this framework will shed light on important aspects of model behaviour and that this will help guide future research efforts 
While performing the evaluation experiments on the LRCN1u model, we noticed that caption agreement does not always improve as the training loss decreases  Ideally, the training objective should be in accordance with how a model is eventually evaluated  In future work, we plan to investigate the feasibility of deliberately encoding the GTD signal in the training process, for instance, by implementing a GTD-aware loss  We also plan to extend the existing ShapeWorldICE benchmark to include more linguistic constructions (such as relative clauses, compound sentences and coreference)  By doing so, we hope to reveal how well existing image captioning models cope with complex generation tasks 
Acknowledgments
We thank the anonymous reviewers for their constructive feedback  HX is grateful for being supported by the CSC Cambridge Scholarship  TS is supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the EPSRC (grant EP/L016427/1) and the University of Edinburgh  AK is grateful for being supported by a Qualcomm Research Studentship and an EPSRC Doctoral Training Studentship 
Context 2: Introduction
To appear in Proceedings of International Workshop on Health Intelligence (W3PHIAI) of the 34th AAAI Conference on Artificial Intelligence, 2020 
Physician burnout is a growing concern, estimated to be experienced by at least 35% of physicians in the developing world and 50% in the United States BIBREF0  BIBREF1 found that for every hour physicians provide direct clinical facetime to patients, nearly two additional hours are spent on EHR (Electronic Health Records) and administrative or desk work  As per the study conducted by Massachusetts General Physicians Organization (MPGO) BIBREF2 and as reported by BIBREF3, the average time spent on administrative tasks increased from 23 7% in 2014 to 27 9% in 2017  Both the surveys found that time spent on administrative tasks was positively associated with higher likelihood of burnout  Top reasons under administrative burden include working on the ambulatory EHR, handling medication reconciliation (sometimes done by aids), medication renewals, and medical billing and coding  The majority of these reasons revolve around documentation of information exchanged between doctors and patients during the clinical encounters  Automatically extracting such clinical information BIBREF4, BIBREF5 can not only help alleviate the documentation burden on the physician, but also allow them to dedicate more time directly with patients 
Among all the clinical information extraction tasks, Medication Regimen (Medication, dosage, and frequency) extraction is particularly interesting due to its ability to help doctors with medication orders cum renewals, medication reconciliation, potentially verifying the reconciliations for errors, and, other medication-centered EHR documentation tasks  In addition, the same information when provided to patients can help them with better recall of doctor instructions which might aid in compliance with the care plan  This is particularly important given that patients forget or wrongly recollect 40-80% BIBREF6 of what is discussed in the clinic, and accessing EHR data has its own challenges 
Spontaneous clinical conversations happening between a doctor and a patient, have several distinguishing characteristics from a normal monologue or prepared speech: it involves multiple speakers with overlapping dialogues, covers a variety of speech patterns, and the vocabulary can range from colloquial to complex domain-specific language  With recent advancements in Conversational Speech Recognition BIBREF7 rendering the systems less prone to errors, the subsequent challenge of understanding and extracting relevant information from the conversations is receiving increasing research focus BIBREF4, BIBREF8 
In this paper, we focus on local information extraction in transcribed clinical conversations  Specifically, we extract dosage (e g  5mg) and frequency (e g  once a day) for the medications (e g  aspirin) from these transcripts, collectively referred to as Medication Regimen (MR) extraction  The information extraction is local as we extract the information from a segment of the transcript and not the entire transcript since doing the latter is difficult owing to the long meandering nature of the conversations often with multiple medication regimens and care plans being discussed 
The challenges associated with the Medication Regimen (MR) extraction task include understanding the spontaneous dialog with clinical vocabulary and understanding the relationship between different entities as the discussion can contain multiple medications and dosages (e g  doctor revising a dosage or reviewing all the current medications) 
We frame this problem as a Question Answering (QA) task by generating questions using templates  We base the QA model on pointer-generator networks BIBREF9 augmented with Co-Attentions BIBREF10  In addition, we develop models combining QA and Information Extraction frameworks using multi-decoder (one each for dosage and frequency) architecture 
Lack of availability of a large volume of data is a typical challenge in healthcare  A conversation corpus by itself is a rare commodity in the healthcare data space because of the cost and difficulty in handing (because of data privacy concerns)  Moreover, transcribing and labeling the conversations is a costly process as it requires domain-specific medical annotation expertise  To address data shortage and improve the model performance, we investigate different high-performance contextual embeddings (ELMO BIBREF11, BERT BIBREF12 and ClinicalBERT BIBREF13), and pretrain the models on a clinical summarization task  We further investigate the effects of training data size on our models 
On the MR extraction task, ELMo with encoder multi-decoder architecture and BERT with encoder-decoder with encoders pretrained on the summarization task perform the best  The best-performing models improve our baseline's dosage and frequency extractions ROUGE-1 F1 scores from 54 28 and 37 13 to 89 57 and 45 94, respectively 
Using our models, we present the first fully automated system to extract MR tags from spontaneous doctor-patient conversations  We evaluate the system (using our best performing models) on the transcripts generated from Automatic Speech Recognition (ASR) APIs offered by Google and IBM  In Google ASR's transcripts, our best model obtained ROUGE-1 F1 of 71 75 for Dosage extraction (which in this specific case equals to the percentage of times dosage is correct, refer Metrics Section for more details) and 40 13 for Frequency extraction tasks  On qualitative evaluation, we find that for 73 58% of the medications the model can find the correct frequency  These results demonstrate that the research on NLP can be used effectively in a real clinical setting to benefit both doctors and patients
Data
Our dataset consists of a total of 6,693 real doctor-patient conversations recorded in a clinical setting using distant microphones of varying quality  The recordings have an average duration of 9min 28s and have a verbatim transcript of 1,500 words on average (written by the experts)  Both the audio and the transcript are de-identified (by removing the identifying information) with digital zeros and [de-identified] tags, respectively  The sentences in the transcript are grounded to the audio with the timestamps of its first and last word 
The transcript of the conversations are annotated with summaries and Medication Regimen tags (MR tags), both grounded using the timestamps of the sentences from the transcript deemed relevant by the expert annotators, refer to Table TABREF1  The transcript for a typical conversation can be quite long, and not easy for many of the high performing deep learning models to act on  Moreover, the medical information about a concept/condition/entity can change during the conversation after a significant time gap  For example, dosage of a medication can be different when discussing current medication the patient is on vs when they are prescribed a different dosage  Hence, we have annotations, that are grounded to a short segment of the transcript 
The summaries (#words - $\mu = 9 7; \sigma = 10 1$) are medically relevant and local  The MR tags are also local and are of the form {Medication Name, Dosage, Frequency}  If dosage ($\mu = 2 0; \sigma = 0$) or frequency ($\mu = 2 1; \sigma = 1 07$) information for a medication is not present in a grounded sentence, the corresponding field in the MR tag will be marked as `none' 
In the MR tags, Medication Name and Dosage (usually a quantity followed by its units) can be relatively easily extracted from the transcript except for the units of the dosage which is sometimes inferred  In contrast, due to high degree of linguistic variation with which Frequency is often expressed, extracting Frequency requires an additional inference step  For example, `take one in the morning and at noon' from the transcript is tagged as `twice a day' in the frequency tag, likewise `take it before sleeping' is tagged as `at night time' 
Out of overall 6,693 files, we set aside a random sample of 423 files (denoted as $\mathcal {D}_{test}$) for final evaluation  The remaining 6,270 files are used for training with 80% train (5016), 10% validation (627), and 10% test (627) split  Overall, the 6,270 files contains 156,186 summaries and 32,000 MR tags out of which 8,654 MR tags contain values for at least one of the Dosage or Frequency, which we used for training to avoid overfitting (the remaining MR tags have both Dosage and Frequency as `none')  Note that we have two test datasets: `10% test' - used to evaluate all the models, and $\mathcal {D}_{test}$ - used to measure the performance of best performing models on ASR transcripts 
Approach
We frame the Medication Regimen extraction problem as a Question Answering (QA) task, which forms the basis for our first approach  It can also be considered as a specific inference or relation extract task, since we extract specific information about an entity (Medication Name), hence our second approach is at the intersection of Question Answering (QA) and Information Extraction (IE) domains  Both the approaches involve using a contiguous segment of the transcript and the Medication Name as input, to find/infer the medication's Dosage and Frequency  When testing the approaches mimicking real-world conditions, we extract Medication Name from the transcript separately using ontology, refer to SECREF19 
In the first approach, we frame the MR task as a QA task and generate questions using the template: “What is the $ <$dosage/frequency$>$ for $<$Medication Name$>$"  Here, we use an abstractive QA model based on pointer-generator networks BIBREF9 augmented with coattention encoder BIBREF10 (QA-PGNet) 
In the second approach, we frame the problem as a conditioned IE task, where the information extracted depends on an entity (Medication Name)  Here, we use a multi-decoder pointer-generator network augmented with coattention encoder (Multi-decoder QA-PGNet)  Instead of using templates to generate questions and using a single decoder to extract different types of information as in the QA approach (which might lead to performance degradation), here we consider separate decoders for extracting specific types of information about an entity $E$ (Medication Name) 
Approach ::: Pointer-generator Network (PGNet)
The network is a sequence-to-sequence attention model that can both copy a word from the input $I$ containing $P$ word tokens or generate a word from its vocabulary $vocab$, to produce the output sequence 
First, the tokens of the $I$ are converted to embeddings and are fed one-by-one to the encoder, a single bi-LSTM layer, which encodes the tokens in $I$ into a sequence of hidden states - $H=encoder(I)$, where $ H=[h_1   h_P]$ 
For each decoder time step $t$, in a loop, we compute, 1) attention $a_t$ (using the last decoder state $s_{t-1}$), over the input tokens $I$, and 2) the decoder state $s_t$ using $a_t$  Then, at each time step, using both $a_t$ and $s_t$ we can find the probability $P_t(w)$, of producing a word $w$ (from both $vocab$ and $I$)  For convenience, we denote the attention and the decoder as $decoder_{pg}(H)=P(w)$, where $P(w)=[P_1(w)   P_T(w)]$  The output can then be decoded from $P(w)$, which is decoded until it produces an `end of output token' or the number of steps reach the maximum allowed limit 
Approach ::: QA PGNet
We first encode both the question - $H_Q = encoder(Q)$, and the input - $H_I = encoder(I)$, separately using encoders (with shared weights)  Then, to condition $I$ on $Q$ (and vice versa), we use the coattention encoder BIBREF10 which attends to both the $I$ and $Q$ simultaneously to generates the coattention context - $C_D = coatt(H_I, H_Q)$  Finally, using the pointer-generator decoder we find the probability distribution of the output sequence - $P(w) = decoder_{pg}([H_I; C_D])$, which is then decoded to generate the answer sequence 
Approach ::: Multi-decoder (MD) QA PGNet
After encoding the inputs into $H_I$ and $H_E$, for extracting $K$ types of information about an entity in an IE fashion, we use the following multi-decoder (MD) setup:
Predictions for each of the $K$ decoders are then decoded using $P^k(w)$ 
All the networks discussed above are trained using a negative log-likelihood loss for the target word at each time step and summed over all the decoder time steps 
Experiments
We initialized MR extraction models' vocabulary from the training dataset after removing words with a frequency lower than 30 in the dataset, resulting in 456 words  Our vocabulary is small because of the size of the dataset, hence we rely on the model's ability to copy words to produce the output effectively  In all our model variations, the embedding and the network's hidden dimension are set to be equal  The networks were trained with a learning rate of 0 0015, dropout of 0 5 on the embedding layer, normal gradient clipping set at 2, batch size of 8, and optimized with Adagrad BIBREF15 and the training was stopped using the $10\%$ validation dataset 
Experiments ::: Data Processing
We did the following basic preprocessing to our data, 1) added `none' to the beginning of the input utterance, so the network could point to it when there was no relevant information in the input, 2) filtered outliers with a large number of grounded transcript sentences ($>$150 words), and 3) converted all text to lower case 
To improve performance, we 1) standardized all numbers (both digits and words) to words concatenated with a hyphen (e g  110 -$>$ one-hundred-ten), in both input and output, 2) removed units from Dosage as sometimes the units were not explicitly mentioned in the transcript segment but were written by the annotators using domain knowledge, 3) prepended all medication mentions with `rx-' tag, as this helps model's performance when multiple medications are discussed in a segment (in both input and output), and 4) when a transcript segment has multiple medications or dosages being discussed we randomly shuffle them (in both input and output) and create a new data point, to increases the number of training data points  Randomly shuffling the entities increases the number of training MR tags from 8,654 to 11,521  Based on the data statistics after data processing, we fixed the maximum encoder steps to 100, dosage decoder steps to 1, and frequency decoder steps to 3 (for both the QA and Multi-decoder QA models)

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is the data de-identified?
Context 2: 
Experiments ::: Metrics
For the MR extraction task, we measure the ROUGE-1 scores BIBREF14 for both the Dosage and Frequency extraction tasks  It should be noted that since Dosage is a single word token (after processing), both the reference and hypothesis are a single token, making its ROUGE-1 F1, Precision and Recall scores equal and be equal to percentage of times we find the correct dosage for the medications 
In our annotations, Frequency has conflicting tags (e g  {`Once a day', `twice a day'} and `daily'), hence metrics like Exact Match will be erroneous  To address this issue, we use the ROUGE scores to compare different models on the 10% test dataset and we use qualitative evaluation to measure the top-performing models on $\mathcal {D}_{test}$ 
Experiments ::: Model variations
We consider QA PGNet and Multi-decoder QA PGNet with lookup table embedding as baseline models and improve on the baselines with other variations described below 
Apart from learning-based baselines, we also create two naive baselines, one each for the Dosage and Frequency extraction tasks  For Dosage extraction, the baseline we consider is `Nearest Number', where we take the number nearest to the Medication Name as the prediction, and `none' if no number is mentioned or if the Medication Name is not detected in the input  For Frequency extraction, the baseline we consider is `Random Top-3' where we predict a random Frequency tag, from top-3 most frequent ones from our dataset - {`none', `daily', `twice a day'} 
Embedding: We developed different variations of our models with a simple lookup table embeddings learned from scratch and using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13 (trained and provided by the authors)  Refer to Table TABREF5 for the performance comparisons 
We derive embeddings from ELMo by learning a linear combination of its last three layer's hidden states (task-specific fine-tuning BIBREF11)  Similarly, for BERT-based embeddings, we take a linear combination of the hidden states from its last four layers, as this combination performs best without increasing the size of the embeddings BIBREF16  Since BERT and ClinicalBERT use word-piece vocabulary and computes sub-word embeddings, we compute word-level embedding by averaging the corresponding sub-word tokens  ELMo and BERT embeddings both have 1024 dimensions, ClinicalBERT have 768 as it is based on BERT base model, and the lookup table have 128 – higher dimension models leads to overfitting 
Pertaining Encoder: We trained the PGNet as a summarization task using the clinical summaries and used the trained model to initialize the encoders (and the embeddings) of the corresponding QA models  We use a vocab size of 4073 words, derived from the training dataset with a frequency threshold of 30 for the task  We trained the models using Adagrad optimizer with a learning rate of 0 015, normal gradient clipping set at 2 and trained for around 150000 iterations (stopped using validation dataset)  On the summarization task PGNet obtained ROUGE-1 F1 scores of 41 42 with ELMo and 39 15 with BERT embeddings  We compare the effects of pretraining the model in Table: TABREF5, models with `pretrained encoder' had their encoders and embeddings pretrained with the summarization task 
Results and Discussion ::: Difference in networks and approaches
Embeddings: On Dosage extraction, in general, ELMo obtains better performance than BERT, refer to Table TABREF5  This could be because we concatenated the numbers with a hyphen, and as ELMo uses character-level tokens it can learn the tagging better than BERT, a similar observation is also noted in BIBREF17  On the other hand, on Frequency extraction, without pretraining, ELMo's performance is lagging by a big margin of $\sim $8 5 ROUGE-1 F1 compared to BERT-based embeddings 
Although in cases without encoder pretraining, ClinicalBERT performed the best in the Frequency extraction task (by a small margin), in general, it does not perform as well as BERT  This could also be a reflection of the fact that the language and style of writing used in clinical notes is very different from the way doctors converse with patients and the embedding dimension difference  Lookup table embedding performed decently in the frequency extraction task but lags behind in the Dosage extraction task 
From the metrics and qualitative inspection, we find that the Frequency extraction is an easier task than the Dosage extraction  This is because, in the conversations, frequency information usually occurs in isolation and near the medications, but a medication's dosage can occur 1) near other medication's dosages, 2) with previous dosages (when a dosage for a medication is revised), and 3) after a large number of words from the medication 
Other Variations: Considering various models' performance (without pretraining) and the resource constraint, we choose ELMo and BERT embeddings to analyze the effects of pretraining the encoder  When the network's encoder (and embedding) is pretrained with the summarization task, we 1) see a small decrease in the average number of iterations required for training, 2) improvement in individual performances of all models for both the sub-tasks, and 3) get best performance metrics across all variations, refer to Table TABREF5  Both in terms of performance and the training speed, there is no clear winner between shared and multi-decoder approaches  Medication tagging and data augmentation increase the best-performing model's ROUGE-1 F1 score by $\sim $1 5 for the Dosage extraction task 
We also measure the performance of Multitask Question Answering Network (MQAN) BIBREF18 a QA model trained by the authors on the Decathlon multitask challenge  Since MQAN was not trained to produce the output sequence in our MR tags, it would not be fair to compute ROUGE scores  Instead, we randomly sample the MQAN's predictions from the 10% test dataset and qualitatively evaluate it  From the evaluations, we find that MQAN can not distinguish between frequency and dosage, and mixed the answers  MQAN correctly predicted the dosage for 29 73% and frequency for 24 24% percent of the medications compared to 84 12% and 76 34% for the encoder pretrained BERT QA PGNet model trained on our dataset  This could be because of the difference in the training dataset, domain and the tasks in the Decathlon challenge compared to ours 
Almost all our models perform better than the naive baselines and the ones using lookup table embeddings, and our best performing models outperform them significantly  Among all the variations, the best performing models are ELMo with Multi-decoder (Dosage extraction) and BERT with shared-decoder QA PGNet architecture with pretrained encoder (Frequency extraction)  We choose these two models for our subsequent analysis 
Results and Discussion ::: Breakdown of Performance
We categorize the 10% test dataset into different categories based on the complexity and type of the data and analyze the breakdown of the system's performance in Table TABREF11  We breakdown the Frequency extraction into two categories, 1) None: ground truth Frequency tag is `none', and 2) NN (Not None): ground truth Frequency tag is not `none'  Similarly, the Dosage extraction into 5 categories, 1) None: ground truth dosage tag is `none', 2) MM (Multiple Medicine): input segment has more than one Medication mentioned, 3) MN (Multiple Numbers): input segment has more than one number present, and 4) NBM (Number between correct Dosage and Medicine) : between the Medication Name and the correct Dosage in the input segment there are other numbers present  Note that the categories of the Dosage extraction task are not exhaustive, and one tag can belong to multiple categories 
From the performance breakdown of Dosage extraction task, we see that 1) the models predict `none' better than other categories, i e , the models are correctly able to identify when a medication's dosage is absent, 2) there is performance dip in hard cases (MM, MN, and NBM), 3) the models are able to figure out the correct dosage (decently) for a medication even when there are multiple numbers/dosage present, and 4) the model struggles the most in the NBM category  The models' low performance in NBM could be because we have a comparatively lower number of examples to train in this category  The Frequency extraction task performs equally well when the tag is `none` or not  In most categories, we see an increase in performance when using pretrained encoders 
Results and Discussion ::: Training Dataset Size
We vary the number of MR tags used to train the model and analyze the model's performance when training the networks, using publicly available contextual embeddings, compared to using pretrained embeddings and encoder (pretrained on the summarization task)  Out of the 5,016 files in the 80% train dataset only 2,476 have atleast one MR tag  Therefore, out of the 2476 files, we randomly choose 100, 500, and 1000 files and trained the best performing model variations to observe the performance differences, refer to Figure FIGREF12  For all these experiments we used the same vocabulary size (456), the same hyper/training parameters, and the same 10% test split of 627 files 
As expected, we see that the encoder pretrained models have higher performance on all the different training data sizes, i e , they achieve higher performance on a lower number of data points, refer to Figure FIGREF12  The difference, as expected, shrinks as the training data size increases 
Results and Discussion ::: Evaluating on ASR transcripts
To test the performance of our models on real-world conditions, we use commercially available ASR services (Google and IBM) to transcribe the $\mathcal {D}_{test}$ files and measure the performance of our models without assuming any annotations (except when calculating the metrics)  It should be noted that this is not the case in our previous evaluations using `10% test' dataset where we use the segmentation information  For ground truth annotations on ASR transcripts, we aligned the MR tags from human written transcripts to the ASR transcript using their grounded timing information  Additionally, since ASR is prone to errors, during the alignment, if a medication from an MR tag is not recognized correctly in the ASR transcript, we remove the corresponding MR tag 
In our evaluations, we use Google Cloud Speech-to-Text (G-STT) and IBM Watson Speech to Text (IBM-STT) as these were among the top-performing ASR APIs on medical speech BIBREF19 and were readily available to us  We used G-STT, with the `video model' with punctuation settings  Unlike our human written transcripts, the transcript provided by G-STT is not verbatim and does not have disfluencies  IBM-STT, on the other hand, does not give punctuation so we used the speaker changes to add end-of-sentence punctuation 
In our $\mathcal {D}_{test}$ dataset, on initial study we see a Word Error Rate of $\sim $50% for the ASR APIs and this number is not accurate because, 1) of the de-identification, 2) disfluencies (verbatim) difference between the human written and ASR transcript, and 3) minor alignment differences between the audio and the ground truth transcript 
During this evaluation, we followed the same preprocessing methods we used during training  Then, we auto segment the transcript into small contiguous segments similar to the grounded sentences in the annotations for tags extraction  To segment the transcript, we follow a simple procedure  First, we detected all the medications in a transcript using RxNorm BIBREF20 via string matching  For all the detected medications, we selected $2 \le x \le 5$ nearby sentences as the input to our model  We increased $x$ iteratively until we encountered a quantity entity – detected using spaCy's entity recognizer, and we set $x$ as 2 if we did not detect any entities in the range 
We show the model's performance on ASR transcripts and human written transcripts with automatic segmentation, and human written transcripts with human (defined) segmentation, in Table TABREF18  The number of recognized medications in IBM-STT is only 95 compared to 725 (human written), we mainly consider the models' performance on G-STT's transcripts (343) 
On the Medications that were recognized correctly, the models can perform decently on ASR transcripts in comparison to human transcripts (within 5 points ROUGE-1 F1 for both tasks, refer to Table TABREF18)  This shows that the models are robust to ASR variations discussed above  The lower performance compared to human transcripts is mainly due to incorrect recognition of Dosage and other medications in the same segments (changing the meaning of the text)  By comparing the performance of the model on the human written transcripts with human (defined) segmentation and the same with auto segmentation, we see a 10 point drop in Dosage and 6 point drop in Frequency extraction tasks  This points out the need for more sophisticated segmentation algorithms 
With G-STT, our best model obtained ROUGE-1 F1 of 71 75 (which equals to percentage of times dosage is correct in this case) for Dosage extraction and 40 13 for Frequency extraction tasks  To measure the percentage of times the correct frequency was extracted by the model, we qualitatively compared the extracted and predicted frequency  We find that for 73 58% of the medications the model can find the correct frequency from the transcripts 
Conclusion
In this paper, we explore the Medication Regimen (MR) extraction task of extracting dosage and frequency for the medications mentioned in a doctor-patient conversation transcript  We explore different variations of abstractive QA models and new architecture in the intersection of QA and IE frameworks and provide a comparative performance analysis of the methods along with other techniques like pretraining to improve the overall performance  Finally, we demonstrate the performance of our best-performing models by automatically extracting MR tags from spontaneous doctor-patient conversations (using commercially available ASR)  Our best model can correctly extract the dosage for 71 75% (interpretation of ROUGE-1 score) and frequency for 73 58% (on qualitative evaluation) of the medications discussed in the transcripts generated using Google Speech-To-Text  In summary, we demonstrate that research on NLP can be translated into real-world clinical settings to realize its benefits for both doctors and patients 
Using ASR transcripts in our training process to improve our performance on both the tasks and extending the medication regimen extraction network to extract other important medical information can be interesting lines of future work 
Acknowledgements
We thank: University of Pittsburgh Medical Center (UPMC), and Abridge AI Inc  for providing access to the de-identified data corpus; Dr  Shivdev Rao, a faculty member and practicing cardiologist in UPMC's Heart and Vascular Institute and Prof  Florian Metze, Associate Research Professor, Carnegie Mellon University for helpful discussions; Ben Schloss, Steven Coleman, and Deborah Osakue for data business development and annotation management 
Context 3: Introduction
Emotion detection has long been a topic of interest to scholars in natural language processing (NLP) domain  Researchers aim to recognize the emotion behind the text and distribute similar ones into the same group  Establishing an emotion classifier can not only understand each user's feeling but also be extended to various application, for example, the motivation behind a user's interests BIBREF0  Based on releasing of large text corpus on social media and the emotion categories proposed by BIBREF1, BIBREF2, numerous models have provided and achieved fabulous precision so far  For example, DeepMoji BIBREF3 which utilized transfer learning concept to enhance emotions and sarcasm understanding behind the target sentence  CARER BIBREF4 learned contextualized affect representations to make itself more sensitive to rare words and the scenario behind the texts 
As methods become mature, text-based emotion detecting applications can be extended from a single utterance to a dialogue contributed by a series of utterances  Table TABREF2 illustrates the difference between single utterance and dialogue emotion recognition  The same utterances in Table TABREF2, even the same person said the same sentence, the emotion it convey may be various, which may depend on different background of the conversation, tone of speaking or personality  Therefore, for emotion detection, the information from preceding utterances in a conversation is relatively critical 
In SocialNLP 2019 EmotionX, the challenge is to recognize emotions for all utterances in EmotionLines dataset, a dataset consists of dialogues  According to the needs for considering context at the same time, we develop two classification models, inspired by bidirectional encoder representations from transformers (BERT) BIBREF5, FriendsBERT and ChatBERT  In this paper, we introduce our approaches including causal utterance modeling, model pre-training, and fine-turning 
Dataset
EmotionLines BIBREF6 is a dialogue dataset composed of two subsets, Friends and EmotionPush, according to the source of the dialogues  The former comes from the scripts of the Friends TV sitcom  The other is made up of Facebook messenger chats  Each subset includes $1,000$ English dialogues, and each dialogue can be further divided into a few consecutive utterances  All the utterances are annotated by five annotators on a crowd-sourcing platform (Amazon Mechanical Turk), and the labeling work is only based on the textual content  Annotator votes for one of the seven emotions, namely Ekman’s six basic emotions BIBREF1, plus the neutral  If none of the emotion gets more than three votes, the utterance will be marked as “non-neutral” 
For the datasets, there are properties worth additional mentioning  Although Friends and EmotionPush share the same data format, they are quite different in nature  Friends is a speech-based dataset which is annotated dialogues from the TV sitcom  It means most of the utterances are generated by the a few main characters  The personality of a character often affects the way of speaking, and therefore “who is the speaker" might provide extra clues for emotion prediction  In contrast, EmotionPush does not have this trait due to the anonymous mechanism  In addition, features such as typo, hyperlink, and emoji that only appear in chat-based data will need some domain-specific techniques to process 
Incidentally, the objective of the challenge is to predict the emotion for each utterance  Just, according to EmotionX 2019 specification, there are only four emotions be selected as our label candidates, which are Joy, Sadness, Anger, and Neutral  These emotions will be considered during performance evaluation  The technical detail will also be introduced and discussed in following Section SECREF13 and Section SECREF26 
Model Description
For this challenge, we adapt BERT which is proposed by BIBREF5 to help understand the context at the same time  Technically, BERT, designed on end-to-end architecture, is a deep pre-trained transformer encoder that dynamically provides language representation and BERT already achieved multiple state-of-the-art results on GLUE benchmark BIBREF7 and many tasks  A quick recap for BERT's architecture and its pre-training tasks will be illustrated in the following subsections 
Model Description ::: Model Architecture
BERT, the Bidirectional Encoder Representations from Transformers, consists of several transformer encoder layers that enable the model to extract very deep language features on both token-level and sentence-level  Each transformer encoder contains multi-head self-attention layers that provide ability to learn multiple attention feature of each word from their bidirectional context  The transformer and its self-attention mechanism are proposed by BIBREF8  This self-attention mechanism can be interpreted as a key-value mapping given query  By given the embedding vector for token input, the query ($Q$), key ($K$) and value ($V$) are produced by the projection from each three parameter matrices where $W^Q \in \mathbb {R}^{d_{{\rm model}} \times d_{k}}, W^K \in \mathbb {R}^{d_{\rm model} \times d_{k}}$ and $W^V \in \mathbb {R}^{d_{\rm model} \times d_{v}}$  The self-attention BIBREF8 is formally represented as:
The $ d_k = d_v = d_{\rm model} = 1024$ in BERT large version and 768 in BERT base version  Once model can extract attention feature, we can extend one self-attention into multi-head self-attention, this extension makes sub-space features can be extracted in same time by this multi-head configuration  Overall, the multi-attention mechanism is adopt for each transformer encoder, and several of encoder layer will be stacked together to form a deep transformer encoder 
For the model input, BERT allow us take one sentence as input sequence or two sentences together as one input sequence, and the maximum length of input sequence is 512  The way that BERT was designed is for giving model the sentence-level and token-level understanding  In two sentences case, a special token ([SEP]) will be inserted between two sentences  In addition, the first input token is also a special token ([CLS]), and its corresponding ouput will be vector place for classification during fine-tuning  The outputs of the last encoder layer corresponding to each input token can be treated as word representations for each token, and the word representation of the first token ([CLS]) will be consider as classification (output) representation for further fine-tuning tasks  In BERT, this vector is denoted as $C \in \mathbb {R}^{d_{\rm model}} $, and a classification layer is denoted as $ W \in \mathbb {R}^{K \times d_{\rm model}}$, where $K$ is number of classification labels  Finally, the prediction $P$ of BERT is represented as $P = {\rm softmax}(CW^T)$ 
Model Description ::: Pre-training Tasks
In pre-training, intead of using unidirectional language models, BERT developed two pre-training tasks: (1) Masked LM (cloze test) and (2) Next Sentence Prediction  At the first pre-training task, bidirectional language modeling can be done at this cloze-like pre-training  In detail, 15% tokens of input sequence will be masked at random and model need to predict those masked tokens  The encoder will try to learn contextual representations from every given tokens due to masking tokens at random  Model will not know which part of the input is going to be masked, so that the information of each masked tokens should be inferred by remaining tokens  At Next Sentence Prediction, two sentences concatenated together will be considered as model input  In order to give model a good nature language understanding, knowing relationship between sentence is one of important abilities  When generating input sequences, 50% of time the sentence B is actually followed by sentence A, and rest 50% of the time the sentence B will be picked randomly from dataset, and model need to predict if the sentence B is next sentence of sentence A  That is, the attention information will be shared between sentences  Such sentence-level understanding may have difficulties to be learned at first pre-training task (Masked LM), therefore, the pre-training task (NSP) is developed as second training goal to capture the cross sentence relationship 
In this competition, limited by the size of dataset and the challenge in contextual emotion recognition, we consider BERT with both two pre-training tasks can give a good starting point to extract emotion changing during dialogue-like conversation  Especially the second pre-training task, it might be more important for dialogue-like conversation where the emotion may various by the context of continuous utterances  That is, given a set of continues conversations, the emotion of current utterance might be influenced by previous utterance  By this assumption and with supporting from the experiment results of BERT, we can take sentence A as one-sentence context and consider sentence B as the target sentence for emotion prediction  The detail will be described in Section SECREF4 
Methodology
The main goal of the present work is to predict the emotion of utterance within the dialogue  Following are four major difficulties we concern about:
The emotion of the utterances depends not only on the text but also on the interaction happened earlier 
The source of the two datasets are different  Friends is speech-based dialogues and EmotionPush is chat-based dialogues  It makes datasets possess different characteristics 
There are only $1,000$ dialogues in both training datasets which are not large enough for the stability of training a complex neural-based model 
The prediction targets (emotion labels) are highly unbalanced 
The proposed approach is summarized in Figure FIGREF3, which aims to overcome these challenges  The framework could be separated into three steps and described as follow:
Methodology ::: Causal Utterance Modeling
Given a dialogue $D^{(i)}$ which includes sequence of utterances denoted as $D^{(i)}=(u^{(i)}_{1}, u^{(i)}_{2},    , u^{(i)}_{n})$, where $i$ is the index in dataset and $n$ is the number of utterances in the given dialogue  In order to conserve the emotional information of both utterance and conversation, we rearrange each two consecutive utterances $u_{t}, u_{t-1}$ into a single sentence representation $x_{t}$ as
The corresponding sentence representation corpus $X^{(i)}$ are denoted as $X^{(i)}=(x^{(i)}_{1}, x^{(i)}_{2},    , x^{(i)}_{n})$  Note that the first utterance within a conversation does not have its causal utterance (previous sentence), therefore, the causal utterance will be set as [None]  A practical example of sentence representation is shown in Table TABREF11 
Since the characteristics of two datasets are not identical, we customize different causal utterance modeling strategies to refine the information in text 
For Friends, there are two specific properties  The first one is that most dialogues are surrounding with the six main characters, including Rachel, Monica, Phoebe, Joey, Chandler, and Ross  The utterance ratio of given by the six roles is up to $83 4\%$  Second, the personal characteristics of the six characters are very clear  Each leading role has its own emotion undulated rule  To make use of these features, we introduce the personality tokenization which help learning the personality of the six characters  Personality tokenization concatenate the speaker and says tokens before the input utterance if the speaker is one of the six characters  The example is shown in Table TABREF12 
For EmotionPush, the text are informal chats which including like slang, acronym, typo, hyperlink, and emoji  Another characteristic is that the specific name entities are tokenized with random index  (e g  “organization_80”, “person_01”, and “time_12”)  We consider some of these informal text are related to expressing emotion such as repeated typing, purposed capitalization, and emoji (e g  “:D”, “:(”, and “<3”))  Therefore, we keep most informal expressions but only process hyperlinks, empty utterance, and name entities by unifying the tokens 
Methodology ::: Model Pre-training
Since the size of both datasets are not large enough for complex neural-based model training as well as BERT model is only pre-train on formal text datasets, the issues of overfitting and domain bias are important considerations for design the pre-training process 
To avoid our model overfitting on the training data and increase the understanding of informal text, we adapted BERT and derived two models, namely FriendsBERT and ChatBERT, with different pre-training tasks before the formal training process for Friends and EmotionPush dataset, respectively  The pre-training strategies are described below 
For pre-training FriendsBERT, we collect the completed scripts of all ten seasons of Friends TV shows from emorynlp which includes 3,107 scenes within 61,309 utterances  All the utterances are followed the preprocessing methods mentions above to compose the corpus for Masked language model pre-training task  The consequent utterances in the same scenes are considered as the consequent sentences to pre-train the Next Sentence Prediction task  In the pre-training process, the training loss is the sum of the mean likelihood of two pre-train tasks 
For pre-training ChatBERT, we pre-train our model on the Twitter dataset, since the text and writing style on Twitter are close to the chat text where both may involved with many informal words or emoticons as well  The Twitter emotion dataset, 8 basic emotions from emotion wheel BIBREF1, was collected by twitter streaming API with specific emotion-related hashtags, such as #anger, #joy, #cry, #sad and etc  The hashtags in tweets are treated as emotion label for model fine-tuning  The tweets were fine-grined processing followed the rules in BIBREF9, BIBREF4, including duplicate tweets removing, the emotion hashtags must appearing in the last position of a tweet, and etc  The statis of tweets were summarized in Table TABREF17  Each tweet and corresponding emotion label composes an emotion classification dataset for pre-training 
Methodology ::: Fine-tuning
Since our emotion recognition task is treated as a sequence-level classification task, the model would be fine-tuned on the processed training data  Following the BERT construction, we take the first embedding vector which corresponds to the special token [CLS] from the final hidden state of the Transformer encoder  This vector represents the embedding vector of the corresponding conversation utterances which is denoted as $\mathbf {C} \in \mathbb {R}^{H}$, where $H$ is the embedding size  A dense neural layer is treated as a classification layer which consists of parameters $\mathbf {W} \in \mathbb {R}^{K\times H}$ and $\mathbf {b} \in \mathbb {R}^{K}$, where $K$ is the number of emotion class  The emotion prediction probabilities $\mathbf {P} \in \mathbb {R}^{K}$ are computed by a softmax activation function as
All the parameters in BERT and the classification layer would be fine-turned together to minimize the Negative Log Likelihood (NLL) loss function, as Equation (DISPLAY_FORM22), based on the ground truth emotion label $c$ 
In order to tackle the problem of highly unbalanced emotion labels, we apply weighted balanced warming on NLL loss function, as Equation (DISPLAY_FORM23), in the first epoch of fine-tuning procedure 
where $\mathbf {w}$ are the weights of corresponding emotion label $c$ which are computed and normalize by the frequency as
By adding the weighted balanced warming on NLL loss, the model could learn to predict the minor emotions (e g  anger and sadness) earlier and make the training process more stable  Since the major evaluation metrics micro F1-score is effect by the number of each label, we only apply the weighted balanced warming in first epoch to optimize the performance 
Experiments
Since the EmotionX challenge only provided the gold labels in training data, we pick the best performance model (weights) to predict the testing data  In this section, we present the experiment and evaluation results 
Experiments ::: Experimental Setup
The EmotionX challenge consists of $1,000$ dialogues for both Friends and EmotionPush  In all of our experiments, each dataset is separated into top 800 dialogues for training and last 200 dialogues for validation  Since the EmotionX challenge considers only the four emotions (anger, joy, neutral, and sadness) in the evaluation stage, we ignore all the data point corresponding to other emotions directly  The details of emotions distribution are shown in Table TABREF18 
The hyperparameters and training setup of our models (FriendsBERT and ChatBERT) are shown in Table TABREF25  Some common and easily implemented methods are selected as the baselines embedding methods and classification models  The baseline embedding methods are including bag-of-words (BOW), term frequency–inverse document frequency (TFIDF), and neural-based word embedding  The classification models are including Logistic Regression (LR), Random Forest (RF), TextCNN BIBREF10 with initial word embedding as GloVe BIBREF11, and our proposed model  All the experiment results are based on the best performances of validation results 
Experiments ::: Performance
The experiment results of validation on Friends are shown in Table TABREF19  The proposed model and baselines are evaluated based on the Precision (P ), Recall (R ), and F1-measure (F1) 
For the traditional baselines, namely BOW and TFIDF, we observe that they achieve surprising high F1 scores around $0 81$, however, the scores for Anger and Sadness are lower  This explains that traditional approaches tend to predict the labels with large sample size, such as Joy and Neutral, but fail to take of scarce samples even when an ensemble random forest classifier is adopted  In order to prevent the unbalanced learning, we choose the weighted loss mechanism for both TextCNN and causal modeling TextCNN (C-TextCNN), these models suffer less than the traditional baselines and achieve a slightly balance performance, where there are around 15% and 7% improvement on Anger and Sadness, respectively  We following adopt the casual utterance modeling to original TextCNN, mapping previous utterance as well as target utterance into model  The causal utterance modeling improve the C-TextCNN over TextCNN for 6%, 2% and 1% on Anger, Joy and overall F1 score  Motivated from these preliminary experiments, the proposed FriendsBERT also adopt the ideas of both weighted loss and causal utterance modeling  As compared to the original BERT, single sentence BERT (FriendsBERT-base-s), the proposed FriendsBERT-base improve 1% for Joy and overall F1, and 2% for Sadness  For the final validation performance, our proposed approach achieves the highest scores, which are $0 85$ and $0 86$ for FriendsBERT-base and FriendsBERT-large, respectively 
Overall, the proposed FriendsBERT successfully captures the sentence-level context-awarded information and outperforms all the baselines, which not only achieves high performance on large sample labels, but also on small sample labels  The similar settings are also adapted to EmotionPush dataset for the final evaluation 
Experiments ::: Evaluation Results
The testing dataset consists of 240 dialogues including $3,296$ and $3,536$ utterances in Friends and EmotionPush respectively  We re-train our FriendsBERT and ChatBERT with top 920 training dialogues and predict the evaluation results using the model performing the best validation results  The results are shown in Table TABREF29 and Table TABREF30  The present method achieves $81 5\%$ and $88 5\%$ micro F1-score on the testing dataset of Friends and EmotionPush, respectively 
Conclusion and Future work
In the present work, we propose FriendsBERT and ChatBERT for the multi-utterance emotion recognition task on EmotionLines dataset  The proposed models are adapted from BERT BIBREF5 with three main improvement during the model training procedure, which are the causal utterance modeling mechanism, specific model pre-training, and adapt weighted loss  The causal utterance modeling takes the advantages of the sentence-level context information during model inference  The specific model pre-training helps to against the bias in different text domain  The weighted loss avoids our model to only predict on large size sample  The effectiveness and generalizability of the proposed methods are demonstrated from the experiments 
In future work, we consider to include the conditional probabilistic constraint $P ({\rm Emo}_{B} | \hat{\rm Emo}_{A})$  Model should predict the emotion based on a certain understanding about context emotions  This might be more reasonable for guiding model than just predicting emotion of ${\rm Sentence}_B$ directly  In addition, due to the limitation of BERT input format, ambiguous number of input sentences is now becoming an important design requirement for our future work  Also, personality embedding development will be another future work of the emotion recognition  The personality embedding will be considered as sentence embedding injected into word embedding, and it seems this additional information can contribute some improvement potentially 

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the sources of the datasets?
Context 4:  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 

ground_truth: Friends TV sitcom, Facebook messenger chats