prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is the data de-identified?
Context 1: Introduction
To appear in Proceedings of International Workshop on Health Intelligence (W3PHIAI) of the 34th AAAI Conference on Artificial Intelligence, 2020 
Physician burnout is a growing concern, estimated to be experienced by at least 35% of physicians in the developing world and 50% in the United States BIBREF0  BIBREF1 found that for every hour physicians provide direct clinical facetime to patients, nearly two additional hours are spent on EHR (Electronic Health Records) and administrative or desk work  As per the study conducted by Massachusetts General Physicians Organization (MPGO) BIBREF2 and as reported by BIBREF3, the average time spent on administrative tasks increased from 23 7% in 2014 to 27 9% in 2017  Both the surveys found that time spent on administrative tasks was positively associated with higher likelihood of burnout  Top reasons under administrative burden include working on the ambulatory EHR, handling medication reconciliation (sometimes done by aids), medication renewals, and medical billing and coding  The majority of these reasons revolve around documentation of information exchanged between doctors and patients during the clinical encounters  Automatically extracting such clinical information BIBREF4, BIBREF5 can not only help alleviate the documentation burden on the physician, but also allow them to dedicate more time directly with patients 
Among all the clinical information extraction tasks, Medication Regimen (Medication, dosage, and frequency) extraction is particularly interesting due to its ability to help doctors with medication orders cum renewals, medication reconciliation, potentially verifying the reconciliations for errors, and, other medication-centered EHR documentation tasks  In addition, the same information when provided to patients can help them with better recall of doctor instructions which might aid in compliance with the care plan  This is particularly important given that patients forget or wrongly recollect 40-80% BIBREF6 of what is discussed in the clinic, and accessing EHR data has its own challenges 
Spontaneous clinical conversations happening between a doctor and a patient, have several distinguishing characteristics from a normal monologue or prepared speech: it involves multiple speakers with overlapping dialogues, covers a variety of speech patterns, and the vocabulary can range from colloquial to complex domain-specific language  With recent advancements in Conversational Speech Recognition BIBREF7 rendering the systems less prone to errors, the subsequent challenge of understanding and extracting relevant information from the conversations is receiving increasing research focus BIBREF4, BIBREF8 
In this paper, we focus on local information extraction in transcribed clinical conversations  Specifically, we extract dosage (e g  5mg) and frequency (e g  once a day) for the medications (e g  aspirin) from these transcripts, collectively referred to as Medication Regimen (MR) extraction  The information extraction is local as we extract the information from a segment of the transcript and not the entire transcript since doing the latter is difficult owing to the long meandering nature of the conversations often with multiple medication regimens and care plans being discussed 
The challenges associated with the Medication Regimen (MR) extraction task include understanding the spontaneous dialog with clinical vocabulary and understanding the relationship between different entities as the discussion can contain multiple medications and dosages (e g  doctor revising a dosage or reviewing all the current medications) 
We frame this problem as a Question Answering (QA) task by generating questions using templates  We base the QA model on pointer-generator networks BIBREF9 augmented with Co-Attentions BIBREF10  In addition, we develop models combining QA and Information Extraction frameworks using multi-decoder (one each for dosage and frequency) architecture 
Lack of availability of a large volume of data is a typical challenge in healthcare  A conversation corpus by itself is a rare commodity in the healthcare data space because of the cost and difficulty in handing (because of data privacy concerns)  Moreover, transcribing and labeling the conversations is a costly process as it requires domain-specific medical annotation expertise  To address data shortage and improve the model performance, we investigate different high-performance contextual embeddings (ELMO BIBREF11, BERT BIBREF12 and ClinicalBERT BIBREF13), and pretrain the models on a clinical summarization task  We further investigate the effects of training data size on our models 
On the MR extraction task, ELMo with encoder multi-decoder architecture and BERT with encoder-decoder with encoders pretrained on the summarization task perform the best  The best-performing models improve our baseline's dosage and frequency extractions ROUGE-1 F1 scores from 54 28 and 37 13 to 89 57 and 45 94, respectively 
Using our models, we present the first fully automated system to extract MR tags from spontaneous doctor-patient conversations  We evaluate the system (using our best performing models) on the transcripts generated from Automatic Speech Recognition (ASR) APIs offered by Google and IBM  In Google ASR's transcripts, our best model obtained ROUGE-1 F1 of 71 75 for Dosage extraction (which in this specific case equals to the percentage of times dosage is correct, refer Metrics Section for more details) and 40 13 for Frequency extraction tasks  On qualitative evaluation, we find that for 73 58% of the medications the model can find the correct frequency  These results demonstrate that the research on NLP can be used effectively in a real clinical setting to benefit both doctors and patients
Data
Our dataset consists of a total of 6,693 real doctor-patient conversations recorded in a clinical setting using distant microphones of varying quality  The recordings have an average duration of 9min 28s and have a verbatim transcript of 1,500 words on average (written by the experts)  Both the audio and the transcript are de-identified (by removing the identifying information) with digital zeros and [de-identified] tags, respectively  The sentences in the transcript are grounded to the audio with the timestamps of its first and last word 
The transcript of the conversations are annotated with summaries and Medication Regimen tags (MR tags), both grounded using the timestamps of the sentences from the transcript deemed relevant by the expert annotators, refer to Table TABREF1  The transcript for a typical conversation can be quite long, and not easy for many of the high performing deep learning models to act on  Moreover, the medical information about a concept/condition/entity can change during the conversation after a significant time gap  For example, dosage of a medication can be different when discussing current medication the patient is on vs when they are prescribed a different dosage  Hence, we have annotations, that are grounded to a short segment of the transcript 
The summaries (#words - $\mu = 9 7; \sigma = 10 1$) are medically relevant and local  The MR tags are also local and are of the form {Medication Name, Dosage, Frequency}  If dosage ($\mu = 2 0; \sigma = 0$) or frequency ($\mu = 2 1; \sigma = 1 07$) information for a medication is not present in a grounded sentence, the corresponding field in the MR tag will be marked as `none' 
In the MR tags, Medication Name and Dosage (usually a quantity followed by its units) can be relatively easily extracted from the transcript except for the units of the dosage which is sometimes inferred  In contrast, due to high degree of linguistic variation with which Frequency is often expressed, extracting Frequency requires an additional inference step  For example, `take one in the morning and at noon' from the transcript is tagged as `twice a day' in the frequency tag, likewise `take it before sleeping' is tagged as `at night time' 
Out of overall 6,693 files, we set aside a random sample of 423 files (denoted as $\mathcal {D}_{test}$) for final evaluation  The remaining 6,270 files are used for training with 80% train (5016), 10% validation (627), and 10% test (627) split  Overall, the 6,270 files contains 156,186 summaries and 32,000 MR tags out of which 8,654 MR tags contain values for at least one of the Dosage or Frequency, which we used for training to avoid overfitting (the remaining MR tags have both Dosage and Frequency as `none')  Note that we have two test datasets: `10% test' - used to evaluate all the models, and $\mathcal {D}_{test}$ - used to measure the performance of best performing models on ASR transcripts 
Approach
We frame the Medication Regimen extraction problem as a Question Answering (QA) task, which forms the basis for our first approach  It can also be considered as a specific inference or relation extract task, since we extract specific information about an entity (Medication Name), hence our second approach is at the intersection of Question Answering (QA) and Information Extraction (IE) domains  Both the approaches involve using a contiguous segment of the transcript and the Medication Name as input, to find/infer the medication's Dosage and Frequency  When testing the approaches mimicking real-world conditions, we extract Medication Name from the transcript separately using ontology, refer to SECREF19 
In the first approach, we frame the MR task as a QA task and generate questions using the template: “What is the $ <$dosage/frequency$>$ for $<$Medication Name$>$"  Here, we use an abstractive QA model based on pointer-generator networks BIBREF9 augmented with coattention encoder BIBREF10 (QA-PGNet) 
In the second approach, we frame the problem as a conditioned IE task, where the information extracted depends on an entity (Medication Name)  Here, we use a multi-decoder pointer-generator network augmented with coattention encoder (Multi-decoder QA-PGNet)  Instead of using templates to generate questions and using a single decoder to extract different types of information as in the QA approach (which might lead to performance degradation), here we consider separate decoders for extracting specific types of information about an entity $E$ (Medication Name) 
Approach ::: Pointer-generator Network (PGNet)
The network is a sequence-to-sequence attention model that can both copy a word from the input $I$ containing $P$ word tokens or generate a word from its vocabulary $vocab$, to produce the output sequence 
First, the tokens of the $I$ are converted to embeddings and are fed one-by-one to the encoder, a single bi-LSTM layer, which encodes the tokens in $I$ into a sequence of hidden states - $H=encoder(I)$, where $ H=[h_1   h_P]$ 
For each decoder time step $t$, in a loop, we compute, 1) attention $a_t$ (using the last decoder state $s_{t-1}$), over the input tokens $I$, and 2) the decoder state $s_t$ using $a_t$  Then, at each time step, using both $a_t$ and $s_t$ we can find the probability $P_t(w)$, of producing a word $w$ (from both $vocab$ and $I$)  For convenience, we denote the attention and the decoder as $decoder_{pg}(H)=P(w)$, where $P(w)=[P_1(w)   P_T(w)]$  The output can then be decoded from $P(w)$, which is decoded until it produces an `end of output token' or the number of steps reach the maximum allowed limit 
Approach ::: QA PGNet
We first encode both the question - $H_Q = encoder(Q)$, and the input - $H_I = encoder(I)$, separately using encoders (with shared weights)  Then, to condition $I$ on $Q$ (and vice versa), we use the coattention encoder BIBREF10 which attends to both the $I$ and $Q$ simultaneously to generates the coattention context - $C_D = coatt(H_I, H_Q)$  Finally, using the pointer-generator decoder we find the probability distribution of the output sequence - $P(w) = decoder_{pg}([H_I; C_D])$, which is then decoded to generate the answer sequence 
Approach ::: Multi-decoder (MD) QA PGNet
After encoding the inputs into $H_I$ and $H_E$, for extracting $K$ types of information about an entity in an IE fashion, we use the following multi-decoder (MD) setup:
Predictions for each of the $K$ decoders are then decoded using $P^k(w)$ 
All the networks discussed above are trained using a negative log-likelihood loss for the target word at each time step and summed over all the decoder time steps 
Experiments
We initialized MR extraction models' vocabulary from the training dataset after removing words with a frequency lower than 30 in the dataset, resulting in 456 words  Our vocabulary is small because of the size of the dataset, hence we rely on the model's ability to copy words to produce the output effectively  In all our model variations, the embedding and the network's hidden dimension are set to be equal  The networks were trained with a learning rate of 0 0015, dropout of 0 5 on the embedding layer, normal gradient clipping set at 2, batch size of 8, and optimized with Adagrad BIBREF15 and the training was stopped using the $10\%$ validation dataset 
Experiments ::: Data Processing
We did the following basic preprocessing to our data, 1) added `none' to the beginning of the input utterance, so the network could point to it when there was no relevant information in the input, 2) filtered outliers with a large number of grounded transcript sentences ($>$150 words), and 3) converted all text to lower case 
To improve performance, we 1) standardized all numbers (both digits and words) to words concatenated with a hyphen (e g  110 -$>$ one-hundred-ten), in both input and output, 2) removed units from Dosage as sometimes the units were not explicitly mentioned in the transcript segment but were written by the annotators using domain knowledge, 3) prepended all medication mentions with `rx-' tag, as this helps model's performance when multiple medications are discussed in a segment (in both input and output), and 4) when a transcript segment has multiple medications or dosages being discussed we randomly shuffle them (in both input and output) and create a new data point, to increases the number of training data points  Randomly shuffling the entities increases the number of training MR tags from 8,654 to 11,521  Based on the data statistics after data processing, we fixed the maximum encoder steps to 100, dosage decoder steps to 1, and frequency decoder steps to 3 (for both the QA and Multi-decoder QA models) 
Experiments ::: Metrics
For the MR extraction task, we measure the ROUGE-1 scores BIBREF14 for both the Dosage and Frequency extraction tasks  It should be noted that since Dosage is a single word token (after processing), both the reference and hypothesis are a single token, making its ROUGE-1 F1, Precision and Recall scores equal and be equal to percentage of times we find the correct dosage for the medications 
In our annotations, Frequency has conflicting tags (e g  {`Once a day', `twice a day'} and `daily'), hence metrics like Exact Match will be erroneous  To address this issue, we use the ROUGE scores to compare different models on the 10% test dataset and we use qualitative evaluation to measure the top-performing models on $\mathcal {D}_{test}$ 
Experiments ::: Model variations
We consider QA PGNet and Multi-decoder QA PGNet with lookup table embedding as baseline models and improve on the baselines with other variations described below 
Apart from learning-based baselines, we also create two naive baselines, one each for the Dosage and Frequency extraction tasks  For Dosage extraction, the baseline we consider is `Nearest Number', where we take the number nearest to the Medication Name as the prediction, and `none' if no number is mentioned or if the Medication Name is not detected in the input  For Frequency extraction, the baseline we consider is `Random Top-3' where we predict a random Frequency tag, from top-3 most frequent ones from our dataset - {`none', `daily', `twice a day'} 
Embedding: We developed different variations of our models with a simple lookup table embeddings learned from scratch and using high-performance contextual embeddings, which are ELMo BIBREF11, BERT BIBREF16 and ClinicalBERT BIBREF13 (trained and provided by the authors)  Refer to Table TABREF5 for the performance comparisons 
We derive embeddings from ELMo by learning a linear combination of its last three layer's hidden states (task-specific fine-tuning BIBREF11)  Similarly, for BERT-based embeddings, we take a linear combination of the hidden states from its last four layers, as this combination performs best without increasing the size of the embeddings BIBREF16  Since BERT and ClinicalBERT use word-piece vocabulary and computes sub-word embeddings, we compute word-level embedding by averaging the corresponding sub-word tokens  ELMo and BERT embeddings both have 1024 dimensions, ClinicalBERT have 768 as it is based on BERT base model, and the lookup table have 128 – higher dimension models leads to overfitting 
Pertaining Encoder: We trained the PGNet as a summarization task using the clinical summaries and used the trained model to initialize the encoders (and the embeddings) of the corresponding QA models  We use a vocab size of 4073 words, derived from the training dataset with a frequency threshold of 30 for the task  We trained the models using Adagrad optimizer with a learning rate of 0 015, normal gradient clipping set at 2 and trained for around 150000 iterations (stopped using validation dataset)  On the summarization task PGNet obtained ROUGE-1 F1 scores of 41 42 with ELMo and 39 15 with BERT embeddings  We compare the effects of pretraining the model in Table: TABREF5, models with `pretrained encoder' had their encoders and embeddings pretrained with the summarization task 
Results and Discussion ::: Difference in networks and approaches
Embeddings: On Dosage extraction, in general, ELMo obtains better performance than BERT, refer to Table TABREF5  This could be because we concatenated the numbers with a hyphen, and as ELMo uses character-level tokens it can learn the tagging better than BERT, a similar observation is also noted in BIBREF17  On the other hand, on Frequency extraction, without pretraining, ELMo's performance is lagging by a big margin of $\sim $8 5 ROUGE-1 F1 compared to BERT-based embeddings 
Although in cases without encoder pretraining, ClinicalBERT performed the best in the Frequency extraction task (by a small margin), in general, it does not perform as well as BERT  This could also be a reflection of the fact that the language and style of writing used in clinical notes is very different from the way doctors converse with patients and the embedding dimension difference  Lookup table embedding performed decently in the frequency extraction task but lags behind in the Dosage extraction task 
From the metrics and qualitative inspection, we find that the Frequency extraction is an easier task than the Dosage extraction  This is because, in the conversations, frequency information usually occurs in isolation and near the medications, but a medication's dosage can occur 1) near other medication's dosages, 2) with previous dosages (when a dosage for a medication is revised), and 3) after a large number of words from the medication 
Other Variations: Considering various models' performance (without pretraining) and the resource constraint, we choose ELMo and BERT embeddings to analyze the effects of pretraining the encoder  When the network's encoder (and embedding) is pretrained with the summarization task, we 1) see a small decrease in the average number of iterations required for training, 2) improvement in individual performances of all models for both the sub-tasks, and 3) get best performance metrics across all variations, refer to Table TABREF5  Both in terms of performance and the training speed, there is no clear winner between shared and multi-decoder approaches  Medication tagging and data augmentation increase the best-performing model's ROUGE-1 F1 score by $\sim $1 5 for the Dosage extraction task 
We also measure the performance of Multitask Question Answering Network (MQAN) BIBREF18 a QA model trained by the authors on the Decathlon multitask challenge  Since MQAN was not trained to produce the output sequence in our MR tags, it would not be fair to compute ROUGE scores  Instead, we randomly sample the MQAN's predictions from the 10% test dataset and qualitatively evaluate it  From the evaluations, we find that MQAN can not distinguish between frequency and dosage, and mixed the answers  MQAN correctly predicted the dosage for 29 73% and frequency for 24 24% percent of the medications compared to 84 12% and 76 34% for the encoder pretrained BERT QA PGNet model trained on our dataset  This could be because of the difference in the training dataset, domain and the tasks in the Decathlon challenge compared to ours 
Almost all our models perform better than the naive baselines and the ones using lookup table embeddings, and our best performing models outperform them significantly  Among all the variations, the best performing models are ELMo with Multi-decoder (Dosage extraction) and BERT with shared-decoder QA PGNet architecture with pretrained encoder (Frequency extraction)  We choose these two models for our subsequent analysis 
Results and Discussion ::: Breakdown of Performance
We categorize the 10% test dataset into different categories based on the complexity and type of the data and analyze the breakdown of the system's performance in Table TABREF11  We breakdown the Frequency extraction into two categories, 1) None: ground truth Frequency tag is `none', and 2) NN (Not None): ground truth Frequency tag is not `none'  Similarly, the Dosage extraction into 5 categories, 1) None: ground truth dosage tag is `none', 2) MM (Multiple Medicine): input segment has more than one Medication mentioned, 3) MN (Multiple Numbers): input segment has more than one number present, and 4) NBM (Number between correct Dosage and Medicine) : between the Medication Name and the correct Dosage in the input segment there are other numbers present  Note that the categories of the Dosage extraction task are not exhaustive, and one tag can belong to multiple categories 
From the performance breakdown of Dosage extraction task, we see that 1) the models predict `none' better than other categories, i e , the models are correctly able to identify when a medication's dosage is absent, 2) there is performance dip in hard cases (MM, MN, and NBM), 3) the models are able to figure out the correct dosage (decently) for a medication even when there are multiple numbers/dosage present, and 4) the model struggles the most in the NBM category  The models' low performance in NBM could be because we have a comparatively lower number of examples to train in this category  The Frequency extraction task performs equally well when the tag is `none` or not  In most categories, we see an increase in performance when using pretrained encoders 
Results and Discussion ::: Training Dataset Size
We vary the number of MR tags used to train the model and analyze the model's performance when training the networks, using publicly available contextual embeddings, compared to using pretrained embeddings and encoder (pretrained on the summarization task)  Out of the 5,016 files in the 80% train dataset only 2,476 have atleast one MR tag  Therefore, out of the 2476 files, we randomly choose 100, 500, and 1000 files and trained the best performing model variations to observe the performance differences, refer to Figure FIGREF12  For all these experiments we used the same vocabulary size (456), the same hyper/training parameters, and the same 10% test split of 627 files 
As expected, we see that the encoder pretrained models have higher performance on all the different training data sizes, i e , they achieve higher performance on a lower number of data points, refer to Figure FIGREF12  The difference, as expected, shrinks as the training data size increases 
Results and Discussion ::: Evaluating on ASR transcripts
To test the performance of our models on real-world conditions, we use commercially available ASR services (Google and IBM) to transcribe the $\mathcal {D}_{test}$ files and measure the performance of our models without assuming any annotations (except when calculating the metrics)  It should be noted that this is not the case in our previous evaluations using `10% test' dataset where we use the segmentation information  For ground truth annotations on ASR transcripts, we aligned the MR tags from human written transcripts to the ASR transcript using their grounded timing information  Additionally, since ASR is prone to errors, during the alignment, if a medication from an MR tag is not recognized correctly in the ASR transcript, we remove the corresponding MR tag 
In our evaluations, we use Google Cloud Speech-to-Text (G-STT) and IBM Watson Speech to Text (IBM-STT) as these were among the top-performing ASR APIs on medical speech BIBREF19 and were readily available to us  We used G-STT, with the `video model' with punctuation settings  Unlike our human written transcripts, the transcript provided by G-STT is not verbatim and does not have disfluencies  IBM-STT, on the other hand, does not give punctuation so we used the speaker changes to add end-of-sentence punctuation 
In our $\mathcal {D}_{test}$ dataset, on initial study we see a Word Error Rate of $\sim $50% for the ASR APIs and this number is not accurate because, 1) of the de-identification, 2) disfluencies (verbatim) difference between the human written and ASR transcript, and 3) minor alignment differences between the audio and the ground truth transcript 
During this evaluation, we followed the same preprocessing methods we used during training  Then, we auto segment the transcript into small contiguous segments similar to the grounded sentences in the annotations for tags extraction  To segment the transcript, we follow a simple procedure  First, we detected all the medications in a transcript using RxNorm BIBREF20 via string matching  For all the detected medications, we selected $2 \le x \le 5$ nearby sentences as the input to our model  We increased $x$ iteratively until we encountered a quantity entity – detected using spaCy's entity recognizer, and we set $x$ as 2 if we did not detect any entities in the range 
We show the model's performance on ASR transcripts and human written transcripts with automatic segmentation, and human written transcripts with human (defined) segmentation, in Table TABREF18  The number of recognized medications in IBM-STT is only 95 compared to 725 (human written), we mainly consider the models' performance on G-STT's transcripts (343) 
On the Medications that were recognized correctly, the models can perform decently on ASR transcripts in comparison to human transcripts (within 5 points ROUGE-1 F1 for both tasks, refer to Table TABREF18)  This shows that the models are robust to ASR variations discussed above  The lower performance compared to human transcripts is mainly due to incorrect recognition of Dosage and other medications in the same segments (changing the meaning of the text)  By comparing the performance of the model on the human written transcripts with human (defined) segmentation and the same with auto segmentation, we see a 10 point drop in Dosage and 6 point drop in Frequency extraction tasks  This points out the need for more sophisticated segmentation algorithms 
With G-STT, our best model obtained ROUGE-1 F1 of 71 75 (which equals to percentage of times dosage is correct in this case) for Dosage extraction and 40 13 for Frequency extraction tasks  To measure the percentage of times the correct frequency was extracted by the model, we qualitatively compared the extracted and predicted frequency  We find that for 73 58% of the medications the model can find the correct frequency from the transcripts 
Conclusion
In this paper, we explore the Medication Regimen (MR) extraction task of extracting dosage and frequency for the medications mentioned in a doctor-patient conversation transcript  We explore different variations of abstractive QA models and new architecture in the intersection of QA and IE frameworks and provide a comparative performance analysis of the methods along with other techniques like pretraining to improve the overall performance  Finally, we demonstrate the performance of our best-performing models by automatically extracting MR tags from spontaneous doctor-patient conversations (using commercially available ASR)  Our best model can correctly extract the dosage for 71 75% (interpretation of ROUGE-1 score) and frequency for 73 58% (on qualitative evaluation) of the medications discussed in the transcripts generated using Google Speech-To-Text  In summary, we demonstrate that research on NLP can be translated into real-world clinical settings to realize its benefits for both doctors and patients 
Using ASR transcripts in our training process to improve our performance on both the tasks and extending the medication regimen extraction network to extract other important medical information can be interesting lines of future work 
Acknowledgements
We thank: University of Pittsburgh Medical Center (UPMC), and Abridge AI Inc  for providing access to the de-identified data corpus; Dr  Shivdev Rao, a faculty member and practicing cardiologist in UPMC's Heart and Vascular Institute and Prof  Florian Metze, Associate Research Professor, Carnegie Mellon University for helpful discussions; Ben Schloss, Steven Coleman, and Deborah Osakue for data business development and annotation management 
Context 2: Background
Teaching machine to read and comprehend a given passage/paragraph and answer its corresponding questions is a challenging task  It is also one of the long-term goals of natural language understanding, and has important applications in e g , building intelligent agents for conversation and customer service support  In a real world setting, it is necessary to judge whether the given questions are answerable given the available knowledge, and then generate correct answers for the ones which are able to infer an answer in the passage or an empty answer (as an unanswerable question) otherwise 
In comparison with many existing MRC systems BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , which extract answers by finding a sub-string in the passages/paragraphs, we propose a model that not only extracts answers but also predicts whether such an answer should exist  Using a multi-task learning approach (c f  BIBREF5 ), we extend the Stochastic Answer Network (SAN) BIBREF1 for MRC answer span detector to include a classifier that whether the question is unanswerable  The unanswerable classifier is a pair-wise classification model BIBREF6 which predicts a label indicating whether the given pair of a passage and a question is unanswerable  The two models share the same lower layer to save the number of parameters, and separate the top layers for different tasks (the span detector and binary classifier) 
Our model is pretty simple and intuitive, yet efficient  Without relying on the large pre-trained language models (ELMo) BIBREF7 , the proposed model achieves competitive results to the state-of-the-art on Stanford Question Answering Dataset (SQuAD) 2 0 
The contribution of this work is summarized as follows  First, we propose a simple yet efficient model for MRC that handles unanswerable questions and is optimized jointly  Second, our model achieves competitive results on SQuAD v2 0 
Model
The Machine Reading Comprehension is a task which takes a question INLINEFORM0 and a passage/paragraph INLINEFORM1 as inputs, and aims to find an answer span INLINEFORM2 in INLINEFORM3   We assume that if the question is answerable, the answer INLINEFORM4 exists in INLINEFORM5 as a contiguous text string; otherwise, INLINEFORM6 is an empty string indicating an unanswerable question  Note that to handle the unanswerable questions, we manually append a dumpy text string NULL at the end of each corresponding passage/paragraph  Formally, the answer is formulated as INLINEFORM7   In case of unanswerable questions, INLINEFORM8 points to the last token of the passage 
Our model is a variation of SAN BIBREF1 , as shown in Figure FIGREF2   The main difference is the additional binary classifier added in the model justifying whether the question is unanswerable  Roughly, the model includes two different layers: the shared layer and task specific layer  The shared layer is almost identical to the lower layers of SAN, which has a lexicon encoding layer, a contextual layer and a memory generation layer  On top of it, there are different answer modules for different tasks  We employ the SAN answer module for the span detector and a one-layer feed forward neural network for the binary classification task  It can also be viewed as a multi-task learning BIBREF8 , BIBREF5 , BIBREF9   We will briefly describe the model from ground up as follows  Detailed descriptions can be found in BIBREF1  
Lexicon Encoding Layer  We map the symbolic/surface feature of INLINEFORM0 and INLINEFORM1 into neural space via word embeddings , 16-dim part-of-speech (POS) tagging embeddings, 8-dim named-entity embeddings and 4-dim hard-rule features  Note that we use small embedding size of POS and NER to reduce model size and they mainly serve the role of coarse-grained word clusters  Additionally, we use question enhanced passages word embeddings which can viewwed as soft matching between questions and passages  At last, we use two separate two-layer position-wise Feed-Forward Networks (FFN) BIBREF11 , BIBREF1 to map both question and passage encodings into the same dimension  As results, we obtain the final lexicon embeddings for the tokens for INLINEFORM2 as a matrix INLINEFORM3 , and tokens in INLINEFORM4 as INLINEFORM5  
Contextual Encoding Layer  A shared two-layers BiLSTM is used on the top to encode the contextual information of both passages and questions  To avoid overfitting, we concatenate a pre-trained 600-dimensional CoVe vectors BIBREF12 trained on German-English machine translation dataset, with the aforementioned lexicon embeddings as the final input of the contextual encoding layer, and also with the output of the first contextual encoding layer as the input of its second encoding layer  Thus, we obtain the final representation of the contextual encoding layer by a concatenation of the outputs of two BiLSTM: INLINEFORM0 for questions and INLINEFORM1 for passages 
Memory Generation Layer  In this layer, we generate a working memory by fusing information from both passages INLINEFORM0 and questions INLINEFORM1   The attention function BIBREF11 is used to compute the similarity score between passages and questions as: INLINEFORM2
Note that INLINEFORM0 and INLINEFORM1 is transformed from INLINEFORM2 and INLINEFORM3 by one layer neural network INLINEFORM4 , respectively  A question-aware passage representation is computed as INLINEFORM5   After that, we use the method of BIBREF13 to apply self attention to the passage: INLINEFORM6
where INLINEFORM0 means that we only drop diagonal elements on the similarity matrix (i e , attention with itself)  At last, INLINEFORM1 and INLINEFORM2 are concatenated and are passed through a BiLSTM to form the final memory: INLINEFORM3  
Span detector  We adopt a multi-turn answer module for the span detector BIBREF1   Formally, at time step INLINEFORM0 in the range of INLINEFORM1 , the state is defined by INLINEFORM2   The initial state INLINEFORM3 is the summary of the INLINEFORM4 : INLINEFORM5 , where INLINEFORM6   Here, INLINEFORM7 is computed from the previous state INLINEFORM8 and memory INLINEFORM9 : INLINEFORM10 and INLINEFORM11   Finally, a bilinear function is used to find the begin and end point of answer spans at each reasoning step INLINEFORM12 : DISPLAYFORM0 DISPLAYFORM1
The final prediction is the average of each time step: INLINEFORM0   We randomly apply dropout on the step level in each time step during training, as done in BIBREF1  
Unanswerable classifier  We adopt a one-layer neural network as our unanswerable binary classifier: DISPLAYFORM0
, where INLINEFORM0 is the summary of the memory: INLINEFORM1 , where INLINEFORM2   INLINEFORM3 denotes the probability of the question which is unanswerable 
Objective The objective function of the joint model has two parts: DISPLAYFORM0
Following BIBREF0 , the span loss function is defined: DISPLAYFORM0
The objective function of the binary classifier is defined: DISPLAYFORM0
where INLINEFORM0 is a binary variable: INLINEFORM1 indicates the question is unanswerable and INLINEFORM2 denotes the question is answerable 
Setup
We evaluate our system on SQuAD 2 0 dataset BIBREF14 , a new MRC dataset which is a combination of Stanford Question Answering Dataset (SQuAD) 1 0 BIBREF15 and additional unanswerable question-answer pairs  The answerable pairs are around 100K; while the unanswerable questions are around 53K  This dataset contains about 23K passages and they come from approximately 500 Wikipedia articles  All the questions and answers are obtained by crowd-sourcing  Two evaluation metrics are used: Exact Match (EM) and Macro-averaged F1 score (F1) BIBREF14  
Implementation details
We utilize spaCy tool to tokenize the both passages and questions, and generate lemma, part-of-speech and named entity tags  The word embeddings are initialized with pre-trained 300-dimensional GloVe BIBREF10   A 2-layer BiLSTM is used encoding the contextual information of both questions and passages  Regarding the hidden size of our model, we search greedily among INLINEFORM0   During training, Adamax BIBREF16 is used as our optimizer  The min-batch size is set to 32  The learning rate is initialized to 0 002 and it is halved after every 10 epochs  The dropout rate is set to 0 1  To prevent overfitting, we also randomly set 0 5% words in both passages and questions as unknown words during the training  Here, we use a special token unk to indicate a word which doesn't appear in GloVe  INLINEFORM1 in Eq EQREF9 is set to 1 
Results
We would like to investigate effectiveness the proposed joint model  To do so, the same shared layer/architecture is employed in the following variants of the proposed model:
The results in terms of EM and F1 is summarized in Table TABREF20   We observe that Joint SAN outperforms the SAN baseline with a large margin, e g , 67 89 vs 69 27 (+1 38) and 70 68 vs 72 20 (+1 52) in terms of EM and F1 scores respectively, so it demonstrates the effectiveness of the joint optimization  By incorporating the output information of classifier into Joint SAN, it obtains a slight improvement, e g , 72 2 vs 72 66 (+0 46) in terms of F1 score  By analyzing the results, we found that in most cases when our model extract an NULL string answer, the classifier also predicts it as an unanswerable question with a high probability 
Table TABREF21 reports comparison results in literature published   Our model achieves state-of-the-art on development dataset in setting without pre-trained large language model (ELMo)  Comparing with the much complicated model R M -Reader + Verifier, which includes several components, our model still outperforms by 0 7 in terms of F1 score  Furthermore, we observe that ELMo gives a great boosting on the performance, e g , 2 8 points in terms of F1 for DocQA  This encourages us to incorporate ELMo into our model in future 
Analysis  To better understand our model, we analyze the accuracy of the classifier in our joint model  We obtain 75 3 classification accuracy on the development with the threshold 0 5  By increasing value of INLINEFORM0 in Eq EQREF9 , the classification accuracy reached to 76 8 ( INLINEFORM1 ), however the final results of our model only have a small improvement (+0 2 in terms of F1 score)  It shows that it is important to make balance between these two components: the span detector and unanswerable classifier 
Conclusion
To sum up, we proposed a simple yet efficient model based on SAN  It showed that the joint learning algorithm boosted the performance on SQuAD 2 0  We also would like to incorporate ELMo into our model in future 
Acknowledgments
We thank Yichong Xu, Shuohang Wang and Sheng Zhang for valuable discussions and comments  We also thank Robin Jia for the help on SQuAD evaluations 

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use attention?
Context 3: Introduction
Social media with abundant user-generated posts provide a rich platform for understanding events, opinions and preferences of groups and individuals  These insights are primarily hidden in unstructured forms of social media posts, such as in free-form text or images without tags  Named entity recognition (NER), the task of recognizing named entities from free-form text, is thus a critical step for building structural information, allowing for its use in personalized assistance, recommendations, advertisement, etc 
While many previous approaches BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 on NER have shown success for well-formed text in recognizing named entities via word context resolution (e g  LSTM with word embeddings) combined with character-level features (e g  CharLSTM/CNN), several additional challenges remain for recognizing named entities from extremely short and coarse text found in social media posts  For instance, short social media posts often do not provide enough textual contexts to resolve polysemous entities (e g  “monopoly is da best ", where `monopoly' may refer to a board game (named entity) or a term in economics)  In addition, noisy text includes a huge number of unknown tokens due to inconsistent lexical notations and frequent mentions of various newly trending entities (e g  “xoxo Marshmelloooo ", where `Marshmelloooo' is a mis-spelling of a known entity `Marshmello', a music producer), making word embeddings based neural networks NER models vulnerable 
To address the challenges above for social media posts, we build upon the state-of-the-art neural architecture for NER with the following two novel approaches (Figure FIGREF1 )  First, we propose to leverage auxiliary modalities for additional context resolution of entities  For example, many popular social media platforms now provide ways to compose a post in multiple modalities - specifically image and text (e g  Snapchat captions, Twitter posts with image URLs), from which we can obtain additional context for understanding posts  While “monopoly" in the previous example is ambiguous in its textual form, an accompanying snap image of a board game can help disambiguate among polysemous entities, thereby correctly recognizing it as a named entity 
Second, we also propose a general modality attention module which chooses per decoding step the most informative modality among available ones (in our case, word embeddings, character embeddings, or visual features) to extract context from  For example, the modality attention module lets the decoder attenuate the word-level signals for unknown word tokens (“Marshmellooooo" with trailing `o's) and amplifies character-level features intsead (capitalized first letter, lexical similarity to other known named entity token `Marshmello', etc ), thereby suppressing noise information (“UNK" token embedding) in decoding steps  Note that most of the previous literature in NER or other NLP tasks combine word and character-level information with naive concatenation, which is vulnerable to noisy social media posts  When an auxiliary image is available, the modality attention module determines to amplify this visual context in disambiguating polysemous entities, or to attenuate visual contexts when they are irrelevant to target named entities, selfies, etc  Note that the proposed modality attention module is distinct from how attention is used in other sequence-to-sequence literature (e g  attending to a specific token within an input sequence)  Section SECREF2 provides the detailed literature review 
Our contributions are three-fold: we propose (1) an LSTM-CNN hybrid multimodal NER network that takes as input both image and text for recognition of a named entity in text input  To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks  (2) We propose a general modality attention module that selectively chooses modalities to extract primary context from, maximizing information gain and suppressing irrelevant contexts from each modality (we treat words, characters, and images as separate modalities)  (3) We show that the proposed approaches outperform the state-of-the-art NER models (both with and without using additional visual contexts) on our new MNER dataset SnapCaptions, a large collection of informal and extremely short social media posts paired with unique images 
Related Work
Neural models for NER have been recently proposed, producing state-of-the-art performance on standard NER tasks  For example, some of the end-to-end NER systems BIBREF4 , BIBREF2 , BIBREF3 , BIBREF0 , BIBREF1 use a recurrent neural network usually with a CRF BIBREF5 , BIBREF6 for sequence labeling, accompanied with feature extractors for words and characters (CNN, LSTMs, etc ), and achieve the state-of-the-art performance mostly without any use of gazetteers information  Note that most of these work aggregate textual contexts via concatenation of word embeddings and character embeddings  Recently, several work have addressed the NER task specifically on noisy short text segments such as Tweets, etc  BIBREF7 , BIBREF8   They report performance gains from leveraging external sources of information such as lexical information (POS tags, etc ) and/or from several preprocessing steps (token substitution, etc )  Our model builds upon these state-of-the-art neural models for NER tasks, and improves the model in two critical ways: (1) incorporation of visual contexts to provide auxiliary information for short media posts, and (2) addition of the modality attention module, which better incorporates word embeddings and character embeddings, especially when there are many missing tokens in the given word embedding matrix  Note that we do not explore the use of gazetteers information or other auxiliary information (POS tags, etc ) BIBREF9 as it is not the focus of our study 
Attention modules are widely applied in several deep learning tasks BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13   For example, they use an attention module to attend to a subset within a single input (a part/region of an image, a specific token in an input sequence of tokens, etc ) at each decoding step in an encoder-decoder framework for image captioning tasks, etc  BIBREF14 explore various attention mechanisms in NLP tasks, but do not incorporate visual components or investigate the impact of such models on noisy social media data  BIBREF15 propose to use attention for a subset of discrete source samples in transfer learning settings  Our modality attention differs from the previous approaches in that we attenuate or amplifies each modality input as a whole among multiple available modalities, and that we use the attention mechanism essentially to map heterogeneous modalities in a single joint embedding space  Our approach also allows for re-use of the same model for predicting labels even when some of the modalities are missing in input, as other modalities would still preserve the same semantics in the embeddings space 
Multimodal learning is studied in various domains and applications, aimed at building a joint model that extracts contextual information from multiple modalities (views) of parallel datasets 
The most relevant task to our multimodal NER system is the task of multimodal machine translation BIBREF16 , BIBREF17 , which aims at building a better machine translation system by taking as input a sentence in a source language as well as a corresponding image  Several standard sequence-to-sequence architectures are explored (a target-language LSTM decoder that takes as input an image first)  Other previous literature include study of Canonical Correlation Analysis (CCA) BIBREF18 to learn feature correlations among multiple modalities, which is widely used in many applications  Other applications include image captioning BIBREF10 , audio-visual recognition BIBREF19 , visual question answering systems BIBREF20 , etc 
To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks 
Proposed Methods
Figure FIGREF2 illustrates the proposed multimodal NER (MNER) model  First, we obtain word embeddings, character embeddings, and visual features (Section SECREF3 )  A Bi-LSTM-CRF model then takes as input a sequence of tokens, each of which comprises a word token, a character sequence, and an image, in their respective representation (Section SECREF4 )  At each decoding step, representations from each modality are combined via the modality attention module to produce an entity label for each token ( SECREF5 )  We formulate each component of the model in the following subsections 
Notations: Let INLINEFORM0 a sequence of input tokens with length INLINEFORM1 , with a corresponding label sequence INLINEFORM2 indicating named entities (e g  in standard BIO formats)  Each input token is composed of three modalities: INLINEFORM3 for word embeddings, character embeddings, and visual embeddings representations, respectively 
Features
Similar to the state-of-the-art NER approaches BIBREF0 , BIBREF1 , BIBREF8 , BIBREF4 , BIBREF2 , BIBREF3 , we use both word embeddings and character embeddings 
Word embeddings are obtained from an unsupervised learning model that learns co-occurrence statistics of words from a large external corpus, yielding word embeddings as distributional semantics BIBREF21   Specifically, we use pre-trained embeddings from GloVE BIBREF22  
Character embeddings are obtained from a Bi-LSTM which takes as input a sequence of characters of each token, similarly to BIBREF0   An alternative approach for obtaining character embeddings is using a convolutional neural network as in BIBREF1 , but we find that Bi-LSTM representation of characters yields empirically better results in our experiments 
Visual embeddings: To extract features from an image, we take the final hidden layer representation of a modified version of the convolutional network model called Inception (GoogLeNet) BIBREF23 , BIBREF24 trained on the ImageNet dataset BIBREF25 to classify multiple objects in the scene  Our implementation of the Inception model has deep 22 layers, training of which is made possible via “network in network" principles and several dimension reduction techniques to improve computing resource utilization  The final layer representation encodes discriminative information describing what objects are shown in an image, which provide auxiliary contexts for understanding textual tokens and entities in accompanying captions 
Incorporating this visual information onto the traditional NER system is an open challenge, and multiple approaches can be considered  For instance, one may provide visual contexts only as an initial input to decoder as in some encoder-decoder image captioning systems BIBREF26   However, we empirically observe that an NER decoder which takes as input the visual embeddings at every decoding step (Section SECREF4 ), combined with the modality attention module (Section SECREF5 ), yields better results 
Lastly, we add a transform layer for each feature INLINEFORM0 before it is fed to the NER entity LSTM 
Bi-LSTM + CRF for Multimodal NER
Our MNER model is built on a Bi-LSTM and CRF hybrid model  We use the following implementation for the entity Bi-LSTM 
it = (Wxiht-1 + Wcict-1)
ct = (1-it) ct-1
+ it tanh(Wxcxt + Whcht-1)
ot = (Wxoxt + Whoht-1 + Wcoct)
ht = LSTM(xt)
= ot tanh(ct)
where INLINEFORM0 is a weighted average of three modalities INLINEFORM1 via the modality attention module, which will be defined in Section SECREF5   Bias terms for gates are omitted here for simplicity of notation 
We then obtain bi-directional entity token representations INLINEFORM0 by concatenating its left and right context representations  To enforce structural correlations between labels in sequence decoding, INLINEFORM1 is then passed to a conditional random field (CRF) to produce a label for each token maximizing the following objective  y* = y p(y|h; WCRF)
p(y|h; WCRF) = t t (yt-1,yt;h) y' t t (y't-1,y't;h)
where INLINEFORM0 is a potential function, INLINEFORM1 is a set of parameters that defines the potential functions and weight vectors for label pairs ( INLINEFORM2 )  Bias terms are omitted for brevity of formulation 
The model can be trained via log-likelihood maximization for the training set INLINEFORM0 :
L(WCRF) = i p(y|h; W)
Modality Attention
The modality attention module learns a unified representation space for multiple available modalities (words, characters, images, etc ), and produces a single vector representation with aggregated knowledge among multiple modalities, based on their weighted importance  We motivate this module from the following observations 
A majority of the previous literature combine the word and character-level contexts by simply concatenating the word and character embeddings at each decoding step, e g  INLINEFORM0 in Eq  SECREF4   However, this naive concatenation of two modalities (word and characters) results in inaccurate decoding, specifically for unknown word token embeddings (an all-zero vector INLINEFORM1 or a random vector INLINEFORM2 is assigned for any unknown token INLINEFORM3 , thus INLINEFORM4 or INLINEFORM5 )  While this concatenation approach does not cause significant errors for well-formatted text, we observe that it induces performance degradation for our social media post datasets which contain a significant number of missing tokens 
Similarly, naive merging of textual and visual information ( INLINEFORM0 ) yields suboptimal results as each modality is treated equally informative, whereas in our datasets some of the images may contain irrelevant contexts to textual modalities  Hence, ideally there needs a mechanism in which the model can effectively turn the switch on and off the modalities adaptive to each sample 
To this end, we propose a general modality attention module, which adaptively attenuates or emphasizes each modality as a whole at each decoding step INLINEFORM0 , and produces a soft-attended context vector INLINEFORM1 as an input token for the entity LSTM  [at(w),at(c),at(v)] = (Wm[xt(w); xt(c); xt(v)] + bm )
t(m) = (at(m))m'{w,c,v}(at(m')) m {w,c,v}
xt = m{w,c,v} t(m)xt(m)
where INLINEFORM0 is an attention vector at each decoding step INLINEFORM1 , and INLINEFORM2 is a final context vector at INLINEFORM3 that maximizes information gain for INLINEFORM4   Note that the optimization of the objective function (Eq  SECREF4 ) with modality attention (Eq  SECREF5 ) requires each modality to have the same dimension ( INLINEFORM5 ), and that the transformation via INLINEFORM6 essentially enforces each modality to be mapped into the same unified subspace, where the weighted average of which encodes discrimitive features for recognition of named entities 
When visual context is not provided with each token (as in the traditional NER task), we can define the modality attention for word and character embeddings only in a similar way: [at(w),at(c)] = (Wm[xt(w); xt(c)] + bm )
t(m) = (at(m))m'{w,c}(at(m')) m {w,c}
xt = m{w,c} t(m)xt(m)
Note that while we apply this modality attention module to the Bi-LSTM+CRF architecture (Section SECREF4 ) for its empirical superiority, the module itself is flexible and thus can work with other NER architectures or for other multimodal applications 
SnapCaptions Dataset
The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC)  These captions are collected exclusively from snaps submitted to public and crowd-sourced stories (aka Snapchat Live Stories or Our Stories)  Examples of such public crowd-sourced stories are “New York Story” or “Thanksgiving Story”, which comprise snaps that are aggregated for various public events, venues, etc  All snaps were posted between year 2016 and 2017, and do not contain raw images or other associated information (only textual captions and obfuscated visual descriptor features extracted from the pre-trained InceptionNet are available)  We split the dataset into train (70%), validation (15%), and test sets (15%)  The captions data have average length of 30 7 characters (5 81 words) with vocabulary size 15,733, where 6,612 are considered unknown tokens from Stanford GloVE embeddings BIBREF22   Named entities annotated in the SnapCaptions dataset include many of new and emerging entities, and they are found in various surface forms (various nicknames, typos, etc ) To the best of our knowledge, SnapCaptions is the only dataset that contains natural image-caption pairs with expert-annotated named entities 
Baselines
Task: given a caption and a paired image (if used), the goal is to label every token in a caption in BIO scheme (B: beginning, I: inside, O: outside) BIBREF27   We report the performance of the following state-of-the-art NER models as baselines, as well as several configurations of our proposed approach to examine contributions of each component (W: word, C: char, V: visual) 
Bi-LSTM/CRF (W only): only takes word token embeddings (Stanford GloVE) as input  The rest of the architecture is kept the same 
Bi-LSTM/CRF + Bi-CharLSTM (C only): only takes a character sequence of each word token as input  (No word embeddings)
Bi-LSTM/CRF + Bi-CharLSTM (W+C) BIBREF0 : takes as input both word embeddings and character embeddings extracted from a Bi-CharLSTM  Entity LSTM takes concatenated vectors of word and character embeddings as input tokens 
Bi-LSTM/CRF + CharCNN (W+C) BIBREF1 : uses character embeddings extracted from a CNN instead 
Bi-LSTM/CRF + CharCNN (W+C) + Multi-task BIBREF8 : trains the model to perform both recognition (into multiple entity types) as well as segmentation (binary) tasks 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM with modality attention (W+C): uses the modality attention to merge word and character embeddings 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception (W+C+V): takes as input visual contexts extracted from InceptionNet as well, concatenated with word and char vectors 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception with modality attention (W+C+V): uses the modality attention to merge word, character, and visual embeddings as input to entity LSTM 
Results: SnapCaptions Dataset
Table TABREF6 shows the NER performance on the Snap Captions dataset  We report both entity types recognition (PER, LOC, ORG, MISC) and named entity segmentation (named entity or not) results 
Parameters: We tune the parameters of each model with the following search space (bold indicate the choice for our final model): character embeddings dimension: {25, 50, 100, 150, 200, 300}, word embeddings size: {25, 50, 100, 150, 200, 300}, LSTM hidden states: {25, 50, 100, 150, 200, 300}, and INLINEFORM0 dimension: {25, 50, 100, 150, 200, 300}  We optimize the parameters with Adagrad BIBREF28 with batch size 10, learning rate 0 02, epsilon INLINEFORM1 , and decay 0 0 
Main Results: When visual context is available (W+C+V), we see that the model performance greatly improves over the textual models (W+C), showing that visual contexts are complimentary to textual information in named entity recognition tasks  In addition, it can be seen that the modality attention module further improves the entity type recognition performance for (W+C+V)  This result indicates that the modality attention is able to focus on the most effective modality (visual, words, or characters) adaptive to each sample to maximize information gain  Note that our text-only model (W+C) with the modality attention module also significantly outperform the state-of-the-art baselines BIBREF8 , BIBREF1 , BIBREF0 that use the same textual modalities (W+C), showing the effectiveness of the modality attention module for textual models as well 
Error Analysis: Table TABREF17 shows example cases where incorporation of visual contexts affects prediction of named entities  For example, the token `curry' in the caption “The curry's " is polysemous and may refer to either a type of food or a famous basketball player `Stephen Curry', and the surrounding textual contexts do not provide enough information to disambiguate it  On the other hand, visual contexts (visual tags: `parade', `urban area',    ) provide similarities to the token's distributional semantics from other training examples (snaps from “NBA Championship Parade Story"), and thus the model successfully predicts the token as a named entity  Similarly, while the text-only model erroneously predicts `Apple' in the caption “Grandma w dat lit Apple Crisp" as an organization (Apple Inc ), the visual contexts (describing objects related to food) help disambiguate the token, making the model predict it correctly as a non-named entity (a fruit)  Trending entities (musicians or DJs such as `CID', `Duke Dumont', `Marshmello', etc ) are also recognized correctly with strengthened contexts from visual information (describing concert scenes) despite lack of surrounding textual contexts  A few cases where visual contexts harmed the performance mostly include visual tags that are unrelated to a token or its surrounding textual contexts 
Visualization of Modality Attention: Figure FIGREF19 visualizes the modality attention module at each decoding step (each column), where amplified modality is represented with darker color, and attenuated modality is represented with lighter color 
For the image-aided model (W+C+V; upper row in Figure FIGREF19 ), we confirm that the modality attention successfully attenuates irrelevant signals (selfies, etc ) and amplifies relevant modality-based contexts in prediction of a given token  In the example of “disney word essential = coffee" with visual tags selfie, phone, person, the modality attention successfully attenuates distracting visual signals and focuses on textual modalities, consequently making correct predictions  The named entities in the examples of “Beautiful night atop The Space Needle" and “Splash Mountain" are challenging to predict because they are composed of common nouns (space, needle, splash, mountain), and thus they often need additional contexts to correctly predict  In the training data, visual contexts make stronger indicators for these named entities (space needle, splash mountain), and the modality attention module successfully attends more to stronger signals 
For text-only model (W+C), we observe that performance gains mostly come from the modality attention module better handling tokens unseen during training or unknown tokens from the pre-trained word embeddings matrix  For example, while WaRriOoOrs and Kooler Matic are missing tokens in the word embeddings matrix, it successfully amplifies character-based contexts (capitalized first letters, similarity to known entities `Golden State Warriors') and suppresses word-based contexts (word embeddings for unknown tokens `WaRriOoOrs'), leading to correct predictions  This result is significant because it shows performance of the model, with an almost identical architecture, can still improve without having to scale the word embeddings matrix indefinitely 
Figure FIGREF19 (b) shows the cases where the modality attention led to incorrect predictions  For example, the model predicts missing tokens HUUUGE and Shampooer incorrectly as named entities by amplifying misleading character-based contexts (capitalized first letters) or visual contexts (concert scenes, associated contexts of which often include named entities in the training dataset) 
Sensitivity to Word Embeddings Vocabulary Size: In order to isolate the effectiveness of the modality attention module on textual models in handling missing tokens, we report the performance with varying word embeddings vocabulary sizes in Table TABREF20   By increasing the number of missing tokens artificially by randomly removing words from the word embeddings matrix (original vocab size: 400K), we observe that while the overall performance degrades, the modality attention module is able to suppress the peformance degradation  Note also that the performance gap generally gets bigger as we decrease the vocabulary size of the word embeddings matrix  This result is significant in that the modality attention is able to improve the model more robust to missing tokens without having to train an indefinitely large word embeddings matrix for arbitrarily noisy social media text datasets 
Conclusions
We proposed a new multimodal NER (MNER: image + text) task on short social media posts  We demonstrated for the first time an effective MNER system, where visual information is combined with textual information to outperform traditional text-based NER baselines  Our work can be applied to myriads of social media posts or other articles across multiple platforms which often include both text and accompanying images  In addition, we proposed the modality attention module, a new neural mechanism which learns optimal integration of different modes of correlated information  In essence, the modality attention learns to attenuate irrelevant or uninformative modal information while amplifying the primary modality to extract better overall representations  We showed that the modality attention based model outperforms other state-of-the-art baselines when text was the only modality available, by better combining word and character level information 

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Does their NER model learn NER from both text and images?
Context 4:  As a training dataset, we use the corpus developed by BIBREF36 , which contains 2,400 essays labeled by one of the five personality traits each 
The fully-connected layer has 150 neurons, which are treated as the features  We concatenate the feature vector of each personality dimension in order to create the final feature vector  Thus, the personality model ultimately extracts a 750-dimensional feature vector (150-dimensional feature vector for each of the five personality traits)  This network is replicated five times, one for each personality trait  In particular, we create a CNN for each personality trait and the aim of each CNN is to classify a sentence into binary classes, i e , whether it expresses a personality trait or not 
Baseline Method and Features
CNN can also be employed on the sarcasm datasets in order to identify sarcastic and non-sarcastic tweets  We term the features extracted from this network baseline features, the method as baseline method and the CNN architecture used in this baseline method as baseline CNN  Since the fully-connected layer has 100 neurons, we have 100 baseline features in our experiment  This method is termed baseline method as it directly aims to classify a sentence as sarcastic vs non-sarcastic  The baseline CNN extracts the inherent semantics from the sarcastic corpus by employing deep domain understanding  The process of using baseline features with other features extracted from the pre-trained model is described in Section SECREF24  
Experimental Results and Discussion
In this section, we present the experimental results using different feature combinations and compare them with the state of the art  For each feature we show the results using only CNN and using CNN-SVM (i e , when the features extracted by CNN are fed to the SVM)  Macro-F1 measure is used as an evaluation scheme in the experiments 
Sarcasm Datasets Used in the Experiment
This dataset was created by BIBREF8   The tweets were downloaded from Twitter using #sarcasm as a marker for sarcastic tweets  It is a monolingual English dataset which consists of a balanced distribution of 50,000 sarcastic tweets and 50,000 non-sarcastic tweets 
Since sarcastic tweets are less frequently used BIBREF8 , we also need to investigate the robustness of the selected features and the model trained on these features on an imbalanced dataset  To this end, we used another English dataset from BIBREF8   It consists of 25,000 sarcastic tweets and 75,000 non-sarcastic tweets 
We have obtained this dataset from The Sarcasm Detector  It contains 120,000 tweets, out of which 20,000 are sarcastic and 100,000 are non-sarcastic  We randomly sampled 10,000 sarcastic and 20,000 non-sarcastic tweets from the dataset  Visualization of both the original and subset data show similar characteristics 
A two-step methodology has been employed in filtering the datasets used in our experiments  Firstly, we identified and removed all the “user", “URL" and “hashtag" references present in the tweets using efficient regular expressions  Special emphasis was given to this step to avoid traces of hashtags, which might trigger the models to provide biased results  Secondly, we used NLTK Twitter Tokenizer to ensure proper tokenization of words along with special symbols and emoticons  Since our deep CNNs extract contextual information present in tweets, we include emoticons as part of the vocabulary  This enables the emoticons to hold a place in the word embedding space and aid in providing information about the emotions present in the sentence 
Merging the Features
Throughout this research, we have carried out several experiments with various feature combinations  For the sake of clarity, we explain below how the features extracted using difference models are merged 
In the standard feature merging process, we first extract the features from all deep CNN based feature extraction models and then we concatenate them  Afterwards, SVM is employed on the resulted feature vector 
In another setting, we use the features extracted from the pre-trained models as the static channels of features in the CNN of the baseline method  These features are appended to the hidden layer of the baseline CNN, preceding the final output softmax layer 
For comparison, we have re-implemented the state-of-the-art methods  Since BIBREF9 did not mention about the sentiment lexicon they use in the experiment, we used SenticNet BIBREF37 in the re-implementation of their method 
Results on Dataset 1
As shown in Table TABREF29 , for every feature CNN-SVM outperforms the performance of the CNN  Following BIBREF6 , we have carried out a 5-fold cross-validation on this dataset  The baseline features ( SECREF16 ) perform best among other features  Among all the pre-trained models, the sentiment model (F1-score: 87 00%) achieves better performance in comparison with the other two pre-trained models  Interestingly, when we merge the baseline features with the features extracted by the pre-trained deep NLP models, we only get 0 11% improvement over the F-score  It means that the baseline features alone are quite capable to detect sarcasm  On the other hand, when we combine sentiment, emotion and personality features, we obtain 90 70% F1-score  This indicates that the pre-trained features are indeed useful for sarcasm detection  We also compare our approach with the best research study conducted on this dataset (Table TABREF30 )  Both the proposed baseline model and the baseline + sentiment + emotion + personality model outperform the state of the art BIBREF9 , BIBREF8   One important difference with the state of the art is that BIBREF8 used relatively larger feature vector size ( INLINEFORM0 500,000) than we used in our experiment (1,100)  This not only prevents our model to overfit the data but also speeds up the computation  Thus, we obtain an improvement in the overall performance with automatic feature extraction using a relatively lower dimensional feature space 
In the literature, word n-grams, skipgrams and character n-grams are used as baseline features  According to Ptacek et al  BIBREF8 , these baseline features along with the other features (sentiment features and part-of-speech based features) produced the best performance  However, Ptacek et al  did not analyze the performance of these features when they were not used with the baseline features  Pre-trained word embeddings play an important role in the performance of the classifier because, when we use randomly generated embeddings, performance falls down to 86 23% using all features 
Results on Dataset 2
5-fold cross-validation has been carried out on Dataset 2  Also for this dataset, we get the best accuracy when we use all features  Baseline features have performed significantly better (F1-score: 92 32%) than all other features  Supporting the observations we have made from the experiments on Dataset 1, we see CNN-SVM outperforming CNN on Dataset 2  However, when we use all the features, CNN alone (F1-score: 89 73%) does not outperform the state of the art BIBREF8 (F1-score: 92 37%)  As shown in Table TABREF30 , CNN-SVM on the baseline + sentiment + emotion + personality feature set outperforms the state of the art (F1-score: 94 80%)  Among the pre-trained models, the sentiment model performs best (F1-score: 87 00%) 
Table TABREF29 shows the performance of different feature combinations  The gap between the F1-scores of only baseline features and all features is larger on the imbalanced dataset than the balanced dataset  This supports our claim that sentiment, emotion and personality features are very useful for sarcasm detection, thanks to the pre-trained models  The F1-score using sentiment features when combined with baseline features is 94 60%  On both of the datasets, emotion and sentiment features perform better than the personality features  Interestingly, using only sentiment, emotion and personality features, we achieve 90 90% F1-score 
Results on Dataset 3
Experimental results on Dataset 3 show the similar trends (Table TABREF30 ) as compared to Dataset 1 and Dataset 2  The highest performance (F1-score 93 30%) is obtained when we combine baseline features with sentiment, emotion and personality features  In this case, also CNN-SVM consistently performs better than CNN for every feature combination  The sentiment model is found to be the best pre-trained model  F1-score of 84 43% is obtained when we merge sentiment, emotion and personality features 
Dataset 3 is more complex and non-linear in nature compared to the other two datasets  As shown in Table TABREF30 , the methods by BIBREF9 and BIBREF8 perform poorly on this dataset  The TP rate achieved by BIBREF9 is only 10 07% and that means their method suffers badly on complex data  The approach of BIBREF8 has also failed to perform well on Dataset 3, achieving 62 37% with a better TP rate of 22 15% than BIBREF9   On the other hand, our proposed model performs consistently well on this dataset achieving 93 30% 
Testing Generalizability of the Models and Discussions
To test the generalization capability of the proposed approach, we perform training on Dataset 1 and test on Dataset 3  The F1-score drops down dramatically to 33 05%  In order to understand this finding, we visualize each dataset using PCA (Figure FIGREF17 )  It depicts that, although Dataset 1 is mostly linearly separable, Dataset 3 is not  A linear kernel that performs well on Dataset 1 fails to provide good performance on Dataset 3  If we use RBF kernel, it overfits the data and produces worse results than what we get using linear kernel  Similar trends are seen in the performance of other two state-of-the-art approaches BIBREF9 , BIBREF8   Thus, we decide to perform training on Dataset 3 and test on the Dataset 1  As expected better performance is obtained with F1-score 76 78%  However, the other two state-of-the-art approaches fail to perform well in this setting  While the method by BIBREF9 obtains F1-score of 47 32%, the approach by BIBREF8 achieves 53 02% F1-score when trained on Dataset 3 and tested on Dataset 1  Below, we discuss about this generalizability issue of the models developed or referred in this paper 
As discussed in the introduction, sarcasm is very much topic-dependent and highly contextual  For example, let us consider the tweet “I am so glad to see Tanzania played very well, I can now sleep well :P"  Unless one knows that Tanzania actually did not play well in that game, it is not possible to spot the sarcastic nature of this sentence  Thus, an n-gram based sarcasm detector trained at time INLINEFORM0 may perform poorly to detect sarcasm in the tweets crawled at time INLINEFORM1 (given that there is a considerable gap between these time stamps) because of the diversity of the topics (new events occur, new topics are discussed) of the tweets  Sentiment and other contextual clues can help to spot the sarcastic nature in this kind of tweets  A highly positive statement which ends with a emoticon expressing joke can be sarcastic 
State-of-the-art methods lack these contextual information which, in our case, we extract using pre-trained sentiment, emotion and personality models  Not only these pre-trained models, the baseline method (baseline CNN architecture) performs better than the state-of-the-art models in this generalizability test setting  In our generalizability test, when the pre-trained features are used with baseline features, we get 4 19% F1-score improvement over the baseline features  On the other hand, when they are not used with the baseline features, together they produce 64 25% F1-score 
Another important fact is that an n-grams model cannot perform well on unseen data unless it is trained on a very large corpus  If most of the n-grams extracted from the unseen data are not in the vocabulary of the already trained n-grams model, in fact, the model will produce a very sparse feature vector representation of the dataset  Instead, we use the word2vec embeddings as the source of the features, as word2vec allows for the computation of similarities between unseen data and training data 
Baseline Features vs Pre-trained Features
Our experimental results show that the baseline features outperform the pre-trained features for sarcasm detection  However, the combination of pre-trained features and baseline features beats both of themselves alone  It is counterintuitive, since experimental results prove that both of those features learn almost the same global and contextual features  In particular, baseline network dominates over pre-trained network as the former learns most of the features learned by the latter  Nonetheless, the combination of baseline and pre-trained classifiers improves the overall performance and generalizability, hence proving their effectiveness in sarcasm detection  Experimental results show that sentiment and emotion features are the most useful features, besides baseline features (Figure FIGREF36 )  Therefore, in order to reach a better understanding of the relation between personality features among themselves and with other pre-trained features, we carried out Spearman correlation testing  Results, displayed in Table TABREF39 , show that those features are highly correlated with each other 
Conclusion
In this work, we developed pre-trained sentiment, emotion and personality models for identifying sarcastic text using CNN, which are found to be very effective for sarcasm detection  In the future, we plan to evaluate the performance of the proposed method on a large corpus and other domain-dependent corpora  Future work will also focus on analyzing past tweets and activities of users in order to better understand their personality and profile and, hence, further improve the disambiguation between sarcastic and non-sarcastic text 

ground_truth: Yes

Yes