prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they combine audio and text sequences in their RNN?
Context 1: Introduction
Recently, deep learning algorithms have successfully addressed problems in various fields, such as image classification, machine translation, speech recognition, text-to-speech generation and other machine learning related areas BIBREF0 , BIBREF1 , BIBREF2   Similarly, substantial improvements in performance have been obtained when deep learning algorithms have been applied to statistical speech processing BIBREF3   These fundamental improvements have led researchers to investigate additional topics related to human nature, which have long been objects of study  One such topic involves understanding human emotions and reflecting it through machine intelligence, such as emotional dialogue models BIBREF4 , BIBREF5  
In developing emotionally aware intelligence, the very first step is building robust emotion classifiers that display good performance regardless of the application; this outcome is considered to be one of the fundamental research goals in affective computing BIBREF6   In particular, the speech emotion recognition task is one of the most important problems in the field of paralinguistics  This field has recently broadened its applications, as it is a crucial factor in optimal human-computer interactions, including dialog systems  The goal of speech emotion recognition is to predict the emotional content of speech and to classify speech according to one of several labels (i e , happy, sad, neutral, and angry)  Various types of deep learning methods have been applied to increase the performance of emotion classifiers; however, this task is still considered to be challenging for several reasons  First, insufficient data for training complex neural network-based models are available, due to the costs associated with human involvement  Second, the characteristics of emotions must be learned from low-level speech signals  Feature-based models display limited skills when applied to this problem 
To overcome these limitations, we propose a model that uses high-level text transcription, as well as low-level audio signals, to utilize the information contained within low-resource datasets to a greater degree  Given recent improvements in automatic speech recognition (ASR) technology BIBREF7 , BIBREF2 , BIBREF8 , BIBREF9 , speech transcription can be carried out using audio signals with considerable skill  The emotional content of speech is clearly indicated by the emotion words contained in a sentence BIBREF10 , such as “lovely” and “awesome,” which carry strong emotions compared to generic (non-emotion) words, such as “person” and “day ” Thus, we hypothesize that the speech emotion recognition model will be benefit from the incorporation of high-level textual input 
In this paper, we propose a novel deep dual recurrent encoder model that simultaneously utilizes audio and text data in recognizing emotions from speech  Extensive experiments are conducted to investigate the efficacy and properties of the proposed model  Our proposed model outperforms previous state-of-the-art methods by 68 8% to 71 8% when applied to the IEMOCAP dataset, which is one of the most well-studied datasets  Based on an error analysis of the models, we show that our proposed model accurately identifies emotion classes  Moreover, the neutral class misclassification bias frequently exhibited by previous models, which focus on audio features, is less pronounced in our model 
Related work
Classical machine learning algorithms, such as hidden Markov models (HMMs), support vector machines (SVMs), and decision tree-based methods, have been employed in speech emotion recognition problems BIBREF11 , BIBREF12 , BIBREF13   Recently, researchers have proposed various neural network-based architectures to improve the performance of speech emotion recognition  An initial study utilized deep neural networks (DNNs) to extract high-level features from raw audio data and demonstrated its effectiveness in speech emotion recognition BIBREF14   With the advancement of deep learning methods, more complex neural-based architectures have been proposed  Convolutional neural network (CNN)-based models have been trained on information derived from raw audio signals using spectrograms or audio features such as Mel-frequency cepstral coefficients (MFCCs) and low-level descriptors (LLDs) BIBREF15 , BIBREF16 , BIBREF17   These neural network-based models are combined to produce higher-complexity models BIBREF18 , BIBREF19 , and these models achieved the best-recorded performance when applied to the IEMOCAP dataset 
Another line of research has focused on adopting variant machine learning techniques combined with neural network-based models  One researcher utilized the multiobject learning approach and used gender and naturalness as auxiliary tasks so that the neural network-based model learned more features from a given dataset BIBREF20   Another researcher investigated transfer learning methods, leveraging external data from related domains BIBREF21  
As emotional dialogue is composed of sound and spoken content, researchers have also investigated the combination of acoustic features and language information, built belief network-based methods of identifying emotional key phrases, and assessed the emotional salience of verbal cues from both phoneme sequences and words BIBREF22 , BIBREF23   However, none of these studies have utilized information from speech signals and text sequences simultaneously in an end-to-end learning neural network-based model to classify emotions 
Model
This section describes the methodologies that are applied to the speech emotion recognition task  We start by introducing the recurrent encoder model for the audio and text modalities individually  We then propose a multimodal approach that encodes both audio and textual information simultaneously via a dual recurrent encoder 
Audio Recurrent Encoder (ARE)
Motivated by the architecture used in BIBREF24 , BIBREF25 , we build an audio recurrent encoder (ARE) to predict the class of a given audio signal  Once MFCC features have been extracted from an audio signal, a subset of the sequential features is fed into the RNN (i e , gated recurrent units (GRUs)), which leads to the formation of the network's internal hidden state INLINEFORM0 to model the time series patterns  This internal hidden state is updated at each time step with the input data INLINEFORM1 and the hidden state of the previous time step INLINEFORM2 as follows: DISPLAYFORM0
where INLINEFORM0 is the RNN function with weight parameter INLINEFORM1 , INLINEFORM2 represents the hidden state at t- INLINEFORM3 time step, and INLINEFORM4 represents the t- INLINEFORM5 MFCC features in INLINEFORM6   After encoding the audio signal INLINEFORM7 with the RNN, the last hidden state of the RNN, INLINEFORM8 , is considered to be the representative vector that contains all of the sequential audio data  This vector is then concatenated with another prosodic feature vector, INLINEFORM9 , to generate a more informative vector representation of the signal, INLINEFORM10   The MFCC and the prosodic features are extracted from the audio signal using the openSMILE toolkit BIBREF26 , INLINEFORM11 , respectively  Finally, the emotion class is predicted by applying the softmax function to the vector INLINEFORM12   For a given audio sample INLINEFORM13 , we assume that INLINEFORM14 is the true label vector, which contains all zeros but contains a one at the correct class, and INLINEFORM15 is the predicted probability distribution from the softmax layer  The training objective then takes the following form: DISPLAYFORM0
where INLINEFORM0 is the calculated representative vector of the audio signal with dimensionality INLINEFORM1   The INLINEFORM2 and the bias INLINEFORM3 are learned model parameters  C is the total number of classes, and N is the total number of samples used in training  The upper part of Figure shows the architecture of the ARE model 
Text Recurrent Encoder (TRE)
We assume that speech transcripts can be extracted from audio signals with high accuracy, given the advancement of ASR technologies BIBREF7   We attempt to use the processed textual information as another modality in predicting the emotion class of a given signal  To use textual information, a speech transcript is tokenized and indexed into a sequence of tokens using the Natural Language Toolkit (NLTK) BIBREF27   Each token is then passed through a word-embedding layer that converts a word index to a corresponding 300-dimensional vector that contains additional contextual meaning between words  The sequence of embedded tokens is fed into a text recurrent encoder (TRE) in such a way that the audio MFCC features are encoded using the ARE represented by equation EQREF2   In this case, INLINEFORM0 is the t- INLINEFORM1 embedded token from the text input  Finally, the emotion class is predicted from the last hidden state of the text-RNN using the softmax function 
We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is last hidden state of the text-RNN, INLINEFORM1 , and the INLINEFORM2 and bias INLINEFORM3 are learned model parameters  The lower part of Figure indicates the architecture of the TRE model 
Multimodal Dual Recurrent Encoder (MDRE)
We present a novel architecture called the multimodal dual recurrent encoder (MDRE) to overcome the limitations of existing approaches  In this study, we consider multiple modalities, such as MFCC features, prosodic features and transcripts, which contain sequential audio information, statistical audio information and textual information, respectively  These types of data are the same as those used in the ARE and TRE cases  The MDRE model employs two RNNs to encode data from the audio signal and textual inputs independently  The audio-RNN encodes MFCC features from the audio signal using equation EQREF2   The last hidden state of the audio-RNN is concatenated with the prosodic features to form the final vector representation INLINEFORM0 , and this vector is then passed through a fully connected neural network layer to form the audio encoding vector A  On the other hand, the text-RNN encodes the word sequence of the transcript using equation EQREF2   The final hidden states of the text-RNN are also passed through another fully connected neural network layer to form a textual encoding vector T  Finally, the emotion class is predicted by applying the softmax function to the concatenation of the vectors A and T  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is the feed-forward neural network with weight parameter INLINEFORM1 , and INLINEFORM2 , INLINEFORM3 are final encoding vectors from the audio-RNN and text-RNN, respectively  INLINEFORM4 and the bias INLINEFORM5 are learned model parameters 
Multimodal Dual Recurrent Encoder with Attention (MDREA)
Inspired by the concept of the attention mechanism used in neural machine translation BIBREF28 , we propose a novel multimodal attention method to focus on the specific parts of a transcript that contain strong emotional information, conditioning on the audio information  Figure shows the architecture of the MDREA model  First, the audio data and text data are encoded with the audio-RNN and text-RNN using equation EQREF2   We then consider the final audio encoding vector INLINEFORM0 as a context vector  As seen in equation EQREF9 , during each time step t, the dot product between the context vector e and the hidden state of the text-RNN at each t-th sequence INLINEFORM1 is evaluated to calculate a similarity score INLINEFORM2   Using this score INLINEFORM3 as a weight parameter, the weighted sum of the sequences of the hidden state of the text-RNN, INLINEFORM4 , is calculated to generate an attention-application vector Z  This attention-application vector is concatenated with the final encoding vector of the audio-RNN INLINEFORM5 (equation EQREF7 ), which will be passed through the softmax function to predict the emotion class  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 and the bias INLINEFORM1 are learned model parameters 
Dataset
We evaluate our model using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) BIBREF18 dataset  This dataset was collected following theatrical theory in order to simulate natural dyadic interactions between actors  We use categorical evaluations with majority agreement  We use only four emotional categories happy, sad, angry, and neutral to compare the performance of our model with other research using the same categories  The IEMOCAP dataset includes five sessions, and each session contains utterances from two speakers (one male and one female)  This data collection process resulted in 10 unique speakers  For consistent comparison with previous work, we merge the excitement dataset with the happiness dataset  The final dataset contains a total of 5531 utterances (1636 happy, 1084 sad, 1103 angry, 1708 neutral) 
Feature extraction
To extract speech information from audio signals, we use MFCC values, which are widely used in analyzing audio signals  The MFCC feature set contains a total of 39 features, which include 12 MFCC parameters (1-12) from the 26 Mel-frequency bands and log-energy parameters, 13 delta and 13 acceleration coefficients The frame size is set to 25 ms at a rate of 10 ms with the Hamming function  According to the length of each wave file, the sequential step of the MFCC features is varied  To extract additional information from the data, we also use prosodic features, which show effectiveness in affective computing  The prosodic features are composed of 35 features, which include the F0 frequency, the voicing probability, and the loudness contours  All of these MFCC and prosodic features are extracted from the data using the OpenSMILE toolkit BIBREF26  
Implementation details
Among the variants of the RNN function, we use GRUs as they yield comparable performance to that of the LSTM and include a smaller number of weight parameters BIBREF29   We use a max encoder step of 750 for the audio input, based on the implementation choices presented in BIBREF30 and 128 for the text input because it covers the maximum length of the transcripts  The vocabulary size of the dataset is 3,747, including the “_UNK_" token, which represents unknown words, and the “_PAD_" token, which is used to indicate padding information added while preparing mini-batch data  The number of hidden units and the number of layers in the RNN for each model (ARE, TRE, MDRE and MDREA) are selected based on extensive hyperparameter search experiments  The weights of the hidden units are initialized using orthogonal weights BIBREF31 ], and the text embedding layer is initialized from pretrained word-embedding vectors BIBREF32  
In preparing the textual dataset, we first use the released transcripts of the IEMOCAP dataset for simplicity  To investigate the practical performance, we then process all of the IEMOCAP audio data using an ASR system (the Google Cloud Speech API) and retrieve the transcripts  The performance of the Google ASR system is reflected by its word error rate (WER) of 5 53% 
Performance evaluation
As the dataset is not explicitly split beforehand into training, development, and testing sets, we perform 5-fold cross validation to determine the overall performance of the model  The data in each fold are split into training, development, and testing datasets (8:0 5:1 5, respectively)  After training the model, we measure the weighted average precision (WAP) over the 5-fold dataset  We train and evaluate the model 10 times per fold, and the model performance is assessed in terms of the mean score and standard deviation 
We examine the WAP values, which are shown in Table 1  First, our ARE model shows the baseline performance because we use minimal audio features, such as the MFCC and prosodic features with simple architectures  On the other hand, the TRE model shows higher performance gain compared to the ARE  From this result, we note that textual data are informative in emotion prediction tasks, and the recurrent encoder model is effective in understanding these types of sequential data  Second, the newly proposed model, MDRE, shows a substantial performance gain  It thus achieves the state-of-the-art performance with a WAP value of 0 718  This result shows that multimodal information is a key factor in affective computing  Lastly, the attention model, MDREA, also outperforms the best existing research results (WAP 0 690 to 0 688) BIBREF19   However, the MDREA model does not match the performance of the MDRE model, even though it utilizes a more complex architecture  We believe that this result arises because insufficient data are available to properly determine the complex model parameters in the MDREA model  Moreover, we presume that this model will show better performance when the audio signals are aligned with the textual sequence while applying the attention mechanism  We leave the implementation of this point as a future research direction 
To investigate the practical performance of the proposed models, we conduct further experiments with the ASR-processed transcript data (see “-ASR” models in Table )  The label accuracy of the processed transcripts is 5 53% WER  The TRE-ASR, MDRE-ASR and MDREA-ASR models reflect degraded performance compared to that of the TRE, MDRE and MDREA models  However, the performance of these models is still competitive; in particular, the MDRE-ASR model outperforms the previous best-performing model, 3CNN-LSTM10H (WAP 0 691 to 0 688) 
Error analysis
We analyze the predictions of the ARE, TRE, and MDRE models  Figure shows the confusion matrix of each model  The ARE model (Fig  ) incorrectly classifies most instances of happy as neutral (43 51%); thus, it shows reduced accuracy (35 15%) in predicting the the happy class  Overall, most of the emotion classes are frequently confused with the neutral class  This observation is in line with the findings of BIBREF30 , who noted that the neutral class is located in the center of the activation-valence space, complicating its discrimination from the other classes 
Interestingly, the TRE model (Fig  ) shows greater prediction gains in predicting the happy class when compared to the ARE model (35 15% to 75 73%)  This result seems plausible because the model can benefit from the differences among the distributions of words in happy and neutral expressions, which gives more emotional information to the model than that of the audio signal data  On the other hand, it is striking that the TRE model incorrectly predicts instances of the sad class as the happy class 16 20% of the time, even though these emotional states are opposites of one another 
The MDRE model (Fig  ) compensates for the weaknesses of the previous two models (ARE and TRE) and benefits from their strengths to a surprising degree  The values arranged along the diagonal axis show that all of the accuracies of the correctly predicted class have increased  Furthermore, the occurrence of the incorrect “sad-to-happy" cases in the TRE model is reduced from 16 20% to 9 15% 
Conclusions
In this paper, we propose a novel multimodal dual recurrent encoder model that simultaneously utilizes text data, as well as audio signals, to permit the better understanding of speech data  Our model encodes the information from audio and text sequences using dual RNNs and then combines the information from these sources using a feed-forward neural model to predict the emotion class  Extensive experiments show that our proposed model outperforms other state-of-the-art methods in classifying the four emotion categories, and accuracies ranging from 68 8% to 71 8% are obtained when the model is applied to the IEMOCAP dataset  In particular, it resolves the issue in which predictions frequently incorrectly yield the neutral class, as occurs in previous models that focus on audio features 
In the future work, we aim to extend the modalities to audio, text and video inputs  Furthermore, we plan to investigate the application of the attention mechanism to data derived from multiple modalities  This approach seems likely to uncover enhanced learning schemes that will increase performance in both speech emotion recognition and other multimodal classification tasks 
Acknowledgments
K  Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea  This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under Industrial Technology Innovation Program (No 10073144) 
Context 2: Introduction
The automatic processing of medical texts and documents plays an increasingly important role in the recent development of the digital health area  To enable dedicated Natural Language Processing (NLP) that is highly accurate with respect to medically relevant categories, manually annotated data from this domain is needed  One category of high interest and relevance are medical entities  Only very few annotated corpora in the medical domain exist  Many of them focus on the relation between chemicals and diseases or proteins and diseases, such as the BC5CDR corpus BIBREF0, the Comparative Toxicogenomics Database BIBREF1, the FSU PRotein GEne corpus BIBREF2 or the ADE (adverse drug effect) corpus BIBREF3  The NCBI Disease Corpus BIBREF4 contains condition mention annotations along with annotations of symptoms  Several new corpora of annotated case reports were made available recently  grouin-etal-2019-clinical presented a corpus with medical entity annotations of clinical cases written in French, copdPhenotype presented a corpus focusing on phenotypic information for chronic obstructive pulmonary disease while 10 1093/database/bay143 presented a corpus focusing on identifying main finding sentences in case reports 
The corpus most comparable to ours is the French corpus of clinical case reports by grouin-etal-2019-clinical  Their annotations are based on UMLS semantic types  Even though there is an overlap in annotated entities, semantic classes are not the same  Lab results are subsumed under findings in our corpus and are not annotated as their own class  Factors extend beyond gender and age and describe any kind of risk factor that contributes to a higher probability of having a certain disease  Our corpus includes additional entity types  We annotate conditions, findings (including medical findings such as blood values), factors, and also modifiers which indicate the negation of other entities as well as case entities, i  e , entities specific to one case report  An overview is available in Table TABREF3 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation tasks
Case reports are standardized in the CARE guidelines BIBREF5  They represent a detailed description of the symptoms, signs, diagnosis, treatment, and follow-up of an individual patient  We focus on documents freely available through PubMed Central (PMC)  The presentation of the patient's case can usually be found in a dedicated section or the abstract  We perform a manual annotation of all mentions of case entities, conditions, findings, factors and modifiers  The scope of our manual annotation is limited to the presentation of a patient's signs and symptoms  In addition, we annotate the title of the case report 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Guidelines
We annotate the following entities:
case entity marks the mention of a patient  A case report can contain more than one case description  Therefore, all the findings, factors and conditions related to one patient are linked to the respective case entity  Within the text, this entity is often represented by the first mention of the patient and overlaps with the factor annotations which can, e  g , mark sex and age (cf  Figure FIGREF12) 
condition marks a medical disease such as pneumothorax or dislocation of the shoulder 
factor marks a feature of a patient which might influence the probability for a specific diagnosis  It can be immutable (e  g , sex and age), describe a specific medical history (e  g , diabetes mellitus) or a behaviour (e  g , smoking) 
finding marks a sign or symptom a patient shows  This can be visible (e  g , rash), described by a patient (e  g , headache) or measurable (e  g , decreased blood glucose level) 
negation modifier explicitly negate the presence of a certain finding usually setting the case apart from common cases 
We also annotate relations between these entities, where applicable  Since we work on case descriptions, the anchor point of these relations is the case that is described  The following relations are annotated:
has relations exist between a case entity and factor, finding or condition entities 
modifies relations exist between negation modifiers and findings 
causes relations exist between conditions and findings 
Example annotations are shown in Figure FIGREF16 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotators
We asked medical doctors experienced in extracting knowledge related to medical entities from texts to annotate the entities described above  Initially, we asked four annotators to test our guidelines on two texts  Subsequently, identified issues were discussed and resolved  Following this pilot annotation phase, we asked two different annotators to annotate two case reports according to our guidelines  The same annotators annotated an overall collection of 53 case reports 
Inter-annotator agreement is calculated based on two case reports  We reach a Cohen's kappa BIBREF6 of 0 68  Disagreements mainly appear for findings that are rather unspecific such as She no longer eats out with friends which can be seen as a finding referring to “avoidance behaviour” 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Annotation Tools and Format
The annotation was performed using WebAnno BIBREF7, a web-based tool for linguistic annotation  The annotators could choose between a pre-annotated version or a blank version of each text  The pre-annotated versions contained suggested entity spans based on string matches from lists of conditions and findings synonym lists  Their quality varied widely throughout the corpus  The blank version was preferred by the annotators  We distribute the corpus in BioC JSON format  BioC was chosen as it allows us to capture the complexities of the annotations in the biomedical domain  It represented each documents properties ranging from full text, individual passages/sentences along with captured annotations and relationships in an organized manner  BioC is based on character offsets of annotations and allows the stacking of different layers 
A Corpus of Medical Case Reports with Medical Entity Annotation ::: Corpus Overview
The corpus consists of 53 documents, which contain an average number of 156 1 sentences per document, each with 19 55 tokens on average  The corpus comprises 8,275 sentences and 167,739 words in total  However, as mentioned above, only case presentation sections, headings and abstracts are annotated  The numbers of annotated entities are summarized in Table TABREF24 
Findings are the most frequently annotated type of entity  This makes sense given that findings paint a clinical picture of the patient's condition  The number of tokens per entity ranges from one token for all types to 5 tokens for cases (average length 3 1), nine tokens for conditions (average length 2 0), 16 tokens for factors (average length 2 5), 25 tokens for findings (average length 2 6) and 18 tokens for modifiers (average length 1 4) (cf  Table TABREF24)  Examples of rather long entities are given in Table TABREF25 
Entities can appear in a discontinuous way  We model this as a relation between two spans which we call “discontinuous” (cf  Figure FIGREF26)
Context 3: Introduction
Offensive content has become pervasive in social media and a reason of concern for government organizations, online communities, and social media platforms  One of the most common strategies to tackle the problem is to train systems capable of recognizing offensive content, which then can be deleted or set aside for human moderation  In the last few years, there have been several studies published on the application of computational methods to deal with this problem  Most prior work focuses on a different aspect of offensive language such as abusive language BIBREF0 , BIBREF1 , (cyber-)aggression BIBREF2 , (cyber-)bullying BIBREF3 , BIBREF4 , toxic comments INLINEFORM0 , hate speech BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , and offensive language BIBREF11   Prior work has focused on these aspects of offensive language in Twitter BIBREF3 , BIBREF7 , BIBREF8 , BIBREF11 , Wikipedia comments, and Facebook posts BIBREF2  
Recently, Waseem et  al  ( BIBREF12 ) acknowledged the similarities among prior work and discussed the need for a typology that differentiates between whether the (abusive) language is directed towards a specific individual or entity or towards a generalized group and whether the abusive content is explicit or implicit  Wiegand et al  ( BIBREF11 ) followed this trend as well on German tweets  In their evaluation, they have a task to detect offensive vs not offensive tweets and a second task for distinguishing between the offensive tweets as profanity, insult, or abuse  However, no prior work has explored the target of the offensive language, which is important in many scenarios, e g , when studying hate speech with respect to a specific target 
Therefore, we expand on these ideas by proposing a a hierarchical three-level annotation model that encompasses:
Using this annotation model, we create a new large publicly available dataset of English tweets  The key contributions of this paper are as follows:
Related Work
Different abusive and offense language identification sub-tasks have been explored in the past few years including aggression identification, bullying detection, hate speech, toxic comments, and offensive language 
Aggression identification: The TRAC shared task on Aggression Identification BIBREF2 provided participants with a dataset containing 15,000 annotated Facebook posts and comments in English and Hindi for training and validation  For testing, two different sets, one from Facebook and one from Twitter were provided  Systems were trained to discriminate between three classes: non-aggressive, covertly aggressive, and overtly aggressive 
Bullying detection: Several studies have been published on bullying detection  One of them is the one by xu2012learning which apply sentiment analysis to detect bullying in tweets  xu2012learning use topic models to to identify relevant topics in bullying  Another related study is the one by dadvar2013improving which use user-related features such as the frequency of profanity in previous messages to improve bullying detection 
Hate speech identification: It is perhaps the most widespread abusive language detection sub-task  There have been several studies published on this sub-task such as kwok2013locate and djuric2015hate who build a binary classifier to distinguish between `clean' comments and comments containing hate speech and profanity  More recently, Davidson et al  davidson2017automated presented the hate speech detection dataset containing over 24,000 English tweets labeled as non offensive, hate speech, and profanity 
Offensive language: The GermEval BIBREF11 shared task focused on Offensive language identification in German tweets  A dataset of over 8,500 annotated tweets was provided for a course-grained binary classification task in which systems were trained to discriminate between offensive and non-offensive tweets and a second task where the organizers broke down the offensive class into three classes: profanity, insult, and abuse 
Toxic comments: The Toxic Comment Classification Challenge was an open competition at Kaggle which provided participants with comments from Wikipedia labeled in six classes: toxic, severe toxic, obscene, threat, insult, identity hate 
While each of these sub-tasks tackle a particular type of abuse or offense, they share similar properties and the hierarchical annotation model proposed in this paper aims to capture this  Considering that, for example, an insult targeted at an individual is commonly known as cyberbulling and that insults targeted at a group are known as hate speech, we pose that OLID's hierarchical annotation model makes it a useful resource for various offensive language identification sub-tasks 
Hierarchically Modelling Offensive Content
In the OLID dataset, we use a hierarchical annotation model split into three levels to distinguish between whether language is offensive or not (A), and type (B) and target (C) of the offensive language  Each level is described in more detail in the following subsections and examples are shown in Table TABREF10  
Level A: Offensive language Detection
Level A discriminates between offensive (OFF) and non-offensive (NOT) tweets 
Not Offensive (NOT): Posts that do not contain offense or profanity;
Offensive (OFF): We label a post as offensive if it contains any form of non-acceptable language (profanity) or a targeted offense, which can be veiled or direct  This category includes insults, threats, and posts containing profane language or swear words 
Level B: Categorization of Offensive Language
Level B categorizes the type of offense and two labels are used: targeted (TIN) and untargeted (INT) insults and threats 
Targeted Insult (TIN): Posts which contain an insult/threat to an individual, group, or others (see next layer);
Untargeted (UNT): Posts containing non-targeted profanity and swearing  Posts with general profanity are not targeted, but they contain non-acceptable language 
Level C: Offensive Language Target Identification
Level C categorizes the targets of insults and threats as individual (IND), group (GRP), and other (OTH) 
Individual (IND): Posts targeting an individual  It can be a a famous person, a named individual or an unnamed participant in the conversation  Insults and threats targeted at individuals are often defined as cyberbulling 
Group (GRP): The target of these offensive posts is a group of people considered as a unity due to the same ethnicity, gender or sexual orientation, political affiliation, religious belief, or other common characteristic  Many of the insults and threats targeted at a group correspond to what is commonly understood as hate speech 
Other (OTH): The target of these offensive posts does not belong to any of the previous two categories (e g  an organization, a situation, an event, or an issue) 
Data Collection
The data included in OLID has been collected from Twitter  We retrieved the data using the Twitter API by searching for keywords and constructions that are often included in offensive messages, such as `she is' or `to:BreitBartNews'  We carried out a first round of trial annotation of 300 instances with six experts  The goal of the trial annotation was to 1) evaluate the proposed tagset; 2) evaluate the data retrieval method; and 3) create a gold standard with instances that could be used as test questions in the training and test setting annotation which was carried out using crowdsourcing  The breakdown of keywords and their offensive content in the trial data of 300 tweets is shown in Table TABREF14   We included a left (@NewYorker) and far-right (@BreitBartNews) news accounts because there tends to be political offense in the comments  One of the best offensive keywords was tweets that were flagged as not being safe by the Twitter `safe' filter (the `-' indicates `not safe')  The vast majority of content on Twitter is not offensive so we tried different strategies to keep a reasonable number of tweets in the offensive class amounting to around 30% of the dataset including excluding some keywords that were not high in offensive content such as `they are` and `to:NewYorker`

ground_truth: combines the information from these sources using a feed-forward neural model

encodes the information from audio and text sequences using dual RNNs and then combines the information from these sources using a feed-forward neural model

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How large is the corpus?
Context 2:  Especially findings often appear as discontinuous entities, we found 543 discontinuous finding relations  The numbers for conditions and factors are lower with seven and two, respectively  Entities can also be nested within one another  This happens either when the span of one annotation is completely embedded in the span of another annotation (fully-nested; cf  Figure FIGREF12), or when there is a partial overlapping between the spans of two different entities (partially-nested; cf  Figure FIGREF12)  There is a high number of inter-sentential relations in the corpus (cf  Table TABREF27)  This can be explained by the fact that the case entity occurs early in each document; furthermore, it is related to finding and factor annotations that are distributed across different sentences 
The most frequently annotated relation in our corpus is the has-relation between a case entity and the findings related to that case  This correlates with the high number of finding entities  The relations contained in our corpus are summarized in Table TABREF27 
Baseline systems for Named Entity Recognition in medical case reports
We evaluate the corpus using Named Entity Recognition (NER), i  e , the task of finding mentions of concepts of interest in unstructured text  We focus on detecting cases, conditions, factors, findings and modifiers in case reports (cf  Section SECREF6)  We approach this as a sequence labeling problem  Four systems were developed to offer comparable robust baselines 
The original documents are pre-processed (sentence splitting and tokenization with ScispaCy)  We do not perform stop word removal or lower-casing of the tokens  The BIO labeling scheme is used to capture the order of tokens belonging to the same entity type and enable span-level detection of entities  Detection of nested and/or discontinuous entities is not supported  The annotated corpus is randomized and split in five folds using scikit-learn BIBREF9  Each fold has a train, test and dev split with the test split defined as  15% of the train split  This ensures comparability between the presented systems 
Baseline systems for Named Entity Recognition in medical case reports ::: Conditional Random Fields
Conditional Random Fields (CRF) BIBREF10 are a standard approach when dealing with sequential data in the context of sequence labeling  We use a combination of linguistic and semantic features, with a context window of size five, to describe each of the tokens and the dependencies between them  Hyper-parameter optimization is performed using randomized search and cross validation  Span-based F1 score is used as the optimization metric 
Baseline systems for Named Entity Recognition in medical case reports ::: BiLSTM-CRF
Prior to the emergence of deep neural language models, BiLSTM-CRF models BIBREF11 had achieved state-of-the-art results for the task of sequence labeling  We use a BiLSTM-CRF model with both word-level and character-level input  BioWordVec BIBREF12 pre-trained word embeddings are used in the embedding layer for the input representation  A bidirectional LSTM layer is applied to a multiplication of the two input representations  Finally, a CRF layer is applied to predict the sequence of labels  Dropout and L1/L2 regularization is used where applicable  He (uniform) initialization BIBREF13 is used to initialize the kernels of the individual layers  As the loss metric, CRF-based loss is used, while optimizing the model based on the CRF Viterbi accuracy  Additionally, span-based F1 score is used to serialize the best performing model  We train for a maximum of 100 epochs, or until an early stopping criterion is reached (no change in validation loss value grater than 0 01 for ten consecutive epochs)  Furthermore, Adam BIBREF14 is used as the optimizer  The learning rate is reduced by a factor of 0 3 in case no significant increase of the optimization metric is achieved in three consecutive epochs 
Baseline systems for Named Entity Recognition in medical case reports ::: Multi-Task Learning
Multi-Task Learning (MTL) BIBREF15 has become popular with the progress in deep learning  This model family is characterized by simultaneous optimization of multiple loss functions and transfer of knowledge achieved this way  The knowledge is transferred through the use of one or multiple shared layers  Through finding supporting patterns in related tasks, MTL provides better generalization on unseen cases and the main tasks we are trying to solve 
We rely on the model presented by bekoulis2018joint and reuse the implementation provided by the authors  The model jointly trains two objectives supported by the dataset: the main task of NER and a supporting task of Relation Extraction (RE)  Two separate models are developed for each of the tasks  The NER task is solved with the help of a BiLSTM-CRF model, similar to the one presented in Section SECREF32 The RE task is solved by using a multi-head selection approach, where each token can have none or more relationships to in-sentence tokens  Additionally, this model also leverages the output of the NER branch model (the CRF prediction) to learn label embeddings  Shared layers consist of a concatenation of word and character embeddings followed by two bidirectional LSTM layers  We keep most of the parameters suggested by the authors and change (1) the number of training epochs to 100 to allow the comparison to other deep learning approaches in this work, (2) use label embeddings of size 64, (3) allow gradient clipping and (4) use $d=0 8$ as the pre-trained word embedding dropout and $d=0 5$ for all other dropouts  $\eta =1^{-3}$ is used as the learning rate with the Adam optimizer and tanh activation functions across layers  Although it is possible to use adversarial training BIBREF16, we omit from using it  We also omit the publication of results for the task of RE as we consider it to be a supporting task and no other competing approaches have been developed 
Baseline systems for Named Entity Recognition in medical case reports ::: BioBERT
Deep neural language models have recently evolved to a successful method for representing text  In particular, Bidirectional Encoder Representations from Transformers (BERT) outperformed previous state-of-the-art methods by a large margin on various NLP tasks BIBREF17  For our experiments, we use BioBERT, an adaptation of BERT for the biomedical domain, pre-trained on PubMed abstracts and PMC full-text articles BIBREF18  The BERT architecture for deriving text representations uses 12 hidden layers, consisting of 768 units each  For NER, token level BIO-tag probabilities are computed with a single output layer based on the representations from the last layer of BERT  We fine-tune the model on the entity recognition task during four training epochs with batch size $b=32$, dropout probability $d=0 1$ and learning rate $\eta =2^{-5}$  These hyper-parameters are proposed by Devlin2018 for BERT fine-tuning 
Baseline systems for Named Entity Recognition in medical case reports ::: Evaluation
To evaluate the performance of the four systems, we calculate the span-level precision (P), recall (R) and F1 scores, along with corresponding micro and macro scores  The reported values are shown in Table TABREF29 and are averaged over five folds, utilising the seqeval framework 
With a macro avg  F1-score of 0 59, MTL achieves the best result with a significant margin compared to CRF, BiLSTM-CRF and BERT  This confirms the usefulness of jointly training multiple objectives (minimizing multiple loss functions), and enabling knowledge transfer, especially in a setting with limited data (which is usually the case in the biomedical NLP domain)  This result also suggest the usefulness of BioBERT for other biomedical datasets as reported by Lee2019  Despite being a rather standard approach, CRF outperforms the more elaborated BiLSTM-CRF, presumably due to data scarcity and class imbalance  We hypothesize that an increase in training data would yield better results for BiLSTM-CRF but not outperform transfer learning approach of MTL (or even BioBERT)  In contrast to other common NER corpora, like CoNLL 2003, even the best baseline system only achieves relatively low scores  This outcome is due to the inherent difficulty of the task (annotators are experienced medical doctors) and the small number of training samples 
Conclusion
We present a new corpus, developed to facilitate the processing of case reports  The corpus focuses on five distinct entity types: cases, conditions, factors, findings and modifiers  Where applicable, relationships between entities are also annotated  Additionally, we annotate discontinuous entities with a special relationship type (discontinuous)  The corpus presented in this paper is the very first of its kind and a valuable addition to the scarce number of corpora available in the field of biomedical NLP  Its complexity, given the discontinuous nature of entities and a high number of nested and multi-label entities, poses new challenges for NLP methods applied for NER and can, hence, be a valuable source for insights into what entities “look like in the wild”  Moreover, it can serve as a playground for new modelling techniques such as the resolution of discontinuous entities as well as multi-task learning given the combination of entities and their relations  We provide an evaluation of four distinct NER systems that will serve as robust baselines for future work but which are, as of yet, unable to solve all the complex challenges this dataset holds  A functional service based on the presented corpus is currently being integrated, as a NER service, in the QURATOR platform BIBREF20 
Acknowledgments
The research presented in this article is funded by the German Federal Ministry of Education and Research (BMBF) through the project QURATOR (Unternehmen Region, Wachstumskern, grant no  03WKDA1A), see http://qurator ai  We want to thank our medical experts for their help annotating the data set, especially Ashlee Finckh and Sophie Klopfenstein 
Context 3:  Although `he is' is lower in offensive content we kept it as a keyword to avoid gender bias  In addition to the keywords in the trial set, we searched for more political keywords which tend to be higher in offensive content, and sampled our dataset such that 50% of the the tweets come from political keywords and 50% come from non-political keywords  In addition to the keywords `gun control', and `to:BreitbartNews', political keywords used to collect these tweets are `MAGA', `antifa', `conservative' and `liberal'  We computed Fliess' INLINEFORM0 on the trial set for the five annotators on 21 of the tweets  INLINEFORM1 is  83 for Layer A (OFF vs NOT) indicating high agreement  As to normalization and anonymization, no user metadata or Twitter IDs have been stored, and URLs and Twitter mentions have been substituted to placeholders  We follow prior work in related areas (burnap2015cyber,davidson2017automated) and annotate our data using crowdsourcing using the platform Figure Eight  We ensure data quality by: 1) we only received annotations from individuals who were experienced in the platform; and 2) we used test questions to discard annotations of individuals who did not reach a certain threshold  Each instance in the dataset was annotated by multiple annotators and inter-annotator agreement has been calculated  We first acquired two annotations for each instance  In case of 100% agreement, we considered these as acceptable annotations, and in case of disagreement, we requested more annotations until the agreement was above 66%  After the crowdsourcing annotation, we used expert adjudication to guarantee the quality of the annotation  The breakdown of the data into training and testing for the labels from each level is shown in Table TABREF15  
Experiments and Evaluation
We assess our dataset using traditional and deep learning methods  Our simplest model is a linear SVM trained on word unigrams  SVMs have produced state-of-the-art results for many text classification tasks BIBREF13   We also train a bidirectional Long Short-Term-Memory (BiLSTM) model, which we adapted from the sentiment analysis system of sentimentSystem,rasooli2018cross and altered to predict offensive labels instead  It consists of (1) an input embedding layer, (2) a bidirectional LSTM layer, (3) an average pooling layer of input features  The concatenation of the LSTM's and average pool layer is passed through a dense layer and the output is passed through a softmax function  We set two input channels for the input embedding layers: pre-trained FastText embeddings BIBREF14 , as well as updatable embeddings learned by the model during training  Finally, we also apply a Convolutional Neural Network (CNN) model based on the architecture of BIBREF15 , using the same multi-channel inputs as the above BiLSTM 
Our models are trained on the training data, and evaluated by predicting the labels for the held-out test set  The distribution is described in Table TABREF15   We evaluate and compare the models using the macro-averaged F1-score as the label distribution is highly imbalanced  Per-class Precision (P), Recall (R), and F1-score (F1), also with other averaged metrics are also reported  The models are compared against baselines of predicting all labels as the majority or minority classes 
Offensive Language Detection
The performance on discriminating between offensive (OFF) and non-offensive (NOT) posts is reported in Table TABREF18   We can see that all systems perform significantly better than chance, with the neural models being substantially better than the SVM  The CNN outperforms the RNN model, achieving a macro-F1 score of 0 80 
Categorization of Offensive Language
In this experiment, the two systems were trained to discriminate between insults and threats (TIN) and untargeted (UNT) offenses, which generally refer to profanity  The results are shown in Table TABREF19  
The CNN system achieved higher performance in this experiment compared to the BiLSTM, with a macro-F1 score of 0 69  All systems performed better at identifying target and threats (TIN) than untargeted offenses (UNT) 
Offensive Language Target Identification
The results of the offensive target identification experiment are reported in Table TABREF20   Here the systems were trained to distinguish between three targets: a group (GRP), an individual (IND), or others (OTH)  All three models achieved similar results far surpassing the random baselines, with a slight performance edge for the neural models 
The performance of all systems for the OTH class is 0  This poor performances can be explained by two main factors  First, unlike the two other classes, OTH is a heterogeneous collection of targets  It includes offensive tweets targeted at organizations, situations, events, etc  making it more challenging for systems to learn discriminative properties of this class  Second, this class contains fewer training instances than the other two  There are only 395 instances in OTH, and 1,075 in GRP, and 2,407 in IND 
Conclusion and Future Work
This paper presents OLID, a new dataset with annotation of type and target of offensive language  OLID is the official dataset of the shared task SemEval 2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) BIBREF16   In OffensEval, each annotation level in OLID is an independent sub-task  The dataset contains 14,100 tweets and is released freely to the research community  To the best of our knowledge, this is the first dataset to contain annotation of type and target of offenses in social media and it opens several new avenues for research in this area  We present baseline experiments using SVMs and neural networks to identify the offensive tweets, discriminate between insults, threats, and profanity, and finally to identify the target of the offensive messages  The results show that this is a challenging task  A CNN-based sentence classifier achieved the best results in all three sub-tasks 
In future work, we would like to make a cross-corpus comparison of OLID and datasets annotated for similar tasks such as aggression identification BIBREF2 and hate speech detection BIBREF8   This comparison is, however, far from trivial as the annotation of OLID is different 
Acknowledgments
The research presented in this paper was partially supported by an ERAS fellowship awarded by the University of Wolverhampton 

ground_truth: 8,275 sentences and 167,739 words in total

The corpus comprises 8,275 sentences and 167,739 words in total.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. In what language are the tweets?
Context 4:  As a training dataset, we use the corpus developed by BIBREF36 , which contains 2,400 essays labeled by one of the five personality traits each 
The fully-connected layer has 150 neurons, which are treated as the features  We concatenate the feature vector of each personality dimension in order to create the final feature vector  Thus, the personality model ultimately extracts a 750-dimensional feature vector (150-dimensional feature vector for each of the five personality traits)  This network is replicated five times, one for each personality trait  In particular, we create a CNN for each personality trait and the aim of each CNN is to classify a sentence into binary classes, i e , whether it expresses a personality trait or not 
Baseline Method and Features
CNN can also be employed on the sarcasm datasets in order to identify sarcastic and non-sarcastic tweets  We term the features extracted from this network baseline features, the method as baseline method and the CNN architecture used in this baseline method as baseline CNN  Since the fully-connected layer has 100 neurons, we have 100 baseline features in our experiment  This method is termed baseline method as it directly aims to classify a sentence as sarcastic vs non-sarcastic  The baseline CNN extracts the inherent semantics from the sarcastic corpus by employing deep domain understanding  The process of using baseline features with other features extracted from the pre-trained model is described in Section SECREF24  
Experimental Results and Discussion
In this section, we present the experimental results using different feature combinations and compare them with the state of the art  For each feature we show the results using only CNN and using CNN-SVM (i e , when the features extracted by CNN are fed to the SVM)  Macro-F1 measure is used as an evaluation scheme in the experiments 
Sarcasm Datasets Used in the Experiment
This dataset was created by BIBREF8   The tweets were downloaded from Twitter using #sarcasm as a marker for sarcastic tweets  It is a monolingual English dataset which consists of a balanced distribution of 50,000 sarcastic tweets and 50,000 non-sarcastic tweets 
Since sarcastic tweets are less frequently used BIBREF8 , we also need to investigate the robustness of the selected features and the model trained on these features on an imbalanced dataset  To this end, we used another English dataset from BIBREF8   It consists of 25,000 sarcastic tweets and 75,000 non-sarcastic tweets 
We have obtained this dataset from The Sarcasm Detector  It contains 120,000 tweets, out of which 20,000 are sarcastic and 100,000 are non-sarcastic  We randomly sampled 10,000 sarcastic and 20,000 non-sarcastic tweets from the dataset  Visualization of both the original and subset data show similar characteristics 
A two-step methodology has been employed in filtering the datasets used in our experiments  Firstly, we identified and removed all the “user", “URL" and “hashtag" references present in the tweets using efficient regular expressions  Special emphasis was given to this step to avoid traces of hashtags, which might trigger the models to provide biased results  Secondly, we used NLTK Twitter Tokenizer to ensure proper tokenization of words along with special symbols and emoticons  Since our deep CNNs extract contextual information present in tweets, we include emoticons as part of the vocabulary  This enables the emoticons to hold a place in the word embedding space and aid in providing information about the emotions present in the sentence 
Merging the Features
Throughout this research, we have carried out several experiments with various feature combinations  For the sake of clarity, we explain below how the features extracted using difference models are merged 
In the standard feature merging process, we first extract the features from all deep CNN based feature extraction models and then we concatenate them  Afterwards, SVM is employed on the resulted feature vector 
In another setting, we use the features extracted from the pre-trained models as the static channels of features in the CNN of the baseline method  These features are appended to the hidden layer of the baseline CNN, preceding the final output softmax layer 
For comparison, we have re-implemented the state-of-the-art methods  Since BIBREF9 did not mention about the sentiment lexicon they use in the experiment, we used SenticNet BIBREF37 in the re-implementation of their method 
Results on Dataset 1
As shown in Table TABREF29 , for every feature CNN-SVM outperforms the performance of the CNN  Following BIBREF6 , we have carried out a 5-fold cross-validation on this dataset  The baseline features ( SECREF16 ) perform best among other features  Among all the pre-trained models, the sentiment model (F1-score: 87 00%) achieves better performance in comparison with the other two pre-trained models  Interestingly, when we merge the baseline features with the features extracted by the pre-trained deep NLP models, we only get 0 11% improvement over the F-score  It means that the baseline features alone are quite capable to detect sarcasm  On the other hand, when we combine sentiment, emotion and personality features, we obtain 90 70% F1-score  This indicates that the pre-trained features are indeed useful for sarcasm detection  We also compare our approach with the best research study conducted on this dataset (Table TABREF30 )  Both the proposed baseline model and the baseline + sentiment + emotion + personality model outperform the state of the art BIBREF9 , BIBREF8   One important difference with the state of the art is that BIBREF8 used relatively larger feature vector size ( INLINEFORM0 500,000) than we used in our experiment (1,100)  This not only prevents our model to overfit the data but also speeds up the computation  Thus, we obtain an improvement in the overall performance with automatic feature extraction using a relatively lower dimensional feature space 
In the literature, word n-grams, skipgrams and character n-grams are used as baseline features  According to Ptacek et al  BIBREF8 , these baseline features along with the other features (sentiment features and part-of-speech based features) produced the best performance  However, Ptacek et al  did not analyze the performance of these features when they were not used with the baseline features  Pre-trained word embeddings play an important role in the performance of the classifier because, when we use randomly generated embeddings, performance falls down to 86 23% using all features 
Results on Dataset 2
5-fold cross-validation has been carried out on Dataset 2  Also for this dataset, we get the best accuracy when we use all features  Baseline features have performed significantly better (F1-score: 92 32%) than all other features  Supporting the observations we have made from the experiments on Dataset 1, we see CNN-SVM outperforming CNN on Dataset 2  However, when we use all the features, CNN alone (F1-score: 89 73%) does not outperform the state of the art BIBREF8 (F1-score: 92 37%)  As shown in Table TABREF30 , CNN-SVM on the baseline + sentiment + emotion + personality feature set outperforms the state of the art (F1-score: 94 80%)  Among the pre-trained models, the sentiment model performs best (F1-score: 87 00%) 
Table TABREF29 shows the performance of different feature combinations  The gap between the F1-scores of only baseline features and all features is larger on the imbalanced dataset than the balanced dataset  This supports our claim that sentiment, emotion and personality features are very useful for sarcasm detection, thanks to the pre-trained models  The F1-score using sentiment features when combined with baseline features is 94 60%  On both of the datasets, emotion and sentiment features perform better than the personality features  Interestingly, using only sentiment, emotion and personality features, we achieve 90 90% F1-score 
Results on Dataset 3
Experimental results on Dataset 3 show the similar trends (Table TABREF30 ) as compared to Dataset 1 and Dataset 2  The highest performance (F1-score 93 30%) is obtained when we combine baseline features with sentiment, emotion and personality features  In this case, also CNN-SVM consistently performs better than CNN for every feature combination  The sentiment model is found to be the best pre-trained model  F1-score of 84 43% is obtained when we merge sentiment, emotion and personality features 
Dataset 3 is more complex and non-linear in nature compared to the other two datasets  As shown in Table TABREF30 , the methods by BIBREF9 and BIBREF8 perform poorly on this dataset  The TP rate achieved by BIBREF9 is only 10 07% and that means their method suffers badly on complex data  The approach of BIBREF8 has also failed to perform well on Dataset 3, achieving 62 37% with a better TP rate of 22 15% than BIBREF9   On the other hand, our proposed model performs consistently well on this dataset achieving 93 30% 
Testing Generalizability of the Models and Discussions
To test the generalization capability of the proposed approach, we perform training on Dataset 1 and test on Dataset 3  The F1-score drops down dramatically to 33 05%  In order to understand this finding, we visualize each dataset using PCA (Figure FIGREF17 )  It depicts that, although Dataset 1 is mostly linearly separable, Dataset 3 is not  A linear kernel that performs well on Dataset 1 fails to provide good performance on Dataset 3  If we use RBF kernel, it overfits the data and produces worse results than what we get using linear kernel  Similar trends are seen in the performance of other two state-of-the-art approaches BIBREF9 , BIBREF8   Thus, we decide to perform training on Dataset 3 and test on the Dataset 1  As expected better performance is obtained with F1-score 76 78%  However, the other two state-of-the-art approaches fail to perform well in this setting  While the method by BIBREF9 obtains F1-score of 47 32%, the approach by BIBREF8 achieves 53 02% F1-score when trained on Dataset 3 and tested on Dataset 1  Below, we discuss about this generalizability issue of the models developed or referred in this paper 
As discussed in the introduction, sarcasm is very much topic-dependent and highly contextual  For example, let us consider the tweet “I am so glad to see Tanzania played very well, I can now sleep well :P"  Unless one knows that Tanzania actually did not play well in that game, it is not possible to spot the sarcastic nature of this sentence  Thus, an n-gram based sarcasm detector trained at time INLINEFORM0 may perform poorly to detect sarcasm in the tweets crawled at time INLINEFORM1 (given that there is a considerable gap between these time stamps) because of the diversity of the topics (new events occur, new topics are discussed) of the tweets  Sentiment and other contextual clues can help to spot the sarcastic nature in this kind of tweets  A highly positive statement which ends with a emoticon expressing joke can be sarcastic 
State-of-the-art methods lack these contextual information which, in our case, we extract using pre-trained sentiment, emotion and personality models  Not only these pre-trained models, the baseline method (baseline CNN architecture) performs better than the state-of-the-art models in this generalizability test setting  In our generalizability test, when the pre-trained features are used with baseline features, we get 4 19% F1-score improvement over the baseline features  On the other hand, when they are not used with the baseline features, together they produce 64 25% F1-score 
Another important fact is that an n-grams model cannot perform well on unseen data unless it is trained on a very large corpus  If most of the n-grams extracted from the unseen data are not in the vocabulary of the already trained n-grams model, in fact, the model will produce a very sparse feature vector representation of the dataset  Instead, we use the word2vec embeddings as the source of the features, as word2vec allows for the computation of similarities between unseen data and training data 
Baseline Features vs Pre-trained Features
Our experimental results show that the baseline features outperform the pre-trained features for sarcasm detection  However, the combination of pre-trained features and baseline features beats both of themselves alone  It is counterintuitive, since experimental results prove that both of those features learn almost the same global and contextual features  In particular, baseline network dominates over pre-trained network as the former learns most of the features learned by the latter  Nonetheless, the combination of baseline and pre-trained classifiers improves the overall performance and generalizability, hence proving their effectiveness in sarcasm detection  Experimental results show that sentiment and emotion features are the most useful features, besides baseline features (Figure FIGREF36 )  Therefore, in order to reach a better understanding of the relation between personality features among themselves and with other pre-trained features, we carried out Spearman correlation testing  Results, displayed in Table TABREF39 , show that those features are highly correlated with each other 
Conclusion
In this work, we developed pre-trained sentiment, emotion and personality models for identifying sarcastic text using CNN, which are found to be very effective for sarcasm detection  In the future, we plan to evaluate the performance of the proposed method on a large corpus and other domain-dependent corpora  Future work will also focus on analyzing past tweets and activities of users in order to better understand their personality and profile and, hence, further improve the disambiguation between sarcastic and non-sarcastic text 

ground_truth: English

English 

English