prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What dataset does this approach achieve state of the art results on?
Context 1: Data
We build and test our MMT models on the Multi30K dataset BIBREF21   Each image in Multi30K contains one English (EN) description taken from Flickr30K BIBREF22 and human translations into German (DE), French (FR) and Czech BIBREF23 , BIBREF24 , BIBREF25   The dataset contains 29,000 instances for training, 1,014 for development, and 1,000 for test  We only experiment with German and French, which are languages for which we have in-house expertise for the type of analysis we present  In addition to the official Multi30K test set (test 2016), we also use the test set from the latest WMT evaluation competition, test 2018 BIBREF25  
Degradation of source
In addition to using the Multi30K dataset as is (standard setup), we probe the ability of our models to address the three linguistic phenomena where additional context has been proved important (Section ): ambiguities, gender-neutral words and noisy input  In a controlled experiment where we aim to remove the influence of frequency biases, we degrade the source sentences by masking words through three strategies to replace words by a placeholder: random source words, ambiguous source words and gender unmarked source words  The procedure is applied to the train, validation and test sets  For the resulting dataset generated for each setting, we compare models having access to text-only context versus additional text and multimodal contexts  We seek to get insights into the contribution of each type of context to address each type of degradation 
In this setting (RND) we simulate erroneous source words by randomly dropping source content words  We first tag the entire source sentences using the spacy toolkit BIBREF26 and then drop nouns, verbs, adjectives and adverbs and replace these with a default BLANK token  By focusing on content words, we differ from previous work that suggests that neural machine translation is robust to non-content word noise in the source BIBREF27  
In this setting (AMB), we rely on the MLT dataset BIBREF11 which provides a list of source words with multiple translations in the Multi30k training set  We replace ambiguous words with the BLANK token in the source language, which results in two language-specific datasets 
In this setting (PERS), we use the Flickr Entities dataset BIBREF28 to identify all the words that were annotated by humans as corresponding to the category person  We then replace such source words with the BLANK token 
The statistics of the resulting datasets for the three degradation strategies are shown in Table TABREF10   We note that RND and PERS are the same for language pairs as the degradation only depends on the source side, while for AMB the words replaced depend on the target language 
Models
Based on the models described in Section we experiment with eight variants: (a) baseline transformer model (base); (b) base with AIC (base+sum); (c) base with AIF using spacial (base+att) or object based (base+obj) image features; (d) standard deliberation model (del); (e) deliberation models enriched with image information: del+sum, del+att and del+obj 
Training
In all cases, we optimise our models with cross entropy loss  For deliberation network models, we first train the standard transformer model until convergence, and use it to initialise the encoder and first-pass decoder  For each of the training samples, we follow BIBREF19 and obtain a set of 10-best samples from the first pass decoder, with a beam search of size 10  We use these as the first-pass decoder samples  We use Adam as optimiser BIBREF29 and train the model until convergence 
Results
In this section we present results of our experiments, first in the original dataset without any source degradation (Section SECREF18 ) and then in the setup with various source degradation strategies (Section SECREF25 ) 
Standard setup
Table TABREF14 shows the results of our main experiments on the 2016 and 2018 test sets for French and German  We use Meteor BIBREF31 as the main metric, as in the WMT tasks BIBREF25   We compare our transformer baseline to transformer models enriched with image information, as well as to the deliberation models, with or without image information 
We first note that our multimodal models achieve the state of the art performance for transformer networks (constrained models) on the English-German dataset, as compared to BIBREF30   Second, our deliberation models lead to significant improvements over this baseline across test sets (average INLINEFORM0 , INLINEFORM1 ) 
Transformer-based models enriched with image information (base+sum, base+att and base+obj), on the other hand, show no major improvements with respect to the base performance  This is also the case for deliberation models with image information (del+sum, del+att, del+obj), which do not show significant improvement over the vanilla deliberation performance (del) 
However, as it has been shown in the WMT shared tasks on MMT BIBREF23 , BIBREF24 , BIBREF25 , automatic metrics often fail to capture nuances in translation quality, such as, the ones we expect the visual modality to help with, which – according to human perception – lead to better translations  To test this assumption in our settings, we performed human evaluation involving professional translators and native speakers of both French and German (three annotators) 
The annotators were asked to rank randomly selected test samples according to how well they convey the meaning of the source, given the image (50 samples per language pair per annotator)  For each source segment, the annotator was shown the outputs of three systems: base+att, the current MMT state-of-the-art BIBREF30 , del and del+obj  A rank could be assigned from 1 to 3, allowing ties BIBREF32   Annotators could assign zero rank to all translations if they were judged incomprehensible  Following the common practice in WMT BIBREF32 , each system was then assigned a score which reflects the proportion of times it was judged to be better or equal other systems 
Table TABREF19 shows the human evaluation results  They are consistent with the automatic evaluation results when it comes to the preference of humans towards the deliberation-based setups, but show a more positive outlook regarding the addition of visual information (del+obj over del) for French 
Manual inspection of translations suggests that deliberation setups tend to improve both the grammaticality and adequacy of the first pass outputs  For German, the most common modifications performed by the second-pass decoder are substitutions of adjectives and verbs (for test 2016, 15% and 12% respectively, of all the edit distance operations)  Changes to adjectives are mainly grammatical, changes to verbs are contextual (e g , changing laufen to rennen, both verbs mean run, but the second refers to running very fast)  For French, 15% of all the changes are substitutions of nouns (for test 2016)  These are again very contextual  For example, the French word travailleur (worker) is replaced by ouvrier (manual worker) in the contexts where tools, machinery or buildings are mentioned  For our analysis we used again spacy 
The information on detected objects is particularly helpful for specific adequacy issues  Figure FIGREF15 demonstrates some such cases  In the first case, the base+att model misses the translation of race car: the German word Rennen translates only the word race  del introduces the word car (Auto) into the translation  Finally, del+obj correctly translates the expression race car (Rennwagen) by exploiting the object information  For French, del translates the source part in a body of water, missing from the base+att translation  del+obj additionally translated the word paddling according to the detected object Paddle 
Source degradation setup
Results of our source degradation experiments are shown in Table TABREF20   A first observation is that – as with the standard setup – the performance of our deliberation models is overall better than that of the base models  The results of the multimodal models differ for German and French  For German, del+obj is the most successful configuration and shows statistically significant improvements over base for all setups  Moreover, for RND and AMB, it shows statistically significant improvements over del  However, especially for RND and AMB, del and del+sum are either the same or slightly worse than base 
For French, all the deliberation models show statistically significant improvements over base (average INLINEFORM0 , INLINEFORM1 ), but the image information added to del only improve scores significantly for test 2018 RND 
This difference in performances for French and German is potentially related to the need of more significant restructurings while translating from English into German  This is where a more complex del+obj architecture is more helpful  This is especially true for RND and AMB setups where blanked words could also be verbs, the part-of-speech most influenced by word order differences between English and German (see the decreasing complexity of translations for del and del+obj for the example (c) in Figure FIGREF21 ) 
To get an insight into the contribution of different contexts to the resolution of blanks, we performed manual analysis of examples coming from the English-German base, del and del+obj setups (50 random examples per setup), where we count correctly translated blanks per system 
The results are shown in Table TABREF27   As expected, they show that the RND and AMB blanks are more difficult to resolve (at most 40% resolved as compared to 61% for PERS)  Translations of the majority of those blanks tend to be guessed by the textual context alone (especially for verbs)  Image information is more helpful for PERS: we observe an increase of 10% in resolved blanks for del+obj as compared to del  However, for PERS the textual context is still enough in the majority of the cases: models tend to associate men with sports or women with cooking and are usually right (see Figure FIGREF21 example (c)) 
The cases where image helps seem to be those with rather generic contexts: see Figure FIGREF21 (b) where enjoying a summer day is not associated with any particular gender and make other models choose homme (man) or femme (woman), and only base+obj chooses enfant (child) (the option closest to the reference) 
In some cases detected objects are inaccurate or not precise enough to be helpful (e g , when an object Person is detected) and can even harm correct translations 
Conclusions
We have proposed a novel approach to multimodal machine translation which makes better use of context, both textual and visual  Our results show that further exploring textual context through deliberation networks already leads to better results than the previous state of the art  Adding visual information, and in particular structural representations of this information, proved beneficial when input text contains noise and the language pair requires substantial restructuring from source to target  Our findings suggest that the combination of a deliberation approach and information from additional modalities is a promising direction for machine translation that is robust to noisy input  Our code and pre-processing scripts are available at https://github com/ImperialNLP/MMT-Delib 
Acknowledgments
The authors thank the anonymous reviewers for their useful feedback  This work was supported by the MultiMT (H2020 ERC Starting Grant No  678017) and MMVC (Newton Fund Institutional Links Grant, ID 352343575) projects  We also thank the annotators for their valuable help 
Context 2: Introduction
How humans process language has become increasingly relevant in natural language processing since physiological data during language understanding is more accessible and recorded with less effort  In this work, we focus on eye-tracking and electroencephalography (EEG) recordings to capture the reading process  On one hand, eye movement data provides millisecond-accurate records about where humans look when they are reading, and is highly correlated with the cognitive load associated with different stages of text processing  On the other hand, EEG records electrical brain activity across the scalp and is a direct measure of physiological processes, including language processing  The combination of both measurement methods enables us to study the language understanding process in a more natural setting, where participants read full sentences at a time, in their own speed  Eye-tracking then permits us to define exact word boundaries in the timeline of a subject reading a sentence, allowing the extraction of brain activity signals for each word 
Human cognitive language processing data is immensely useful for NLP: Not only can it be leveraged to improve NLP applications (e g  barrett2016weakly for part-of-speech tagging or klerke2016improving for sentence compression), but also to evaluate state-of-the-art machine learning systems  For example, hollenstein2019cognival evaluate word embeddings, or schwartz2019inducing fine-tune language models with brain-relevant bias 
Additionally, the availability of labelled data plays a crucial role in all supervised machine learning applications  Physiological data can be used to understand and improve the labelling process (e g  tokunaga2017eye), and, for instance, to build cost models for active learning scenarios BIBREF0  Is it possible to replace this expensive manual work with models trained on physiological activity data recorded from humans while reading? That is to say, can we find and extract relevant aspects of text understanding and annotation directly from the source, i e  eye-tracking and brain activity signals during reading?
Motivated by these questions and our previously released dataset, ZuCo 1 0 BIBREF1, we developed this new corpus, where we specifically aim to collect recordings during natural reading as well as during annotation 
We provide the first dataset of simultaneous eye movement and brain activity recordings to analyze and compare normal reading to task-specific reading during annotation  The Zurich Cognitive Language Processing Corpus (ZuCo) 2 0, including raw and preprocessed eye-tracking and electroencephalography (EEG) data of 18 subjects, as well as the recording and preprocessing scripts, is publicly available at https://osf io/2urht/  It contains physiological data of each subject reading 739 English sentences from Wikipedia (see example in Figure FIGREF1)  We want to highlight the re-use potential of this data  In addition to the psycholinguistic motivation, this corpus is especially tailored for training and evaluating machine learning algorithms for NLP purposes  We conduct a detailed technical validation of the data as proof of the quality of the recordings 
Related Work
Some eye-tracking corpora of natural reading (e g  the Dundee BIBREF2, Provo BIBREF3 and GECO corpus BIBREF4), and a few EEG corpora (for example, the UCL corpus BIBREF5) are available  It has been shown that this type of cognitive processing data is useful for improving and evaluating NLP methods (e g  barrett2018sequence,hollenstein2019cognival, hale2018finding)  However, before the Zurich Cognitive Language Processing Corpus (ZuCo 1 0), there was no available data for simultaneous eye-tracking and EEG recordings of natural reading  dimigen2011coregistration studied the linguistic effects of eye movements and EEG co-registration in natural reading and showed that they accurately represent lexical processing  Moreover, the simultaneous recordings are crucial to extract word-level brain activity signals 
While the above mentioned studies analyze and leverage natural reading, some NLP work has used eye-tracking during annotation (but, as of yet, not EEG data)  mishra2016predicting and joshi2014measuring recorded eye-tracking during binary sentiment annotation (positive/negative)  This data was used to determine the annotation complexity of the text passages based on eye movement metrics and for sarcasm detection BIBREF6  Moreover, eye-tracking has been used to analyze the word sense annotation process in Hindi BIBREF7, named entity annotation in Japanese BIBREF8, and to leverage annotator gaze behaviour for coreference resolution BIBREF9  Finally, tomanek2010cognitive used eye-tracking data during entity annotation to build a cost model for active learning  However, until now there is no available data or research that analyzes the differences in the human processing of normal reading versus annotation 
Related Work ::: ZuCo1 0
In previous work, we recorded a first dataset of simultaneous eye-tracking and EEG during natural reading BIBREF1  ZuCo 1 0 consists of three reading tasks, two of which contain very similar reading material and experiments as presented in the current work  However, the main difference and reason for recording ZuCo 2 0, consists in the experiment procedure  For ZuCo 1 0 the normal reading and task-specific reading paradigms were recorded in different sessions on different days  Therefore, the recorded data is not appropriate as a means of comparison between natural reading and annotation, since the differences in the brain activity data might result mostly from the different sessions due to the sensitivity of EEG  This, and extending the dataset with more sentences and more subjects, were the main factors for recording the current corpus  We purposefully maintained an overlap of some sentences between both datasets to allow additional analyses (details are described in Section SECREF7) 
Corpus Construction
In this section we describe the contents and experimental design of the ZuCo 2 0 corpus 
Corpus Construction ::: Participants
We recorded data from 19 participants and discarded the data of one of them due to technical difficulties with the eye-tracking calibration  Hence, we share the data of 18 participants  All participants are healthy adults (mean age = 34 (SD=8 3), 10 females)  Their native language is English, originating from Australia, Canada, UK, USA or South Africa  Two participants are left-handed and three participants wear glasses for reading  Details on subject demographics can be found in Table TABREF4  All participants gave written consent for their participation and the re-use of the data prior to the start of the experiments  The study was approved by the Ethics Commission of the University of Zurich 
Corpus Construction ::: Reading materials
During the recording session, the participants read 739 sentences that were selected from the Wikipedia corpus provided by culotta2006integrating  This corpus was chosen because it provides annotations of semantic relations  We included seven of the originally defined relation types: political_affiliation, education, founder, wife/husband, job_title, nationality, and employer  The sentences were chosen in the same length range as ZuCo 1 0, and with similar Flesch reading ease scores  The dataset statistics are shown in Table TABREF2 
Of the 739 sentences, the participants read 349 sentences in a normal reading paradigm, and 390 sentences in a task-specific reading paradigm, in which they had to determine whether a certain relation type occurred in the sentence or not  Table TABREF3 shows the distribution of the different relation types in the sentences of the task-specific annotation paradigm 
Purposefully, there are 63 duplicates between the normal reading and the task-specific sentences (8% of all sentences)  The intention of these duplicate sentences is to provide a set of sentences read twice by all participants with a different task in mind  Hence, this enables the comparison of eye-tracking and brain activity data when reading normally and when annotating specific relations (see examples in Section SECREF4) 
Furthermore, there is also an overlap in the sentences between ZuCo 1 0 and ZuCo 2 0  100 normal reading and 85 task-specific sentences recorded for this dataset were already recorded in ZuCo 1 0  This allows for comparisons between the different recording procedures (i e  session-specific effects) and between more participants (subject-specific effects) 
Corpus Construction ::: Experimental design
As mentioned above, we recorded two different reading tasks for the ZuCo 2 0 dataset  During both tasks the participants were able to read in their own speed, using a control pad to move to the next sentence and to answer the control questions, which allowed for natural reading  Since each subject reads at their own personal pace, the reading speed between varies between subjects  Table TABREF4 shows the average reading speed for each task, i e  the average number of seconds a subject spends per sentence before switching to the next one 
All 739 sentences were recorded in a single session for each participant  The duration of the recording sessions was between 100 and 180 minutes, depending on the time required to set up and calibrate the devices, and the personal reading speed of the participants 
We recorded 14 blocks of approx  50 sentences, alternating between tasks: 50 sentences of normal reading, followed by 50 sentences of task-specific reading  The order of blocks and sentences within blocks was identical for all subjects  Each sentence block was preceded by a practice round of three sentences 
Corpus Construction ::: Experimental design ::: Normal reading (NR)
In the first task, participants were instructed to read the sentences naturally, without any specific task other than comprehension  Participants were told to read the sentences normally without any special instructions  Figure FIGREF8 (left) shows an example sentence as it was depicted on the screen during recording  As shown in Figure FIGREF8 (middle), the control condition for this task consisted of single-choice questions about the content of the previous sentence  12% of randomly selected sentences were followed by such a comprehension question with three answer options on a separate screen 
Corpus Construction ::: Experimental design ::: Task-specific reading (TSR)
In the second task, the participants were instructed to search for a specific relation in each sentence they read  Instead of comprehension questions, the participants had to decide for each sentence whether it contains the relation or not, i e  they were actively annotating each sentence  Figure FIGREF8 (right) shows an example screen for this task  17% of the sentences did not include the relation type and were used as control conditions  All sentences within one block involved the same relation type
Context 3: Introduction
Sarcasm is defined as “a sharp, bitter, or cutting expression or remark; a bitter gibe or taunt”  As the fields of affective computing and sentiment analysis have gained increasing popularity BIBREF0 , it is a major concern to detect sarcastic, ironic, and metaphoric expressions  Sarcasm, especially, is key for sentiment analysis as it can completely flip the polarity of opinions  Understanding the ground truth, or the facts about a given event, allows for the detection of contradiction between the objective polarity of the event (usually negative) and its sarcastic characteristic by the author (usually positive), as in “I love the pain of breakup”  Obtaining such knowledge is, however, very difficult 
In our experiments, we exposed the classifier to such knowledge extracted indirectly from Twitter  Namely, we used Twitter data crawled in a time period, which likely contain both the sarcastic and non-sarcastic accounts of an event or similar events  We believe that unambiguous non-sarcastic sentences provided the classifier with the ground-truth polarity of those events, which the classifier could then contrast with the opposite estimations in sarcastic sentences  Twitter is a more suitable resource for this purpose than blog posts, because the polarity of short tweets is easier to detect (as all the information necessary to detect polarity is likely to be contained in the same sentence) and because the Twitter API makes it easy to collect a large corpus of tweets containing both sarcastic and non-sarcastic examples of the same event 
Sometimes, however, just knowing the ground truth or simple facts on the topic is not enough, as the text may refer to other events in order to express sarcasm  For example, the sentence “If Hillary wins, she will surely be pleased to recall Monica each time she enters the Oval Office :P :D”, which refers to the 2016 US presidential election campaign and to the events of early 1990's related to the US president Clinton, is sarcastic because Hillary, a candidate and Clinton's wife, would in fact not be pleased to recall her husband's alleged past affair with Monica Lewinsky  The system, however, would need a considerable amount of facts, commonsense knowledge, anaphora resolution, and logical reasoning to draw such a conclusion  In this paper, we will not deal with such complex cases 
Existing works on sarcasm detection have mainly focused on unigrams and the use of emoticons BIBREF1 , BIBREF2 , BIBREF3 , unsupervised pattern mining approach BIBREF4 , semi-supervised approach BIBREF5 and n-grams based approach BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 with sentiment features  Instead, we propose a framework that learns sarcasm features automatically from a sarcasm corpus using a convolutional neural network (CNN)  We also investigate whether features extracted using the pre-trained sentiment, emotion and personality models can improve sarcasm detection performance  Our approach uses relatively lower dimensional feature vectors and outperforms the state of the art on different datasets  In summary, the main contributions of this paper are the following:
The rest of the paper is organized as follows: Section SECREF2 proposes a brief literature review on sarcasm detection; Section SECREF4 presents the proposed approach; experimental results and thorough discussion on the experiments are given in Section SECREF5 ; finally, Section SECREF6 concludes the paper 
Related Works
NLP research is gradually evolving from lexical to compositional semantics BIBREF10 through the adoption of novel meaning-preserving and context-aware paradigms such as convolutional networks BIBREF11 , recurrent belief networks BIBREF12 , statistical learning theory BIBREF13 , convolutional multiple kernel learning BIBREF14 , and commonsense reasoning BIBREF15   But while other NLP tasks have been extensively investigated, sarcasm detection is a relatively new research topic which has gained increasing interest only recently, partly thanks to the rise of social media analytics and sentiment analysis  Sentiment analysis BIBREF16 and using multimodal information as a new trend BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , BIBREF14 is a popular branch of NLP research that aims to understand sentiment of documents automatically using combination of various machine learning approaches BIBREF21 , BIBREF22 , BIBREF20 , BIBREF23  
An early work in this field was done by BIBREF6 on a dataset of 6,600 manually annotated Amazon reviews using a kNN-classifier over punctuation-based and pattern-based features, i e , ordered sequence of high frequency words  BIBREF1 used support vector machine (SVM) and logistic regression over a feature set of unigrams, dictionary-based lexical features and pragmatic features (e g , emoticons) and compared the performance of the classifier with that of humans  BIBREF24 described a set of textual features for recognizing irony at a linguistic level, especially in short texts created via Twitter, and constructed a new model that was assessed along two dimensions: representativeness and relevance  BIBREF5 used the presence of a positive sentiment in close proximity of a negative situation phrase as a feature for sarcasm detection  BIBREF25 used the Balanced Window algorithm for classifying Dutch tweets as sarcastic vs  non-sarcastic; n-grams (uni, bi and tri) and intensifiers were used as features for classification 
BIBREF26 compared the performance of different classifiers on the Amazon review dataset using the imbalance between the sentiment expressed by the review and the user-given star rating  Features based on frequency (gap between rare and common words), written spoken gap (in terms of difference between usage), synonyms (based on the difference in frequency of synonyms) and ambiguity (number of words with many synonyms) were used by BIBREF3 for sarcasm detection in tweets  BIBREF9 proposed the use of implicit incongruity and explicit incongruity based features along with lexical and pragmatic features, such as emoticons and punctuation marks  Their method is very much similar to the method proposed by BIBREF5 except BIBREF9 used explicit incongruity features  Their method outperforms the approach by BIBREF5 on two datasets 
BIBREF8 compared the performance with different language-independent features and pre-processing techniques for classifying text as sarcastic and non-sarcastic  The comparison was done over three Twitter dataset in two different languages, two of these in English with a balanced and an imbalanced distribution and the third one in Czech  The feature set included n-grams, word-shape patterns, pointedness and punctuation-based features 
In this work, we use features extracted from a deep CNN for sarcasm detection  Some of the key differences between the proposed approach and existing methods include the use of a relatively smaller feature set, automatic feature extraction, the use of deep networks, and the adoption of pre-trained NLP models 
Sentiment Analysis and Sarcasm Detection
Sarcasm detection is an important subtask of sentiment analysis BIBREF27   Since sarcastic sentences are subjective, they carry sentiment and emotion-bearing information  Most of the studies in the literature BIBREF28 , BIBREF29 , BIBREF9 , BIBREF30 include sentiment features in sarcasm detection with the use of a state-of-the-art sentiment lexicon  Below, we explain how sentiment information is key to express sarcastic opinions and the approach we undertake to exploit such information for sarcasm detection 
In general, most sarcastic sentences contradict the fact  In the sentence “I love the pain present in the breakups" (Figure FIGREF4 ), for example, the word “love" contradicts “pain present in the breakups”, because in general no-one loves to be in pain  In this case, the fact (i e , “pain in the breakups") and the contradictory statement to that fact (i e , “I love") express sentiment explicitly  Sentiment shifts from positive to negative but, according to sentic patterns BIBREF31 , the literal sentiment remains positive  Sentic patterns, in fact, aim to detect the polarity expressed by the speaker; thus, whenever the construction “I love” is encountered, the sentence is positive no matter what comes after it (e g , “I love the movie that you hate”)  In this case, however, the sentence carries sarcasm and, hence, reflects the negative sentiment of the speaker 
In another example (Figure FIGREF4 ), the fact, i e , “I left the theater during the interval", has implicit negative sentiment  The statement “I love the movie" contradicts the fact “I left the theater during the interval"; thus, the sentence is sarcastic  Also in this case the sentiment shifts from positive to negative and hints at the sarcastic nature of the opinion 
The above discussion has made clear that sentiment (and, in particular, sentiment shifts) can largely help to detect sarcasm  In order to include sentiment shifting into the proposed framework, we train a sentiment model for sentiment-specific feature extraction  Training with a CNN helps to combine the local features in the lower layers into global features in the higher layers  We do not make use of sentic patterns BIBREF31 in this paper but we do plan to explore that research direction as a part of our future work  In the literature, it is found that sarcasm is user-specific too, i e , some users have a particular tendency to post more sarcastic tweets than others  This acts as a primary intuition for us to extract personality-based features for sarcasm detection 
The Proposed Framework
As discussed in the literature BIBREF5 , sarcasm detection may depend on sentiment and other cognitive aspects  For this reason, we incorporate both sentiment and emotion clues in our framework  Along with these, we also argue that personality of the opinion holder is an important factor for sarcasm detection  In order to address all of these variables, we create different models for each of them, namely: sentiment, emotion and personality  The idea is to train each model on its corresponding benchmark dataset and, hence, use such pre-trained models together to extract sarcasm-related features from the sarcasm datasets 
Now, the viable research question here is - Do these models help to improve sarcasm detection performance?' Literature shows that they improve the performance but not significantly  Thus, do we need to consider those factors in spotting sarcastic sentences? Aren't n-grams enough for sarcasm detection? Throughout the rest of this paper, we address these questions in detail  The training of each model is done using a CNN  Below, we explain the framework in detail  Then, we discuss the pre-trained models  Figure FIGREF6 presents a visualization of the proposed framework 
General CNN Framework
CNN can automatically extract key features from the training data  It grasps contextual local features from a sentence and, after several convolution operations, it forms a global feature vector out of those local features  CNN does not need the hand-crafted features used in traditional supervised classifiers  Such hand-crafted features are difficult to compute and a good guess for encoding the features is always necessary in order to get satisfactory results  CNN, instead, uses a hierarchy of local features which are important to learn context  The hand-crafted features often ignore such a hierarchy of local features 
Features extracted by CNN can therefore be used instead of hand-crafted features, as they carry more useful information  The idea behind convolution is to take the dot product of a vector of INLINEFORM0 weights INLINEFORM1 also known as kernel vector with each INLINEFORM2 -gram in the sentence INLINEFORM3 to obtain another sequence of features INLINEFORM4   DISPLAYFORM0
Thus, a max pooling operation is applied over the feature map and the maximum value INLINEFORM0 is taken as the feature corresponding to this particular kernel vector  Similarly, varying kernel vectors and window sizes are used to obtain multiple features BIBREF32   For each word INLINEFORM1 in the vocabulary, a INLINEFORM2 -dimensional vector representation is given in a look up table that is learned from the data BIBREF33   The vector representation of a sentence, hence, is a concatenation of vectors for individual words  Similarly, we can have look up tables for other features  One might want to provide features other than words if these features are suspected to be helpful  The convolution kernels are then applied to word vectors instead of individual words 
We use these features to train higher layers of the CNN, in order to represent bigger groups of words in sentences  We denote the feature learned at hidden neuron INLINEFORM0 in layer INLINEFORM1 as INLINEFORM2   Multiple features may be learned in parallel in the same CNN layer  The features learned in each layer are used to train the next layer: DISPLAYFORM0
where * indicates convolution and INLINEFORM0 is a weight kernel for hidden neuron INLINEFORM1 and INLINEFORM2 is the total number of hidden neurons  The CNN sentence model preserves the order of words by adopting convolution kernels of gradually increasing sizes that span an increasing number of words and ultimately the entire sentence  As mentioned above, each word in a sentence is represented using word embeddings 
We employ the publicly available word2vec vectors, which were trained on 100 billion words from Google News  The vectors are of dimensionality 300, trained using the continuous bag-of-words architecture BIBREF33   Words not present in the set of pre-trained words are initialized randomly  However, while training the neural network, we use non-static representations  These include the word vectors, taken as input, into the list of parameters to be learned during training 
Two primary reasons motivated us to use non-static channels as opposed to static ones  Firstly, the common presence of informal language and words in tweets resulted in a relatively high random initialization of word vectors due to the unavailability of these words in the word2vec dictionary  Secondly, sarcastic sentences are known to include polarity shifts in sentimental and emotional degrees  For example, “I love the pain present in breakups" is a sarcastic sentence with a significant change in sentimental polarity  As word2vec was not trained to incorporate these nuances, we allow our models to update the embeddings during training in order to include them  Each sentence is wrapped to a window of INLINEFORM0 , where INLINEFORM1 is the maximum number of words amongst all sentences in the dataset  We use the output of the fully-connected layer of the network as our feature vector 
We have done two kinds of experiments: firstly, we used CNN for the classification; secondly, we extracted features from the fully-connected layer of the CNN and fed them to an SVM for the final classification  The latter CNN-SVM scheme is quite useful for text classification as shown by Poria et al  BIBREF18   We carry out n-fold cross-validation on the dataset using CNN  In every fold iteration, in order to obtain the training and test features, the output of the fully-connected layer is treated as features to be used for the final classification using SVM  Table TABREF12 shows the training settings for each CNN model developed in this work  ReLU is used as the non-linear activation function of the network  The network configurations of all models developed in this work are given in Table TABREF12  
Sentiment Feature Extraction Model
As discussed above, sentiment clues play an important role for sarcastic sentence detection  In our work, we train a CNN (see Section SECREF5 for details) on a sentiment benchmark dataset  This pre-trained model is then used to extract features from the sarcastic datasets  In particular, we use Semeval 2014 BIBREF34 Twitter Sentiment Analysis Dataset for the training  This dataset contains 9,497 tweets out of which 5,895 are positive, 3,131 are negative and 471 are neutral  The fully-connected layer of the CNN used for sentiment feature extraction has 100 neurons, so 100 features are extracted from this pre-trained model  The final softmax determines whether a sentence is positive, negative or neutral  Thus, we have three neurons in the softmax layer 
Emotion Feature Extraction Model
We use the CNN structure as described in Section SECREF5 for emotional feature extraction  As a dataset for extracting emotion-related features, we use the corpus developed by BIBREF35   This dataset consists of blog posts labeled by their corresponding emotion categories  As emotion taxonomy, the authors used six basic emotions, i e , Anger, Disgust, Surprise, Sadness, Joy and Fear  In particular, the blog posts were split into sentences and each sentence was labeled  The dataset contains 5,205 sentences labeled by one of the emotion labels  After employing this model on the sarcasm dataset, we obtained a 150-dimensional feature vector from the fully-connected layer  As the aim of training is to classify each sentence into one of the six emotion classes, we used six neurons in the softmax layer 
Personality Feature Extraction Model
Detecting personality from text is a well-known challenging problem  In our work, we use five personality traits described by BIBREF36 , i e , Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, sometimes abbreviated as OCEAN (by their first letters)

ground_truth: the English-German dataset

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Did they experiment with this new dataset?
Context 2:  The blocks started with a practice round, which described the relation and was followed by three sample sentences, so that the participants would be familiar with the respective relation type 
Corpus Construction ::: Linguistic assessment
As a linguistic assessment, the vocabulary and language proficiency of the participants was tested with the LexTALE test (Lexical Test for Advanced Learners of English, lemhofer2012introducing)  This is an unspeeded lexical decision task designed for intermediate to highly proficient language users  The average LexTALE score over all participants was 88 54%  Moreover, we also report the scores the participants achieved with their answers to the reading comprehension control questions and their relation annotations  The detailed scores for all participants are also presented in Table TABREF4 
Corpus Construction ::: Data acquisition
Data acquisition took place in a sound-attenuated and dark experiment room  Participants were seated at a distance of 68cm from a 24-inch monitor with a resolution of 800x600 pixels  A stable head position was ensured via a chin rest  Participants were instructed to stay as still as possible during the tasks to avoid motor EEG artifacts  Participants were also offered snacks and water during the breaks and were encouraged to rest  All sentences were presented at the same position on the screen and could span multiple lines  The sentences were presented in black on a light grey background with font size 20-point Arial, resulting in a letter height of 0 8 mm  The experiment was programmed in MATLAB 2016b BIBREF10, using PsychToolbox BIBREF11  Participants completed the tasks sitting alone in the room, while two research assistants were monitoring their progress in the adjoining room  All recording scripts including detailed participant instructions are available alongside the data 
Corpus Construction ::: Data acquisition ::: Eye-tracking acquisition
Eye position and pupil size were recorded with an infrared video-based eye tracker (EyeLink 1000 Plus, SR Research) at a sampling rate of 500 Hz  The eye tracker was calibrated with a 9-point grid at the beginning of the session and re-validated before each block of sentences 
Corpus Construction ::: Data acquisition ::: EEG acquisition
High-density EEG data were recorded at a sampling rate of 500 Hz with a bandpass of 0 1 to 100 Hz, using a 128-channel EEG Geodesic Hydrocel system (Electrical Geodesics)  The recording reference was set at electrode Cz  The head circumference of each participant was measured to select an appropriately sized EEG net  To ensure good contact, the impedance of each electrode was checked prior to recording, and was kept below 40 kOhm  Electrode impedance levels were checked after every third block of 50 sentences (approx  every 30 mins) and reduced if necessary 
Corpus Construction ::: Preprocessing and feature extraction ::: Eye-tracking
The eye-tracking data consists of (x,y) gaze location entries for all individual fixations (Figure FIGREF1b)  Coordinates were given in pixels with respect to the monitor coordinates (the upper left corner of the screen was (0,0) and down/right was positive)  We provide this raw data as well as various engineered eye-tracking features  For this feature extraction only fixations within the boundaries of each displayed word were extracted  Data points distinctly not associated with reading (minimum distance of 50 pixels to the text) were excluded  Additionally, fixations shorter than 100 ms were excluded from the analyses, because these are unlikely to reflect fixations relevant for reading BIBREF12  On the basis of the GECO and ZuCo 1 0 corpora, we extracted the following features: (i) gaze duration (GD), the sum of all fixations on the current word in the first-pass reading before the eye moves out of the word; (ii) total reading time (TRT), the sum of all fixation durations on the current word, including regressions; (iii) first fixation duration (FFD), the duration of the first fixation on the prevailing word; (iv) single fixation duration (SFD), the duration of the first and only fixation on the current word; and (v) go-past time (GPT), the sum of all fixations prior to progressing to the right of the current word, including regressions to previous words that originated from the current word  For each of these eye-tracking features we additionally computed the pupil size  Furthermore, we extracted the number of fixations and mean pupil size for each word and sentence 
Corpus Construction ::: Preprocessing and feature extraction ::: EEG
The EEG data shared in this project are available as raw data, but also preprocessed with Automagic (version 1 4 6, pedroni2019automagic), a tool for automatic EEG data cleaning and validation  105 EEG channels (i e  electrodes) were used from the scalp recordings  9 EOG channels were used for artifact removal and additional 14 channels lying mainly on the neck and face were discarded before data analysis  Bad channels were identified and interpolated  We used the Multiple Artifact Rejection Algorithm (MARA), a supervised machine learning algorithm that evaluates ICA components, for automatic artifact rejection  MARA has been trained on manual component classifications, and thus captures a wide range of artifacts  MARA is especially effective at detecting and removing eye and muscle artifact components  The effect of this preprocessing can be seen in Figure FIGREF1d 
After preprocessing, we synchronized the EEG and eye-tracking data to enable EEG analyses time-locked to the onsets of fixations  To compute oscillatory power measures, we band-pass filtered the continuous EEG signals across an entire reading task for five different frequency bands resulting in a time-series for each frequency band  The independent frequency bands were determined as follows: theta$_1$ (4–6 Hz), theta$_2$ (6 5–8 Hz), alpha$_1$ (8 5–10 Hz), alpha$_2$ (10 5–13 Hz), beta$_1$ (13 5–18 Hz), beta$_2$ (18 5–30 Hz), gamma$_1$ (30 5–40 Hz), and gamma$_2$ (40–49 5 Hz)  We then applied a Hilbert transformation to each of these time-series  We specifically chose the Hilbert transformation to maintain the temporal information of the amplitude of the frequency bands, to enable the power of the different frequencies for time segments defined through the fixations from the eye-tracking recording  Thus, for each eye-tracking feature we computed the corresponding EEG feature in each frequency band  Furthermore, we extracted sentence-level EEG features by calculating the power in each frequency band, and additionally, the difference of the power spectra between frontal left and right homologue electrodes pairs  For each eye-tracking based EEG feature, all channels were subject to an artifact rejection criterion of $90\mu V$ to exclude trials with transient noise 
Data Validation
The aim of the technical validation of the data is to guarantee good recording quality and to replicate findings of previous studies investigating co-registration of EEG and eye movement data during natural reading tasks (e g  dimigen2011coregistration)  We also compare the results to ZuCo 1 0 BIBREF1, which allows a more direct comparison due to the analogous recording procedure 
Data Validation ::: Eye-tracking
We validated the recorded eye-tracking data by analyzing the fixations made by all subjects through their reading speed and omission rate on sentence level  The omission rate is defined as the percentage of words that is not fixated in a sentence  Figure FIGREF10 (middle) shows the mean reading speed over all subjects, measured in seconds per sentence and Figure FIGREF10 (right) shows the mean omission rates aggregated over all subjects for each task  Clearly, the participants made less fixations during the task-specific reading, which lead to faster reading speed 
Moreover, we corroborated these sentence-level metrics by visualizing the skipping proportion on word level (Figure FIGREF13)  The skipping proportion is the average rate of words being skipped (i e  not being fixated) in a sentence  As expected, this also increases in the task-specific reading 
Although the reading material is from the same source and of the same length range (see Figure FIGREF10 (left)), in the first task (NR) passive reading was recorded, while in the second task (TSR) the subjects had to annotate a specific relation type in each sentence  Thus, the task-specific annotation reading lead to shorter passes because the goal was merely to recognize a relation in the text, but not necessarily to process the every word in each sentence  This distinct reading behavior is shown in Figure FIGREF15, where fixations occur until the end of the sentence during normal reading, while during task-specific reading the fixations stop after the decisive words to detect a given relation type  Finally, we also analyzed the average reading times for each of the extracted eye-tracking features  The means and distributions for both tasks are shown in Figure FIGREF21  These results are in line with the recorded data in ZuCo 1 0, as well as with the features extracted in the GECO corpus BIBREF4 
Data Validation ::: EEG
As a first validation step, we extracted fixation-related potentials (FRPs), where the EEG signal during all fixations of one task are averaged  Figure FIGREF24 shows the time-series of the resulting FRPs for two electrodes (PO8 and Cz), as well as topographies of the voltage distributions across the scalp at selected points in time  The five components (for which the scalp topographies are plotted) are highly similar in the time-course of the chosen electrodes to dimigen2011coregistration as well as to ZuCo 1 0 
Moreover, these previous studies were able to show an effect of fixation duration on the resulting FRPs  To show this dependency we followed two approaches  First, for each reading task, all single-trial FRPs were ordered by fixation duration and a vertical sliding time-window was used to smooth the data BIBREF13  Figure FIGREF25 (bottom) shows the resulting plots  In line with this previous work, a first positivation can be identified at 100 ms post-fixation onset  A second positive peak is located dependent on the duration of the fixation, which can be explained by the time-jittered succeeding fixation  The second approach is based on henderson2013co in which single trial EEG segments are clustered by the duration of the current fixation  As shown in Figure FIGREF25 (top), we chose four clusters and averaged the data within each cluster to four distinct FRPs, depending on the fixation duration  Again, the same positivation peaks become apparent  Both findings are consistent with the previous work mentioned and with our findings from ZuCo 1 0 
Conclusion
We presented a new, freely available corpus of eye movement and electrical brain activity recordings during natural reading as well as during annotation  This is the first dataset that allows for the comparison between these two reading paradigms  We described the materials and experiment design in detail and conducted an extensive validation to ensure the quality of the recorded data  Since this corpus is tailored to cognitively-inspired NLP, the applications and re-use potentials of this data are extensive  The provided word-level and sentence-level eye-tracking and EEG features can be used to improve and evaluate NLP and machine learning methods, for instance, to evaluate linguistic phenomena in neural models via psycholinguistic data  In addition, because the sentences contains semantic relation labels and the annotations of all participants, it can also be widely used for relation extraction and classification  Finally, the two carefully constructed reading paradigms allow for the comparison between normal reading and reading during annotation, which can be relevant to improve the manual labelling process as well as the quality of the annotations for supervised machine learning 

ground_truth: No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the network's baseline features?
Context 3:  As a training dataset, we use the corpus developed by BIBREF36 , which contains 2,400 essays labeled by one of the five personality traits each 
The fully-connected layer has 150 neurons, which are treated as the features  We concatenate the feature vector of each personality dimension in order to create the final feature vector  Thus, the personality model ultimately extracts a 750-dimensional feature vector (150-dimensional feature vector for each of the five personality traits)  This network is replicated five times, one for each personality trait  In particular, we create a CNN for each personality trait and the aim of each CNN is to classify a sentence into binary classes, i e , whether it expresses a personality trait or not 
Baseline Method and Features
CNN can also be employed on the sarcasm datasets in order to identify sarcastic and non-sarcastic tweets  We term the features extracted from this network baseline features, the method as baseline method and the CNN architecture used in this baseline method as baseline CNN  Since the fully-connected layer has 100 neurons, we have 100 baseline features in our experiment  This method is termed baseline method as it directly aims to classify a sentence as sarcastic vs non-sarcastic  The baseline CNN extracts the inherent semantics from the sarcastic corpus by employing deep domain understanding  The process of using baseline features with other features extracted from the pre-trained model is described in Section SECREF24  
Experimental Results and Discussion
In this section, we present the experimental results using different feature combinations and compare them with the state of the art  For each feature we show the results using only CNN and using CNN-SVM (i e , when the features extracted by CNN are fed to the SVM)  Macro-F1 measure is used as an evaluation scheme in the experiments 
Sarcasm Datasets Used in the Experiment
This dataset was created by BIBREF8   The tweets were downloaded from Twitter using #sarcasm as a marker for sarcastic tweets  It is a monolingual English dataset which consists of a balanced distribution of 50,000 sarcastic tweets and 50,000 non-sarcastic tweets 
Since sarcastic tweets are less frequently used BIBREF8 , we also need to investigate the robustness of the selected features and the model trained on these features on an imbalanced dataset  To this end, we used another English dataset from BIBREF8   It consists of 25,000 sarcastic tweets and 75,000 non-sarcastic tweets 
We have obtained this dataset from The Sarcasm Detector  It contains 120,000 tweets, out of which 20,000 are sarcastic and 100,000 are non-sarcastic  We randomly sampled 10,000 sarcastic and 20,000 non-sarcastic tweets from the dataset  Visualization of both the original and subset data show similar characteristics 
A two-step methodology has been employed in filtering the datasets used in our experiments  Firstly, we identified and removed all the “user", “URL" and “hashtag" references present in the tweets using efficient regular expressions  Special emphasis was given to this step to avoid traces of hashtags, which might trigger the models to provide biased results  Secondly, we used NLTK Twitter Tokenizer to ensure proper tokenization of words along with special symbols and emoticons  Since our deep CNNs extract contextual information present in tweets, we include emoticons as part of the vocabulary  This enables the emoticons to hold a place in the word embedding space and aid in providing information about the emotions present in the sentence 
Merging the Features
Throughout this research, we have carried out several experiments with various feature combinations  For the sake of clarity, we explain below how the features extracted using difference models are merged 
In the standard feature merging process, we first extract the features from all deep CNN based feature extraction models and then we concatenate them  Afterwards, SVM is employed on the resulted feature vector 
In another setting, we use the features extracted from the pre-trained models as the static channels of features in the CNN of the baseline method  These features are appended to the hidden layer of the baseline CNN, preceding the final output softmax layer 
For comparison, we have re-implemented the state-of-the-art methods  Since BIBREF9 did not mention about the sentiment lexicon they use in the experiment, we used SenticNet BIBREF37 in the re-implementation of their method 
Results on Dataset 1
As shown in Table TABREF29 , for every feature CNN-SVM outperforms the performance of the CNN  Following BIBREF6 , we have carried out a 5-fold cross-validation on this dataset  The baseline features ( SECREF16 ) perform best among other features  Among all the pre-trained models, the sentiment model (F1-score: 87 00%) achieves better performance in comparison with the other two pre-trained models  Interestingly, when we merge the baseline features with the features extracted by the pre-trained deep NLP models, we only get 0 11% improvement over the F-score  It means that the baseline features alone are quite capable to detect sarcasm  On the other hand, when we combine sentiment, emotion and personality features, we obtain 90 70% F1-score  This indicates that the pre-trained features are indeed useful for sarcasm detection  We also compare our approach with the best research study conducted on this dataset (Table TABREF30 )  Both the proposed baseline model and the baseline + sentiment + emotion + personality model outperform the state of the art BIBREF9 , BIBREF8   One important difference with the state of the art is that BIBREF8 used relatively larger feature vector size ( INLINEFORM0 500,000) than we used in our experiment (1,100)  This not only prevents our model to overfit the data but also speeds up the computation  Thus, we obtain an improvement in the overall performance with automatic feature extraction using a relatively lower dimensional feature space 
In the literature, word n-grams, skipgrams and character n-grams are used as baseline features  According to Ptacek et al  BIBREF8 , these baseline features along with the other features (sentiment features and part-of-speech based features) produced the best performance  However, Ptacek et al  did not analyze the performance of these features when they were not used with the baseline features  Pre-trained word embeddings play an important role in the performance of the classifier because, when we use randomly generated embeddings, performance falls down to 86 23% using all features 
Results on Dataset 2
5-fold cross-validation has been carried out on Dataset 2  Also for this dataset, we get the best accuracy when we use all features  Baseline features have performed significantly better (F1-score: 92 32%) than all other features  Supporting the observations we have made from the experiments on Dataset 1, we see CNN-SVM outperforming CNN on Dataset 2  However, when we use all the features, CNN alone (F1-score: 89 73%) does not outperform the state of the art BIBREF8 (F1-score: 92 37%)  As shown in Table TABREF30 , CNN-SVM on the baseline + sentiment + emotion + personality feature set outperforms the state of the art (F1-score: 94 80%)  Among the pre-trained models, the sentiment model performs best (F1-score: 87 00%) 
Table TABREF29 shows the performance of different feature combinations  The gap between the F1-scores of only baseline features and all features is larger on the imbalanced dataset than the balanced dataset  This supports our claim that sentiment, emotion and personality features are very useful for sarcasm detection, thanks to the pre-trained models  The F1-score using sentiment features when combined with baseline features is 94 60%  On both of the datasets, emotion and sentiment features perform better than the personality features  Interestingly, using only sentiment, emotion and personality features, we achieve 90 90% F1-score 
Results on Dataset 3
Experimental results on Dataset 3 show the similar trends (Table TABREF30 ) as compared to Dataset 1 and Dataset 2  The highest performance (F1-score 93 30%) is obtained when we combine baseline features with sentiment, emotion and personality features  In this case, also CNN-SVM consistently performs better than CNN for every feature combination  The sentiment model is found to be the best pre-trained model  F1-score of 84 43% is obtained when we merge sentiment, emotion and personality features 
Dataset 3 is more complex and non-linear in nature compared to the other two datasets  As shown in Table TABREF30 , the methods by BIBREF9 and BIBREF8 perform poorly on this dataset  The TP rate achieved by BIBREF9 is only 10 07% and that means their method suffers badly on complex data  The approach of BIBREF8 has also failed to perform well on Dataset 3, achieving 62 37% with a better TP rate of 22 15% than BIBREF9   On the other hand, our proposed model performs consistently well on this dataset achieving 93 30% 
Testing Generalizability of the Models and Discussions
To test the generalization capability of the proposed approach, we perform training on Dataset 1 and test on Dataset 3  The F1-score drops down dramatically to 33 05%  In order to understand this finding, we visualize each dataset using PCA (Figure FIGREF17 )  It depicts that, although Dataset 1 is mostly linearly separable, Dataset 3 is not  A linear kernel that performs well on Dataset 1 fails to provide good performance on Dataset 3  If we use RBF kernel, it overfits the data and produces worse results than what we get using linear kernel  Similar trends are seen in the performance of other two state-of-the-art approaches BIBREF9 , BIBREF8   Thus, we decide to perform training on Dataset 3 and test on the Dataset 1  As expected better performance is obtained with F1-score 76 78%  However, the other two state-of-the-art approaches fail to perform well in this setting  While the method by BIBREF9 obtains F1-score of 47 32%, the approach by BIBREF8 achieves 53 02% F1-score when trained on Dataset 3 and tested on Dataset 1  Below, we discuss about this generalizability issue of the models developed or referred in this paper 
As discussed in the introduction, sarcasm is very much topic-dependent and highly contextual  For example, let us consider the tweet “I am so glad to see Tanzania played very well, I can now sleep well :P"  Unless one knows that Tanzania actually did not play well in that game, it is not possible to spot the sarcastic nature of this sentence  Thus, an n-gram based sarcasm detector trained at time INLINEFORM0 may perform poorly to detect sarcasm in the tweets crawled at time INLINEFORM1 (given that there is a considerable gap between these time stamps) because of the diversity of the topics (new events occur, new topics are discussed) of the tweets  Sentiment and other contextual clues can help to spot the sarcastic nature in this kind of tweets  A highly positive statement which ends with a emoticon expressing joke can be sarcastic 
State-of-the-art methods lack these contextual information which, in our case, we extract using pre-trained sentiment, emotion and personality models  Not only these pre-trained models, the baseline method (baseline CNN architecture) performs better than the state-of-the-art models in this generalizability test setting  In our generalizability test, when the pre-trained features are used with baseline features, we get 4 19% F1-score improvement over the baseline features  On the other hand, when they are not used with the baseline features, together they produce 64 25% F1-score 
Another important fact is that an n-grams model cannot perform well on unseen data unless it is trained on a very large corpus  If most of the n-grams extracted from the unseen data are not in the vocabulary of the already trained n-grams model, in fact, the model will produce a very sparse feature vector representation of the dataset  Instead, we use the word2vec embeddings as the source of the features, as word2vec allows for the computation of similarities between unseen data and training data 
Baseline Features vs Pre-trained Features
Our experimental results show that the baseline features outperform the pre-trained features for sarcasm detection  However, the combination of pre-trained features and baseline features beats both of themselves alone  It is counterintuitive, since experimental results prove that both of those features learn almost the same global and contextual features  In particular, baseline network dominates over pre-trained network as the former learns most of the features learned by the latter  Nonetheless, the combination of baseline and pre-trained classifiers improves the overall performance and generalizability, hence proving their effectiveness in sarcasm detection  Experimental results show that sentiment and emotion features are the most useful features, besides baseline features (Figure FIGREF36 )  Therefore, in order to reach a better understanding of the relation between personality features among themselves and with other pre-trained features, we carried out Spearman correlation testing  Results, displayed in Table TABREF39 , show that those features are highly correlated with each other 
Conclusion
In this work, we developed pre-trained sentiment, emotion and personality models for identifying sarcastic text using CNN, which are found to be very effective for sarcasm detection  In the future, we plan to evaluate the performance of the proposed method on a large corpus and other domain-dependent corpora  Future work will also focus on analyzing past tweets and activities of users in order to better understand their personality and profile and, hence, further improve the disambiguation between sarcastic and non-sarcastic text 

ground_truth:  The features extracted from CNN.