prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How are models evaluated in this human-machine communication game?
Context 1: Introduction
Suppose a user wants to write a sentence “I will be 10 minutes late ” Ideally, she would type just a few keywords such as “10 minutes late” and an autocomplete system would be able to infer the intended sentence (Figure FIGREF1)  Existing left-to-right autocomplete systems BIBREF0, BIBREF1 can often be inefficient, as the prefix of a sentence (e g  “I will be”) fails to capture the core meaning of the sentence  Besides the practical goal of building a better autocomplete system, we are interested in exploring the tradeoffs inherent to such communication schemes between the efficiency of typing keywords, accuracy of reconstruction, and interpretability of keywords 
One approach to learn such schemes is to collect a supervised dataset of keywords-sentence pairs as a training set, but (i) it would be expensive to collect such data from users, and (ii) a static dataset would not capture a real user's natural predilection to adapt to the system BIBREF2  Another approach is to avoid supervision and jointly learn a user-system communication scheme to directly optimize the combination of efficiency and accuracy  However, learning in this way can lead to communication schemes that are uninterpretable to humans BIBREF3, BIBREF4 (see Appendix for additional related work) 
In this work, we propose a simple, unsupervised approach to an autocomplete system that is efficient, accurate, and interpretable  For interpretability, we restrict keywords to be subsequences of their source sentences based on the intuition that humans can infer most of the original meaning from a few keywords  We then apply multi-objective optimization approaches to directly control and achieve desirable tradeoffs between efficiency and accuracy 
We observe that naively optimizing a linear combination of efficiency and accuracy terms is unstable and leads to suboptimal schemes  Thus, we propose a new objective which optimizes for communication efficiency under an accuracy constraint  We show this new objective is more stable and efficient than the linear objective at all accuracy levels 
As a proof-of-concept, we build an autocomplete system within this framework which allows a user to write sentences by specifying keywords  We empirically show that our framework produces communication schemes that are 52 16% more accurate than rule-based baselines when specifying 77 37% of sentences, and 11 73% more accurate than a naive, weighted optimization approach when specifying 53 38% of sentences  Finally, we demonstrate that humans can easily adapt to the keyword-based autocomplete system and save nearly 50% of time compared to typing a full sentence in our user study 
Approach
Consider a communication game in which the goal is for a user to communicate a target sequence $x= (x_1,    , x_m)$ to a system by passing a sequence of keywords $z= (z_1,    , z_n)$  The user generates keywords $z$ using an encoding strategy $q_{\alpha }(z\mid x)$, and the system attempts to guess the target sequence $x$ via a decoding strategy $p_{\beta }(x\mid z)$ 
A good communication scheme $(q_{\alpha }, p_{\beta })$ should be both efficient and accurate  Specifically, we prefer schemes that use fewer keywords (cost), and the target sentence $x$ to be reconstructed with high probability (loss) where
Based on our assumption that humans have an intuitive sense of retaining important keywords, we restrict the set of schemes to be a (potentially noncontiguous) subsequence of the target sentence  Our hypothesis is that such subsequence schemes naturally ensure interpretability, as efficient human and machine communication schemes are both likely to involve keeping important content words 
Approach ::: Modeling with autoencoders 
To learn communication schemes without supervision, we model the cooperative communication between a user and system through an encoder-decoder framework  Concretely, we model the user's encoding strategy $q_{\alpha }(z\mid x)$ with an encoder which encodes the target sentence $x$ into the keywords $z$ by keeping a subset of the tokens  This stochastic encoder $q_{\alpha }(z\mid x)$ is defined by a model which returns the probability of each token retained in the final subsequence $z$  Then, we sample from Bernoulli distributions according to these probabilities to either keep or drop the tokens independently (see Appendix for an example) 
We model the autocomplete system's decoding strategy $p_{\beta }(x\mid z)$ as a probabilistic model which conditions on the keywords $z$ and returns a distribution over predictions $x$  We use a standard sequence-to-sequence model with attention and copying for the decoder, but any model architecture can be used (see Appendix for details) 
Approach ::: Multi-objective optimization 
Our goal now is to learn encoder-decoder pairs which optimally balance the communication cost and reconstruction loss  The simplest approach to balancing efficiency and accuracy is to weight $\mathrm {cost}(x, \alpha )$ and $\mathrm {loss}(x, \alpha , \beta )$ linearly using a weight $\lambda $ as follows,
where the expectation is taken over the population distribution of source sentences $x$, which is omitted to simplify notation  However, we observe that naively weighting and searching over $\lambda $ is suboptimal and highly unstable—even slight changes to the weighting results in degenerate schemes which keep all or none of its tokens  This instability motivates us to develop a new stable objective 
Our main technical contribution is to draw inspiration from the multi-objective optimization literature and view the tradeoff as a sequence of constrained optimization problems, where we minimize the expected cost subject to varying expected reconstruction error constraints $\epsilon $,
This greatly improves the stability of the training procedure  We empirically observe that the model initially keeps most of the tokens to meet the constraints, and slowly learns to drop uninformative words from the keywords to minimize the cost  Furthermore, $\epsilon $ in Eq (DISPLAY_FORM6) allows us to directly control the maximum reconstruction error of resulting schemes, whereas $\lambda $ in Eq (DISPLAY_FORM5) is not directly related to any of our desiderata 
To optimize the constrained objective, we consider the Lagrangian of Eq (DISPLAY_FORM6),
Much like the objective in Eq (DISPLAY_FORM5) we can compute unbiased gradients by replacing the expectations with their averages over random minibatches  Although gradient descent guarantees convergence on Eq (DISPLAY_FORM7) only when the objective is convex, we find that not only is the optimization stable, the resulting solution achieves better performance than the weighting approach in Eq (DISPLAY_FORM5) 
Approach ::: Optimization 
Optimization with respect to $q_{\alpha }(z\mid x)$ is challenging as $z$ is discrete, and thus, we cannot differentiate $\alpha $ through $z$ via the chain rule  Because of this, we use the stochastic REINFORCE estimate BIBREF5 as follows:
We perform joint updates on $(\alpha , \beta , \lambda )$, where $\beta $ and $\lambda $ are updated via standard gradient computations, while $\alpha $ uses an unbiased, stochastic gradient estimate where we approximate the expectation in Eq (DISPLAY_FORM9)  We use a single sample from $q_{\alpha }(z\mid x)$ and moving-average of rewards as a baseline to reduce variance 
Experiments
We evaluate our approach by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews BIBREF6 (see Appendix for details)  We quantify the efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords  The accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence 
Experiments ::: Effectiveness of constrained objective 
We first show that the linear objective in Eq (DISPLAY_FORM5) is suboptimal compared to the constrained objective in Eq (DISPLAY_FORM6)  Figure FIGREF10 compares the achievable accuracy and efficiency tradeoffs for the two objectives, which shows that the constrained objective results in more efficient schemes than the linear objective at every accuracy level (e g  11 73% more accurate at a 53 38% retention rate) 
We also observe that the linear objective is highly unstable as a function of the tradeoff parameter $\lambda $ and requires careful tuning  Even slight changes to $\lambda $ results in degenerate schemes that keep all or none of the tokens (e g  $\lambda \le 4 2$ and $\lambda \ge 4 4$)  On the other hand, the constrained objective is substantially more stable as a function of $\epsilon $ (e g  points for $\epsilon $ are more evenly spaced than $\lambda $) 
Experiments ::: Efficiency-accuracy tradeoff 
We quantify the efficiency-accuracy tradeoff compared to two rule-based baselines: Unif and Stopword  The Unif encoder randomly keeps tokens to generate keywords with the probability $\delta $  The Stopword encoder keeps all tokens but drops stop words (e g  `the', `a', `or') all the time ($\delta =0$) or half of the time ($\delta =0 5$)  The corresponding decoders for these encoders are optimized using gradient descent to minimize the reconstruction error (i e  $\mathrm {loss}(x, \alpha , \beta )$) 
Figure FIGREF10 shows that two baselines achieve similar tradeoff curves, while the constrained model achieves a substantial 52 16% improvement in accuracy at a 77 37% retention rate compared to Unif, thereby showing the benefits of jointly training the encoder and decoder 
Experiments ::: Robustness and analysis 
We provide additional experimental results on the robustness of learned communication schemes as well as in-depth analysis on the correlation between the retention rates of tokens and their properties, which we defer to Appendix and for space 
Experiments ::: User study 
We recruited 100 crowdworkers on Amazon Mechanical Turk (AMT) and measured completion times and accuracies for typing randomly sampled sentences from the Yelp corpus  Each user was shown alternating autocomplete and writing tasks across 50 sentences (see Appendix for user interface)  For the autocomplete task, we gave users a target sentence and asked them to type a set of keywords into the system  The users were shown the top three suggestions from the autocomplete system, and were asked to mark whether each of these three suggestions was semantically equivalent to the target sentence  For the writing task, we gave users a target sentence and asked them to either type the sentence verbatim or a sentence that preserves the meaning of the target sentence 
Table TABREF13 shows two examples of the autocomplete task and actual user-provided keywords  Each column contains a set of keywords and its corresponding top three suggestions generated by the autocomplete system with beam search  We observe that the system is likely to propose generic sentences for under-specified keywords (left column) and almost the same sentences for over-specified keywords (right column)  For properly specified keywords (middle column), the system completes sentences accordingly by adding a verb, adverb, adjective, preposition, capitalization, and punctuation 
Overall, the autocomplete system achieved high accuracy in reconstructing the keywords  Users marked the top suggestion from the autocomplete system to be semantically equivalent to the target $80 6$% of the time, and one of the top 3 was semantically equivalent $90 11$% of the time  The model also achieved a high exact match accuracy of 18 39%  Furthermore, the system was efficient, as users spent $3 86$ seconds typing keywords compared to $5 76$ seconds for full sentences on average  The variance of the typing time was $0 08$ second for keywords and $0 12$ second for full sentences, indicating that choosing and typing keywords for the system did not incur much overhead 
Experiments ::: Acknowledgments
We thank the reviewers and Yunseok Jang for their insightful comments  This work was supported by NSF CAREER Award IIS-1552635 and an Intuit Research Award 
Experiments ::: Reproducibility
All code, data and experiments are available on CodaLab at https://bit ly/353fbyn 
Context 2: Introduction
Since humans amass more and more generally available data in the form of unstructured text it would be very useful to teach machines to read and comprehend such data and then use this understanding to answer our questions  A significant amount of research has recently focused on answering one particular kind of questions the answer to which depends on understanding a context document  These are cloze-style questions BIBREF0 which require the reader to fill in a missing word in a sentence  An important advantage of such questions is that they can be generated automatically from a suitable text corpus which allows us to produce a practically unlimited amount of them  That opens the task to notoriously data-hungry deep-learning techniques which now seem to outperform all alternative approaches 
Two such large-scale datasets have recently been proposed by researchers from Google DeepMind and Facebook AI: the CNN/Daily Mail dataset BIBREF1 and the Children's Book Test (CBT) BIBREF2 respectively  These have attracted a lot of attention from the research community BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 with a new state-of-the-art model coming out every few weeks 
However if our goal is a production-level system actually capable of helping humans, we want the model to use all available resources as efficiently as possible  Given that
we believe that if the community is striving to bring the performance as far as possible, it should move its work to larger data 
This thinking goes in line with recent developments in the area of language modelling  For a long time models were being compared on several "standard" datasets with publications often presenting minuscule improvements in performance  Then the large-scale One Billion Word corpus dataset appeared BIBREF15 and it allowed Jozefowicz et al  to train much larger LSTM models BIBREF16 that almost halved the state-of-the-art perplexity on this dataset 
We think it is time to make a similar step in the area of text comprehension  Hence we are introducing the BookTest, a new dataset very similar to the Children's Book test but more than 60 times larger to enable training larger models even in the domain of text comprehension  Furthermore the methodology used to create our data can later be used to create even larger datasets when the need arises thanks to further technological progress 
We show that if we evaluate a model trained on the new dataset on the now standard Children's Book Test dataset, we see an improvement in accuracy much larger than other research groups achieved by enhancing the model architecture itself (while still using the original CBT training data)  By training on the new dataset, we reduce the prediction error by almost one third  On the named-entity version of CBT this brings the ensemble of our models to the level of human baseline as reported by Facebook BIBREF2   However in the final section we show in our own human study that there is still room for improvement on the CBT beyond the performance of our model 
Task Description
A natural way of testing a reader's comprehension of a text is to ask her a question the answer to which can be deduced from the text  Hence the task we are trying to solve consists of answering a cloze-style question, the answer to which depends on the understanding of a context document provided with the question  The model is also provided with a set of possible answers from which the correct one is to be selected  This can be formalized as follows:
The training data consist of tuples INLINEFORM0 , where INLINEFORM1 is a question, INLINEFORM2 is a document that contains the answer to question INLINEFORM3 , INLINEFORM4 is a set of possible answers and INLINEFORM5 is the ground-truth answer  Both INLINEFORM6 and INLINEFORM7 are sequences of words from vocabulary INLINEFORM8   We also assume that all possible answers are words from the vocabulary, that is INLINEFORM9   In the CBT and CNN/Daily Mail datasets it is also true that the ground-truth answer INLINEFORM10 appears in the document  This is exploited by many machine learning models BIBREF2 , BIBREF4 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF10 , BIBREF11 , BIBREF12 , however some do not explicitly depend on this property BIBREF1 , BIBREF3 , BIBREF5 , BIBREF9
Current Landscape
We will now briefly review what datasets for text comprehension have been published up to date and look at models which have been recently applied to solving the task we have just described 
Datasets
A crucial condition for applying deep-learning techniques is to have a huge amount of data available for training  For question answering this specifically means having a large number of document-question-answer triples available  While there is an unlimited amount of text available, coming up with relevant questions and the corresponding answers can be extremely labour-intensive if done by human annotators  There were efforts to provide such human-generated datasets, e g  Microsoft's MCTest BIBREF17 , however their scale is not suitable for deep learning without pre-training on other data BIBREF18 (such as using pre-trained word embedding vectors) 
Google DeepMind managed to avoid this scale issue with their way of generating document-question-answer triples automatically, closely followed by Facebook with a similar method  Let us now briefly introduce the two resulting datasets whose properties are summarized in Table TABREF8  
These two datasets BIBREF1 exploit a useful feature of online news articles – many articles include a short summarizing sentence near the top of the page  Since all information in the summary sentence is also presented in the article body, we get a nice cloze-style question about the article contents by removing a word from the short summary 
The dataset's authors also replaced all named entities in the dataset by anonymous tokens which are further shuffled for each new batch  This forces the model to rely solely on information from the context document, not being able to transfer any meaning of the named entities between documents 
This restricts the task to one specific aspect of context-dependent question answering which may be useful however it moves the task further from the real application scenario, where we would like the model to use all information available to answer questions  Furthermore Chen et al  BIBREF5 have suggested that this can make about 17% of the questions unanswerable even by humans  They also claim that more than a half of the question sentences are mere paraphrases or exact matches of a single sentence from the context document  This raises a question to what extent the dataset can test deeper understanding of the articles 
The Children's Book Test BIBREF2 uses a different source - books freely available thanks to Project Gutenberg  Since no summary is available, each example consists of a context document formed from 20 consecutive sentences from the story together with a question formed from the subsequent sentence 
The dataset comes in four flavours depending on what type of word is omitted from the question sentence  Based on human evaluation done in BIBREF2 it seems that NE and CN are more context dependent than the other two types – prepositions and verbs  Therefore we (and all of the recent publications) focus only on these two word types 
Several new datasets related to the (now almost standard) ones above emerged recently  We will now briefly present them and explain how the dataset we are introducing in this article differs from them 
The LAMBADA dataset BIBREF19 is designed to measure progress in understanding common-sense questions about short stories that can be easily answered by humans but cannot be answered by current standard machine-learning models (e g  plain LSTM language models)  This dataset is useful for measuring the gap between humans and machine learning algorithms  However, by contrast to our BookTest dataset, it will not allow us to track progress towards the performance of the baseline systems or on examples where machine learning may show super-human performance  Also LAMBADA is just a diagnostic dataset and does not provide ready-to-use question-answering training data, just a plain-text corpus which may moreover include copyrighted books making its use potentially problematic for some purposes  We are providing ready training data consisting of copyright-free books only 
The SQuAD dataset BIBREF20 based on Wikipedia and the Who-did-What dataset BIBREF21 based on Gigaword news articles are factoid question-answering datasets where a multi-word answer should be extracted from a context document  This is in contrast to the previous datasets, including CNN/DM, CBT, LAMBADA and our new dataset, which require only single-word answers  Both these datasets however provide less than 130,000 training questions, two orders of magnitude less than our dataset does 
The Story Cloze Test BIBREF22 provides a crowd-sourced corpus of 49,255 commonsense stories for training and 3,744 testing stories with right and wrong endings  Hence the dataset is again rather small  Similarly to LAMBADA, the Story Cloze Test was designed to be easily answerable by humans 
In the WikiReading BIBREF23 dataset the context document is formed from a Wikipedia article and the question-answer pair is taken from the corresponding WikiData page  For each entity (e g  Hillary Clinton), WikiData contain a number of property-value pairs (e g  place of birth: Chicago) which form the datasets's question-answer pairs  The dataset is certainly relevant to the community, however the questions are of very limited variety with only 20 properties (and hence unique questions) covering INLINEFORM0 of the dataset  Furthermore many of the frequent properties are mentioned at a set spot within the article (e g  the date of birth is almost always in brackets behind the name of a person) which may make the task easier for machines  We are trying to provide a more varied dataset 
Although there are several datasets related to task we are aiming to solve, they differ sufficiently for our dataset to bring new value to the community  Its biggest advantage is its size which can furthermore be easily upscaled without expensive human annotation  Finally while we are emphasizing the differences, models could certainly benefit from as diverse a collection of datasets as possible 
Machine Learning Models
A first major work applying deep-learning techniques to text comprehension was Hermann et al  BIBREF1   This work was followed by the application of Memory Networks to the same task BIBREF2   Later three models emerged around the same time BIBREF3 , BIBREF4 , BIBREF5 including our psr model BIBREF4   The AS Reader inspired several subsequent models that use it as a sub-component in a diverse ensemble BIBREF8 ; extend it with a hierarchical structure BIBREF6 , BIBREF24 , BIBREF7 ; compute attention over the context document for every word in the query BIBREF10 or use two-way context-query attention mechanism for every word in the context and the query BIBREF11 that is similar in its spirit to models recently proposed in different domains, e g  BIBREF25 in information retrieval  Other neural approaches to text comprehension are explored in BIBREF9 , BIBREF12  
Possible Directions for Improvements
Accuracy in any machine learning tasks can be enhanced either by improving a machine learning model or by using more in-domain training data  Current state of the art models BIBREF6 , BIBREF7 , BIBREF8 , BIBREF11 improve over AS Reader's accuracy on CBT NE and CN datasets by 1-2 percent absolute  This suggests that with current techniques there is only limited room for improvement on the algorithmic side 
The other possibility to improve performance is simply to use more training data  The importance of training data was highlighted by the frequently quoted Mercer's statement that “There is no data like more data ” The observation that having more data is often more important than having better algorithms has been frequently stressed since then BIBREF13 , BIBREF14  
As a step in the direction of exploiting the potential of more data in the domain of text comprehension, we created a new dataset called BookTest similar to, but much larger than the widely used CBT and CNN/DM datasets 
BookTest
Similarly to the CBT, our BookTest dataset is derived from books available through project Gutenberg  We used 3555 copyright-free books to extract CN examples and 10507 books for NE examples, for comparison the CBT dataset was extracted from just 108 books 
When creating our dataset we follow the same procedure as was used to create the CBT dataset BIBREF2   That is, we detect whether each sentence contains either a named entity or a common noun that already appeared in one of the preceding twenty sentences  This word is then replaced by a gap tag (XXXXX) in this sentence which is hence turned into a cloze-style question  The preceding 20 sentences are used as the context document  For common noun and named entity detection we use the Stanford POS tagger BIBREF27 and Stanford NER BIBREF28 
Context 3: Introduction
Sarcasm is an intensive, indirect and complex construct that is often intended to express contempt or ridicule   Sarcasm, in speech, is multi-modal, involving tone, body-language and gestures along with linguistic artifacts used in speech  Sarcasm in text, on the other hand, is more restrictive when it comes to such non-linguistic modalities  This makes recognizing textual sarcasm more challenging for both humans and machines 
Sarcasm detection plays an indispensable role in applications like online review summarizers, dialog systems, recommendation systems and sentiment analyzers  This makes automatic detection of sarcasm an important problem  However, it has been quite difficult to solve such a problem with traditional NLP tools and techniques  This is apparent from the results reported by the survey from DBLP:journals/corr/JoshiBC16  The following discussion brings more insights into this 
Consider a scenario where an online reviewer gives a negative opinion about a movie through sarcasm: “This is the kind of movie you see because the theater has air conditioning”  It is difficult for an automatic sentiment analyzer to assign a rating to the movie and, in the absence of any other information, such a system may not be able to comprehend that prioritizing the air-conditioning facilities of the theater over the movie experience indicates a negative sentiment towards the movie  This gives an intuition to why, for sarcasm detection, it is necessary to go beyond textual analysis 
We aim to address this problem by exploiting the psycholinguistic side of sarcasm detection, using cognitive features extracted with the help of eye-tracking  A motivation to consider cognitive features comes from analyzing human eye-movement trajectories that supports the conjecture: Reading sarcastic texts induces distinctive eye movement patterns, compared to literal texts  The cognitive features, derived from human eye movement patterns observed during reading, include two primary feature types:
The cognitive features, along with textual features used in best available sarcasm detectors, are used to train binary classifiers against given sarcasm labels  Our experiments show significant improvement in classification accuracy over the state of the art, by performing such augmentation 
Related Work
Sarcasm, in general, has been the focus of research for quite some time  In one of the pioneering works jorgensen1984test explained how sarcasm arises when a figurative meaning is used opposite to the literal meaning of the utterance  In the word of clark1984pretense, sarcasm processing involves canceling the indirectly negated message and replacing it with the implicated one  giora1995irony, on the other hand, define sarcasm as a mode of indirect negation that requires processing of both negated and implicated messages  ivanko2003context define sarcasm as a six tuple entity consisting of a speaker, a listener, Context, Utterance, Literal Proposition and Intended Proposition and study the cognitive aspects of sarcasm processing 
Computational linguists have previously addressed this problem using rule based and statistical techniques, that make use of : (a) Unigrams and Pragmatic features BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 (b) Stylistic patterns BIBREF4 and patterns related to situational disparity BIBREF5 and (c) Hastag interpretations BIBREF6 , BIBREF7  
Most of the previously done work on sarcasm detection uses distant supervision based techniques (ex: leveraging hashtags) and stylistic/pragmatic features (emoticons, laughter expressions such as “lol” etc)  But, detecting sarcasm in linguistically well-formed structures, in absence of explicit cues or information (like emoticons), proves to be hard using such linguistic/stylistic features alone 
With the advent of sophisticated eye-trackers and electro/magneto-encephalographic (EEG/MEG) devices, it has been possible to delve deep into the cognitive underpinnings of sarcasm understanding  Filik2014, using a series of eye-tracking and EEG experiments try to show that for unfamiliar ironies, the literal interpretation would be computed first  They also show that a mismatch with context would lead to a re-interpretation of the statement, as being ironic  Camblin2007103 show that in multi-sentence passages, discourse congruence has robust effects on eye movements  This also implies that disrupted processing occurs for discourse incongruent words, even though they are perfectly congruous at the sentence level  In our previous work BIBREF8 , we augment cognitive features, derived from eye-movement patterns of readers, with textual features to detect whether a human reader has realized the presence of sarcasm in text or not 
The recent advancements in the literature discussed above, motivate us to explore gaze-based cognition for sarcasm detection  As far as we know, our work is the first of its kind 
Eye-tracking Database for Sarcasm Analysis
Sarcasm often emanates from incongruity BIBREF9 , which enforces the brain to reanalyze it BIBREF10   This, in turn, affects the way eyes move through the text  Hence, distinctive eye-movement patterns may be observed in the case of successful processing of sarcasm in text in contrast to literal texts  This hypothesis forms the crux of our method for sarcasm detection and we validate this using our previously released freely available sarcasm dataset BIBREF8 enriched with gaze information 
Document Description
The database consists of 1,000 short texts, each having 10-40 words  Out of these, 350 are sarcastic and are collected as follows: (a) 103 sentences are from two popular sarcastic quote websites, (b) 76 sarcastic short movie reviews are manually extracted from the Amazon Movie Corpus BIBREF11 by two linguists  (c) 171 tweets are downloaded using the hashtag #sarcasm from Twitter  The 650 non-sarcastic texts are either downloaded from Twitter or extracted from the Amazon Movie Review corpus  The sentences do not contain words/phrases that are highly topic or culture specific  The tweets were normalized to make them linguistically well formed to avoid difficulty in interpreting social media lingo  Every sentence in our dataset carries positive or negative opinion about specific “aspects”  For example, the sentence “The movie is extremely well cast” has positive sentiment about the aspect “cast” 
The annotators were seven graduate students with science and engineering background, and possess good English proficiency  They were given a set of instructions beforehand and are advised to seek clarifications before they proceed  The instructions mention the nature of the task, annotation input method, and necessity of head movement minimization during the experiment 
Task Description
The task assigned to annotators was to read sentences one at a time and label them with with binary labels indicating the polarity (i e , positive/negative)  Note that, the participants were not instructed to annotate whether a sentence is sarcastic or not , to rule out the Priming Effect (i e , if sarcasm is expected beforehand, processing incongruity becomes relatively easier BIBREF12 )  The setup ensures its “ecological validity” in two ways: (1) Readers are not given any clue that they have to treat sarcasm with special attention  This is done by setting the task to polarity annotation (instead of sarcasm detection)

ground_truth: by training an autocomplete system on 500K randomly sampled sentences from Yelp reviews

efficiency of a communication scheme $(q_{\alpha },p_{\beta })$ by the retention rate of tokens, which is measured as the fraction of tokens that are kept in the keywords, accuracy of a scheme is measured as the fraction of sentences generated by greedily decoding the model that exactly matches the target sentence

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does their ensemble method work?
Context 2: 
The training dataset consists of the original CBT NE and CN data extended with new NE and CN examples  The new BookTest dataset hence contains INLINEFORM0 training examples and INLINEFORM1 tokens 
The validation dataset consists of INLINEFORM0 NE and INLINEFORM1 CN questions  We have one test set for NE and one for CN, each containing INLINEFORM2 examples  The training, validation and test sets were generated from non-overlapping sets of books 
When generating the dataset we removed all editions of books used to create CBT validation and test sets from our training dataset  Therefore the models trained on the BookTest corpus can be evaluated on the original CBT data and they can be compared with recent text-comprehension models utilizing this dataset BIBREF2 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11  
Baselines
We will now use our psr model to evaluate the performance gain from increasing the dataset size 
AS Reader
In BIBREF4 we introduced the psr , which at the time of publication significantly outperformed all other architectures on the CNN, DM and CBT datasets  This model is built to leverage the fact that the answer is a single word from the context document  Similarly to many other models it uses attention over the document – intuitively a measure of how relevant each word is to answering the question  However while most previous models used this attention as weights to calculate a blended representation of the answer word, we simply sum the attention across all occurrences of each unique words and then simply select the word with the highest sum as the final answer  While simple, this trick seems both to improve accuracy and to speed-up training  It was adopted by many subsequent models BIBREF8 , BIBREF6 , BIBREF7 , BIBREF10 , BIBREF11 , BIBREF24  
Let us now describe the model in more detail  Figure FIGREF21 may help you in understanding the following paragraphs 
The words from the document and the question are first converted into vector embeddings using a look-up matrix INLINEFORM0   The document is then read by a bidirectional GRU network BIBREF29   A concatenation of the hidden states of the forward and backward GRUs at each word is then used as a contextual embedding of this word, intuitively representing the context in which the word is appearing  We can also understand it as representing the set of questions to which this word may be an answer 
Similarly the question is read by a bidirectional GRU but in this case only the final hidden states are concatenated to form the question embedding 
The attention over each word in the context is then calculated as the dot product of its contextual embedding with the question embedding  This attention is then normalized by the softmax function and summed across all occurrences of each answer candidate  The candidate with most accumulated attention is selected as the final answer 
For a more detailed description of the model including equations check BIBREF4   More details about the training setup and model hyperparameters can be found in the Appendix 
During our past experiments on the CNN, DM and CBT datasets BIBREF4 each unique word from the training, validation and test datasets had its row in the look-up matrix INLINEFORM0   However as we radically increased the dataset size, this would result in an extremely large number of model parameters so we decided to limit the vocabulary size to INLINEFORM1 most frequent words  For each example, each unique out-of-vocabulary word is now mapped on one of 1000 anonymous tokens which are randomly initialized and untrained  Fixing the embeddings of these anonymous tags proved to significantly improve the performance 
While mostly using the original AS Reader model, we have also tried introducing a minor tweak in some instances of the model  We tried initializing the context encoder GRU's hidden state by letting the encoder read the question first before proceeding to read the context document  Intuitively this allows the encoder to know in advance what to look for when reading over the context document 
Including models of this kind in the ensemble helped to improve the performance 
Results
Table TABREF25 shows the accuracy of the psr and other architectures on the CBT validation and test data  The last two rows show the performance of the psr trained on the BookTest dataset; all the other models have been trained on the original CBT training data 
If we take the best psr ensemble trained on CBT as a baseline, improving the model architecture as in BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , continuing to use the original CBT training data, lead to improvements of INLINEFORM0 and INLINEFORM1 absolute on named entities and common nouns respectively  By contrast, inflating the training dataset provided a boost of INLINEFORM2 while using the same model  The ensemble of our models even exceeded the human baseline provided by Facebook BIBREF2 on the Common Noun dataset 
Our model takes approximately two weeks to converge when trained on the BookTest dataset on a single Nvidia Tesla K40 GPU 
Discussion
Embracing the abundance of data may mean focusing on other aspects of system design than with smaller data  Here are some of the challenges that we need to face in this situation 
Firstly, since the amount of data is practically unlimited – we could even generate them on the fly resulting in continuous learning similar to the Never-Ending Language Learning by Carnegie Mellon University BIBREF30 – it is now the speed of training that determines how much data the model is able to see  Since more training data significantly help the model performance, focusing on speeding up the algorithm may be more important than ever before  This may for instance influence the decision whether to use regularization such as dropout which does seem to somewhat improve the model performance, however usually at a cost of slowing down training 
Thanks to its simplicity, the psr seems to be training fast - for example around seven times faster than the models proposed by Chen et al  BIBREF5   Hence the psr may be particularly suitable for training on large datasets 
The second challenge is how to generalize the performance gains from large data to a specific target domain  While there are huge amounts of natural language data in general, it may not be the case in the domain where we may want to ultimately apply our model 
Hence we are usually not facing a scenario of simply using a larger amount of the same training data, but rather extending training to a related domain of data, hoping that some of what the model learns on the new data will still help it on the original task 
This is highlighted by our observations from applying a model trained on the BookTest to Children's Book Test test data  If we move model training from joint CBT NE+CN training data to a subset of the BookTest of the same size (230k examples), we see a drop in accuracy of around 10% on the CBT test datasets 
Hence even though the Children's Book Test and BookTest datasets are almost as close as two disjoint datasets can get, the transfer is still very imperfect   Rightly choosing data to augment the in-domain training data is certainly a problem worth exploring in future work 
Our results show that given enough data the AS Reader was able to exceed the human performance on CBT CN reported by Facebook  However we hypothesized that the system is still not achieving its full potential so we decided to examine the room for improvement in our own small human study 
Human Study
After adding more data we have the performance on the CBT validation and test datasets soaring  However is there still potential for much further growth beyond the results we have observed?
We decided to explore the remaining space for improvement on the CBT by testing humans on a random subset of 50 named entity and 50 common noun validation questions that the psr ensemble could not answer correctly  These questions were answered by 10 non-native English speakers from our research laboratory, each on a disjoint subset of questions   Participants had unlimited time to answer the questions and were told that these questions were not correctly answered by a machine, providing additional motivation to prove they are better than computers  The results of the human study are summarized in Table TABREF28   They show that a majority of questions that our system could not answer so far are in fact answerable  This suggests that 1) the original human baselines might have been underestimated, however, it might also be the case that there are some examples that can be answered by machines and not by humans; 2) there is still space for improvement 
A system that would answer correctly every time when either our ensemble or human answered correctly would achieve accuracy over 92% percent on both validation and test NE datasets and over 96% on both CN datasets  Hence it still makes sense to use CBT dataset to study further improvements of text-comprehension systems 
Conclusion
Few ways of improving model performance are as solidly established as using more training data  Yet we believe this principle has been somewhat neglected by recent research in text comprehension  While there is a practically unlimited amount of data available in this field, most research was performed on unnecessarily small datasets 
As a gentle reminder to the community we have shown that simply infusing a model with more data can yield performance improvements of up to INLINEFORM0 where several attempts to improve the model architecture on the same training data have given gains of at most INLINEFORM1 compared to our best ensemble result  Yes, experiments on small datasets certainly can bring useful insights  However we believe that the community should also embrace the real-world scenario of data abundance 
The BookTest dataset we are proposing gives the reading-comprehension community an opportunity to make a step in that direction 
Training Details
The training details are similar to those in BIBREF4 however we are including them here for completeness 
To train the model we used stochastic gradient descent with the ADAM update rule BIBREF32 and learning rates of INLINEFORM0 , INLINEFORM1 and INLINEFORM2   The best learning rate in our experiments was INLINEFORM3   We minimized negative log-likelihood as the training objective 
The initial weights in the word-embedding matrix were drawn randomly uniformly from the interval INLINEFORM0   Weights in the GRU networks were initialized by random orthogonal matrices BIBREF34 and biases were initialized to zero  We also used a gradient clipping BIBREF33 threshold of 10 and batches of sizes between 32 or 256  Increasing the batch from 32 to 128 seems to significantly improve performance on the large dataset - something we did not observe on the original CBT data  Increasing the batch size much above 128 is currently difficult due to memory constraints of the GPU 
During training we randomly shuffled all examples at the beginning of each epoch  To speed up training, we always pre-fetched 10 batches worth of examples and sorted them according to document length  Hence each batch contained documents of roughly the same length 
We also did not use pre-trained word embeddings 
We did not perform any text pre-processing since the datasets were already tokenized 
During training we evaluated the model performance every 12 hours and at the end of each epoch and stopped training when the error on the 20k BookTest validation set started increasing  We explored the hyperparameter space by training 67 different models The region of the parameter space that we explored together with the parameters of the model with best validation accuracy are summarized in Table TABREF29  
Our model was implemented using Theano BIBREF31 and Blocks BIBREF35  
The ensembles were formed by simply averaging the predictions from the constituent single models  These single models were selected using the following algorithm 
We started with the best performing model according to validation performance  Then in each step we tried adding the best performing model that had not been previously tried  We kept it in the ensemble if it did improve its validation performance and discarded it otherwise  This way we gradually tried each model once  We call the resulting model a greedy ensemble  We used the INLINEFORM0 BookTest validation dataset for this procedure 
The algorithm was offered 10 models and selected 5 of them for the final ensemble 
Context 3:  (2) Sarcastic sentences are mixed with non sarcastic text, which does not give prior knowledge about whether the forthcoming text will be sarcastic or not 
The eye-tracking experiment is conducted by following the standard norms in eye-movement research BIBREF13   At a time, one sentence is displayed to the reader along with the “aspect” with respect to which the annotation has to be provided  While reading, an SR-Research Eyelink-1000 eye-tracker (monocular remote mode, sampling rate 500Hz) records several eye-movement parameters like fixations (a long stay of gaze) and saccade (quick jumping of gaze between two positions of rest) and pupil size 
The accuracy of polarity annotation varies between 72%-91% for sarcastic texts and 75%-91% for non-sarcastic text, showing the inherent difficulty of sentiment annotation, when sarcasm is present in the text under consideration  Annotation errors may be attributed to: (a) lack of patience/attention while reading, (b) issues related to text comprehension, and (c) confusion/indecisiveness caused due to lack of context 
For our analysis, we do not discard the incorrect annotations present in the database  Since our system eventually aims to involve online readers for sarcasm detection, it will be hard to segregate readers who misinterpret the text  We make a rational assumption that, for a particular text, most of the readers, from a fairly large population, will be able to identify sarcasm  Under this assumption, the eye-movement parameters, averaged across all readers in our setting, may not be significantly distorted by a few readers who would have failed to identify sarcasm  This assumption is applicable for both regular and multi-instance based classifiers explained in section SECREF6  
Analysis of Eye-movement Data
We observe distinct behavior during sarcasm reading, by analyzing the “fixation duration on the text” (also referred to as “dwell time” in the literature) and “scanpaths” of the readers 
Variation in the Average Fixation Duration per Word
Since sarcasm in text can be expected to induce cognitive load, it is reasonable to believe that it would require more processing time BIBREF14   Hence, fixation duration normalized over total word count should usually be higher for a sarcastic text than for a non-sarcastic one  We observe this for all participants in our dataset, with the average fixation duration per word for sarcastic texts being at least 1 5 times more than that of non-sarcastic texts  To test the statistical significance, we conduct a two-tailed t-test (assuming unequal variance) to compare the average fixation duration per word for sarcastic and non-sarcastic texts  The hypothesized mean difference is set to 0 and the error tolerance limit ( INLINEFORM0 ) is set to 0 05  The t-test analysis, presented in Table TABREF11 , shows that for all participants, a statistically significant difference exists between the average fixation duration per word for sarcasm (higher average fixation duration) and non-sarcasm (lower average fixation duration)  This affirms that the presence of sarcasm affects the duration of fixation on words 
It is important to note that longer fixations may also be caused by other linguistic subtleties (such as difficult words, ambiguity and syntactically complex structures) causing delay in comprehension, or occulomotor control problems forcing readers to spend time adjusting eye-muscles  So, an elevated average fixation duration per word may not sufficiently indicate the presence of sarcasm  But we would also like to share that, for our dataset, when we considered readability (Flesch readability ease-score BIBREF15 ), number of words in a sentence and average character per word along with the sarcasm label as the predictors of average fixation duration following a linear mixed effect model BIBREF16 , sarcasm label turned out to be the most significant predictor with a maximum slope  This indicates that average fixation duration per word has a strong connection with the text being sarcastic, at least in our dataset 
We now analyze scanpaths to gain more insights into the sarcasm comprehension process 
Analysis of Scanpaths
Scanpaths are line-graphs that contain fixations as nodes and saccades as edges; the radii of the nodes represent the fixation duration  A scanpath corresponds to a participant's eye-movement pattern while reading a particular sentence  Figure FIGREF14 presents scanpaths of three participants for the sarcastic sentence S1 and the non-sarcastic sentence S2  The x-axis of the graph represents the sequence of words a reader reads, and the y-axis represents a temporal sequence in milliseconds 
Consider a sarcastic text containing incongruous phrases A and B  Our qualitative scanpath-analysis reveals that scanpaths with respect to sarcasm processing have two typical characteristics  Often, a long regression - a saccade that goes to a previously visited segment - is observed when a reader starts reading B after skimming through A  In a few cases, the fixation duration on A and B are significantly higher than the average fixation duration per word  In sentence S1, we see long and multiple regressions from the two incongruous phrases “misconception” and “cherish”, and a few instances where phrases “always cherish” and “original misconception” are fixated longer than usual  Such eye-movement behaviors are not seen for S2 
Though sarcasm induces distinctive scanpaths like the ones depicted in Figure FIGREF14 in the observed examples, presence of such patterns is not sufficient to guarantee sarcasm; such patterns may also possibly arise from literal texts  We believe that a combination of linguistic features, readability of text and features derived from scanpaths would help discriminative machine learning models learn sarcasm better 
Features for Sarcasm Detection
We describe the features used for sarcasm detection in Table   The features enlisted under lexical,implicit incongruity and explicit incongruity are borrowed from various literature (predominantly from joshi2015harnessing)  These features are essential to separate sarcasm from other forms semantic incongruity in text (for example ambiguity arising from semantic ambiguity or from metaphors)  Two additional textual features viz  readability and word count of the text are also taken under consideration  These features are used to reduce the effect of text hardness and text length on the eye-movement patterns 
Simple Gaze Based Features
Readers' eye-movement behavior, characterized by fixations, forward saccades, skips and regressions, can be directly quantified by simple statistical aggregation (i e , either computing features for individual participants and then averaging or performing a multi-instance based learning as explained in section SECREF6 )  Since these eye-movement attributes relate to the cognitive process in reading BIBREF17 , we consider these as features in our model  Some of these features have been reported by sarcasmunderstandability for modeling sarcasm understandability of readers  However, as far as we know, these features are being introduced in NLP tasks like textual sarcasm detection for the first time  The values of these features are believed to increase with the increase in the degree of surprisal caused by incongruity in text (except skip count, which will decrease) 
Complex Gaze Based Features
For these features, we rely on a graph structure, namely “saliency graphs", derived from eye-gaze information and word sequences in the text 
For each reader and each sentence, we construct a “saliency graph”, representing the reader's attention characteristics  A saliency graph for a sentence INLINEFORM0 for a reader INLINEFORM1 , represented as INLINEFORM2 , is a graph with vertices ( INLINEFORM3 ) and edges ( INLINEFORM4 ) where each vertex INLINEFORM5 corresponds to a word in INLINEFORM6 (may not be unique) and there exists an edge INLINEFORM7 between vertices INLINEFORM8 and INLINEFORM9 if R performs at least one saccade between the words corresponding to INLINEFORM10 and INLINEFORM11  
Figure FIGREF15 shows an example of a saliency graph A saliency graph may be weighted, but not necessarily connected, for a given text (as there may be words in the given text with no fixation on them)

ground_truth: simply averaging the predictions from the constituent single models

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What traditional linguistics features did they use?
Context 3:  The “complex” gaze features derived from saliency graphs are also motivated by the theory of incongruity  For instance, Edge Density of a saliency graph increases with the number of distinct saccades, which could arise from the complexity caused by presence of sarcasm  Similarly, the highest weighted degree of a graph is expected to be higher, if the node corresponds to a phrase, incongruous to some other phrase in the text 
The Sarcasm Classifier
We interpret sarcasm detection as a binary classification problem  The training data constitutes 994 examples created using our eye-movement database for sarcasm detection  To check the effectiveness of our feature set, we observe the performance of multiple classification techniques on our dataset through a stratified 10-fold cross validation  We also compare the classification accuracy of our system and the best available systems proposed by riloff2013sarcasm and joshi2015harnessing on our dataset  Using Weka BIBREF18 and LibSVM BIBREF19 APIs, we implement the following classifiers:
Results
Table TABREF17 shows the classification results considering various feature combinations for different classifiers and other systems  These are:
Unigram (with principal components of unigram feature vectors),
Sarcasm (the feature-set reported by joshi2015harnessing subsuming unigram features and features from other reported systems)
Gaze (the simple and complex cognitive features we introduce, along with readability and word count features), and
Gaze+Sarcasm (the complete set of features) 
For all regular classifiers, the gaze features are averaged across participants and augmented with linguistic and sarcasm related features  For the MILR classifier, the gaze features derived from each participant are augmented with linguistic features and thus, a multi instance “bag” of features is formed for each sentence in the training data  This multi-instance dataset is given to an MILR classifier, which follows the standard multi instance assumption to derive class-labels for each bag 
For all the classifiers, our feature combination outperforms the baselines (considering only unigram features) as well as BIBREF3 , with the MILR classifier getting an F-score improvement of 3 7% and Kappa difference of 0 08  We also achieve an improvement of 2% over the baseline, using SVM classifier, when we employ our feature set  We also observe that the gaze features alone, also capture the differences between sarcasm and non-sarcasm classes with a high-precision but a low recall 
To see if the improvement obtained is statistically significant over the state-of-the art system with textual sarcasm features alone, we perform McNemar test  The output of the SVM classifier using only linguistic features used for sarcasm detection by joshi2015harnessing and the output of the MILR classifier with the complete set of features are compared, setting threshold INLINEFORM0   There was a significant difference in the classifier's accuracy with p(two-tailed) = 0 02 with an odds-ratio of 1 43, showing that the classification accuracy improvement is unlikely to be observed by chance in 95% confidence interval 
Considering Reading Time as a Cognitive Feature along with Sarcasm Features
One may argue that, considering simple measures of reading effort like “reading time” as cognitive feature instead of the expensive eye-tracking features for sarcasm detection may be a cost-effective solution  To examine this, we repeated our experiments with “reading time” considered as the only cognitive feature, augmented with the textual features  The F-scores of all the classifiers turn out to be close to that of the classifiers considering sarcasm feature alone and the difference in the improvement is not statistically significant ( INLINEFORM0 )  One the other hand, F-scores with gaze features are superior to the F-scores when reading time is considered as a cognitive feature 
How Effective are the Cognitive Features
We examine the effectiveness of cognitive features on the classification accuracy by varying the input training data size  To examine this, we create a stratified (keeping the class ratio constant) random train-test split of 80%:20%  We train our classifier with 100%, 90%, 80% and 70% of the training data with our whole feature set, and the feature combination from joshi2015harnessing  The goodness of our system is demonstrated by improvements in F-score and Kappa statistics, shown in Figure FIGREF22  
We further analyze the importance of features by ranking the features based on (a) Chi squared test, and (b) Information Gain test, using Weka's attribute selection module  Figure FIGREF23 shows the top 20 ranked features produced by both the tests  For both the cases, we observe 16 out of top 20 features to be gaze features  Further, in each of the cases, Average Fixation Duration per Word and Largest Regression Position are seen to be the two most significant features 
Example Cases
Table TABREF21 shows a few example cases from the experiment with stratified 80%-20% train-test split 
Example sentence 1 is sarcastic, and requires extra-linguistic knowledge (about poor living conditions at Manchester)  Hence, the sarcasm detector relying only on textual features is unable to detect the underlying incongruity  However, our system predicts the label successfully, possibly helped by the gaze features 
Similarly, for sentence 2, the false sense of presence of incongruity (due to phrases like “Helped me” and “Can't stop”) affects the system with only linguistic features  Our system, though, performs well in this case also 
Sentence 3 presents a false-negative case where it was hard for even humans to get the sarcasm  This is why our gaze features (and subsequently the complete set of features) account for erroneous prediction 
In sentence 4, gaze features alone false-indicate presence of incongruity, whereas the system predicts correctly when gaze and linguistic features are taken together 
From these examples, it can be inferred that, only gaze features would not have sufficed to rule out the possibility of detecting other forms of incongruity that do not result in sarcasm 
Error Analysis
Errors committed by our system arise from multiple factors, starting from limitations of the eye-tracker hardware to errors committed by linguistic tools and resources  Also, aggregating various eye-tracking parameters to extract the cognitive features may have caused information loss in the regular classification setting 
Conclusion
In the current work, we created a novel framework to detect sarcasm, that derives insights from human cognition, that manifests over eye movement patterns  We hypothesized that distinctive eye-movement patterns, associated with reading sarcastic text, enables improved detection of sarcasm  We augmented traditional linguistic features with cognitive features obtained from readers' eye-movement data in the form of simple gaze-based features and complex features derived from a graph structure  This extended feature-set improved the success rate of the sarcasm detector by 3 7%, over the best available system  Using cognitive features in an NLP Processing system like ours is the first proposal of its kind 
Our general approach may be useful in other NLP sub-areas like sentiment and emotion analysis, text summarization and question answering, where considering textual clues alone does not prove to be sufficient  We propose to augment this work in future by exploring deeper graph and gaze features  We also propose to develop models for the purpose of learning complex gaze feature representation, that accounts for the power of individual eye movement patterns along with the aggregated patterns of eye movements 
Acknowledgments
We thank the members of CFILT Lab, especially Jaya Jha and Meghna Singh, and the students of IIT Bombay for their help and support 

ground_truth: Unanswerable