prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are strong baselines model is compared to?
Context 1: Introduction
Chinese word segmentation (CWS) is a task for Chinese natural language process to delimit word boundary  CWS is a basic and essential task for Chinese which is written without explicit word delimiters and different from alphabetical languages like English  BIBREF0 treats Chinese word segmentation (CWS) as a sequence labeling task with character position tags, which is followed by BIBREF1, BIBREF2, BIBREF3  Traditional CWS models depend on the design of features heavily which effects the performance of model  To minimize the effort in feature engineering, some CWS models BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11 are developed following neural network architecture for sequence labeling tasks BIBREF12  Neural CWS models perform strong ability of feature representation, employing unigram and bigram character embedding as input and approach good performance 
The CWS task is often modeled as one graph model based on a scoring model that means it is composed of two parts, one part is an encoder which is used to generate the representation of characters from the input sequence, the other part is a decoder which performs segmentation according to the encoder scoring  Table TABREF1 summarizes typical CWS models according to their decoding ways for both traditional and neural models  Markov models such as BIBREF13 and BIBREF4 depend on the maximum entropy model or maximum entropy Markov model both with a Viterbi decoder  Besides, conditional random field (CRF) or Semi-CRF for sequence labeling has been used for both traditional and neural models though with different representations BIBREF2, BIBREF15, BIBREF10, BIBREF17, BIBREF18  Generally speaking, the major difference between traditional and neural network models is about the way to represent input sentences 
Recent works about neural CWS which focus on benchmark dataset, namely SIGHAN Bakeoff BIBREF21, may be put into the following three categories roughly 
Encoder  Practice in various natural language processing tasks has been shown that effective representation is essential to the performance improvement  Thus for better CWS, it is crucial to encode the input character, word or sentence into effective representation  Table TABREF2 summarizes regular feature sets for typical CWS models including ours as well  The building blocks that encoders use include recurrent neural network (RNN) and convolutional neural network (CNN), and long-term memory network (LSTM) 
Graph model  As CWS is a kind of structure learning task, the graph model determines which type of decoder should be adopted for segmentation, also it may limit the capability of defining feature, as shown in Table 2, not all graph models can support the word features  Thus recent work focused on finding more general or flexible graph model to make model learn the representation of segmentation more effective as BIBREF9, BIBREF11 
External data and pre-trained embedding  Whereas both encoder and graph model are about exploring a way to get better performance only by improving the model strength itself  Using external resource such as pre-trained embeddings or language representation is an alternative for the same purpose BIBREF22, BIBREF23  SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21  In this work, we will focus on the closed test setting by finding a better model design for further CWS performance improvement 
Shown in Table TABREF1, different decoders have particular decoding algorithms to match the respective CWS models  Markov models and CRF-based models often use Viterbi decoders with polynomial time complexity  In general graph model, search space may be too large for model to search  Thus it forces graph models to use an approximate beam search strategy  Beam search algorithm has a kind low-order polynomial time complexity  Especially, when beam width $b$=1, the beam search algorithm will reduce to greedy algorithm with a better time complexity $O(Mn)$ against the general beam search time complexity $O(Mnb^2)$, where $n$ is the number of units in one sentences, $M$ is a constant representing the model complexity  Greedy decoding algorithm can bring the fastest speed of decoding while it is not easy to guarantee the precision of decoding when the encoder is not strong enough 
In this paper, we focus on more effective encoder design which is capable of offering fast and accurate Chinese word segmentation with only unigram feature and greedy decoding  Our proposed encoder will only consist of attention mechanisms as building blocks but nothing else  Motivated by the Transformer BIBREF24 and its strength of capturing long-range dependencies of input sentences, we use a self-attention network to generate the representation of input which makes the model encode sentences at once without feeding input iteratively  Considering the weakness of the Transformer to model relative and absolute position information directly BIBREF25 and the importance of localness information, position information and directional information for CWS, we further improve the architecture of standard multi-head self-attention of the Transformer with a directional Gaussian mask and get a variant called Gaussian-masked directional multi-head attention  Based on the newly improved attention mechanism, we expand the encoder of the Transformer to capture different directional information  With our powerful encoder, our model uses only simple unigram features to generate representation of sentences 
For decoder which directly performs the segmentation, we use the bi-affinal attention scorer, which has been used in dependency parsing BIBREF26 and semantic role labeling BIBREF27, to implement greedy decoding on finding the boundaries of words  In our proposed model, greedy decoding ensures a fast segmentation while powerful encoder design ensures a good enough segmentation performance even working with greedy decoder together  Our model will be strictly evaluated on benchmark datasets from SIGHAN Bakeoff shared task on CWS in terms of closed test setting, and the experimental results show that our proposed model achieves new state-of-the-art 
The technical contributions of this paper can be summarized as follows 
We propose a CWS model with only attention structure  The encoder and decoder are both based on attention structure 
With a powerful enough encoder, we for the first time show that unigram (character) featues can help yield strong performance instead of diverse $n$-gram (character and word) features in most of previous work 
To capture the representation of localness information and directional information, we propose a variant of directional multi-head self-attention to further enhance the state-of-the-art Transformer encoder 
Models
The CWS task is often modelled as one graph model based on an encoder-based scoring model  The model for CWS task is composed of an encoder to represent the input and a decoder based on the encoder to perform actual segmentation  Figure FIGREF6 is the architecture of our model  The model feeds sentence into encoder  Embedding captures the vector $e=(e_1,   ,e_n)$ of the input character sequences of $c=(c_1,   ,c_n)$  The encoder maps vector sequences of $ {e}=(e_1,  ,e_n)$ to two sequences of vector which are $ {v^b}=(v_1^b,   ,v_n^b)$ and ${v^f}=(v_1^f,   v_n^f)$ as the representation of sentences  With $v^b$ and $v^f$, the bi-affinal scorer calculates the probability of each segmentation gaps and predicts the word boundaries of input  Similar as the Transformer, the encoder is an attention network with stacked self-attention and point-wise, fully connected layers while our encoder includes three independent directional encoders 
Models ::: Encoder Stacks
In the Transformer, the encoder is composed of a stack of N identical layers and each layer has one multi-head self-attention layer and one position-wise fully connected feed-forward layer  One residual connection is around two sub-layers and followed by layer normalization BIBREF24  This architecture provides the Transformer a good ability to generate representation of sentence 
With the variant of multi-head self-attention, we design a Gaussian-masked directional encoder to capture representation of different directions to improve the ability of capturing the localness information and position information for the importance of adjacent characters  One unidirectional encoder can capture information of one particular direction 
For CWS tasks, one gap of characters, which is from a word boundary, can divide one sequence into two parts, one part in front of the gap and one part in the rear of it  The forward encoder and backward encoder are used to capture information of two directions which correspond to two parts divided by the gap 
One central encoder is paralleled with forward and backward encoders to capture the information of entire sentences  The central encoder is a special directional encoder for forward and backward information of sentences  The central encoder can fuse the information and enable the encoder to capture the global information 
The encoder outputs one forward information and one backward information of each positions  The representation of sentence generated by center encoder will be added to these information directly:
where $v^{b}=(v^b_1,   ,v^b_n)$ is the backward information, $v^{f}=(v^f_1,   ,v^f_n)$ is the forward information, $r^{b}=(r^b_1,   ,r^b_n)$ is the output of backward encoder, $r^{c}=(r^c_1,   ,r^c_n)$ is the output of center encoder and $r^{f}=(r^f_1,   ,r^f_n)$ is the output of forward encoder 
Models ::: Gaussian-Masked Directional Multi-Head Attention
Similar as scaled dot-product attention BIBREF24, Gaussian-masked directional attention can be described as a function to map queries and key-value pairs to the representation of input  Here queries, keys and values are all vectors  Standard scaled dot-product attention is calculated by dotting query $Q$ with all keys $K$, dividing each values by $\sqrt{d_k}$, where $\sqrt{d_k}$ is the dimension of keys, and apply a softmax function to generate the weights in the attention:
Different from scaled dot-product attention, Gaussian-masked directional attention expects to pay attention to the adjacent characters of each positions and cast the localness relationship between characters as a fix Gaussian weight for attention  We assume that the Gaussian weight only relys on the distance between characters 
Firstly we introduce the Gaussian weight matrix $G$ which presents the localness relationship between each two characters:
where $g_{ij}$ is the Gaussian weight between character $i$ and $j$, $dis_{ij}$ is the distance between character $i$ and $j$, $\Phi (x)$ is the cumulative distribution function of Gaussian, $\sigma $ is the standard deviation of Gaussian function and it is a hyperparameter in our method  Equation (DISPLAY_FORM13) can ensure the Gaussian weight equals 1 when $dis_{ij}$ is 0  The larger distance between charactersis, the smaller the weight is, which makes one character can affect its adjacent characters more compared with other characters 
To combine the Gaussian weight to the self-attention, we produce the Hadamard product of Gaussian weight matrix $G$ and the score matrix produced by $Q{K^{T}}$
where $AG$ is the Gaussian-masked attention  It ensures that the relationship between two characters with long distances is weaker than adjacent characters 
The scaled dot-product attention models the relationship between two characters without regard to their distances in one sequence  For CWS task, the weight between adjacent characters should be more important while it is hard for self-attention to achieve the effect explicitly because the self-attention cannot get the order of sentences directly  The Gaussian-masked attention adjusts the weight between characters and their adjacent character to a larger value which stands for the effect of adjacent characters 
For forward and backward encoder, the self-attention sublayer needs to use a triangular matrix mask to let the self-attention focus on different weights:
where $pos_i$ is the position of character $c_i$  The triangular matrix for forward and backward encode are:
$\left[ \begin{matrix} 1 & 0 & 0 & \cdots &0\\ 1 & 1 & 0 & \cdots &0\\ 1 & 1 & 1 & \cdots &0\\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 1 & 1 & 1 & \cdots & 1\\ \end{matrix} \right]$ $\left[ \begin{matrix} 1 & 1 & 1 & \cdots &1 \\ 0 & 1 & 1 & \cdots &1 \\ 0 & 0& 1 & \cdots &1 \\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 0 & 0 & 0 & \cdots & 1\\ \end{matrix}\right]$
Similar as BIBREF24, we use multi-head attention to capture information from different dimension positions as Figure FIGREF16 and get Gaussian-masked directional multi-head attention  With multi-head attention architecture, the representation of input can be captured by
where $MH$ is the Gaussian-masked multi-head attention, ${W_i^q, W_i^k,W_i^v} \in \mathbb {R}^{d_k \times d_h}$ is the parameter matrices to generate heads, $d_k$ is the dimension of model and $d_h$ is the dimension of one head 
Models ::: Bi-affinal Attention Scorer
Regarding word boundaries as gaps between any adjacent words converts the character labeling task to the gap labeling task  Different from character labeling task, gap labeling task requires information of two adjacent characters  The relationship between adjacent characters can be represented as the type of gap  The characteristic of word boundaries makes bi-affine attention an appropriate scorer for CWS task 
Bi-affinal attention scorer is the component that we use to label the gap  Bi-affinal attention is developed from bilinear attention which has been used in dependency parsing BIBREF26 and SRL BIBREF27  The distribution of labels in a labeling task is often uneven which makes the output layer often include a fixed bias term for the prior probability of different labels BIBREF27  Bi-affine attention uses bias terms to alleviate the burden of the fixed bias term and get the prior probability which makes it different from bilinear attention  The distribution of the gap is uneven that is similar as other labeling task which fits bi-affine 
Bi-affinal attention scorer labels the target depending on information of independent unit and the joint information of two units  In bi-affinal attention, the score $s_{ij}$ of characters $c_i$ and $c_j$ $(i < j)$ is calculated by:
where $v_i^f$ is the forward information of $c_i$ and $v_i^b$ is the backward information of $c_j$  In Equation (DISPLAY_FORM21), $W$, $U$ and $b$ are all parameters that can be updated in training  $W$ is a matrix with shape $(d_i \times N\times d_j)$ and $U$ is a $(N\times (d_i + d_j))$ matrix where $d_i$ is the dimension of vector $v_i^f$ and $N$ is the number of labels 
In our model, the biaffine scorer uses the forward information of character in front of the gap and the backward information of the character behind the gap to distinguish the position of characters  Figure FIGREF22 is an example of labeling gap  The method of using biaffine scorer ensures that the boundaries of words can be determined by adjacent characters with different directional information  The score vector of the gap is formed by the probability of being a boundary of word  Further, the model generates all boundaries using activation function in a greedy decoding way 
Experiments ::: Experimental Settings ::: Data
We train and evaluate our model on datasets from SIGHAN Bakeoff 2005 BIBREF21 which has four datasets, PKU, MSR, AS and CITYU  Table TABREF23 shows the statistics of train data  We use F-score to evaluate CWS models  To train model with pre-trained embeddings in AS and CITYU, we use OpenCC to transfer data from traditional Chinese to simplified Chinese 
Experiments ::: Experimental Settings ::: Pre-trained Embedding
We only use unigram feature so we only trained character embeddings  Our pre-trained embedding are pre-trained on Chinese Wikipedia corpus by word2vec BIBREF29 toolkit  The corpus used for pre-trained embedding is all transferred to simplified Chinese and not segmented  On closed test, we use embeddings initialized randomly 
Experiments ::: Experimental Settings ::: Hyperparameters
For different datasets, we use two kinds of hyperparameters which are presented in Table TABREF24  We use hyperparameters in Table TABREF24 for small corpora (PKU and CITYU) and normal corpora (MSR and AS)  We set the standard deviation of Gaussian function in Equation (DISPLAY_FORM13) to 2  Each training batch contains sentences with at most 4096 tokens 
Experiments ::: Experimental Settings ::: Optimizer
To train our model, we use the Adam BIBREF30 optimizer with $\beta _1=0 9$, $\beta _2=0 98$ and $\epsilon =10^{-9}$  The learning rate schedule is the same as BIBREF24:
where $d$ is the dimension of embeddings, $step$ is the step number of training and $warmup_step$ is the step number of warmup  When the number of steps is smaller than the step of warmup, the learning rate increases linearly and then decreases 
Experiments ::: Hardware and Implements
We trained our models on a single CPU (Intel i7-5960X) with an nVidia 1080 Ti GPU  We implement our model in Python with Pytorch 1 0 
Experiments ::: Results
Tables TABREF25 and TABREF26 reports the performance of recent models and ours in terms of closed test setting  Without the assistance of unsupervised segmentation features userd in BIBREF20, our model outperforms all the other models in MSR and AS except BIBREF18 and get comparable performance in PKU and CITYU  Note that all the other models for this comparison adopt various $n$-gram features while only our model takes unigram ones 
With unsupervised segmentation features introduced by BIBREF20, our model gets a higher result  Specially, the results in MSR and AS achieve new state-of-the-art and approaching previous state-of-the-art in CITYU and PKU  The unsupervised segmentation features are derived from the given training dataset, thus using them does not violate the rule of closed test of SIGHAN Bakeoff 
Table TABREF36 compares our model and recent neural models in terms of open test setting in which any external resources, especially pre-trained embeddings or language models can be used  In MSR and AS, our model gets a comparable result while our results in CITYU and PKU are not remarkable 
However, it is well known that it is always hard to compare models when using open test setting, especially with pre-trained embedding  Not all models may use the same method and data to pre-train  Though pre-trained embedding or language model can improve the performance, the performance improvement itself may be from multiple sources  It often that there is a success of pre-trained embedding to improve the performance, while it cannot prove that the model is better 
Compared with other LSTM models, our model performs better in AS and MSR than in CITYU and PKU  Considering the scale of different corpora, we believe that the size of corpus affects our model and the larger size is, the better model performs  For small corpus, the model tends to be overfitting 
Tables TABREF25 and TABREF26 also show the decoding time in different datasets  Our model finishes the segmentation with the least decoding time in all four datasets, thanks to the architecture of model which only takes attention mechanism as basic block 
Related Work ::: Chinese Word Segmentation
CWS is a task for Chinese natural language process to delimit word boundary  BIBREF0 for the first time formulize CWS as a sequence labeling task  BIBREF3 show that different character tag sets can make essential impact for CWS  BIBREF2 use CRFs as a model for CWS, achieving new state-of-the-art  Works of statistical CWS has built the basis for neural CWS 
Neural word segmentation has been widely used to minimize the efforts in feature engineering which was important in statistical CWS  BIBREF4 introduce the neural model with sliding-window based sequence labeling  BIBREF6 propose a gated recursive neural network (GRNN) for CWS to incorporate complicated combination of contextual character and n-gram features  BIBREF7 use LSTM to learn long distance information  BIBREF9 propose a neural framework that eliminates context windows and utilize complete segmentation history  BIBREF33 explore a joint model that performs segmentation, POS-Tagging and chunking simultaneously  BIBREF34 propose a feature-enriched neural model for joint CWS and part-of-speech tagging  BIBREF35 present a joint model to enhance the segmentation of Chinese microtext by performing CWS and informal word detection simultaneously  BIBREF17 propose a character-based convolutional neural model to capture $n$-gram features automatically and an effective approach to incorporate word embeddings  BIBREF11 improve the model in BIBREF9 and propose a greedy neural word segmenter with balanced word and character embedding inputs  BIBREF23 propose a novel neural network model to incorporate unlabeled and partially-labeled data  BIBREF36 propose two methods that extend the Bi-LSTM to perform incorporating dictionaries into neural networks for CWS  BIBREF37 propose Switch-LSTMs to segment words and provided a more flexible solution for multi-criteria CWS which is easy to transfer the learned knowledge to new criteria 
Related Work ::: Transformer
Transformer BIBREF24 is an attention-based neural machine translation model  The Transformer is one kind of self-attention networks (SANs) which is proposed in BIBREF38  Encoder of the Transformer consists of one self-attention layer and a position-wise feed-forward layer  Decoder of the Transformer contains one self-attention layer, one encoder-decoder attention layer and one position-wise feed-forward layer  The Transformer uses residual connections around the sublayers and then followed by a layer normalization layer 
Scaled dot-product attention is the key component in the Transformer  The input of attention contains queries, keys, and values of input sequences  The attention is generated using queries and keys like Equation (DISPLAY_FORM11)  Structure of scaled dot-product attention allows the self-attention layer generate the representation of sentences at once and contain the information of the sentence which is different from RNN that process characters of sentences one by one  Standard self-attention is similar as Gaussian-masked direction attention while it does not have directional mask and gaussian mask  BIBREF24 also propose multi-head attention which is better to generate representation of sentence by dividing queries, keys and values to different heads and get information from different subspaces 
Conclusion
In this paper, we propose an attention mechanism only based Chinese word segmentation model  Our model uses self-attention from the Transformer encoder to take sequence input and bi-affine attention scorer to predict the label of gaps  To improve the ability of capturing the localness and directional information of self-attention based encoder, we propose a variant of self-attention called Gaussian-masked directional multi-head attention to replace the standard self-attention  We also extend the Transformer encoder to capture directional features  Our model uses only unigram features instead of multiple $n$-gram features in previous work  Our model is evaluated on standard benchmark dataset, SIGHAN Bakeoff 2005, which shows not only our model performs segmentation faster than any previous models but also gives new higher or comparable segmentation performance against previous state-of-the-art models 
Context 2: Introduction
Reading Comprehension (RC) has become a central task in natural language processing, with great practical value in various industries  In recent years, many large-scale RC datasets in English BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 have nourished the development of numerous powerful and diverse RC models BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11  The state-of-the-art model BIBREF12 on SQuAD, one of the most widely used RC benchmarks, even surpasses human-level performance  Nonetheless, RC on languages other than English has been limited due to the absence of sufficient training data  Although some efforts have been made to create RC datasets for Chinese BIBREF13, BIBREF14 and Korean BIBREF15, it is not feasible to collect RC datasets for every language since annotation efforts to collect a new RC dataset are often far from trivial  Therefore, the setup of transfer learning, especially zero-shot learning, is of extraordinary importance 
Existing methods BIBREF16 of cross-lingual transfer learning on RC datasets often count on machine translation (MT) to translate data from source language into target language, or vice versa  These methods may not require a well-annotated RC dataset for the target language, whereas a high-quality MT model is needed as a trade-off, which might not be available when it comes to low-resource languages 
In this paper, we leverage pre-trained multilingual language representation, for example, BERT learned from multilingual un-annotated sentences (multi-BERT), in cross-lingual zero-shot RC  We fine-tune multi-BERT on the training set in source language, then test the model in target language, with a number of combinations of source-target language pair to explore the cross-lingual ability of multi-BERT  Surprisingly, we find that the models have the ability to transfer between low lexical similarity language pair, such as English and Chinese  Recent studies BIBREF17, BIBREF12, BIBREF18 show that cross-lingual language models have the ability to enable preliminary zero-shot transfer on simple natural language understanding tasks, but zero-shot transfer of RC has not been studied  To our knowledge, this is the first work systematically exploring the cross-lingual transferring ability of multi-BERT on RC tasks 
Zero-shot Transfer with Multi-BERT
Multi-BERT has showcased its ability to enable cross-lingual zero-shot learning on the natural language understanding tasks including XNLI BIBREF19, NER, POS, Dependency Parsing, and so on  We now seek to know if a pre-trained multi-BERT has ability to solve RC tasks in the zero-shot setting 
Zero-shot Transfer with Multi-BERT ::: Experimental Setup and Data
We have training and testing sets in three different languages: English, Chinese and Korean  The English dataset is SQuAD BIBREF2  The Chinese dataset is DRCD BIBREF14, a Chinese RC dataset with 30,000+ examples in the training set and 10,000+ examples in the development set  The Korean dataset is KorQuAD BIBREF15, a Korean RC dataset with 60,000+ examples in the training set and 10,000+ examples in the development set, created in exactly the same procedure as SQuAD  We always use the development sets of SQuAD, DRCD and KorQuAD for testing since the testing sets of the corpora have not been released yet 
Next, to construct a diverse cross-lingual RC dataset with compromised quality, we translated the English and Chinese datasets into more languages, with Google Translate  An obvious issue with this method is that some examples might no longer have a recoverable span  To solve the problem, we use fuzzy matching to find the most possible answer, which calculates minimal edit distance between translated answer and all possible spans  If the minimal edit distance is larger than min(10, lengths of translated answer - 1), we drop the examples during training, and treat them as noise when testing  In this way, we can recover more than 95% of examples  The following generated datasets are recovered with same setting 
The pre-trained multi-BERT is the official released one  This multi-lingual version of BERT were pre-trained on corpus in 104 languages  Data in different languages were simply mixed in batches while pre-training, without additional effort to align between languages  When fine-tuning, we simply adopted the official training script of BERT, with default hyperparameters, to fine-tune each model until training loss converged 
Zero-shot Transfer with Multi-BERT ::: Experimental Results
Table TABREF6 shows the result of different models trained on either Chinese or English and tested on Chinese  In row (f), multi-BERT is fine-tuned on English but tested on Chinese, which achieves competitive performance compared with QANet trained on Chinese  We also find that multi-BERT trained on English has relatively lower EM compared with the model with comparable F1 scores  This shows that the model learned with zero-shot can roughly identify the answer spans in context but less accurate  In row (c), we fine-tuned a BERT model pre-trained on English monolingual corpus (English BERT) on Chinese RC training data directly by appending fastText-initialized Chinese word embeddings to the original word embeddings of English-BERT  Its F1 score is even lower than that of zero-shot transferring multi-BERT (rows (c) v s  (e))  The result implies multi-BERT does acquire better cross-lingual capability through pre-training on multilingual corpus  Table TABREF8 shows the results of multi-BERT fine-tuned on different languages and then tested on English , Chinese and Korean  The top half of the table shows the results of training data without translation  It is not surprising that when the training and testing sets are in the same language, the best results are achieved, and multi-BERT shows transfer capability when training and testing sets are in different languages, especially between Chinese and Korean 
In the lower half of Table TABREF8, the results are obtained by the translated training data  First, we found that when testing on English and Chinese, translation always degrades the performance (En v s  En-XX, Zh v s  Zh-XX)  Even though we translate the training data into the same language as testing data, using the untranslated data still yield better results  For example, when testing on English, the F1 score of the model training on Chinese (Zh) is 53 8, while the F1 score is only 44 1 for the model training on Zh-En  This shows that translation degrades the quality of data  There are some exceptions when testing on Korean  Translating the English training data into Chinese, Japanese and Korean still improve the performance on Korean  We also found that when translated into the same language, the English training data is always better than the Chinese data (En-XX v s  Zh-XX), with only one exception (En-Fr v s  Zh-Fr when testing on KorQuAD)  This may be because we have less Chinese training data than English  These results show that the quality and the size of dataset are much more important than whether the training and testing are in the same language or not 
Zero-shot Transfer with Multi-BERT ::: Discussion ::: The Effect of Machine Translation
Table TABREF8 shows that fine-tuning on un-translated target language data achieves much better performance than data translated into the target language  Because the above statement is true across all the languages, it is a strong evidence that translation degrades the performance We notice that the translated corpus and untranslated corpus are not the same  This may be another factor that influences the results  Conducting an experiment between un-translated and back-translated data may deal with this problem 
Zero-shot Transfer with Multi-BERT ::: Discussion ::: The Effect of Other Factors
Here we discuss the case that the training data are translated  We consider each result is affected by at least three factors: (1) training corpus, (2) data size, (3) whether the source corpus is translated into the target language  To study the effect of data-size, we conducted an extra experiment where we down-sampled the size of English data to be the same as Chinese corpus, and used the down-sampled corpus to train  Then We carried out one-way ANOVA test and found out the significance of the three factors are ranked as below: (1) > (2) >> (3)  The analysis supports that the characteristics of training data is more important than translated into target language or not  Therefore, although translation degrades the performance, whether translating the corpus into the target language is not critical 
What Does Zero-shot Transfer Model Learn? ::: Unseen Language Dataset
It has been shown that extractive QA tasks like SQuAD may be tackled by some language independent strategies, for example, matching words in questions and context BIBREF20  Is zero-shot learning feasible because the model simply learns this kind of language independent strategies on one language and apply to the other?
To verify whether multi-BERT largely counts on a language independent strategy, we test the model on the languages unseen during pre-training  To make sure the languages have never been seen before, we artificially make unseen languages by permuting the whole vocabulary of existing languages  That is, all the words in the sentences of a specific language are replaced by other words in the same language to form the sentences in the created unseen language  It is assumed that if multi-BERT used to find answers by language independent strategy, then multi-BERT should also do well on unseen languages  Table TABREF14 shows that the performance of multi-BERT drops drastically on the dataset  It implies that multi-BERT might not totally rely on pattern matching when finding answers 
What Does Zero-shot Transfer Model Learn? ::: Embedding in Multi-BERT
PCA projection of hidden representations of the last layer of multi-BERT before and after fine-tuning are shown in Fig  FIGREF15  The red points represent Chinese tokens, and the blue points are for English  The results show that tokens from different languages might be embedded into the same space with close spatial distribution  Even though during the fine-tuning only the English data is used, the embedding of the Chinese token changed accordingly  We also quantitatively evaluate the similarities between the embedding of the languages  The results can be found in the Appendix 
What Does Zero-shot Transfer Model Learn? ::: Code-switching Dataset
We observe linguistic-agnostic representations in the last subsection  If tokens are represented in a language-agnostic way, the model may be able to handle code-switching data  Because there is no code-switching data for RC, we create artificial code-switching datasets by replacing some of the words in contexts or questions with their synonyms in another language  The synonyms are found by word-by-word translation with given dictionaries  We use the bilingual dictionaries collected and released in facebookresearch/MUSE GitHub repository  We substitute the words if and only if the words are in the bilingual dictionaries 
Table TABREF14 shows that on all the code-switching datasets, the EM/F1 score drops, indicating that the semantics of representations are not totally disentangled from language  However, the examples of the answers of the model (Table TABREF21) show that multi-BERT could find the correct answer spans although some keywords in the spans have been translated into another language 
What Does Zero-shot Transfer Model Learn? ::: Typology-manipulated Dataset
There are various types of typology in languages  For example, in English the typology order is subject-verb-object (SVO) order, but in Japanese and Korean the order is subject-object-verb (SOV)  We construct a typology-manipulated dataset to examine if the typology order of the training data influences the transfer learning results  If the model only learns the semantic mapping between different languages, changing English typology order from SVO to SOV should improve the transfer ability from English to Japanese  The method used to generate datasets is the same as BIBREF21 
The source code is from a GitHub repository named Shaul1321/rnn_typology, which labels given sentences to CoNLL format with StanfordCoreNLP and then re-arranges them greedily 
Table TABREF23 shows that when we change the English typology order to SOV or OSV order, the performance on Korean is improved and worsen on English and Chinese, but very slightly  The results show that the typology manipulation on the training set has little influence  It is possible that multi-BERT normalizes the typology order of different languages to some extent 
Conclusion
In this paper, we systematically explore zero-shot cross-lingual transfer learning on RC with multi-BERT  The experimental results on English, Chinese and Korean corpora show that even when the languages for training and testing are different, reasonable performance can be obtained  Furthermore, we created several artificial data to study the cross-lingual ability of multi-BERT in the presence of typology variation and code-switching  We showed that only token-level pattern matching is not sufficient for multi-BERT to answer questions and typology variation and code-switching only caused minor effects on testing performance 
Supplemental Material ::: Internal Representation of multi-BERT
The architecture of multi-BERT is a Transformer encoder BIBREF25  While fine-tuning on SQuAD-like dataset, the bottom layers of multi-BERT are initialized from Google-pretrained parameters, with an added output layer initialized from random parameters  Tokens representations from the last layer of bottom-part of multi-BERT are inputs to the output layer and then the output layer outputs a distribution over all tokens that indicates the probability of a token being the START/END of an answer span 
Supplemental Material ::: Internal Representation of multi-BERT ::: Cosine Similarity
As all translated versions of SQuAD/DRCD are parallel to each other  Given a source-target language pair, we calculate cosine similarity of the mean pooling of tokens representation within corresponding answer-span as a measure of how much they look like in terms of the internal representation of multi-BERT  The results are shown in Fig  FIGREF26 
Supplemental Material ::: Internal Representation of multi-BERT ::: SVCCA
Singular Vector Canonical Correlation Analysis (SVCCA) is a general method to compare the correlation of two sets of vector representations  SVCCA has been proposed to compare learned representations across language models BIBREF24  Here we adopt SVCCA to measure the linear similarity of two sets of representations in the same multi-BERT from different translated datasets, which are parallel to each other  The results are shown in Fig FIGREF28 
Supplemental Material ::: Improve Transfering
In the paper, we show that internal representations of multi-BERT are linear-mappable to some extent between different languages  This implies that multi-BERT model might encode semantic and syntactic information in language-agnostic ways and explains how zero-shot transfer learning could be done 
To take a step further, while transfering model from source dataset to target dataset, we align representations in two proposed way, to improve performance on target dataset 
Supplemental Material ::: Improve Transfering ::: Linear Mapping Method
Algorithms proposed in BIBREF23, BIBREF22, BIBREF26 to unsupervisedly learn linear mapping between two sets of embeddings are used here to align representations of source (training data) to those of target  We obtain the mapping generated by embeddings from one specific layer of pre-trained multi-BERT then we apply this mapping to transform the internal representations of multi-BERT while fine-tuning on training data 
Supplemental Material ::: Improve Transfering ::: Adversarial Method
In Adversarial Method, we add an additional transform layer to transform representations and a discrimination layer to discriminate between transformed representations from source language (training set) and target language (development set)  And the GAN loss is applied in the total loss of fine-tuning 
Supplemental Material ::: Improve Transfering ::: Discussion
As table TABREF33 shows, there are no improvements among above methods  Some linear mapping methods even causes devastating effect on EM/F1 scores 

ground_truth: Baseline models are:
- Chen et al., 2015a
- Chen et al., 2015b
- Liu et al., 2016
- Cai and Zhao, 2016
- Cai et al., 2017
- Zhou et al., 2017
- Ma et al., 2018
- Wang et al., 2019

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the model performance on target language reading comprehension?
Context 4: Introduction
Propaganda aims at influencing people's mindset with the purpose of advancing a specific agenda  In the Internet era, thanks to the mechanism of sharing in social networks, propaganda campaigns have the potential of reaching very large audiences BIBREF0, BIBREF1, BIBREF2 
Propagandist news articles use specific techniques to convey their message, such as whataboutism, red Herring, and name calling, among many others (cf  Section SECREF3)  Whereas proving intent is not easy, we can analyse the language of a claim/article and look for the use of specific propaganda techniques  Going at this fine-grained level can yield more reliable systems and it also makes it possible to explain to the user why an article was judged as propagandist by an automatic system 
With this in mind, we organised the shared task on fine-grained propaganda detection at the NLP4IF@EMNLP-IJCNLP 2019 workshop  The task is based on a corpus of news articles annotated with an inventory of 18 propagandist techniques at the fragment level  We hope that the corpus would raise interest outside of the community of researchers studying propaganda  For example, the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
Related Work
Propaganda has been tackled mostly at the article level  BIBREF3 created a corpus of news articles labelled as propaganda, trusted, hoax, or satire  BIBREF4 experimented with a binarized version of that corpus: propaganda vs  the other three categories  BIBREF5 annotated a large binary corpus of propagandist vs  non-propagandist articles and proposed a feature-based system for discriminating between them  In all these cases, the labels were obtained using distant supervision, assuming that all articles from a given news outlet share the label of that outlet, which inevitably introduces noise BIBREF6 
A related field is that of computational argumentation which, among others, deals with some logical fallacies related to propaganda  BIBREF7 presented a corpus of Web forum discussions with instances of ad hominem fallacy  BIBREF8, BIBREF9 introduced Argotario, a game to educate people to recognize and create fallacies, a by-product of which is a corpus with $1 3k$ arguments annotated with five fallacies such as ad hominem, red herring and irrelevant authority, which directly relate to propaganda 
Unlike BIBREF8, BIBREF9, BIBREF7, our corpus uses 18 techniques annotated on the same set of news articles  Moreover, our annotations aim at identifying the minimal fragments related to a technique instead of flagging entire arguments 
The most relevant related work is our own, which is published in parallel to this paper at EMNLP-IJCNLP 2019 BIBREF10 and describes a corpus that is a subset of the one used for this shared task 
Propaganda Techniques
Propaganda uses psychological and rhetorical techniques to achieve its objective  Such techniques include the use of logical fallacies and appeal to emotions  For the shared task, we use 18 techniques that can be found in news articles and can be judged intrinsically, without the need to retrieve supporting information from external resources  We refer the reader to BIBREF10 for more details on the propaganda techniques; below we report the list of techniques:
Propaganda Techniques ::: 1  Loaded language 
Using words/phrases with strong emotional implications (positive or negative) to influence an audience BIBREF11 
Propaganda Techniques ::: 2  Name calling or labeling 
Labeling the object of the propaganda as something the target audience fears, hates, finds undesirable or otherwise loves or praises BIBREF12 
Propaganda Techniques ::: 3  Repetition 
Repeating the same message over and over again, so that the audience will eventually accept it BIBREF13, BIBREF12 
Propaganda Techniques ::: 4  Exaggeration or minimization 
Either representing something in an excessive manner: making things larger, better, worse, or making something seem less important or smaller than it actually is BIBREF14, e g , saying that an insult was just a joke 
Propaganda Techniques ::: 5  Doubt 
Questioning the credibility of someone or something 
Propaganda Techniques ::: 6  Appeal to fear/prejudice 
Seeking to build support for an idea by instilling anxiety and/or panic in the population towards an alternative, possibly based on preconceived judgments 
Propaganda Techniques ::: 7  Flag-waving 
Playing on strong national feeling (or with respect to a group, e g , race, gender, political preference) to justify or promote an action or idea BIBREF15 
Propaganda Techniques ::: 8  Causal oversimplification 
Assuming one cause when there are multiple causes behind an issue  We include scapegoating as well: the transfer of the blame to one person or group of people without investigating the complexities of an issue 
Propaganda Techniques ::: 9  Slogans 
A brief and striking phrase that may include labeling and stereotyping  Slogans tend to act as emotional appeals BIBREF16 
Propaganda Techniques ::: 10  Appeal to authority 
Stating that a claim is true simply because a valid authority/expert on the issue supports it, without any other supporting evidence BIBREF17  We include the special case where the reference is not an authority/expert, although it is referred to as testimonial in the literature BIBREF14 
Propaganda Techniques ::: 11  Black-and-white fallacy, dictatorship 
Presenting two alternative options as the only possibilities, when in fact more possibilities exist BIBREF13  As an extreme case, telling the audience exactly what actions to take, eliminating any other possible choice (dictatorship) 
Propaganda Techniques ::: 12  Thought-terminating cliché 
Words or phrases that discourage critical thought and meaningful discussion about a given topic  They are typically short and generic sentences that offer seemingly simple answers to complex questions or that distract attention away from other lines of thought BIBREF18 
Propaganda Techniques ::: 13  Whataboutism 
Discredit an opponent's position by charging them with hypocrisy without directly disproving their argument BIBREF19 
Propaganda Techniques ::: 14  Reductio ad Hitlerum 
Persuading an audience to disapprove an action or idea by suggesting that the idea is popular with groups hated in contempt by the target audience  It can refer to any person or concept with a negative connotation BIBREF20 
Propaganda Techniques ::: 15  Red herring 
Introducing irrelevant material to the issue being discussed, so that everyone's attention is diverted away from the points made BIBREF11  Those subjected to a red herring argument are led away from the issue that had been the focus of the discussion and urged to follow an observation or claim that may be associated with the original claim, but is not highly relevant to the issue in dispute BIBREF20 
Propaganda Techniques ::: 16  Bandwagon 
Attempting to persuade the target audience to join in and take the course of action because “everyone else is taking the same action” BIBREF15 
Propaganda Techniques ::: 17  Obfuscation, intentional vagueness, confusion 
Using deliberately unclear words, to let the audience have its own interpretation BIBREF21, BIBREF11  For instance, when an unclear phrase with multiple possible meanings is used within the argument and, therefore, it does not really support the conclusion 
Propaganda Techniques ::: 18  Straw man 
When an opponent's proposition is substituted with a similar one which is then refuted in place of the original BIBREF22 
Tasks
The shared task features two subtasks:
Tasks ::: Fragment-Level Classification task (FLC) 
Given a news article, detect all spans of the text in which a propaganda technique is used  In addition, for each span the propaganda technique applied must be identified

ground_truth: Table TABREF6, Table TABREF8

when testing on English, the F1 score of the model training on Chinese (Zh) is 53.8,  F1 score is only 44.1 for the model training on Zh-En

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What traditional linguistics features did they use?
Context 3: Introduction
Sarcasm is an intensive, indirect and complex construct that is often intended to express contempt or ridicule   Sarcasm, in speech, is multi-modal, involving tone, body-language and gestures along with linguistic artifacts used in speech  Sarcasm in text, on the other hand, is more restrictive when it comes to such non-linguistic modalities  This makes recognizing textual sarcasm more challenging for both humans and machines 
Sarcasm detection plays an indispensable role in applications like online review summarizers, dialog systems, recommendation systems and sentiment analyzers  This makes automatic detection of sarcasm an important problem  However, it has been quite difficult to solve such a problem with traditional NLP tools and techniques  This is apparent from the results reported by the survey from DBLP:journals/corr/JoshiBC16  The following discussion brings more insights into this 
Consider a scenario where an online reviewer gives a negative opinion about a movie through sarcasm: “This is the kind of movie you see because the theater has air conditioning”  It is difficult for an automatic sentiment analyzer to assign a rating to the movie and, in the absence of any other information, such a system may not be able to comprehend that prioritizing the air-conditioning facilities of the theater over the movie experience indicates a negative sentiment towards the movie  This gives an intuition to why, for sarcasm detection, it is necessary to go beyond textual analysis 
We aim to address this problem by exploiting the psycholinguistic side of sarcasm detection, using cognitive features extracted with the help of eye-tracking  A motivation to consider cognitive features comes from analyzing human eye-movement trajectories that supports the conjecture: Reading sarcastic texts induces distinctive eye movement patterns, compared to literal texts  The cognitive features, derived from human eye movement patterns observed during reading, include two primary feature types:
The cognitive features, along with textual features used in best available sarcasm detectors, are used to train binary classifiers against given sarcasm labels  Our experiments show significant improvement in classification accuracy over the state of the art, by performing such augmentation 
Related Work
Sarcasm, in general, has been the focus of research for quite some time  In one of the pioneering works jorgensen1984test explained how sarcasm arises when a figurative meaning is used opposite to the literal meaning of the utterance  In the word of clark1984pretense, sarcasm processing involves canceling the indirectly negated message and replacing it with the implicated one  giora1995irony, on the other hand, define sarcasm as a mode of indirect negation that requires processing of both negated and implicated messages  ivanko2003context define sarcasm as a six tuple entity consisting of a speaker, a listener, Context, Utterance, Literal Proposition and Intended Proposition and study the cognitive aspects of sarcasm processing 
Computational linguists have previously addressed this problem using rule based and statistical techniques, that make use of : (a) Unigrams and Pragmatic features BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 (b) Stylistic patterns BIBREF4 and patterns related to situational disparity BIBREF5 and (c) Hastag interpretations BIBREF6 , BIBREF7  
Most of the previously done work on sarcasm detection uses distant supervision based techniques (ex: leveraging hashtags) and stylistic/pragmatic features (emoticons, laughter expressions such as “lol” etc)  But, detecting sarcasm in linguistically well-formed structures, in absence of explicit cues or information (like emoticons), proves to be hard using such linguistic/stylistic features alone 
With the advent of sophisticated eye-trackers and electro/magneto-encephalographic (EEG/MEG) devices, it has been possible to delve deep into the cognitive underpinnings of sarcasm understanding  Filik2014, using a series of eye-tracking and EEG experiments try to show that for unfamiliar ironies, the literal interpretation would be computed first  They also show that a mismatch with context would lead to a re-interpretation of the statement, as being ironic  Camblin2007103 show that in multi-sentence passages, discourse congruence has robust effects on eye movements  This also implies that disrupted processing occurs for discourse incongruent words, even though they are perfectly congruous at the sentence level  In our previous work BIBREF8 , we augment cognitive features, derived from eye-movement patterns of readers, with textual features to detect whether a human reader has realized the presence of sarcasm in text or not 
The recent advancements in the literature discussed above, motivate us to explore gaze-based cognition for sarcasm detection  As far as we know, our work is the first of its kind 
Eye-tracking Database for Sarcasm Analysis
Sarcasm often emanates from incongruity BIBREF9 , which enforces the brain to reanalyze it BIBREF10   This, in turn, affects the way eyes move through the text  Hence, distinctive eye-movement patterns may be observed in the case of successful processing of sarcasm in text in contrast to literal texts  This hypothesis forms the crux of our method for sarcasm detection and we validate this using our previously released freely available sarcasm dataset BIBREF8 enriched with gaze information 
Document Description
The database consists of 1,000 short texts, each having 10-40 words  Out of these, 350 are sarcastic and are collected as follows: (a) 103 sentences are from two popular sarcastic quote websites, (b) 76 sarcastic short movie reviews are manually extracted from the Amazon Movie Corpus BIBREF11 by two linguists  (c) 171 tweets are downloaded using the hashtag #sarcasm from Twitter  The 650 non-sarcastic texts are either downloaded from Twitter or extracted from the Amazon Movie Review corpus  The sentences do not contain words/phrases that are highly topic or culture specific  The tweets were normalized to make them linguistically well formed to avoid difficulty in interpreting social media lingo  Every sentence in our dataset carries positive or negative opinion about specific “aspects”  For example, the sentence “The movie is extremely well cast” has positive sentiment about the aspect “cast” 
The annotators were seven graduate students with science and engineering background, and possess good English proficiency  They were given a set of instructions beforehand and are advised to seek clarifications before they proceed  The instructions mention the nature of the task, annotation input method, and necessity of head movement minimization during the experiment 
Task Description
The task assigned to annotators was to read sentences one at a time and label them with with binary labels indicating the polarity (i e , positive/negative)  Note that, the participants were not instructed to annotate whether a sentence is sarcastic or not , to rule out the Priming Effect (i e , if sarcasm is expected beforehand, processing incongruity becomes relatively easier BIBREF12 )  The setup ensures its “ecological validity” in two ways: (1) Readers are not given any clue that they have to treat sarcasm with special attention  This is done by setting the task to polarity annotation (instead of sarcasm detection)  (2) Sarcastic sentences are mixed with non sarcastic text, which does not give prior knowledge about whether the forthcoming text will be sarcastic or not 
The eye-tracking experiment is conducted by following the standard norms in eye-movement research BIBREF13   At a time, one sentence is displayed to the reader along with the “aspect” with respect to which the annotation has to be provided  While reading, an SR-Research Eyelink-1000 eye-tracker (monocular remote mode, sampling rate 500Hz) records several eye-movement parameters like fixations (a long stay of gaze) and saccade (quick jumping of gaze between two positions of rest) and pupil size 
The accuracy of polarity annotation varies between 72%-91% for sarcastic texts and 75%-91% for non-sarcastic text, showing the inherent difficulty of sentiment annotation, when sarcasm is present in the text under consideration  Annotation errors may be attributed to: (a) lack of patience/attention while reading, (b) issues related to text comprehension, and (c) confusion/indecisiveness caused due to lack of context 
For our analysis, we do not discard the incorrect annotations present in the database  Since our system eventually aims to involve online readers for sarcasm detection, it will be hard to segregate readers who misinterpret the text  We make a rational assumption that, for a particular text, most of the readers, from a fairly large population, will be able to identify sarcasm  Under this assumption, the eye-movement parameters, averaged across all readers in our setting, may not be significantly distorted by a few readers who would have failed to identify sarcasm  This assumption is applicable for both regular and multi-instance based classifiers explained in section SECREF6  
Analysis of Eye-movement Data
We observe distinct behavior during sarcasm reading, by analyzing the “fixation duration on the text” (also referred to as “dwell time” in the literature) and “scanpaths” of the readers 
Variation in the Average Fixation Duration per Word
Since sarcasm in text can be expected to induce cognitive load, it is reasonable to believe that it would require more processing time BIBREF14   Hence, fixation duration normalized over total word count should usually be higher for a sarcastic text than for a non-sarcastic one  We observe this for all participants in our dataset, with the average fixation duration per word for sarcastic texts being at least 1 5 times more than that of non-sarcastic texts  To test the statistical significance, we conduct a two-tailed t-test (assuming unequal variance) to compare the average fixation duration per word for sarcastic and non-sarcastic texts  The hypothesized mean difference is set to 0 and the error tolerance limit ( INLINEFORM0 ) is set to 0 05  The t-test analysis, presented in Table TABREF11 , shows that for all participants, a statistically significant difference exists between the average fixation duration per word for sarcasm (higher average fixation duration) and non-sarcasm (lower average fixation duration)  This affirms that the presence of sarcasm affects the duration of fixation on words 
It is important to note that longer fixations may also be caused by other linguistic subtleties (such as difficult words, ambiguity and syntactically complex structures) causing delay in comprehension, or occulomotor control problems forcing readers to spend time adjusting eye-muscles  So, an elevated average fixation duration per word may not sufficiently indicate the presence of sarcasm  But we would also like to share that, for our dataset, when we considered readability (Flesch readability ease-score BIBREF15 ), number of words in a sentence and average character per word along with the sarcasm label as the predictors of average fixation duration following a linear mixed effect model BIBREF16 , sarcasm label turned out to be the most significant predictor with a maximum slope  This indicates that average fixation duration per word has a strong connection with the text being sarcastic, at least in our dataset 
We now analyze scanpaths to gain more insights into the sarcasm comprehension process 
Analysis of Scanpaths
Scanpaths are line-graphs that contain fixations as nodes and saccades as edges; the radii of the nodes represent the fixation duration  A scanpath corresponds to a participant's eye-movement pattern while reading a particular sentence  Figure FIGREF14 presents scanpaths of three participants for the sarcastic sentence S1 and the non-sarcastic sentence S2  The x-axis of the graph represents the sequence of words a reader reads, and the y-axis represents a temporal sequence in milliseconds 
Consider a sarcastic text containing incongruous phrases A and B  Our qualitative scanpath-analysis reveals that scanpaths with respect to sarcasm processing have two typical characteristics  Often, a long regression - a saccade that goes to a previously visited segment - is observed when a reader starts reading B after skimming through A  In a few cases, the fixation duration on A and B are significantly higher than the average fixation duration per word  In sentence S1, we see long and multiple regressions from the two incongruous phrases “misconception” and “cherish”, and a few instances where phrases “always cherish” and “original misconception” are fixated longer than usual  Such eye-movement behaviors are not seen for S2 
Though sarcasm induces distinctive scanpaths like the ones depicted in Figure FIGREF14 in the observed examples, presence of such patterns is not sufficient to guarantee sarcasm; such patterns may also possibly arise from literal texts  We believe that a combination of linguistic features, readability of text and features derived from scanpaths would help discriminative machine learning models learn sarcasm better 
Features for Sarcasm Detection
We describe the features used for sarcasm detection in Table   The features enlisted under lexical,implicit incongruity and explicit incongruity are borrowed from various literature (predominantly from joshi2015harnessing)  These features are essential to separate sarcasm from other forms semantic incongruity in text (for example ambiguity arising from semantic ambiguity or from metaphors)  Two additional textual features viz  readability and word count of the text are also taken under consideration  These features are used to reduce the effect of text hardness and text length on the eye-movement patterns 
Simple Gaze Based Features
Readers' eye-movement behavior, characterized by fixations, forward saccades, skips and regressions, can be directly quantified by simple statistical aggregation (i e , either computing features for individual participants and then averaging or performing a multi-instance based learning as explained in section SECREF6 )  Since these eye-movement attributes relate to the cognitive process in reading BIBREF17 , we consider these as features in our model  Some of these features have been reported by sarcasmunderstandability for modeling sarcasm understandability of readers  However, as far as we know, these features are being introduced in NLP tasks like textual sarcasm detection for the first time  The values of these features are believed to increase with the increase in the degree of surprisal caused by incongruity in text (except skip count, which will decrease) 
Complex Gaze Based Features
For these features, we rely on a graph structure, namely “saliency graphs", derived from eye-gaze information and word sequences in the text 
For each reader and each sentence, we construct a “saliency graph”, representing the reader's attention characteristics  A saliency graph for a sentence INLINEFORM0 for a reader INLINEFORM1 , represented as INLINEFORM2 , is a graph with vertices ( INLINEFORM3 ) and edges ( INLINEFORM4 ) where each vertex INLINEFORM5 corresponds to a word in INLINEFORM6 (may not be unique) and there exists an edge INLINEFORM7 between vertices INLINEFORM8 and INLINEFORM9 if R performs at least one saccade between the words corresponding to INLINEFORM10 and INLINEFORM11  
Figure FIGREF15 shows an example of a saliency graph A saliency graph may be weighted, but not necessarily connected, for a given text (as there may be words in the given text with no fixation on them)  The “complex” gaze features derived from saliency graphs are also motivated by the theory of incongruity  For instance, Edge Density of a saliency graph increases with the number of distinct saccades, which could arise from the complexity caused by presence of sarcasm  Similarly, the highest weighted degree of a graph is expected to be higher, if the node corresponds to a phrase, incongruous to some other phrase in the text 
The Sarcasm Classifier
We interpret sarcasm detection as a binary classification problem  The training data constitutes 994 examples created using our eye-movement database for sarcasm detection  To check the effectiveness of our feature set, we observe the performance of multiple classification techniques on our dataset through a stratified 10-fold cross validation  We also compare the classification accuracy of our system and the best available systems proposed by riloff2013sarcasm and joshi2015harnessing on our dataset  Using Weka BIBREF18 and LibSVM BIBREF19 APIs, we implement the following classifiers:
Results
Table TABREF17 shows the classification results considering various feature combinations for different classifiers and other systems  These are:
Unigram (with principal components of unigram feature vectors),
Sarcasm (the feature-set reported by joshi2015harnessing subsuming unigram features and features from other reported systems)
Gaze (the simple and complex cognitive features we introduce, along with readability and word count features), and
Gaze+Sarcasm (the complete set of features) 
For all regular classifiers, the gaze features are averaged across participants and augmented with linguistic and sarcasm related features  For the MILR classifier, the gaze features derived from each participant are augmented with linguistic features and thus, a multi instance “bag” of features is formed for each sentence in the training data  This multi-instance dataset is given to an MILR classifier, which follows the standard multi instance assumption to derive class-labels for each bag 
For all the classifiers, our feature combination outperforms the baselines (considering only unigram features) as well as BIBREF3 , with the MILR classifier getting an F-score improvement of 3 7% and Kappa difference of 0 08  We also achieve an improvement of 2% over the baseline, using SVM classifier, when we employ our feature set  We also observe that the gaze features alone, also capture the differences between sarcasm and non-sarcasm classes with a high-precision but a low recall 
To see if the improvement obtained is statistically significant over the state-of-the art system with textual sarcasm features alone, we perform McNemar test  The output of the SVM classifier using only linguistic features used for sarcasm detection by joshi2015harnessing and the output of the MILR classifier with the complete set of features are compared, setting threshold INLINEFORM0   There was a significant difference in the classifier's accuracy with p(two-tailed) = 0 02 with an odds-ratio of 1 43, showing that the classification accuracy improvement is unlikely to be observed by chance in 95% confidence interval 
Considering Reading Time as a Cognitive Feature along with Sarcasm Features
One may argue that, considering simple measures of reading effort like “reading time” as cognitive feature instead of the expensive eye-tracking features for sarcasm detection may be a cost-effective solution  To examine this, we repeated our experiments with “reading time” considered as the only cognitive feature, augmented with the textual features  The F-scores of all the classifiers turn out to be close to that of the classifiers considering sarcasm feature alone and the difference in the improvement is not statistically significant ( INLINEFORM0 )  One the other hand, F-scores with gaze features are superior to the F-scores when reading time is considered as a cognitive feature 
How Effective are the Cognitive Features
We examine the effectiveness of cognitive features on the classification accuracy by varying the input training data size  To examine this, we create a stratified (keeping the class ratio constant) random train-test split of 80%:20%  We train our classifier with 100%, 90%, 80% and 70% of the training data with our whole feature set, and the feature combination from joshi2015harnessing  The goodness of our system is demonstrated by improvements in F-score and Kappa statistics, shown in Figure FIGREF22  
We further analyze the importance of features by ranking the features based on (a) Chi squared test, and (b) Information Gain test, using Weka's attribute selection module  Figure FIGREF23 shows the top 20 ranked features produced by both the tests  For both the cases, we observe 16 out of top 20 features to be gaze features  Further, in each of the cases, Average Fixation Duration per Word and Largest Regression Position are seen to be the two most significant features 
Example Cases
Table TABREF21 shows a few example cases from the experiment with stratified 80%-20% train-test split 
Example sentence 1 is sarcastic, and requires extra-linguistic knowledge (about poor living conditions at Manchester)  Hence, the sarcasm detector relying only on textual features is unable to detect the underlying incongruity  However, our system predicts the label successfully, possibly helped by the gaze features 
Similarly, for sentence 2, the false sense of presence of incongruity (due to phrases like “Helped me” and “Can't stop”) affects the system with only linguistic features  Our system, though, performs well in this case also 
Sentence 3 presents a false-negative case where it was hard for even humans to get the sarcasm  This is why our gaze features (and subsequently the complete set of features) account for erroneous prediction 
In sentence 4, gaze features alone false-indicate presence of incongruity, whereas the system predicts correctly when gaze and linguistic features are taken together 
From these examples, it can be inferred that, only gaze features would not have sufficed to rule out the possibility of detecting other forms of incongruity that do not result in sarcasm 
Error Analysis
Errors committed by our system arise from multiple factors, starting from limitations of the eye-tracker hardware to errors committed by linguistic tools and resources  Also, aggregating various eye-tracking parameters to extract the cognitive features may have caused information loss in the regular classification setting 
Conclusion
In the current work, we created a novel framework to detect sarcasm, that derives insights from human cognition, that manifests over eye movement patterns  We hypothesized that distinctive eye-movement patterns, associated with reading sarcastic text, enables improved detection of sarcasm  We augmented traditional linguistic features with cognitive features obtained from readers' eye-movement data in the form of simple gaze-based features and complex features derived from a graph structure  This extended feature-set improved the success rate of the sarcasm detector by 3 7%, over the best available system  Using cognitive features in an NLP Processing system like ours is the first proposal of its kind 
Our general approach may be useful in other NLP sub-areas like sentiment and emotion analysis, text summarization and question answering, where considering textual clues alone does not prove to be sufficient  We propose to augment this work in future by exploring deeper graph and gaze features  We also propose to develop models for the purpose of learning complex gaze feature representation, that accounts for the power of individual eye movement patterns along with the aggregated patterns of eye movements 
Acknowledgments
We thank the members of CFILT Lab, especially Jaya Jha and Meghna Singh, and the students of IIT Bombay for their help and support 

ground_truth: Unanswerable