prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are strong baselines model is compared to?
Context 1: Introduction
Chinese word segmentation (CWS) is a task for Chinese natural language process to delimit word boundary  CWS is a basic and essential task for Chinese which is written without explicit word delimiters and different from alphabetical languages like English  BIBREF0 treats Chinese word segmentation (CWS) as a sequence labeling task with character position tags, which is followed by BIBREF1, BIBREF2, BIBREF3  Traditional CWS models depend on the design of features heavily which effects the performance of model  To minimize the effort in feature engineering, some CWS models BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11 are developed following neural network architecture for sequence labeling tasks BIBREF12  Neural CWS models perform strong ability of feature representation, employing unigram and bigram character embedding as input and approach good performance 
The CWS task is often modeled as one graph model based on a scoring model that means it is composed of two parts, one part is an encoder which is used to generate the representation of characters from the input sequence, the other part is a decoder which performs segmentation according to the encoder scoring  Table TABREF1 summarizes typical CWS models according to their decoding ways for both traditional and neural models  Markov models such as BIBREF13 and BIBREF4 depend on the maximum entropy model or maximum entropy Markov model both with a Viterbi decoder  Besides, conditional random field (CRF) or Semi-CRF for sequence labeling has been used for both traditional and neural models though with different representations BIBREF2, BIBREF15, BIBREF10, BIBREF17, BIBREF18  Generally speaking, the major difference between traditional and neural network models is about the way to represent input sentences 
Recent works about neural CWS which focus on benchmark dataset, namely SIGHAN Bakeoff BIBREF21, may be put into the following three categories roughly 
Encoder  Practice in various natural language processing tasks has been shown that effective representation is essential to the performance improvement  Thus for better CWS, it is crucial to encode the input character, word or sentence into effective representation  Table TABREF2 summarizes regular feature sets for typical CWS models including ours as well  The building blocks that encoders use include recurrent neural network (RNN) and convolutional neural network (CNN), and long-term memory network (LSTM) 
Graph model  As CWS is a kind of structure learning task, the graph model determines which type of decoder should be adopted for segmentation, also it may limit the capability of defining feature, as shown in Table 2, not all graph models can support the word features  Thus recent work focused on finding more general or flexible graph model to make model learn the representation of segmentation more effective as BIBREF9, BIBREF11 
External data and pre-trained embedding  Whereas both encoder and graph model are about exploring a way to get better performance only by improving the model strength itself  Using external resource such as pre-trained embeddings or language representation is an alternative for the same purpose BIBREF22, BIBREF23  SIGHAN Bakeoff defines two types of evaluation settings, closed test limits all the data for learning should not be beyond the given training set, while open test does not take this limitation BIBREF21  In this work, we will focus on the closed test setting by finding a better model design for further CWS performance improvement 
Shown in Table TABREF1, different decoders have particular decoding algorithms to match the respective CWS models  Markov models and CRF-based models often use Viterbi decoders with polynomial time complexity  In general graph model, search space may be too large for model to search  Thus it forces graph models to use an approximate beam search strategy  Beam search algorithm has a kind low-order polynomial time complexity  Especially, when beam width $b$=1, the beam search algorithm will reduce to greedy algorithm with a better time complexity $O(Mn)$ against the general beam search time complexity $O(Mnb^2)$, where $n$ is the number of units in one sentences, $M$ is a constant representing the model complexity  Greedy decoding algorithm can bring the fastest speed of decoding while it is not easy to guarantee the precision of decoding when the encoder is not strong enough 
In this paper, we focus on more effective encoder design which is capable of offering fast and accurate Chinese word segmentation with only unigram feature and greedy decoding  Our proposed encoder will only consist of attention mechanisms as building blocks but nothing else  Motivated by the Transformer BIBREF24 and its strength of capturing long-range dependencies of input sentences, we use a self-attention network to generate the representation of input which makes the model encode sentences at once without feeding input iteratively  Considering the weakness of the Transformer to model relative and absolute position information directly BIBREF25 and the importance of localness information, position information and directional information for CWS, we further improve the architecture of standard multi-head self-attention of the Transformer with a directional Gaussian mask and get a variant called Gaussian-masked directional multi-head attention  Based on the newly improved attention mechanism, we expand the encoder of the Transformer to capture different directional information  With our powerful encoder, our model uses only simple unigram features to generate representation of sentences 
For decoder which directly performs the segmentation, we use the bi-affinal attention scorer, which has been used in dependency parsing BIBREF26 and semantic role labeling BIBREF27, to implement greedy decoding on finding the boundaries of words  In our proposed model, greedy decoding ensures a fast segmentation while powerful encoder design ensures a good enough segmentation performance even working with greedy decoder together  Our model will be strictly evaluated on benchmark datasets from SIGHAN Bakeoff shared task on CWS in terms of closed test setting, and the experimental results show that our proposed model achieves new state-of-the-art 
The technical contributions of this paper can be summarized as follows 
We propose a CWS model with only attention structure  The encoder and decoder are both based on attention structure 
With a powerful enough encoder, we for the first time show that unigram (character) featues can help yield strong performance instead of diverse $n$-gram (character and word) features in most of previous work 
To capture the representation of localness information and directional information, we propose a variant of directional multi-head self-attention to further enhance the state-of-the-art Transformer encoder 
Models
The CWS task is often modelled as one graph model based on an encoder-based scoring model  The model for CWS task is composed of an encoder to represent the input and a decoder based on the encoder to perform actual segmentation  Figure FIGREF6 is the architecture of our model  The model feeds sentence into encoder  Embedding captures the vector $e=(e_1,   ,e_n)$ of the input character sequences of $c=(c_1,   ,c_n)$  The encoder maps vector sequences of $ {e}=(e_1,  ,e_n)$ to two sequences of vector which are $ {v^b}=(v_1^b,   ,v_n^b)$ and ${v^f}=(v_1^f,   v_n^f)$ as the representation of sentences  With $v^b$ and $v^f$, the bi-affinal scorer calculates the probability of each segmentation gaps and predicts the word boundaries of input  Similar as the Transformer, the encoder is an attention network with stacked self-attention and point-wise, fully connected layers while our encoder includes three independent directional encoders 
Models ::: Encoder Stacks
In the Transformer, the encoder is composed of a stack of N identical layers and each layer has one multi-head self-attention layer and one position-wise fully connected feed-forward layer  One residual connection is around two sub-layers and followed by layer normalization BIBREF24  This architecture provides the Transformer a good ability to generate representation of sentence 
With the variant of multi-head self-attention, we design a Gaussian-masked directional encoder to capture representation of different directions to improve the ability of capturing the localness information and position information for the importance of adjacent characters  One unidirectional encoder can capture information of one particular direction 
For CWS tasks, one gap of characters, which is from a word boundary, can divide one sequence into two parts, one part in front of the gap and one part in the rear of it  The forward encoder and backward encoder are used to capture information of two directions which correspond to two parts divided by the gap 
One central encoder is paralleled with forward and backward encoders to capture the information of entire sentences  The central encoder is a special directional encoder for forward and backward information of sentences  The central encoder can fuse the information and enable the encoder to capture the global information 
The encoder outputs one forward information and one backward information of each positions  The representation of sentence generated by center encoder will be added to these information directly:
where $v^{b}=(v^b_1,   ,v^b_n)$ is the backward information, $v^{f}=(v^f_1,   ,v^f_n)$ is the forward information, $r^{b}=(r^b_1,   ,r^b_n)$ is the output of backward encoder, $r^{c}=(r^c_1,   ,r^c_n)$ is the output of center encoder and $r^{f}=(r^f_1,   ,r^f_n)$ is the output of forward encoder 
Models ::: Gaussian-Masked Directional Multi-Head Attention
Similar as scaled dot-product attention BIBREF24, Gaussian-masked directional attention can be described as a function to map queries and key-value pairs to the representation of input  Here queries, keys and values are all vectors  Standard scaled dot-product attention is calculated by dotting query $Q$ with all keys $K$, dividing each values by $\sqrt{d_k}$, where $\sqrt{d_k}$ is the dimension of keys, and apply a softmax function to generate the weights in the attention:
Different from scaled dot-product attention, Gaussian-masked directional attention expects to pay attention to the adjacent characters of each positions and cast the localness relationship between characters as a fix Gaussian weight for attention  We assume that the Gaussian weight only relys on the distance between characters 
Firstly we introduce the Gaussian weight matrix $G$ which presents the localness relationship between each two characters:
where $g_{ij}$ is the Gaussian weight between character $i$ and $j$, $dis_{ij}$ is the distance between character $i$ and $j$, $\Phi (x)$ is the cumulative distribution function of Gaussian, $\sigma $ is the standard deviation of Gaussian function and it is a hyperparameter in our method  Equation (DISPLAY_FORM13) can ensure the Gaussian weight equals 1 when $dis_{ij}$ is 0  The larger distance between charactersis, the smaller the weight is, which makes one character can affect its adjacent characters more compared with other characters 
To combine the Gaussian weight to the self-attention, we produce the Hadamard product of Gaussian weight matrix $G$ and the score matrix produced by $Q{K^{T}}$
where $AG$ is the Gaussian-masked attention  It ensures that the relationship between two characters with long distances is weaker than adjacent characters 
The scaled dot-product attention models the relationship between two characters without regard to their distances in one sequence  For CWS task, the weight between adjacent characters should be more important while it is hard for self-attention to achieve the effect explicitly because the self-attention cannot get the order of sentences directly  The Gaussian-masked attention adjusts the weight between characters and their adjacent character to a larger value which stands for the effect of adjacent characters 
For forward and backward encoder, the self-attention sublayer needs to use a triangular matrix mask to let the self-attention focus on different weights:
where $pos_i$ is the position of character $c_i$  The triangular matrix for forward and backward encode are:
$\left[ \begin{matrix} 1 & 0 & 0 & \cdots &0\\ 1 & 1 & 0 & \cdots &0\\ 1 & 1 & 1 & \cdots &0\\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 1 & 1 & 1 & \cdots & 1\\ \end{matrix} \right]$ $\left[ \begin{matrix} 1 & 1 & 1 & \cdots &1 \\ 0 & 1 & 1 & \cdots &1 \\ 0 & 0& 1 & \cdots &1 \\ \vdots &\vdots &\vdots &\ddots &\vdots \\ 0 & 0 & 0 & \cdots & 1\\ \end{matrix}\right]$
Similar as BIBREF24, we use multi-head attention to capture information from different dimension positions as Figure FIGREF16 and get Gaussian-masked directional multi-head attention  With multi-head attention architecture, the representation of input can be captured by
where $MH$ is the Gaussian-masked multi-head attention, ${W_i^q, W_i^k,W_i^v} \in \mathbb {R}^{d_k \times d_h}$ is the parameter matrices to generate heads, $d_k$ is the dimension of model and $d_h$ is the dimension of one head 
Models ::: Bi-affinal Attention Scorer
Regarding word boundaries as gaps between any adjacent words converts the character labeling task to the gap labeling task  Different from character labeling task, gap labeling task requires information of two adjacent characters  The relationship between adjacent characters can be represented as the type of gap  The characteristic of word boundaries makes bi-affine attention an appropriate scorer for CWS task 
Bi-affinal attention scorer is the component that we use to label the gap  Bi-affinal attention is developed from bilinear attention which has been used in dependency parsing BIBREF26 and SRL BIBREF27  The distribution of labels in a labeling task is often uneven which makes the output layer often include a fixed bias term for the prior probability of different labels BIBREF27  Bi-affine attention uses bias terms to alleviate the burden of the fixed bias term and get the prior probability which makes it different from bilinear attention  The distribution of the gap is uneven that is similar as other labeling task which fits bi-affine 
Bi-affinal attention scorer labels the target depending on information of independent unit and the joint information of two units  In bi-affinal attention, the score $s_{ij}$ of characters $c_i$ and $c_j$ $(i < j)$ is calculated by:
where $v_i^f$ is the forward information of $c_i$ and $v_i^b$ is the backward information of $c_j$  In Equation (DISPLAY_FORM21), $W$, $U$ and $b$ are all parameters that can be updated in training  $W$ is a matrix with shape $(d_i \times N\times d_j)$ and $U$ is a $(N\times (d_i + d_j))$ matrix where $d_i$ is the dimension of vector $v_i^f$ and $N$ is the number of labels 
In our model, the biaffine scorer uses the forward information of character in front of the gap and the backward information of the character behind the gap to distinguish the position of characters  Figure FIGREF22 is an example of labeling gap  The method of using biaffine scorer ensures that the boundaries of words can be determined by adjacent characters with different directional information  The score vector of the gap is formed by the probability of being a boundary of word  Further, the model generates all boundaries using activation function in a greedy decoding way 
Experiments ::: Experimental Settings ::: Data
We train and evaluate our model on datasets from SIGHAN Bakeoff 2005 BIBREF21 which has four datasets, PKU, MSR, AS and CITYU  Table TABREF23 shows the statistics of train data  We use F-score to evaluate CWS models  To train model with pre-trained embeddings in AS and CITYU, we use OpenCC to transfer data from traditional Chinese to simplified Chinese 
Experiments ::: Experimental Settings ::: Pre-trained Embedding
We only use unigram feature so we only trained character embeddings  Our pre-trained embedding are pre-trained on Chinese Wikipedia corpus by word2vec BIBREF29 toolkit  The corpus used for pre-trained embedding is all transferred to simplified Chinese and not segmented  On closed test, we use embeddings initialized randomly 
Experiments ::: Experimental Settings ::: Hyperparameters
For different datasets, we use two kinds of hyperparameters which are presented in Table TABREF24  We use hyperparameters in Table TABREF24 for small corpora (PKU and CITYU) and normal corpora (MSR and AS)  We set the standard deviation of Gaussian function in Equation (DISPLAY_FORM13) to 2  Each training batch contains sentences with at most 4096 tokens 
Experiments ::: Experimental Settings ::: Optimizer
To train our model, we use the Adam BIBREF30 optimizer with $\beta _1=0 9$, $\beta _2=0 98$ and $\epsilon =10^{-9}$  The learning rate schedule is the same as BIBREF24:
where $d$ is the dimension of embeddings, $step$ is the step number of training and $warmup_step$ is the step number of warmup  When the number of steps is smaller than the step of warmup, the learning rate increases linearly and then decreases 
Experiments ::: Hardware and Implements
We trained our models on a single CPU (Intel i7-5960X) with an nVidia 1080 Ti GPU  We implement our model in Python with Pytorch 1 0 
Experiments ::: Results
Tables TABREF25 and TABREF26 reports the performance of recent models and ours in terms of closed test setting  Without the assistance of unsupervised segmentation features userd in BIBREF20, our model outperforms all the other models in MSR and AS except BIBREF18 and get comparable performance in PKU and CITYU  Note that all the other models for this comparison adopt various $n$-gram features while only our model takes unigram ones 
With unsupervised segmentation features introduced by BIBREF20, our model gets a higher result  Specially, the results in MSR and AS achieve new state-of-the-art and approaching previous state-of-the-art in CITYU and PKU  The unsupervised segmentation features are derived from the given training dataset, thus using them does not violate the rule of closed test of SIGHAN Bakeoff 
Table TABREF36 compares our model and recent neural models in terms of open test setting in which any external resources, especially pre-trained embeddings or language models can be used  In MSR and AS, our model gets a comparable result while our results in CITYU and PKU are not remarkable 
However, it is well known that it is always hard to compare models when using open test setting, especially with pre-trained embedding  Not all models may use the same method and data to pre-train  Though pre-trained embedding or language model can improve the performance, the performance improvement itself may be from multiple sources  It often that there is a success of pre-trained embedding to improve the performance, while it cannot prove that the model is better 
Compared with other LSTM models, our model performs better in AS and MSR than in CITYU and PKU  Considering the scale of different corpora, we believe that the size of corpus affects our model and the larger size is, the better model performs  For small corpus, the model tends to be overfitting 
Tables TABREF25 and TABREF26 also show the decoding time in different datasets  Our model finishes the segmentation with the least decoding time in all four datasets, thanks to the architecture of model which only takes attention mechanism as basic block 
Related Work ::: Chinese Word Segmentation
CWS is a task for Chinese natural language process to delimit word boundary  BIBREF0 for the first time formulize CWS as a sequence labeling task  BIBREF3 show that different character tag sets can make essential impact for CWS  BIBREF2 use CRFs as a model for CWS, achieving new state-of-the-art  Works of statistical CWS has built the basis for neural CWS 
Neural word segmentation has been widely used to minimize the efforts in feature engineering which was important in statistical CWS  BIBREF4 introduce the neural model with sliding-window based sequence labeling  BIBREF6 propose a gated recursive neural network (GRNN) for CWS to incorporate complicated combination of contextual character and n-gram features  BIBREF7 use LSTM to learn long distance information  BIBREF9 propose a neural framework that eliminates context windows and utilize complete segmentation history  BIBREF33 explore a joint model that performs segmentation, POS-Tagging and chunking simultaneously  BIBREF34 propose a feature-enriched neural model for joint CWS and part-of-speech tagging  BIBREF35 present a joint model to enhance the segmentation of Chinese microtext by performing CWS and informal word detection simultaneously  BIBREF17 propose a character-based convolutional neural model to capture $n$-gram features automatically and an effective approach to incorporate word embeddings  BIBREF11 improve the model in BIBREF9 and propose a greedy neural word segmenter with balanced word and character embedding inputs  BIBREF23 propose a novel neural network model to incorporate unlabeled and partially-labeled data  BIBREF36 propose two methods that extend the Bi-LSTM to perform incorporating dictionaries into neural networks for CWS  BIBREF37 propose Switch-LSTMs to segment words and provided a more flexible solution for multi-criteria CWS which is easy to transfer the learned knowledge to new criteria 
Related Work ::: Transformer
Transformer BIBREF24 is an attention-based neural machine translation model  The Transformer is one kind of self-attention networks (SANs) which is proposed in BIBREF38  Encoder of the Transformer consists of one self-attention layer and a position-wise feed-forward layer  Decoder of the Transformer contains one self-attention layer, one encoder-decoder attention layer and one position-wise feed-forward layer  The Transformer uses residual connections around the sublayers and then followed by a layer normalization layer 
Scaled dot-product attention is the key component in the Transformer  The input of attention contains queries, keys, and values of input sequences  The attention is generated using queries and keys like Equation (DISPLAY_FORM11)  Structure of scaled dot-product attention allows the self-attention layer generate the representation of sentences at once and contain the information of the sentence which is different from RNN that process characters of sentences one by one  Standard self-attention is similar as Gaussian-masked direction attention while it does not have directional mask and gaussian mask  BIBREF24 also propose multi-head attention which is better to generate representation of sentence by dividing queries, keys and values to different heads and get information from different subspaces 
Conclusion
In this paper, we propose an attention mechanism only based Chinese word segmentation model  Our model uses self-attention from the Transformer encoder to take sequence input and bi-affine attention scorer to predict the label of gaps  To improve the ability of capturing the localness and directional information of self-attention based encoder, we propose a variant of self-attention called Gaussian-masked directional multi-head attention to replace the standard self-attention  We also extend the Transformer encoder to capture directional features  Our model uses only unigram features instead of multiple $n$-gram features in previous work  Our model is evaluated on standard benchmark dataset, SIGHAN Bakeoff 2005, which shows not only our model performs segmentation faster than any previous models but also gives new higher or comparable segmentation performance against previous state-of-the-art models 
Context 3: Introduction
Word Sense Disambiguation (WSD) is a fundamental task and long-standing challenge in Natural Language Processing (NLP), which aims to find the exact sense of an ambiguous word in a particular context BIBREF0  Previous WSD approaches can be grouped into two main categories: knowledge-based and supervised methods 
Knowledge-based WSD methods rely on lexical resources like WordNet BIBREF1 and usually exploit two kinds of lexical knowledge  The gloss, which defines a word sense meaning, is first utilized in Lesk algorithm BIBREF2 and then widely taken into account in many other approaches BIBREF3, BIBREF4  Besides, structural properties of semantic graphs are mainly used in graph-based algorithms BIBREF5, BIBREF6 
Traditional supervised WSD methods BIBREF7, BIBREF8, BIBREF9 focus on extracting manually designed features and then train a dedicated classifier (word expert) for every target lemma 
Although word expert supervised WSD methods perform better, they are less flexible than knowledge-based methods in the all-words WSD task BIBREF10  Recent neural-based methods are devoted to dealing with this problem  BIBREF11 present a supervised classifier based on Bi-LSTM, which shares parameters among all word types except the last layer  BIBREF10 convert WSD task to a sequence labeling task, thus building a unified model for all polysemous words  However, neither of them can totally beat the best word expert supervised methods 
More recently, BIBREF12 propose to leverage the gloss information from WordNet and model the semantic relationship between the context and gloss in an improved memory network  Similarly, BIBREF13 introduce a (hierarchical) co-attention mechanism to generate co-dependent representations for the context and gloss  Their attempts prove that incorporating gloss knowledge into supervised WSD approach is helpful, but they still have not achieved much improvement, because they may not make full use of gloss knowledge 
In this paper, we focus on how to better leverage gloss information in a supervised neural WSD system  Recently, the pre-trained language models, such as ELMo BIBREF14 and BERT BIBREF15, have shown their effectiveness to alleviate the effort of feature engineering  Especially, BERT has achieved excellent results in question answering (QA) and natural language inference (NLI)  We construct context-gloss pairs from glosses of all possible senses (in WordNet) of the target word, thus treating WSD task as a sentence-pair classification problem  We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task  In particular, our contribution is two-fold:
1  We construct context-gloss pairs and propose three BERT-based models for WSD 
2  We fine-tune the pre-trained BERT model, and the experimental results on several English all-words WSD benchmark datasets show that our approach significantly outperforms the state-of-the-art systems 
Methodology
In this section, we describe our method in detail 
Methodology ::: Task Definition
In WSD, a sentence $s$ usually consists of a series of words: $\lbrace w_1,\cdots ,w_m\rbrace $, and some of the words $\lbrace w_{i_1},\cdots ,w_{i_k}\rbrace $ are targets $\lbrace t_1,\cdots ,t_k\rbrace $ need to be disambiguated  For each target $t$, its candidate senses $\lbrace c_1,\cdots ,c_n\rbrace $ come from entries of its lemma in a pre-defined sense inventory (usually WordNet)  Therefore, WSD task aims to find the most suitable entry (symbolized as unique sense key) for each target in a sentence  See a sentence example in Table TABREF1 
Methodology ::: BERT
BERT BIBREF15 is a new language representation model, and its architecture is a multi-layer bidirectional Transformer encoder  BERT model is pre-trained on a large corpus and two novel unsupervised prediction tasks, i e , masked language model and next sentence prediction tasks are used in pre-training  When incorporating BERT into downstream tasks, the fine-tuning procedure is recommended  We fine-tune the pre-trained BERT model on WSD task 
Methodology ::: BERT ::: BERT(Token-CLS)
Since every target in a sentence needs to be disambiguated to find its exact sense, WSD task can be regarded as a token-level classification task  To incorporate BERT to WSD task, we take the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer for every target lemma, which is the same as the last layer of the Bi-LSTM model BIBREF11 
Methodology ::: GlossBERT
BERT can explicitly model the relationship of a pair of texts, which has shown to be beneficial to many pair-wise natural language understanding tasks  In order to fully leverage gloss information, we propose GlossBERT to construct context-gloss pairs from all possible senses of the target word in WordNet, thus treating WSD task as a sentence-pair classification problem 
We describe our construction method with an example (See Table TABREF1)  There are four targets in this sentence, and here we take target word research as an example:
Methodology ::: GlossBERT ::: Context-Gloss Pairs
The sentence containing target words is denoted as context sentence  For each target word, we extract glosses of all $N$ possible senses (here $N=4$) of the target word (research) in WordNet to obtain the gloss sentence  [CLS] and [SEP] marks are added to the context-gloss pairs to make it suitable for the input of BERT model  A similar idea is also used in aspect-based sentiment analysis BIBREF16 
Methodology ::: GlossBERT ::: Context-Gloss Pairs with Weak Supervision
Based on the previous construction method, we add weak supervised signals to the context-gloss pairs (see the highlighted part in Table TABREF1)  The signal in the gloss sentence aims to point out the target word, and the signal in the context sentence aims to emphasize the target word considering the situation that a target word may occur more than one time in the same sentence 
Therefore, each target word has $N$ context-gloss pair training instances ($label\in \lbrace yes, no\rbrace $)  When testing, we output the probability of $label=yes$ of each context-gloss pair and choose the sense corresponding to the highest probability as the prediction label of the target word  We experiment with three GlossBERT models:
Methodology ::: GlossBERT ::: GlossBERT(Token-CLS)
We use context-gloss pairs as input  We highlight the target word by taking the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer ($label\in \lbrace yes, no\rbrace $) 
Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS)
We use context-gloss pairs as input  We take the final hidden state of the first token [CLS] as the representation of the whole sequence and add a classification layer ($label\in \lbrace yes, no\rbrace $), which does not highlight the target word 
Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS-WS)
We use context-gloss pairs with weak supervision as input  We take the final hidden state of the first token [CLS] and add a classification layer ($label\in \lbrace yes, no\rbrace $), which weekly highlight the target word by the weak supervision 
Experiments ::: Datasets
The statistics of the WSD datasets are shown in Table TABREF12 
Experiments ::: Datasets ::: Training Dataset
Following previous work BIBREF13, BIBREF12, BIBREF10, BIBREF17, BIBREF9, BIBREF7, we choose SemCor3 0 as training corpus, which is the largest corpus manually annotated with WordNet sense for WSD 
Experiments ::: Datasets ::: Evaluation Datasets
We evaluate our method on several English all-words WSD datasets  For a fair comparison, we use the benchmark datasets proposed by BIBREF17 which include five standard all-words fine-grained WSD datasets from the Senseval and SemEval competitions: Senseval-2 (SE2), Senseval-3 (SE3), SemEval-2007 (SE07), SemEval-2013 (SE13) and SemEval-2015 (SE15)  Following BIBREF13, BIBREF12 and BIBREF10, we choose SE07, the smallest among these test sets, as the development set 
Experiments ::: Datasets ::: WordNet
Since BIBREF17 map all the sense annotations in these datasets from their original versions to WordNet 3 0, we extract word sense glosses from WordNet 3 0 
Experiments ::: Settings
We use the pre-trained uncased BERT$_\mathrm {BASE}$ model for fine-tuning, because we find that BERT$_\mathrm {LARGE}$ model performs slightly worse than BERT$_\mathrm {BASE}$ in this task  The number of Transformer blocks is 12, the number of the hidden layer is 768, the number of self-attention heads is 12, and the total number of parameters of the pre-trained model is 110M  When fine-tuning, we use the development set (SE07) to find the optimal settings for our experiments  We keep the dropout probability at 0 1, set the number of epochs to 4  The initial learning rate is 2e-5, and the batch size is 64 
Experiments ::: Results
Table TABREF19 shows the performance of our method on the English all-words WSD benchmark datasets  We compare our approach with previous methods 
The first block shows the MFS baseline, which selects the most frequent sense in the training corpus for each target word 
The second block shows two knowledge-based systems  Lesk$_{ext+emb}$ BIBREF4 is a variant of Lesk algorithm BIBREF2 by calculating the gloss-context overlap of the target word  Babelfy BIBREF6 is a unified graph-based approach which exploits the semantic network structure from BabelNet 
The third block shows two word expert traditional supervised systems  IMS BIBREF7 is a flexible framework which trains SVM classifiers and uses local features  And IMS$_{+emb}$ BIBREF9 is the best configuration of the IMS framework, which also integrates word embeddings as features 
The fourth block shows several recent neural-based methods  Bi-LSTM BIBREF11 is a baseline for neural models  Bi-LSTM$_{+ att  + LEX + POS}$ BIBREF10 is a multi-task learning framework for WSD, POS tagging, and LEX with self-attention mechanism, which converts WSD to a sequence learning task  GAS$_{ext}$ BIBREF12 is a variant of GAS which is a gloss-augmented variant of the memory network by extending gloss knowledge  CAN$^s$ and HCAN BIBREF13 are sentence-level and hierarchical co-attention neural network models which leverage gloss knowledge 
In the last block, we report the performance of our method  BERT(Token-CLS) is our baseline, which does not incorporate gloss information, and it performs slightly worse than previous traditional supervised methods and recent neural-based methods  It proves that directly using BERT cannot obtain performance growth  The other three methods outperform other models by a substantial margin, which proves that the improvements come from leveraging BERT to better exploit gloss information  It is worth noting that our method achieves significant improvements in SE07 and Verb over previous methods, which have the highest ambiguity level among all datasets and all POS tags respectively according to BIBREF17 
Moreover, GlossBERT(Token-CLS) performs better than GlossBERT(Sent-CLS), which proves that highlighting the target word in the sentence is important  However, the weakly highlighting method GlossBERT(Sent-CLS-WS) performs best in most circumstances, which may result from its combination of the advantages of the other two methods 
Experiments ::: Discussion
There are two main reasons for the great improvements of our experimental results  First, we construct context-gloss pairs and convert WSD problem to a sentence-pair classification task which is similar to NLI tasks and train only one classifier, which is equivalent to expanding the corpus  Second, we leverage BERT BIBREF15 to better exploit the gloss information  BERT model shows its advantage in dealing with sentence-pair classification tasks by its amazing improvement on QA and NLI tasks  This advantage comes from both of its two novel unsupervised prediction tasks 
Compared with traditional word expert supervised methods, our GlossBERT shows its effectiveness to alleviate the effort of feature engineering and does not require training a dedicated classifier for every target lemma  Up to now, it can be said that the neural network method can totally beat the traditional word expert method  Compared with recent neural-based methods, our solution is more intuitive and can make better use of gloss knowledge  Besides, our approach demonstrates that when we fine-tune BERT on a downstream task, converting it into a sentence-pair classification task may be a good choice 
Conclusion
In this paper, we seek to better leverage gloss knowledge in a supervised neural WSD system  We propose a new solution to WSD by constructing context-gloss pairs and then converting WSD to a sentence-pair classification task  We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task 
Acknowledgments
We would like to thank the anonymous reviewers for their valuable comments  The research work is supported by National Natural Science Foundation of China (No  61751201 and 61672162), Shanghai Municipal Science and Technology Commission (16JC1420401 and 17JC1404100), Shanghai Municipal Science and Technology Major Project (No 2018SHZDZX01) and ZJLab 

ground_truth: Baseline models are:
- Chen et al., 2015a
- Chen et al., 2015b
- Liu et al., 2016
- Cai and Zhao, 2016
- Cai et al., 2017
- Zhou et al., 2017
- Ma et al., 2018
- Wang et al., 2019

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. By how much did they improve?
Context 2: Introduction
Automatic classification of sentiment has mainly focused on categorizing tweets in either two (binary sentiment analysis) or three (ternary sentiment analysis) categories BIBREF0   In this work we study the problem of fine-grained sentiment classification where tweets are classified according to a five-point scale ranging from VeryNegative to VeryPositive  To illustrate this, Table TABREF3 presents examples of tweets associated with each of these categories  Five-point scales are widely adopted in review sites like Amazon and TripAdvisor, where a user's sentiment is ordered with respect to its intensity  From a sentiment analysis perspective, this defines a classification problem with five categories  In particular, Sebastiani et al  BIBREF1 defined such classification problems whose categories are explicitly ordered to be ordinal classification problems  To account for the ordering of the categories, learners are penalized according to how far from the true class their predictions are 
Although considering different scales, the various settings of sentiment classification are related  First, one may use the same feature extraction and engineering approaches to represent the text spans such as word membership in lexicons, morpho-syntactic statistics like punctuation or elongated word counts BIBREF2 , BIBREF3   Second, one would expect that knowledge from one task can be transfered to the others and this would benefit the performance  Knowing that a tweet is “Positive” in the ternary setting narrows the classification decision between the VeryPositive and Positive categories in the fine-grained setting  From a research perspective this raises the question of whether and how one may benefit when tackling such related tasks and how one can transfer knowledge from one task to another during the training phase 
Our focus in this work is to exploit the relation between the sentiment classification settings and demonstrate the benefits stemming from combining them  To this end, we propose to formulate the different classification problems as a multitask learning problem and jointly learn them  Multitask learning BIBREF4 has shown great potential in various domains and its benefits have been empirically validated BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 using different types of data and learning approaches  An important benefit of multitask learning is that it provides an elegant way to access resources developed for similar tasks  By jointly learning correlated tasks, the amount of usable data increases  For instance, while for ternary classification one can label data using distant supervision with emoticons BIBREF9 , there is no straightforward way to do so for the fine-grained problem  However, the latter can benefit indirectly, if the ternary and fine-grained tasks are learned jointly 
The research question that the paper attempts to answer is the following: Can twitter sentiment classification problems, and fine-grained sentiment classification in particular, benefit from multitask learning? To answer the question, the paper brings the following two main contributions: (i) we show how jointly learning the ternary and fine-grained sentiment classification problems in a multitask setting improves the state-of-the-art performance, and (ii) we demonstrate that recurrent neural networks outperform models previously proposed without access to huge corpora while being flexible to incorporate different sources of data 
Multitask Learning for Twitter Sentiment Classification
In his work, Caruana BIBREF4 proposed a multitask approach in which a learner takes advantage of the multiplicity of interdependent tasks while jointly learning them  The intuition is that if the tasks are correlated, the learner can learn a model jointly for them while taking into account the shared information which is expected to improve its generalization ability  People express their opinions online on various subjects (events, products  ), on several languages and in several styles (tweets, paragraph-sized reviews  ), and it is exactly this variety that motivates the multitask approaches  Specifically for Twitter for instance, the different settings of classification like binary, ternary and fine-grained are correlated since their difference lies in the sentiment granularity of the classes which increases while moving from binary to fine-grained problems 
There are two main decisions to be made in our approach: the learning algorithm, which learns a decision function, and the data representation  With respect to the former, neural networks are particularly suitable as one can design architectures with different properties and arbitrary complexity  Also, as training neural network usually relies on back-propagation of errors, one can have shared parts of the network trained by estimating errors on the joint tasks and others specialized for particular tasks  Concerning the data representation, it strongly depends on the data type available  For the task of sentiment classification of tweets with neural networks, distributed embeddings of words have shown great potential  Embeddings are defined as low-dimensional, dense representations of words that can be obtained in an unsupervised fashion by training on large quantities of text BIBREF10  
Concerning the neural network architecture, we focus on Recurrent Neural Networks (RNNs) that are capable of modeling short-range and long-range dependencies like those exhibited in sequence data of arbitrary length like text  While in the traditional information retrieval paradigm such dependencies are captured using INLINEFORM0 -grams and skip-grams, RNNs learn to capture them automatically BIBREF11   To circumvent the problems with capturing long-range dependencies and preventing gradients from vanishing, the long short-term memory network (LSTM) was proposed BIBREF12   In this work, we use an extended version of LSTM called bidirectional LSTM (biLSTM)  While standard LSTMs access information only from the past (previous words), biLSTMs capture both past and future information effectively BIBREF13 , BIBREF11   They consist of two LSTM networks, for propagating text forward and backwards with the goal being to capture the dependencies better  Indeed, previous work on multitask learning showed the effectiveness of biLSTMs in a variety of problems: BIBREF14 tackled sequence prediction, while BIBREF6 and BIBREF15 used biLSTMs for Named Entity Recognition and dependency parsing respectively 
Figure FIGREF2 presents the architecture we use for multitask learning  In the top-left of the figure a biLSTM network (enclosed by the dashed line) is fed with embeddings INLINEFORM0 that correspond to the INLINEFORM1 words of a tokenized tweet  Notice, as discussed above, the biLSTM consists of two LSTMs that are fed with the word sequence forward and backwards  On top of the biLSTM network one (or more) hidden layers INLINEFORM2 transform its output  The output of INLINEFORM3 is led to the softmax layers for the prediction step  There are INLINEFORM4 softmax layers and each is used for one of the INLINEFORM5 tasks of the multitask setting  In tasks such as sentiment classification, additional features like membership of words in sentiment lexicons or counts of elongated/capitalized words can be used to enrich the representation of tweets before the classification step BIBREF3   The lower part of the network illustrates how such sources of information can be incorporated to the process  A vector “Additional Features” for each tweet is transformed from the hidden layer(s) INLINEFORM6 and then is combined by concatenation with the transformed biLSTM output in the INLINEFORM7 layer 
Experimental setup
Our goal is to demonstrate how multitask learning can be successfully applied on the task of sentiment classification of tweets  The particularities of tweets are to be short and informal text spans  The common use of abbreviations, creative language etc , makes the sentiment classification problem challenging  To validate our hypothesis, that learning the tasks jointly can benefit the performance, we propose an experimental setting where there are data from two different twitter sentiment classification problems: a fine-grained and a ternary  We consider the fine-grained task to be our primary task as it is more challenging and obtaining bigger datasets, e g  by distant supervision, is not straightforward and, hence we report the performance achieved for this task 
Ternary and fine-grained sentiment classification were part of the SemEval-2016 “Sentiment Analysis in Twitter” task BIBREF16   We use the high-quality datasets the challenge organizers released  The dataset for fine-grained classification is split in training, development, development_test and test parts  In the rest, we refer to these splits as train, development and test, where train is composed by the training and the development instances  Table TABREF7 presents an overview of the data  As discussed in BIBREF16 and illustrated in the Table, the fine-grained dataset is highly unbalanced and skewed towards the positive sentiment: only INLINEFORM0 of the training examples are labeled with one of the negative classes 
Feature representation We report results using two different feature sets  The first one, dubbed nbow, is a neural bag-of-words that uses text embeddings to generate low-dimensional, dense representations of the tweets  To construct the nbow representation, given the word embeddings dictionary where each word is associated with a vector, we apply the average compositional function that averages the embeddings of the words that compose a tweet  Simple compositional functions like average were shown to be robust and efficient in previous work BIBREF17   Instead of training embeddings from scratch, we use the pre-trained on tweets GloVe embeddings of BIBREF10   In terms of resources required, using only nbow is efficient as it does not require any domain knowledge  However, previous research on sentiment analysis showed that using extra resources, like sentiment lexicons, can benefit significantly the performance BIBREF3 , BIBREF2   To validate this and examine at which extent neural networks and multitask learning benefit from such features we evaluate the models using an augmented version of nbow, dubbed nbow+  The feature space of the latter, is augmented using 1,368 extra features consisting mostly of counts of punctuation symbols ('!?#@'), emoticons, elongated words and word membership features in several sentiment lexicons  Due to space limitations, for a complete presentation of these features, we refer the interested reader to BIBREF2 , whose open implementation we used to extract them 
Evaluation measure To reproduce the setting of the SemEval challenges BIBREF16 , we optimize our systems using as primary measure the macro-averaged Mean Absolute Error ( INLINEFORM0 ) given by: INLINEFORM1
where INLINEFORM0 is the number of categories, INLINEFORM1 is the set of instances whose true class is INLINEFORM2 , INLINEFORM3 is the true label of the instance INLINEFORM4 and INLINEFORM5 the predicted label  The measure penalizes decisions far from the true ones and is macro-averaged to account for the fact that the data are unbalanced  Complementary to INLINEFORM6 , we report the performance achieved on the micro-averaged INLINEFORM7 measure, which is a commonly used measure for classification 
The models To evaluate the multitask learning approach, we compared it with several other models  Support Vector Machines (SVMs) are maximum margin classification algorithms that have been shown to achieve competitive performance in several text classification problems BIBREF16   SVM INLINEFORM0 stands for an SVM with linear kernel and an one-vs-rest approach for the multi-class problem  Also, SVM INLINEFORM1 is an SVM with linear kernel that employs the crammer-singer strategy BIBREF18 for the multi-class problem  Logistic regression (LR) is another type of linear classification method, with probabilistic motivation  Again, we use two types of Logistic Regression depending on the multi-class strategy: LR INLINEFORM2 that uses an one-vs-rest approach and multinomial Logistic Regression also known as the MaxEnt classifier that uses a multinomial criterion 
Both SVMs and LRs as discussed above treat the problem as a multi-class one, without considering the ordering of the classes  For these four models, we tuned the hyper-parameter INLINEFORM0 that controls the importance of the L INLINEFORM1 regularization part in the optimization problem with grid-search over INLINEFORM2 using 10-fold cross-validation in the union of the training and development data and then retrained the models with the selected values  Also, to account for the unbalanced classification problem we used class weights to penalize more the errors made on the rare classes  These weights were inversely proportional to the frequency of each class  For the four models we used the implementations of Scikit-learn BIBREF19  
For multitask learning we use the architecture shown in Figure FIGREF2 , which we implemented with Keras BIBREF20   The embeddings are initialized with the 50-dimensional GloVe embeddings while the output of the biLSTM network is set to dimension 50  The activation function of the hidden layers is the hyperbolic tangent  The weights of the layers were initialized from a uniform distribution, scaled as described in BIBREF21   We used the Root Mean Square Propagation optimization method  We used dropout for regularizing the network  We trained the network using batches of 128 examples as follows: before selecting the batch, we perform a Bernoulli trial with probability INLINEFORM0 to select the task to train for  With probability INLINEFORM1 we pick a batch for the fine-grained sentiment classification problem, while with probability INLINEFORM2 we pick a batch for the ternary problem  As shown in Figure FIGREF2 , the error is backpropagated until the embeddings, that we fine-tune during the learning process  Notice also that the weights of the network until the layer INLINEFORM3 are shared and therefore affected by both tasks 
To tune the neural network hyper-parameters we used 5-fold cross validation  We tuned the probability INLINEFORM0 of dropout after the hidden layers INLINEFORM1 and for the biLSTM for INLINEFORM2 , the size of the hidden layer INLINEFORM3 and the probability INLINEFORM4 of the Bernoulli trials from INLINEFORM5   During training, we monitor the network's performance on the development set and apply early stopping if the performance on the validation set does not improve for 5 consecutive epochs 
Experimental results Table TABREF9 illustrates the performance of the models for the different data representations  The upper part of the Table summarizes the performance of the baselines  The entry “Balikas et al ” stands for the winning system of the 2016 edition of the challenge BIBREF2 , which to the best of our knowledge holds the state-of-the-art  Due to the stochasticity of training the biLSTM models, we repeat the experiment 10 times and report the average and the standard deviation of the performance achieved 
Several observations can be made from the table  First notice that, overall, the best performance is achieved by the neural network architecture that uses multitask learning  This entails that the system makes use of the available resources efficiently and improves the state-of-the-art performance  In conjunction with the fact that we found the optimal probability INLINEFORM0 , this highlights the benefits of multitask learning over single task learning  Furthermore, as described above, the neural network-based models have only access to the training data as the development are hold for early stopping  On the other hand, the baseline systems were retrained on the union of the train and development sets  Hence, even with fewer resources available for training on the fine-grained problem, the neural networks outperform the baselines  We also highlight the positive effect of the additional features that previous research proposed  Adding the features both in the baselines and in the biLSTM-based architectures improves the INLINEFORM1 scores by several points 
Lastly, we compare the performance of the baseline systems with the performance of the state-of-the-art system of BIBREF2   While BIBREF2 uses n-grams (and character-grams) with INLINEFORM0 , the baseline systems (SVMs, LRs) used in this work use the nbow+ representation, that relies on unigrams  Although they perform on par, the competitive performance of nbow highlights the potential of distributed representations for short-text classification  Further, incorporating structure and distributed representations leads to the gains of the biLSTM network, in the multitask and single task setting 
Similar observations can be drawn from Figure FIGREF10 that presents the INLINEFORM0 scores  Again, the biLSTM network with multitask learning achieves the best performance  It is also to be noted that although the two evaluation measures are correlated in the sense that the ranking of the models is the same, small differences in the INLINEFORM1 have large effect on the scores of the INLINEFORM2 measure 
Conclusion
In this paper, we showed that by jointly learning the tasks of ternary and fine-grained classification with a multitask learning model, one can greatly improve the performance on the second  This opens several avenues for future research  Since sentiment is expressed in different textual types like tweets and paragraph-sized reviews, in different languages (English, German,   ) and in different granularity levels (binary, ternary,  ) one can imagine multitask approaches that could benefit from combining such resources  Also, while we opted for biLSTM networks here, one could use convolutional neural networks or even try to combine different types of networks and tasks to investigate the performance effect of multitask learning  Lastly, while our approach mainly relied on the foundations of BIBREF4 , the internal mechanisms and the theoretical guarantees of multitask learning remain to be better understood 
Acknowledgements
This work is partially supported by the CIFRE N 28/2015 

ground_truth: They decrease MAE in 0.34

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do they use large or small BERT?
Context 4:  Figure FIGREF14 presents scanpaths of three participants for the sarcastic sentence S1 and the non-sarcastic sentence S2  The x-axis of the graph represents the sequence of words a reader reads, and the y-axis represents a temporal sequence in milliseconds 
Consider a sarcastic text containing incongruous phrases A and B  Our qualitative scanpath-analysis reveals that scanpaths with respect to sarcasm processing have two typical characteristics  Often, a long regression - a saccade that goes to a previously visited segment - is observed when a reader starts reading B after skimming through A  In a few cases, the fixation duration on A and B are significantly higher than the average fixation duration per word  In sentence S1, we see long and multiple regressions from the two incongruous phrases “misconception” and “cherish”, and a few instances where phrases “always cherish” and “original misconception” are fixated longer than usual  Such eye-movement behaviors are not seen for S2 
Though sarcasm induces distinctive scanpaths like the ones depicted in Figure FIGREF14 in the observed examples, presence of such patterns is not sufficient to guarantee sarcasm; such patterns may also possibly arise from literal texts  We believe that a combination of linguistic features, readability of text and features derived from scanpaths would help discriminative machine learning models learn sarcasm better 
Features for Sarcasm Detection
We describe the features used for sarcasm detection in Table   The features enlisted under lexical,implicit incongruity and explicit incongruity are borrowed from various literature (predominantly from joshi2015harnessing)  These features are essential to separate sarcasm from other forms semantic incongruity in text (for example ambiguity arising from semantic ambiguity or from metaphors)  Two additional textual features viz  readability and word count of the text are also taken under consideration  These features are used to reduce the effect of text hardness and text length on the eye-movement patterns 
Simple Gaze Based Features
Readers' eye-movement behavior, characterized by fixations, forward saccades, skips and regressions, can be directly quantified by simple statistical aggregation (i e , either computing features for individual participants and then averaging or performing a multi-instance based learning as explained in section SECREF6 )  Since these eye-movement attributes relate to the cognitive process in reading BIBREF17 , we consider these as features in our model  Some of these features have been reported by sarcasmunderstandability for modeling sarcasm understandability of readers  However, as far as we know, these features are being introduced in NLP tasks like textual sarcasm detection for the first time  The values of these features are believed to increase with the increase in the degree of surprisal caused by incongruity in text (except skip count, which will decrease) 
Complex Gaze Based Features
For these features, we rely on a graph structure, namely “saliency graphs", derived from eye-gaze information and word sequences in the text 
For each reader and each sentence, we construct a “saliency graph”, representing the reader's attention characteristics  A saliency graph for a sentence INLINEFORM0 for a reader INLINEFORM1 , represented as INLINEFORM2 , is a graph with vertices ( INLINEFORM3 ) and edges ( INLINEFORM4 ) where each vertex INLINEFORM5 corresponds to a word in INLINEFORM6 (may not be unique) and there exists an edge INLINEFORM7 between vertices INLINEFORM8 and INLINEFORM9 if R performs at least one saccade between the words corresponding to INLINEFORM10 and INLINEFORM11  
Figure FIGREF15 shows an example of a saliency graph A saliency graph may be weighted, but not necessarily connected, for a given text (as there may be words in the given text with no fixation on them)  The “complex” gaze features derived from saliency graphs are also motivated by the theory of incongruity  For instance, Edge Density of a saliency graph increases with the number of distinct saccades, which could arise from the complexity caused by presence of sarcasm  Similarly, the highest weighted degree of a graph is expected to be higher, if the node corresponds to a phrase, incongruous to some other phrase in the text 
The Sarcasm Classifier
We interpret sarcasm detection as a binary classification problem  The training data constitutes 994 examples created using our eye-movement database for sarcasm detection  To check the effectiveness of our feature set, we observe the performance of multiple classification techniques on our dataset through a stratified 10-fold cross validation  We also compare the classification accuracy of our system and the best available systems proposed by riloff2013sarcasm and joshi2015harnessing on our dataset  Using Weka BIBREF18 and LibSVM BIBREF19 APIs, we implement the following classifiers:
Results
Table TABREF17 shows the classification results considering various feature combinations for different classifiers and other systems  These are:
Unigram (with principal components of unigram feature vectors),
Sarcasm (the feature-set reported by joshi2015harnessing subsuming unigram features and features from other reported systems)
Gaze (the simple and complex cognitive features we introduce, along with readability and word count features), and
Gaze+Sarcasm (the complete set of features) 
For all regular classifiers, the gaze features are averaged across participants and augmented with linguistic and sarcasm related features  For the MILR classifier, the gaze features derived from each participant are augmented with linguistic features and thus, a multi instance “bag” of features is formed for each sentence in the training data  This multi-instance dataset is given to an MILR classifier, which follows the standard multi instance assumption to derive class-labels for each bag 
For all the classifiers, our feature combination outperforms the baselines (considering only unigram features) as well as BIBREF3 , with the MILR classifier getting an F-score improvement of 3 7% and Kappa difference of 0 08  We also achieve an improvement of 2% over the baseline, using SVM classifier, when we employ our feature set  We also observe that the gaze features alone, also capture the differences between sarcasm and non-sarcasm classes with a high-precision but a low recall 
To see if the improvement obtained is statistically significant over the state-of-the art system with textual sarcasm features alone, we perform McNemar test  The output of the SVM classifier using only linguistic features used for sarcasm detection by joshi2015harnessing and the output of the MILR classifier with the complete set of features are compared, setting threshold INLINEFORM0   There was a significant difference in the classifier's accuracy with p(two-tailed) = 0 02 with an odds-ratio of 1 43, showing that the classification accuracy improvement is unlikely to be observed by chance in 95% confidence interval 
Considering Reading Time as a Cognitive Feature along with Sarcasm Features
One may argue that, considering simple measures of reading effort like “reading time” as cognitive feature instead of the expensive eye-tracking features for sarcasm detection may be a cost-effective solution  To examine this, we repeated our experiments with “reading time” considered as the only cognitive feature, augmented with the textual features  The F-scores of all the classifiers turn out to be close to that of the classifiers considering sarcasm feature alone and the difference in the improvement is not statistically significant ( INLINEFORM0 )  One the other hand, F-scores with gaze features are superior to the F-scores when reading time is considered as a cognitive feature 
How Effective are the Cognitive Features
We examine the effectiveness of cognitive features on the classification accuracy by varying the input training data size  To examine this, we create a stratified (keeping the class ratio constant) random train-test split of 80%:20%  We train our classifier with 100%, 90%, 80% and 70% of the training data with our whole feature set, and the feature combination from joshi2015harnessing  The goodness of our system is demonstrated by improvements in F-score and Kappa statistics, shown in Figure FIGREF22  
We further analyze the importance of features by ranking the features based on (a) Chi squared test, and (b) Information Gain test, using Weka's attribute selection module  Figure FIGREF23 shows the top 20 ranked features produced by both the tests  For both the cases, we observe 16 out of top 20 features to be gaze features  Further, in each of the cases, Average Fixation Duration per Word and Largest Regression Position are seen to be the two most significant features 
Example Cases
Table TABREF21 shows a few example cases from the experiment with stratified 80%-20% train-test split 
Example sentence 1 is sarcastic, and requires extra-linguistic knowledge (about poor living conditions at Manchester)  Hence, the sarcasm detector relying only on textual features is unable to detect the underlying incongruity  However, our system predicts the label successfully, possibly helped by the gaze features 
Similarly, for sentence 2, the false sense of presence of incongruity (due to phrases like “Helped me” and “Can't stop”) affects the system with only linguistic features  Our system, though, performs well in this case also 
Sentence 3 presents a false-negative case where it was hard for even humans to get the sarcasm  This is why our gaze features (and subsequently the complete set of features) account for erroneous prediction 
In sentence 4, gaze features alone false-indicate presence of incongruity, whereas the system predicts correctly when gaze and linguistic features are taken together 
From these examples, it can be inferred that, only gaze features would not have sufficed to rule out the possibility of detecting other forms of incongruity that do not result in sarcasm 
Error Analysis
Errors committed by our system arise from multiple factors, starting from limitations of the eye-tracker hardware to errors committed by linguistic tools and resources  Also, aggregating various eye-tracking parameters to extract the cognitive features may have caused information loss in the regular classification setting 
Conclusion
In the current work, we created a novel framework to detect sarcasm, that derives insights from human cognition, that manifests over eye movement patterns  We hypothesized that distinctive eye-movement patterns, associated with reading sarcastic text, enables improved detection of sarcasm  We augmented traditional linguistic features with cognitive features obtained from readers' eye-movement data in the form of simple gaze-based features and complex features derived from a graph structure  This extended feature-set improved the success rate of the sarcasm detector by 3 7%, over the best available system  Using cognitive features in an NLP Processing system like ours is the first proposal of its kind 
Our general approach may be useful in other NLP sub-areas like sentiment and emotion analysis, text summarization and question answering, where considering textual clues alone does not prove to be sufficient  We propose to augment this work in future by exploring deeper graph and gaze features  We also propose to develop models for the purpose of learning complex gaze feature representation, that accounts for the power of individual eye movement patterns along with the aggregated patterns of eye movements 
Acknowledgments
We thank the members of CFILT Lab, especially Jaya Jha and Meghna Singh, and the students of IIT Bombay for their help and support 

ground_truth: small BERT

small BERT