prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What data was presented to the subjects to elicit event-related responses?
Context 1: Introduction
Decoding intended speech or motor activity from brain signals is one of the major research areas in Brain Computer Interface (BCI) systems BIBREF0 , BIBREF1   In particular, speech-related BCI technologies attempt to provide effective vocal communication strategies for controlling external devices through speech commands interpreted from brain signals BIBREF2   Not only do they provide neuro-prosthetic help for people with speaking disabilities and neuro-muscular disorders like locked-in-syndrome, nasopharyngeal cancer, and amytotropic lateral sclerosis (ALS), but also equip people with a better medium to communicate and express thoughts, thereby improving the quality of rehabilitation and clinical neurology BIBREF3 , BIBREF4   Such devices also have applications in entertainment, preventive treatments, personal communication, games, etc  Furthermore, BCI technologies can be utilized in silent communication, as in noisy environments, or situations where any sort of audio-visual communication is infeasible 
Among the various brain activity-monitoring modalities in BCI, electroencephalography (EEG) BIBREF5 , BIBREF6 has demonstrated promising potential to differentiate between various brain activities through measurement of related electric fields  EEG is non-invasive, portable, low cost, and provides satisfactory temporal resolution  This makes EEG suitable to realize BCI systems  EEG data, however, is challenging: these data are high dimensional, have poor SNR, and suffer from low spatial resolution and a multitude of artifacts  For these reasons, it is not particularly obvious how to decode the desired information from raw EEG signals  Although the area of BCI based speech intent recognition has received increasing attention among the research community in the past few years, most research has focused on classification of individual speech categories in terms of discrete vowels, phonemes and words BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15   This includes categorization of imagined EEG signal into binary vowel categories like /a/, /u/ and rest BIBREF7 , BIBREF8 , BIBREF9 ; binary syllable classes like /ba/ and /ku/ BIBREF1 , BIBREF10 , BIBREF11 , BIBREF12 ; a handful of control words like 'up', 'down', 'left', 'right' and 'select' BIBREF15 or others like 'water', 'help', 'thanks', 'food', 'stop' BIBREF13 , Chinese characters BIBREF14 , etc  Such works mostly involve traditional signal processing or manual feature handcrafting along with linear classifiers (e g , SVMs)  In our recent work BIBREF16 , we introduced deep learning models for classification of vowels and words that achieved 23 45% improvement of accuracy over the baseline 
Production of articulatory speech is an extremely complicated process, thereby rendering understanding of the discriminative EEG manifold corresponding to imagined speech highly challenging  As a result, most of the existing approaches failed to achieve satisfactory accuracy on decoding speech tokens from the speech imagery EEG data  Perhaps, for these reasons, very little work has been devoted to relating the brain signals to the underlying articulation  The few exceptions include BIBREF17 , BIBREF18   In BIBREF17 , Zhao et al  used manually handcrafted features from EEG data, combined with speech audio and facial features to achieve classification of the phonological categories varying based on the articulatory steps  However, the imagined speech classification accuracy based on EEG data alone, as reported in BIBREF17 , BIBREF18 , are not satisfactory in terms of accuracy and reliability  We now turn to describing our proposed models 
Proposed Framework
Cognitive learning process underlying articulatory speech production involves incorporation of intermediate feedback loops and utilization of past information stored in the form of memory as well as hierarchical combination of several feature extractors  To this end, we develop our mixed neural network architecture composed of three supervised and a single unsupervised learning step, discussed in the next subsections and shown in Fig  FIGREF1   We formulate the problem of categorizing EEG data based on speech imagery as a non-linear mapping INLINEFORM0 of a multivariate time-series input sequence INLINEFORM1 to fixed output INLINEFORM2 , i e, mathematically INLINEFORM3 : INLINEFORM4 , where c and t denote the EEG channels and time instants respectively 
Preprocessing step
We follow similar pre-processing steps on raw EEG data as reported in BIBREF17 (ocular artifact removal using blind source separation, bandpass filtering and subtracting mean value from each channel) except that we do not perform Laplacian filtering step since such high-pass filtering may decrease information content from the signals in the selected bandwidth 
Joint variability of electrodes
Multichannel EEG data is high dimensional multivariate time series data whose dimensionality depends on the number of electrodes  It is a major hurdle to optimally encode information from these EEG data into lower dimensional space  In fact, our investigation based on a development set (as we explain later) showed that well-known deep neural networks (e g , fully connected networks such as convolutional neural networks, recurrent neural networks and autoencoders) fail to individually learn such complex feature representations from single-trial EEG data  Besides, we found that instead of using the raw multi-channel high-dimensional EEG requiring large training times and resource requirements, it is advantageous to first reduce its dimensionality by capturing the information transfer among the electrodes  Instead of the conventional approach of selecting a handful of channels as BIBREF17 , BIBREF18 , we address this by computing the channel cross-covariance, resulting in positive, semi-definite matrices encoding the connectivity of the electrodes  We define channel cross-covariance (CCV) between any two electrodes INLINEFORM0 and INLINEFORM1 as: INLINEFORM2   Next, we reject the channels which have significantly lower cross-covariance than auto-covariance values (where auto-covariance implies CCV on same electrode)  We found this measure to be essential as the higher cognitive processes underlying speech planning and synthesis involve frequent information exchange between different parts of the brain  Hence, such matrices often contain more discriminative features and hidden information than mere raw signals  This is essentially different than our previous work BIBREF16 where we extract per-channel 1-D covariance information and feed it to the networks  We present our sample 2-D EEG cross-covariance matrices (of two individuals) in Fig  FIGREF2  
CNN & LSTM
In order to decode spatial connections between the electrodes from the channel covariance matrix, we use a CNN BIBREF19 , in particular a four-layered 2D CNN stacking two convolutional and two fully connected hidden layers  The INLINEFORM0 feature map at a given CNN layer with input INLINEFORM1 , weight matrix INLINEFORM2 and bias INLINEFORM3 is obtained as: INLINEFORM4   At this first level of hierarchy, the network is trained with the corresponding labels as target outputs, optimizing a cross-entropy cost function  In parallel, we apply a four-layered recurrent neural network on the channel covariance matrices to explore the hidden temporal features of the electrodes  Namely, we exploit an LSTM BIBREF20 consisting of two fully connected hidden layers, stacked with two LSTM layers and trained in a similar manner as CNN 
Deep autoencoder for spatio-temporal information
As we found the individually-trained parallel networks (CNN and LSTM) to be useful (see Table TABREF12 ), we suspected the combination of these two networks could provide a more powerful discriminative spatial and temporal representation of the data than each independent network  As such, we concatenate the last fully-connected layer from the CNN with its counterpart in the LSTM to compose a single feature vector based on these two penultimate layers  Ultimately, this forms a joint spatio-temporal encoding of the cross-covariance matrix 
In order to further reduce the dimensionality of the spatio-temporal encodings and cancel background noise effects BIBREF21 , we train an unsupervised deep autoenoder (DAE) on the fused heterogeneous features produced by the combined CNN and LSTM information  The DAE forms our second level of hierarchy, with 3 encoding and 3 decoding layers, and mean squared error (MSE) as the cost function 
Classification with Extreme Gradient Boost
At the third level of hierarchy, the discrete latent vector representation of the deep autoencoder is fed into an Extreme Gradient Boost based classification layer BIBREF22 , BIBREF23 motivated by BIBREF21   It is a regularized gradient boosted decision tree that performs well on structured problems  Since our EEG-phonological pairwise classification has an internal structure involving individual phonemes and words, it seems to be a reasonable choice of classifier  The classifier receives its input from the latent vectors of the deep autoencoder and is trained in a supervised manner to output the final predicted classes corresponding to the speech imagery 
Dataset
We evaluate our model on a publicly available dataset, KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)  The dataset consists of 14 participants, with each prompt presented 11 times to each individual  Since our intention is to classify the phonological categories from human thoughts, we discard the facial and audio information and only consider the EEG data corresponding to imagined speech  It is noteworthy that given the mixed nature of EEG signals, it is reportedly challenging to attain a pairwise EEG-phoneme mapping BIBREF18   In order to explore the problem space, we thus specifically target five binary classification problems addressed in BIBREF17 , BIBREF18 , i e presence/absence of consonants, phonemic nasal, bilabial, high-front vowels and high-back vowels 
Training and hyperparameter selection
We performed two sets of experiments with the single-trial EEG data  In PHASE-ONE, our goals was to identify the best architectures and hyperparameters for our networks with a reasonable number of runs  For PHASE-ONE, we randomly shuffled and divided the data (1913 signals from 14 individuals) into train (80%), development (10%) and test sets (10%)  In PHASE-TWO, in order to perform a fair comparison with the previous methods reported on the same dataset, we perform a leave-one-subject out cross-validation experiment using the best settings we learn from PHASE-ONE 
The architectural parameters and hyperparameters listed in Table TABREF6 were selected through an exhaustive grid-search based on the validation set of PHASE-ONE  We conducted a series of empirical studies starting from single hidden-layered networks for each of the blocks and, based on the validation accuracy, we increased the depth of each given network and selected the optimal parametric set from all possible combinations of parameters  For the gradient boosting classification, we fixed the maximum depth at 10, number of estimators at 5000, learning rate at 0 1, regularization coefficient at 0 3, subsample ratio at 0 8, and column-sample/iteration at 0 4  We did not find any notable change of accuracy while varying other hyperparameters while training gradient boost classifier 
Performance analysis and discussion
To demonstrate the significance of the hierarchical CNN-LSTM-DAE method, we conducted separate experiments with the individual networks in PHASE-ONE of experiments and summarized the results in Table TABREF12 From the average accuracy scores, we observe that the mixed network performs much better than individual blocks which is in agreement with the findings in BIBREF21   A detailed analysis on repeated runs further shows that in most of the cases, LSTM alone does not perform better than chance  CNN, on the other hand, is heavily biased towards the class label which sees more training data corresponding to it  Though the situation improves with combined CNN-LSTM, our analysis clearly shows the necessity of a better encoding scheme to utilize the combined features rather than mere concatenation of the penultimate features of both networks 
The very fact that our combined network improves the classification accuracy by a mean margin of 14 45% than the CNN-LSTM network indeed reveals that the autoencoder contributes towards filtering out the unrelated and noisy features from the concatenated penultimate feature set  It also proves that the combined supervised and unsupervised neural networks, trained hierarchically, can learn the discriminative manifold better than the individual networks and it is crucial for improving the classification accuracy  In addition to accuracy, we also provide the kappa coefficients BIBREF24 of our method in Fig  FIGREF14   Here, a higher mean kappa value corresponding to a task implies that the network is able to find better discriminative information from the EEG data beyond random decisions  The maximum above-chance accuracy (75 92%) is recorded for presence/absence of the vowel task and the minimum (49 14%) is recorded for the INLINEFORM0  
To further investigate the feature representation achieved by our model, we plot T-distributed Stochastic Neighbor Embedding (tSNE) corresponding to INLINEFORM0 and V/C classification tasks in Fig  FIGREF8   We particularly select these two tasks as our model exhibits respectively minimum and maximum performance for these two  The tSNE visualization reveals that the second set of features are more easily separable than the first one, thereby giving a rationale for our performance 
Next, we provide performance comparison of the proposed approach with the baseline methods for PHASE-TWO of our study (cross-validation experiment) in Table TABREF15   Since the model encounters the unseen data of a new subject for testing, and given the high inter-subject variability of the EEG data, a reduction in the accuracy was expected  However, our network still managed to achieve an improvement of 18 91, 9 95, 67 15, 2 83 and 13 70 % over BIBREF17   Besides, our best model shows more reliability compared to previous works: The standard deviation of our model's classification accuracy across all the tasks is reduced from 22 59% BIBREF17 and 17 52% BIBREF18 to a mere 5 41% 
Conclusion and future direction
In an attempt to move a step towards understanding the speech information encoded in brain signals, we developed a novel mixed deep neural network scheme for a number of binary classification tasks from speech imagery EEG data  Unlike previous approaches which mostly deal with subject-dependent classification of EEG into discrete vowel or word labels, this work investigates a subject-invariant mapping of EEG data with different phonological categories, varying widely in terms of underlying articulator motions (eg: involvement or non-involvement of lips and velum, variation of tongue movements etc)  Our model takes an advantage of feature extraction capability of CNN, LSTM as well as the deep learning benefit of deep autoencoders  We took BIBREF17 , BIBREF18 as the baseline works investigating the same problem and compared our performance with theirs  Our proposed method highly outperforms the existing methods across all the five binary classification tasks by a large average margin of 22 51% 
Acknowledgments
This work was funded by the Natural Sciences and Engineering Research Council (NSERC) of Canada and Canadian Institutes for Health Research (CIHR) 
Context 2: Introduction
Word Sense Disambiguation (WSD) is a fundamental task and long-standing challenge in Natural Language Processing (NLP), which aims to find the exact sense of an ambiguous word in a particular context BIBREF0  Previous WSD approaches can be grouped into two main categories: knowledge-based and supervised methods 
Knowledge-based WSD methods rely on lexical resources like WordNet BIBREF1 and usually exploit two kinds of lexical knowledge  The gloss, which defines a word sense meaning, is first utilized in Lesk algorithm BIBREF2 and then widely taken into account in many other approaches BIBREF3, BIBREF4  Besides, structural properties of semantic graphs are mainly used in graph-based algorithms BIBREF5, BIBREF6 
Traditional supervised WSD methods BIBREF7, BIBREF8, BIBREF9 focus on extracting manually designed features and then train a dedicated classifier (word expert) for every target lemma 
Although word expert supervised WSD methods perform better, they are less flexible than knowledge-based methods in the all-words WSD task BIBREF10  Recent neural-based methods are devoted to dealing with this problem  BIBREF11 present a supervised classifier based on Bi-LSTM, which shares parameters among all word types except the last layer  BIBREF10 convert WSD task to a sequence labeling task, thus building a unified model for all polysemous words  However, neither of them can totally beat the best word expert supervised methods 
More recently, BIBREF12 propose to leverage the gloss information from WordNet and model the semantic relationship between the context and gloss in an improved memory network  Similarly, BIBREF13 introduce a (hierarchical) co-attention mechanism to generate co-dependent representations for the context and gloss  Their attempts prove that incorporating gloss knowledge into supervised WSD approach is helpful, but they still have not achieved much improvement, because they may not make full use of gloss knowledge 
In this paper, we focus on how to better leverage gloss information in a supervised neural WSD system  Recently, the pre-trained language models, such as ELMo BIBREF14 and BERT BIBREF15, have shown their effectiveness to alleviate the effort of feature engineering  Especially, BERT has achieved excellent results in question answering (QA) and natural language inference (NLI)  We construct context-gloss pairs from glosses of all possible senses (in WordNet) of the target word, thus treating WSD task as a sentence-pair classification problem  We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task  In particular, our contribution is two-fold:
1  We construct context-gloss pairs and propose three BERT-based models for WSD 
2  We fine-tune the pre-trained BERT model, and the experimental results on several English all-words WSD benchmark datasets show that our approach significantly outperforms the state-of-the-art systems 
Methodology
In this section, we describe our method in detail 
Methodology ::: Task Definition
In WSD, a sentence $s$ usually consists of a series of words: $\lbrace w_1,\cdots ,w_m\rbrace $, and some of the words $\lbrace w_{i_1},\cdots ,w_{i_k}\rbrace $ are targets $\lbrace t_1,\cdots ,t_k\rbrace $ need to be disambiguated  For each target $t$, its candidate senses $\lbrace c_1,\cdots ,c_n\rbrace $ come from entries of its lemma in a pre-defined sense inventory (usually WordNet)  Therefore, WSD task aims to find the most suitable entry (symbolized as unique sense key) for each target in a sentence  See a sentence example in Table TABREF1 
Methodology ::: BERT
BERT BIBREF15 is a new language representation model, and its architecture is a multi-layer bidirectional Transformer encoder  BERT model is pre-trained on a large corpus and two novel unsupervised prediction tasks, i e , masked language model and next sentence prediction tasks are used in pre-training  When incorporating BERT into downstream tasks, the fine-tuning procedure is recommended  We fine-tune the pre-trained BERT model on WSD task 
Methodology ::: BERT ::: BERT(Token-CLS)
Since every target in a sentence needs to be disambiguated to find its exact sense, WSD task can be regarded as a token-level classification task  To incorporate BERT to WSD task, we take the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer for every target lemma, which is the same as the last layer of the Bi-LSTM model BIBREF11 
Methodology ::: GlossBERT
BERT can explicitly model the relationship of a pair of texts, which has shown to be beneficial to many pair-wise natural language understanding tasks  In order to fully leverage gloss information, we propose GlossBERT to construct context-gloss pairs from all possible senses of the target word in WordNet, thus treating WSD task as a sentence-pair classification problem 
We describe our construction method with an example (See Table TABREF1)  There are four targets in this sentence, and here we take target word research as an example:
Methodology ::: GlossBERT ::: Context-Gloss Pairs
The sentence containing target words is denoted as context sentence  For each target word, we extract glosses of all $N$ possible senses (here $N=4$) of the target word (research) in WordNet to obtain the gloss sentence  [CLS] and [SEP] marks are added to the context-gloss pairs to make it suitable for the input of BERT model  A similar idea is also used in aspect-based sentiment analysis BIBREF16 
Methodology ::: GlossBERT ::: Context-Gloss Pairs with Weak Supervision
Based on the previous construction method, we add weak supervised signals to the context-gloss pairs (see the highlighted part in Table TABREF1)  The signal in the gloss sentence aims to point out the target word, and the signal in the context sentence aims to emphasize the target word considering the situation that a target word may occur more than one time in the same sentence 
Therefore, each target word has $N$ context-gloss pair training instances ($label\in \lbrace yes, no\rbrace $)  When testing, we output the probability of $label=yes$ of each context-gloss pair and choose the sense corresponding to the highest probability as the prediction label of the target word  We experiment with three GlossBERT models:
Methodology ::: GlossBERT ::: GlossBERT(Token-CLS)
We use context-gloss pairs as input  We highlight the target word by taking the final hidden state of the token corresponding to the target word (if more than one token, we average them) and add a classification layer ($label\in \lbrace yes, no\rbrace $) 
Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS)
We use context-gloss pairs as input  We take the final hidden state of the first token [CLS] as the representation of the whole sequence and add a classification layer ($label\in \lbrace yes, no\rbrace $), which does not highlight the target word 
Methodology ::: GlossBERT ::: GlossBERT(Sent-CLS-WS)
We use context-gloss pairs with weak supervision as input  We take the final hidden state of the first token [CLS] and add a classification layer ($label\in \lbrace yes, no\rbrace $), which weekly highlight the target word by the weak supervision
Context 3: Introduction
We understand from Zipf's Law that in any natural language corpus a majority of the vocabulary word types will either be absent or occur in low frequency  Estimating the statistical properties of these rare word types is naturally a difficult task  This is analogous to the curse of dimensionality when we deal with sequences of tokens - most sequences will occur only once in the training data  Neural network architectures overcome this problem by defining non-linear compositional models over vector space representations of tokens and hence assign non-zero probability even to sequences not seen during training BIBREF0 , BIBREF1   In this work, we explore a similar approach to learning distributed representations of social media posts by composing them from their constituent characters, with the goal of generalizing to out-of-vocabulary words as well as sequences at test time 
Traditional Neural Network Language Models (NNLMs) treat words as the basic units of language and assign independent vectors to each word type  To constrain memory requirements, the vocabulary size is fixed before-hand; therefore, rare and out-of-vocabulary words are all grouped together under a common type `UNKNOWN'  This choice is motivated by the assumption of arbitrariness in language, which means that surface forms of words have little to do with their semantic roles  Recently, BIBREF2 challenge this assumption and present a bidirectional Long Short Term Memory (LSTM) BIBREF3 for composing word vectors from their constituent characters which can memorize the arbitrary aspects of word orthography as well as generalize to rare and out-of-vocabulary words 
Encouraged by their findings, we extend their approach to a much larger unicode character set, and model long sequences of text as functions of their constituent characters (including white-space)  We focus on social media posts from the website Twitter, which are an excellent testing ground for character based models due to the noisy nature of text  Heavy use of slang and abundant misspellings means that there are many orthographically and semantically similar tokens, and special characters such as emojis are also immensely popular and carry useful semantic information  In our moderately sized training dataset of 2 million tweets, there were about 0 92 million unique word types  It would be expensive to capture all these phenomena in a word based model in terms of both the memory requirement (for the increased vocabulary) and the amount of training data required for effective learning  Additional benefits of the character based approach include language independence of the methods, and no requirement of NLP preprocessing such as word-segmentation 
A crucial step in learning good text representations is to choose an appropriate objective function to optimize  Unsupervised approaches attempt to reconstruct the original text from its latent representation BIBREF4 , BIBREF0   Social media posts however, come with their own form of supervision annotated by millions of users, in the form of hashtags which link posts about the same topic together  A natural assumption is that the posts with the same hashtags should have embeddings which are close to each other  Hence, we formulate our training objective to maximize cross-entropy loss at the task of predicting hashtags for a post from its latent representation 
We propose a Bi-directional Gated Recurrent Unit (Bi-GRU) BIBREF5 neural network for learning tweet representations  Treating white-space as a special character itself, the model does a forward and backward pass over the entire sequence, and the final GRU states are linearly combined to get the tweet embedding  Posterior probabilities over hashtags are computed by projecting this embedding to a softmax output layer  Compared to a word-level baseline this model shows improved performance at predicting hashtags for a held-out set of posts  Inspired by recent work in learning vector space text representations, we name our model tweet2vec 
Related Work
Using neural networks to learn distributed representations of words dates back to BIBREF0   More recently, BIBREF4 released word2vec - a collection of word vectors trained using a recurrent neural network  These word vectors are in widespread use in the NLP community, and the original work has since been extended to sentences BIBREF1 , documents and paragraphs BIBREF6 , topics BIBREF7 and queries BIBREF8   All these methods require storing an extremely large table of vectors for all word types and cannot be easily generalized to unseen words at test time BIBREF2   They also require preprocessing to find word boundaries which is non-trivial for a social network domain like Twitter 
In BIBREF2 , the authors present a compositional character model based on bidirectional LSTMs as a potential solution to these problems  A major benefit of this approach is that large word lookup tables can be compacted into character lookup tables and the compositional model scales to large data sets better than other state-of-the-art approaches  While BIBREF2 generate word embeddings from character representations, we propose to generate vector representations of entire tweets from characters in our tweet2vec model 
Our work adds to the growing body of work showing the applicability of character models for a variety of NLP tasks such as Named Entity Recognition BIBREF9 , POS tagging BIBREF10 , text classification BIBREF11 and language modeling BIBREF12 , BIBREF13  
Previously, BIBREF14 dealt with the problem of estimating rare word representations by building them from their constituent morphemes  While they show improved performance over word-based models, their approach requires a morpheme parser for preprocessing which may not perform well on noisy text like Twitter  Also the space of all morphemes, though smaller than the space of all words, is still large enough that modelling all morphemes is impractical 
Hashtag prediction for social media has been addressed earlier, for example in BIBREF15 , BIBREF16   BIBREF15 also use a neural architecture, but compose text embeddings from a lookup table of words  They also show that the learned embeddings can generalize to an unrelated task of document recommendation, justifying the use of hashtags as supervision for learning text representations 
Tweet2Vec
Bi-GRU Encoder: Figure 1 shows our model for encoding tweets  It uses a similar structure to the C2W model in BIBREF2 , with LSTM units replaced with GRU units 
The input to the network is defined by an alphabet of characters $C$ (this may include the entire unicode character set)  The input tweet is broken into a stream of characters $c_1, c_2,     c_m$ each of which is represented by a 1-by- $|C|$ encoding  These one-hot vectors are then projected to a character space by multiplying with the matrix $P_C \in \mathbb {R}^{|C| \times d_c}$ , where $d_c$ is the dimension of the character vector space  Let $x_1, x_2,     x_m$ be the sequence of character vectors for the input tweet after the lookup  The encoder consists of a forward-GRU and a backward-GRU  Both have the same architecture, except the backward-GRU processes the sequence in reverse order  Each of the GRU units process these vectors sequentially, and starting with the initial state $h_0$ compute the sequence $h_1, h_2,     h_m$ as follows: $ r_t &= \sigma (W_r x_t + U_r h_{t-1} + b_r), \\ z_t &= \sigma (W_z x_t + U_z h_{t-1} + b_z), \\ \tilde{h}_t &= tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h), \\ h_t &= (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t  $
Here $r_t$ , $z_t$ are called the reset and update gates respectively, and $\tilde{h}_t$ is the candidate output state which is converted to the actual output state $h_t$   $W_r, W_z, W_h$ are $d_h \times d_c$ matrices and $U_r, U_z, U_h$ are $d_h \times d_h$ matrices, where $d_h$ is the hidden state dimension of the GRU  The final states $h_m^f$ from the forward-GRU, and $z_t$0 from the backward GRU are combined using a fully-connected layer to the give the final tweet embedding $z_t$1 :
$$e_t = W^f h_m^f + W^b h_0^b$$   (Eq  3)
Here $W^f, W^b$ are $d_t \times d_h$ and $b$ is $d_t \times 1$ bias term, where $d_t$ is the dimension of the final tweet embedding  In our experiments we set $d_t=d_h$ 

ground_truth: 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)

KARA ONE BIBREF17 , composed of multimodal data for stimulus-based, imagined and articulated speech state corresponding to 7 phonemic/syllabic ( /iy/, /piy/, /tiy/, /diy/, /uw/, /m/, /n/ ) as well as 4 words(pat, pot, knew and gnaw)

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is SemCor3.0 reflective of English language data in general?
Context 2: 
Experiments ::: Datasets
The statistics of the WSD datasets are shown in Table TABREF12 
Experiments ::: Datasets ::: Training Dataset
Following previous work BIBREF13, BIBREF12, BIBREF10, BIBREF17, BIBREF9, BIBREF7, we choose SemCor3 0 as training corpus, which is the largest corpus manually annotated with WordNet sense for WSD 
Experiments ::: Datasets ::: Evaluation Datasets
We evaluate our method on several English all-words WSD datasets  For a fair comparison, we use the benchmark datasets proposed by BIBREF17 which include five standard all-words fine-grained WSD datasets from the Senseval and SemEval competitions: Senseval-2 (SE2), Senseval-3 (SE3), SemEval-2007 (SE07), SemEval-2013 (SE13) and SemEval-2015 (SE15)  Following BIBREF13, BIBREF12 and BIBREF10, we choose SE07, the smallest among these test sets, as the development set 
Experiments ::: Datasets ::: WordNet
Since BIBREF17 map all the sense annotations in these datasets from their original versions to WordNet 3 0, we extract word sense glosses from WordNet 3 0 
Experiments ::: Settings
We use the pre-trained uncased BERT$_\mathrm {BASE}$ model for fine-tuning, because we find that BERT$_\mathrm {LARGE}$ model performs slightly worse than BERT$_\mathrm {BASE}$ in this task  The number of Transformer blocks is 12, the number of the hidden layer is 768, the number of self-attention heads is 12, and the total number of parameters of the pre-trained model is 110M  When fine-tuning, we use the development set (SE07) to find the optimal settings for our experiments  We keep the dropout probability at 0 1, set the number of epochs to 4  The initial learning rate is 2e-5, and the batch size is 64 
Experiments ::: Results
Table TABREF19 shows the performance of our method on the English all-words WSD benchmark datasets  We compare our approach with previous methods 
The first block shows the MFS baseline, which selects the most frequent sense in the training corpus for each target word 
The second block shows two knowledge-based systems  Lesk$_{ext+emb}$ BIBREF4 is a variant of Lesk algorithm BIBREF2 by calculating the gloss-context overlap of the target word  Babelfy BIBREF6 is a unified graph-based approach which exploits the semantic network structure from BabelNet 
The third block shows two word expert traditional supervised systems  IMS BIBREF7 is a flexible framework which trains SVM classifiers and uses local features  And IMS$_{+emb}$ BIBREF9 is the best configuration of the IMS framework, which also integrates word embeddings as features 
The fourth block shows several recent neural-based methods  Bi-LSTM BIBREF11 is a baseline for neural models  Bi-LSTM$_{+ att  + LEX + POS}$ BIBREF10 is a multi-task learning framework for WSD, POS tagging, and LEX with self-attention mechanism, which converts WSD to a sequence learning task  GAS$_{ext}$ BIBREF12 is a variant of GAS which is a gloss-augmented variant of the memory network by extending gloss knowledge  CAN$^s$ and HCAN BIBREF13 are sentence-level and hierarchical co-attention neural network models which leverage gloss knowledge 
In the last block, we report the performance of our method  BERT(Token-CLS) is our baseline, which does not incorporate gloss information, and it performs slightly worse than previous traditional supervised methods and recent neural-based methods  It proves that directly using BERT cannot obtain performance growth  The other three methods outperform other models by a substantial margin, which proves that the improvements come from leveraging BERT to better exploit gloss information  It is worth noting that our method achieves significant improvements in SE07 and Verb over previous methods, which have the highest ambiguity level among all datasets and all POS tags respectively according to BIBREF17 
Moreover, GlossBERT(Token-CLS) performs better than GlossBERT(Sent-CLS), which proves that highlighting the target word in the sentence is important  However, the weakly highlighting method GlossBERT(Sent-CLS-WS) performs best in most circumstances, which may result from its combination of the advantages of the other two methods 
Experiments ::: Discussion
There are two main reasons for the great improvements of our experimental results  First, we construct context-gloss pairs and convert WSD problem to a sentence-pair classification task which is similar to NLI tasks and train only one classifier, which is equivalent to expanding the corpus  Second, we leverage BERT BIBREF15 to better exploit the gloss information  BERT model shows its advantage in dealing with sentence-pair classification tasks by its amazing improvement on QA and NLI tasks  This advantage comes from both of its two novel unsupervised prediction tasks 
Compared with traditional word expert supervised methods, our GlossBERT shows its effectiveness to alleviate the effort of feature engineering and does not require training a dedicated classifier for every target lemma  Up to now, it can be said that the neural network method can totally beat the traditional word expert method  Compared with recent neural-based methods, our solution is more intuitive and can make better use of gloss knowledge  Besides, our approach demonstrates that when we fine-tune BERT on a downstream task, converting it into a sentence-pair classification task may be a good choice 
Conclusion
In this paper, we seek to better leverage gloss knowledge in a supervised neural WSD system  We propose a new solution to WSD by constructing context-gloss pairs and then converting WSD to a sentence-pair classification task  We fine-tune the pre-trained BERT model and achieve new state-of-the-art results on WSD task 
Acknowledgments
We would like to thank the anonymous reviewers for their valuable comments  The research work is supported by National Natural Science Foundation of China (No  61751201 and 61672162), Shanghai Municipal Science and Technology Commission (16JC1420401 and 17JC1404100), Shanghai Municipal Science and Technology Major Project (No 2018SHZDZX01) and ZJLab 

ground_truth: Yes

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What other tasks do they test their method on?
Context 3:  All parameters are learned using gradient descent 
Softmax: Finally, the tweet embedding is passed through a linear layer whose output is the same size as the number of hashtags $L$ in the data set  We use a softmax layer to compute the posterior hashtag probabilities:
$$P(y=j |e) = \frac{exp(w_j^Te + b_j)}{\sum _{i=1}^L exp(w_i^Te + b_j)} $$   (Eq  4)
Objective Function: We optimize the categorical cross-entropy loss between predicted and true hashtags:
$$J = \frac{1}{B} \sum _{i=1}^{B} \sum _{j=1}^{L} -t_{i,j}log(p_{i,j}) + \lambda \Vert \Theta \Vert ^2 $$   (Eq  5)
Here $B$ is the batch size, $L$ is the number of classes, $p_{i,j}$ is the predicted probability that the $i$ -th tweet has hashtag $j$ , and $t_{i,j} \in \lbrace 0,1\rbrace $ denotes the ground truth of whether the $j$ -th hashtag is in the $i$ -th tweet  We use L2-regularization weighted by $\lambda $  
Word Level Baseline
Since our objective is to compare character-based and word-based approaches, we have also implemented a simple word-level encoder for tweets  The input tweet is first split into tokens along white-spaces  A more sophisticated tokenizer may be used, but for a fair comparison we wanted to keep language specific preprocessing to a minimum  The encoder is essentially the same as tweet2vec, with the input as words instead of characters  A lookup table stores word vectors for the $V$ (20K here) most common words, and the rest are grouped together under the `UNK' token 
Data
Our dataset consists of a large collection of global posts from Twitter between the dates of June 1, 2013 to June 5, 2013  Only English language posts (as detected by the lang field in Twitter API) and posts with at least one hashtag are retained  We removed infrequent hashtags ( $<500$ posts) since they do not have enough data for good generalization  We also removed very frequent tags ( $>19K$ posts) which were almost always from automatically generated posts (ex: #androidgame) which are trivial to predict  The final dataset contains 2 million tweets for training, 10K for validation and 50K for testing, with a total of 2039 distinct hashtags  We use simple regex to preprocess the post text and remove hashtags (since these are to be predicted) and HTML tags, and replace user-names and URLs with special tokens  We also removed retweets and convert the text to lower-case 
Implementation Details
Word vectors and character vectors are both set to size $d_L=150$ for their respective models  There were 2829 unique characters in the training set and we model each of these independently in a character look-up table  Embedding sizes were chosen such that each model had roughly the same number of parameters (Table 2 )  Training is performed using mini-batch gradient descent with Nesterov's momentum  We use a batch size $B=64$ , initial learning rate $\eta _0=0 01$ and momentum parameter $\mu _0=0 9$   L2-regularization with $\lambda =0 001$ was applied to all models  Initial weights were drawn from 0-mean gaussians with $\sigma =0 1$ and initial biases were set to 0  The hyperparameters were tuned one at a time keeping others fixed, and values with the lowest validation cost were chosen  The resultant combination was used to train the models until performance on validation set stopped increasing  During training, the learning rate is halved everytime the validation set precision increases by less than 0 01 % from one epoch to the next  The models converge in about 20 epochs  Code for training both the models is publicly available on github 
Results
We test the character and word-level variants by predicting hashtags for a held-out test set of posts  Since there may be more than one correct hashtag per post, we generate a ranked list of tags for each post from the output posteriors, and report average precision@1, recall@10 and mean rank of the correct hashtags  These are listed in Table 3  
To see the performance of each model on posts containing rare words (RW) and frequent words (FW) we selected two test sets each containing 2,000 posts  We populated these sets with posts which had the maximum and minimum number of out-of-vocabulary words respectively, where vocabulary is defined by the 20K most frequent words  Overall, tweet2vec outperforms the word model, doing significantly better on RW test set and comparably on FW set  This improved performance comes at the cost of increased training time (see Table 2 ), since moving from words to characters results in longer input sequences to the GRU 
We also study the effect of model size on the performance of these models  For the word model we set vocabulary size $V$ to 8K, 15K and 20K respectively  For tweet2vec we set the GRU hidden state size to 300, 400 and 500 respectively  Figure 2 shows precision 1 of the two models as the number of parameters is increased, for each test set described above  There is not much variation in the performance, and moreover tweet2vec always outperforms the word based model for the same number of parameters 
Table 4 compares the models as complexity of the task is increased  We created 3 datasets (small, medium and large) with an increasing number of hashtags to be predicted  This was done by varying the lower threshold of the minimum number of tags per post for it to be included in the dataset  Once again we observe that tweet2vec outperforms its word-based counterpart for each of the three settings 
Finally, table 1 shows some predictions from the word level model and tweet2vec  We selected these to highlight some strengths of the character based approach - it is robust to word segmentation errors and spelling mistakes, effectively interprets emojis and other special characters to make predictions, and also performs comparably to the word-based approach for in-vocabulary tokens 
Conclusion
We have presented tweet2vec - a character level encoder for social media posts trained using supervision from associated hashtags  Our result shows that tweet2vec outperforms the word based approach, doing significantly better when the input post contains many rare words  We have focused only on English language posts, but the character model requires no language specific preprocessing and can be extended to other languages  For future work, one natural extension would be to use a character-level decoder for predicting the hashtags  This will allow generation of hashtags not seen in the training dataset  Also, it will be interesting to see how our tweet2vec embeddings can be used in domains where there is a need for semantic understanding of social media, such as tracking infectious diseases BIBREF17   Hence, we provide an off-the-shelf encoder trained on medium dataset described above to compute vector-space representations of tweets along with our code on github 
Acknowledgments
We would like to thank Alex Smola, Yun Fu, Hsiao-Yu Fish Tung, Ruslan Salakhutdinov, and Barnabas Poczos for useful discussions  We would also like to thank Juergen Pfeffer for providing access to the Twitter data, and the reviewers for their comments 

ground_truth: None