prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Do the hashtag and SemEval datasets contain only English data?
Context 1: Introduction
A hashtag is a keyphrase represented as a sequence of alphanumeric characters plus underscore, preceded by the # symbol  Hashtags play a central role in online communication by providing a tool to categorize the millions of posts generated daily on Twitter, Instagram, etc  They are useful in search, tracking content about a certain topic BIBREF0 , BIBREF1 , or discovering emerging trends BIBREF2  
Hashtags often carry very important information, such as emotion BIBREF3 , sentiment BIBREF4 , sarcasm BIBREF5 , and named entities BIBREF6 , BIBREF7   However, inferring the semantics of hashtags is non-trivial since many hashtags contain multiple tokens joined together, which frequently leads to multiple potential interpretations (e g , lion head vs  lionhead)  Table TABREF3 shows several examples of single- and multi-token hashtags  While most hashtags represent a mix of standard tokens, named entities and event names are prevalent and pose challenges to both human and automatic comprehension, as these are more likely to be rare tokens  Hashtags also tend to be shorter to allow fast typing, to attract attention or to satisfy length limitations imposed by some social media platforms  Thus, they tend to contain a large number of abbreviations or non-standard spelling variations (e g , #iloveu4eva) BIBREF8 , BIBREF9 , which hinders their understanding 
The goal of our study is to build efficient methods for automatically splitting a hashtag into a meaningful word sequence  Our contributions are:
Our new dataset includes segmentation for 12,594 unique hashtags and their associated tweets annotated in a multi-step process for higher quality than the previous dataset of 1,108 hashtags BIBREF10   We frame the segmentation task as a pairwise ranking problem, given a set of candidate segmentations  We build several neural architectures using this problem formulation which use corpus-based, linguistic and thesaurus based features  We further propose a multi-task learning approach which jointly learns segment ranking and single- vs  multi-token hashtag classification  The latter leads to an error reduction of 24 6% over the current state-of-the-art  Finally, we demonstrate the utility of our method by using hashtag segmentation in the downstream task of sentiment analysis  Feeding the automatically segmented hashtags to a state-of-the-art sentiment analysis method on the SemEval 2017 benchmark dataset results in a 2 6% increase in the official metric for the task 
Background and Preliminaries
Current approaches for hashtag segmentation can be broadly divided into three categories: (a) gazeteer and rule based BIBREF11 , BIBREF12 , BIBREF13 , (b) word boundary detection BIBREF14 , BIBREF15 , and (c) ranking with language model and other features BIBREF16 , BIBREF10 , BIBREF0 , BIBREF17 , BIBREF18   Hashtag segmentation approaches draw upon work on compound splitting for languages such as German or Finnish BIBREF19 and word segmentation BIBREF20 for languages with no spaces between words such as Chinese BIBREF21 , BIBREF22   Similar to our work, BIBREF10 BansalBV15 extract an initial set of candidate segmentations using a sliding window, then rerank them using a linear regression model trained on lexical, bigram and other corpus-based features  The current state-of-the-art approach BIBREF14 , BIBREF15 uses maximum entropy and CRF models with a combination of language model and hand-crafted features to predict if each character in the hashtag is the beginning of a new word 
Generating Candidate Segmentations  Microsoft Word Breaker BIBREF16 is, among the existing methods, a strong baseline for hashtag segmentation, as reported in BIBREF14 and BIBREF10   It employs a beam search algorithm to extract INLINEFORM0 best segmentations as ranked by the n-gram language model probability: INLINEFORM1
where INLINEFORM0 is the word sequence of segmentation INLINEFORM1 and INLINEFORM2 is the window size  More sophisticated ranking strategies, such as Binomial and word length distribution based ranking, did not lead to a further improvement in performance BIBREF16   The original Word Breaker was designed for segmenting URLs using language models trained on web data  In this paper, we reimplemented and tailored this approach to segmenting hashtags by using a language model specifically trained on Twitter data (implementation details in § SECREF26 )  The performance of this method itself is competitive with state-of-the-art methods (evaluation results in § SECREF46 )  Our proposed pairwise ranking method will effectively take the top INLINEFORM3 segmentations generated by this baseline as candidates for reranking 
However, in prior work, the ranking scores of each segmentation were calculated independently, ignoring the relative order among the top INLINEFORM0 candidate segmentations  To address this limitation, we utilize a pairwise ranking strategy for the first time for this task and propose neural architectures to model this 
Multi-task Pairwise Neural Ranking
We propose a multi-task pairwise neural ranking approach to better incorporate and distinguish the relative order between the candidate segmentations of a given hashtag  Our model adapts to address single- and multi-token hashtags differently via a multi-task learning strategy without requiring additional annotations  In this section, we describe the task setup and three variants of pairwise neural ranking models (Figure FIGREF11 ) 
Segmentation as Pairwise Ranking
The goal of hashtag segmentation is to divide a given hashtag INLINEFORM0 into a sequence of meaningful words INLINEFORM1   For a hashtag of INLINEFORM2 characters, there are a total of INLINEFORM3 possible segmentations but only one, or occasionally two, of them ( INLINEFORM4 ) are considered correct (Table TABREF9 ) 
We transform this task into a pairwise ranking problem: given INLINEFORM0 candidate segmentations { INLINEFORM1 }, we rank them by comparing each with the rest in a pairwise manner  More specifically, we train a model to predict a real number INLINEFORM2 for any two candidate segmentations INLINEFORM3 and INLINEFORM4 of hashtag INLINEFORM5 , which indicates INLINEFORM6 is a better segmentation than INLINEFORM7 if positive, and vice versa  To quantify the quality of a segmentation in training, we define a gold scoring function INLINEFORM8 based on the similarities with the ground-truth segmentation INLINEFORM9 : INLINEFORM10
We use the Levenshtein distance (minimum number of single-character edits) in this paper, although it is possible to use other similarity measurements as alternatives  We use the top INLINEFORM0 segmentations generated by Microsoft Word Breaker (§ SECREF2 ) as initial candidates 
Pairwise Neural Ranking Model
For an input candidate segmentation pair INLINEFORM0 , we concatenate their feature vectors INLINEFORM1 and INLINEFORM2 , and feed them into a feedforward network which emits a comparison score INLINEFORM3   The feature vector INLINEFORM4 or INLINEFORM5 consists of language model probabilities using Good-Turing BIBREF23 and modified Kneser-Ney smoothing BIBREF24 , BIBREF25 , lexical and linguistic features (more details in § SECREF23 )  For training, we use all the possible pairs INLINEFORM6 of the INLINEFORM7 candidates as the input and their gold scores INLINEFORM8 as the target  The training objective is to minimize the Mean Squared Error (MSE): DISPLAYFORM0
where INLINEFORM0 is the number of training examples 
To aggregate the pairwise comparisons, we follow a greedy algorithm proposed by BIBREF26 cohen1998learning and used for preference ranking BIBREF27   For each segmentation INLINEFORM0 in the candidate set INLINEFORM1 , we calculate a single score INLINEFORM2 , and find the segmentation INLINEFORM3 corresponding to the highest score  We repeat the same procedure after removing INLINEFORM4 from INLINEFORM5 , and continue until INLINEFORM6 reduces to an empty set  Figure FIGREF11 (a) shows the architecture of this model 
Margin Ranking (MR) Loss
As an alternative to the pairwise ranker (§ SECREF15 ), we propose a pairwise model which learns from candidate pairs INLINEFORM0 but ranks each individual candidate directly rather than relatively  We define a new scoring function INLINEFORM1 which assigns a higher score to the better candidate, i e , INLINEFORM2 , if INLINEFORM3 is a better candidate than INLINEFORM4 and vice-versa  Instead of concatenating the features vectors INLINEFORM5 and INLINEFORM6 , we feed them separately into two identical feedforward networks with shared parameters  During testing, we use only one of the networks to rank the candidates based on the INLINEFORM7 scores  For training, we add a ranking layer on top of the networks to measure the violations in the ranking order and minimize the Margin Ranking Loss (MR): DISPLAYFORM0
where INLINEFORM0 is the number of training samples  The architecture of this model is presented in Figure FIGREF11 (b) 
Adaptive Multi-task Learning
Both models in § SECREF15 and § SECREF17 treat all the hashtags uniformly  However, different features address different types of hashtags  By design, the linguistic features capture named entities and multi-word hashtags that exhibit word shape patterns, such as camel case  The ngram probabilities with Good-Turing smoothing gravitate towards multi-word segmentations with known words, as its estimate for unseen ngrams depends on the fraction of ngrams seen once which can be very low BIBREF28   The modified Kneser-Ney smoothing is more likely to favor segmentations that contain rare words, and single-word segmentations in particular  Please refer to § SECREF46 for a more detailed quantitative and qualitative analysis 
To leverage this intuition, we introduce a binary classification task to help the model differentiate single-word from multi-word hashtags  The binary classifier takes hashtag features INLINEFORM0 as the input and outputs INLINEFORM1 , which represents the probability of INLINEFORM2 being a multi-word hashtag  INLINEFORM3 is used as an adaptive gating value in our multi-task learning setup  The gold labels for this task are obtained at no extra cost by simply verifying whether the ground-truth segmentation has multiple words  We train the pairwise segmentation ranker and the binary single- vs  multi-token hashtag classifier jointly, by minimizing INLINEFORM4 for the pairwise ranker and the Binary Cross Entropy Error ( INLINEFORM5 ) for the classifier: DISPLAYFORM0
where INLINEFORM0 is the adaptive gating value, INLINEFORM1 indicates if INLINEFORM2 is actually a multi-word hashtag and INLINEFORM3 is the number of training examples  INLINEFORM4 and INLINEFORM5 are the weights for each loss  For our experiments, we apply equal weights 
More specifically, we divide the segmentation feature vector INLINEFORM0 into two subsets: (a) INLINEFORM1 with modified Kneser-Ney smoothing features, and (b) INLINEFORM2 with Good-Turing smoothing and linguistic features  For an input candidate segmentation pair INLINEFORM3 , we construct two pairwise vectors INLINEFORM4 and INLINEFORM5 by concatenation, then combine them based on the adaptive gating value INLINEFORM6 before feeding them into the feedforward network INLINEFORM7 for pairwise ranking: DISPLAYFORM0
We use summation with padding, as we find this simple ensemble method achieves similar performance in our experiments as the more complex multi-column networks BIBREF29   Figure FIGREF11 (c) shows the architecture of this model  An analogue multi-task formulation can also be used for the Margin Ranking loss as: DISPLAYFORM0
Features
We use a combination of corpus-based and linguistic features to rank the segmentations  For a candidate segmentation INLINEFORM0 , its feature vector INLINEFORM1 includes the number of words in the candidate, the length of each word, the proportion of words in an English dictionary or Urban Dictionary BIBREF30 , ngram counts from Google Web 1TB corpus BIBREF31 , and ngram probabilities from trigram language models trained on the Gigaword corpus BIBREF32 and 1 1 billion English tweets from 2010, respectively  We train two language models on each corpus: one with Good-Turing smoothing using SRILM BIBREF33 and the other with modified Kneser-Ney smoothing using KenLM BIBREF34   We also add boolean features, such as if the candidate is a named-entity present in the list of Wikipedia titles, and if the candidate segmentation INLINEFORM2 and its corresponding hashtag INLINEFORM3 satisfy certain word-shapes (more details in appendix SECREF61 ) 
Similarly, for hashtag INLINEFORM0 , we extract the feature vector INLINEFORM1 consisting of hashtag length, ngram count of the hashtag in Google 1TB corpus BIBREF31 , and boolean features indicating if the hashtag is in an English dictionary or Urban Dictionary, is a named-entity, is in camel case, ends with a number, and has all the letters as consonants  We also include features of the best-ranked candidate by the Word Breaker model 
Implementation Details
We use the PyTorch framework to implement our multi-task pairwise ranking model  The pairwise ranker consists of an input layer, three hidden layers with eight nodes in each layer and hyperbolic tangent ( INLINEFORM0 ) activation, and a single linear output node  The auxiliary classifier consists of an input layer, one hidden layer with eight nodes and one output node with sigmoid activation  We use the Adam algorithm BIBREF35 for optimization and apply a dropout of 0 5 to prevent overfitting  We set the learning rate to 0 01 and 0 05 for the pairwise ranker and auxiliary classifier respectively  For each experiment, we report results obtained after 100 epochs 
For the baseline model used to extract the INLINEFORM0 initial candidates, we reimplementated the Word Breaker BIBREF16 as described in § SECREF2 and adapted it to use a language model trained on 1 1 billion tweets with Good-Turing smoothing using SRILM BIBREF33 to give a better performance in segmenting hashtags (§ SECREF46 )  For all our experiments, we set INLINEFORM1  
Hashtag Segmentation Data
We use two datasets for experiments (Table TABREF29 ): (a) STAN INLINEFORM0 , created by BIBREF10 BansalBV15, which consists of 1,108 unique English hashtags from 1,268 randomly selected tweets in the Stanford Sentiment Analysis Dataset BIBREF36 along with their crowdsourced segmentations and our additional corrections; and (b) STAN INLINEFORM1 , our new expert curated dataset, which includes all 12,594 unique English hashtags and their associated tweets from the same Stanford dataset 
Experiments
In this section, we present experimental results that compare our proposed method with the other state-of-the-art approaches on hashtag segmentation datasets  The next section will show experiments of applying hashtag segmentation to the popular task of sentiment analysis 
Existing Methods
We compare our pairwise neural ranker with the following baseline and state-of-the-art approaches:
The original hashtag as a single token;
A rule-based segmenter, which employs a set of word-shape rules with an English dictionary BIBREF13 ;
A Viterbi model which uses word frequencies from a book corpus BIBREF0 ;
The specially developed GATE Hashtag Tokenizer from the open source toolkit, which combines dictionaries and gazetteers in a Viterbi-like algorithm BIBREF11 ;
A maximum entropy classifier (MaxEnt) trained on the STAN INLINEFORM0 training dataset  It predicts whether a space should be inserted at each position in the hashtag and is the current state-of-the-art BIBREF14 ;
Our reimplementation of the Word Breaker algorithm which uses beam search and a Twitter ngram language model BIBREF16 ;
A pairwise linear ranker which we implemented for comparison purposes with the same features as our neural model, but using perceptron as the underlying classifier BIBREF38 and minimizing the hinge loss between INLINEFORM0 and a scoring function similar to INLINEFORM1   It is trained on the STAN INLINEFORM2 dataset 
Evaluation Metrics
We evaluate the performance by the top INLINEFORM0 ( INLINEFORM1 ) accuracy (A@1, A@2), average token-level F INLINEFORM2 score (F INLINEFORM3 @1), and mean reciprocal rank (MRR)  In particular, the accuracy and MRR are calculated at the segmentation-level, which means that an output segmentation is considered correct if and only if it fully matches the human segmentation  Average token-level F INLINEFORM4 score accounts for partially correct segmentation in the multi-token hashtag cases 
Results
Tables TABREF32 and TABREF33 show the results on the STAN INLINEFORM0 and STAN INLINEFORM1 datasets, respectively  All of our pairwise neural rankers are trained on the 2,518 manually segmented hashtags in the training set of STAN INLINEFORM2 and perform favorably against other state-of-the-art approaches  Our best model (MSE+multitask) that utilizes different features adaptively via a multi-task learning procedure is shown to perform better than simply combining all the features together (MR and MSE)  We highlight the 24 6% error reduction on STAN INLINEFORM3 and 16 5% on STAN INLINEFORM4 of our approach over the previous SOTA BIBREF14 on the Multi-token hashtags, and the importance of having a separate evaluation of multi-word cases as it is trivial to obtain 100% accuracy for Single-token hashtags  While our hashtag segmentation model is achieving a very high accuracy@2, to be practically useful, it remains a challenge to get the top one predication exactly correct  Some hashtags are very difficult to interpret, e g , #BTVbrownSMB refers to the Social Media Breakfast (SMB) in Burlington, Vermont (BTV) 
The improved Word Breaker with our addition of a Twitter-specific language model is a very strong baseline, which echos the findings of the original Word Breaker paper BIBREF16 that having a large in-domain language model is extremely helpful for word segmentation tasks  It is worth noting that the other state-of-the-art system BIBREF14 also utilized a 4-gram language model trained on 476 million tweets from 2009 
Analysis and Discussion
To empirically illustrate the effectiveness of different features on different types of hashtags, we show the results for models using individual feature sets in pairwise ranking models (MSE) in Table TABREF45   Language models with modified Kneser-Ney smoothing perform best on single-token hashtags, while Good-Turing and Linguistic features work best on multi-token hashtags, confirming our intuition about their usefulness in a multi-task learning approach  Table TABREF47 shows a qualitative analysis with the first column ( INLINEFORM0 INLINEFORM1 INLINEFORM2 ) indicating which features lead to correct or wrong segmentations, their count in our data and illustrative examples with human segmentation 
As expected, longer hashtags with more than three tokens pose greater challenges and the segmentation-level accuracy of our best model (MSE+multitask) drops to 82 1%  For many error cases, our model predicts a close-to-correct segmentation, e g , #youbrownknowyoubrownupttoobrownearly, #iseebrownlondoniseebrownfrance, which is also reflected by the higher token-level F INLINEFORM0 scores across hashtags with different lengths (Figure FIGREF51 ) 
Since our approach heavily relies on building a Twitter language model, we experimented with its sizes and show the results in Figure FIGREF52   Our approach can perform well even with access to a smaller amount of tweets  The drop in F INLINEFORM0 score for our pairwise neural ranker is only 1 4% and 3 9% when using the language models trained on 10% and 1% of the total 1 1 billion tweets, respectively 
Language use in Twitter changes with time BIBREF9   Our pairwise ranker uses language models trained on the tweets from the year 2010  We tested our approach on a set of 500 random English hashtags posted in tweets from the year 2019 and show the results in Table TABREF55   With a segmentation-level accuracy of 94 6% and average token-level F INLINEFORM0 score of 95 6%, our approach performs favorably on 2019 hashtags 
Extrinsic Evaluation: Twitter Sentiment Analysis
We attempt to demonstrate the effectiveness of our hashtag segmentation system by studying its impact on the task of sentiment analysis in Twitter BIBREF39 , BIBREF40 , BIBREF41   We use our best model (MSE+multitask), under the name HashtagMaster, in the following experiments 
Experimental Setup
We compare the performance of the BiLSTM+Lex BIBREF42 sentiment analysis model under three configurations: (a) tweets with hashtags removed, (b) tweets with hashtags as single tokens excluding the # symbol, and (c) tweets with hashtags as segmented by our system, HashtagMaster  BiLSTM+Lex is a state-of-the-art open source system for predicting tweet-level sentiment BIBREF43   It learns a context-sensitive sentiment intensity score by leveraging a Twitter-based sentiment lexicon BIBREF44   We use the same settings as described by BIBREF42 teng-vo-zhang:2016:EMNLP2016 to train the model 
We use the dataset from the Sentiment Analysis in Twitter shared task (subtask A) at SemEval 2017 BIBREF41   Given a tweet, the goal is to predict whether it expresses POSITIVE, NEGATIVE or NEUTRAL sentiment  The training and development sets consist of 49,669 tweets and we use 40,000 for training and the rest for development  There are a total of 12,284 tweets containing 12,128 hashtags in the SemEval 2017 test set, and our hashtag segmenter ended up splitting 6,975 of those hashtags present in 3,384 tweets 
Results and Analysis
In Table TABREF59 , we report the results based on the 3,384 tweets where HashtagMaster predicted a split, as for the rest of tweets in the test set, the hashtag segmenter would neither improve nor worsen the sentiment prediction  Our hashtag segmenter successfully improved the sentiment analysis performance by 2% on average recall and F INLINEFORM0 comparing to having hashtags unsegmented  This improvement is seemingly small but decidedly important for tweets where sentiment-related information is embedded in multi-word hashtags and sentiment prediction would be incorrect based only on the text (see Table TABREF60 for examples)  In fact, 2,605 out of the 3,384 tweets have multi-word hashtags that contain words in the Twitter-based sentiment lexicon BIBREF44 and 125 tweets contain sentiment words only in the hashtags but not in the rest of the tweet  On the entire test set of 12,284 tweets, the increase in the average recall is 0 5% 
Other Related Work
Automatic hashtag segmentation can improve the performance of many applications besides sentiment analysis, such as text classification BIBREF13 , named entity linking BIBREF10 and modeling user interests for recommendations BIBREF45   It can also help in collecting data of higher volume and quality by providing a more nuanced interpretation of its content, as shown for emotion analysis BIBREF46 , sarcasm and irony detection BIBREF11 , BIBREF47   Better semantic analysis of hashtags can also potentially be applied to hashtag annotation BIBREF48 , to improve distant supervision labels in training classifiers for tasks such as sarcasm BIBREF5 , sentiment BIBREF4 , emotions BIBREF3 ; and, more generally, as labels for pre-training representations of words BIBREF49 , sentences BIBREF50 , and images BIBREF51  
Conclusion
We proposed a new pairwise neural ranking model for hashtag segmention and showed significant performance improvements over the state-of-the-art  We also constructed a larger and more curated dataset for analyzing and benchmarking hashtag segmentation methods  We demonstrated that hashtag segmentation helps with downstream tasks such as sentiment analysis  Although we focused on English hashtags, our pairwise ranking approach is language-independent and we intend to extend our toolkit to languages other than English as future work 
Acknowledgments
We thank Ohio Supercomputer Center BIBREF52 for computing resources and the NVIDIA for providing GPU hardware  We thank Alan Ritter, Quanze Chen, Wang Ling, Pravar Mahajan, and Dushyanta Dhyani for valuable discussions  We also thank the annotators: Sarah Flanagan, Kaushik Mani, and Aswathnarayan Radhakrishnan  This material is based in part on research sponsored by the NSF under grants IIS-1822754 and IIS-1755898, DARPA through the ARO under agreement number W911NF-17-C-0095, through a Figure-Eight (CrowdFlower) AI for Everyone Award and a Criteo Faculty Research Award to Wei Xu  The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of the U S  Government 
Word-shape rules
Our model uses the following word shape rules as boolean features  If the candidate segmentation INLINEFORM0 and its corresponding hashtag INLINEFORM1 satisfies a word shape rule, then the boolean feature is set to True 
Context 2: Introduction
The Transformer architecture BIBREF0 for deep neural networks has quickly risen to prominence in NLP through its efficiency and performance, leading to improvements in the state of the art of Neural Machine Translation BIBREF1, BIBREF2, as well as inspiring other powerful general-purpose models like BERT BIBREF3 and GPT-2 BIBREF4  At the heart of the Transformer lie multi-head attention mechanisms: each word is represented by multiple different weighted averages of its relevant context  As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9 
The attention distribution of each head is predicted typically using the softmax normalizing transform  As a result, all context words have non-zero attention weight  Recent work on single attention architectures suggest that using sparse normalizing transforms in attention mechanisms such as sparsemax – which can yield exactly zero probabilities for irrelevant words – may improve performance and interpretability BIBREF12, BIBREF13, BIBREF14  Qualitative analysis of attention heads BIBREF0 suggests that, depending on what phenomena they capture, heads tend to favor flatter or more peaked distributions 
Recent works have proposed sparse Transformers BIBREF10 and adaptive span Transformers BIBREF11  However, the “sparsity" of those models only limits the attention to a contiguous span of past tokens, while in this work we propose a highly adaptive Transformer model that is capable of attending to a sparse set of words that are not necessarily contiguous  Figure FIGREF1 shows the relationship of these methods with ours 
Our contributions are the following:
We introduce sparse attention into the Transformer architecture, showing that it eases interpretability and leads to slight accuracy gains 
We propose an adaptive version of sparse attention, where the shape of each attention head is learnable and can vary continuously and dynamically between the dense limit case of softmax and the sparse, piecewise-linear sparsemax case 
We make an extensive analysis of the added interpretability of these models, identifying both crisper examples of attention head behavior observed in previous work, as well as novel behaviors unraveled thanks to the sparsity and adaptivity of our proposed model 
Background ::: The Transformer
In NMT, the Transformer BIBREF0 is a sequence-to-sequence (seq2seq) model which maps an input sequence to an output sequence through hierarchical multi-head attention mechanisms, yielding a dynamic, context-dependent strategy for propagating information within and across sentences  It contrasts with previous seq2seq models, which usually rely either on costly gated recurrent operations BIBREF15, BIBREF16 or static convolutions BIBREF17 
Given $n$ query contexts and $m$ sequence items under consideration, attention mechanisms compute, for each query, a weighted representation of the items  The particular attention mechanism used in BIBREF0 is called scaled dot-product attention, and it is computed in the following way:
where $\mathbf {Q} \in \mathbb {R}^{n \times d}$ contains representations of the queries, $\mathbf {K}, \mathbf {V} \in \mathbb {R}^{m \times d}$ are the keys and values of the items attended over, and $d$ is the dimensionality of these representations  The $\mathbf {\pi }$ mapping normalizes row-wise using softmax, $\mathbf {\pi }(\mathbf {Z})_{ij} = \operatornamewithlimits{\mathsf {softmax}}(\mathbf {z}_i)_j$, where
In words, the keys are used to compute a relevance score between each item and query  Then, normalized attention weights are computed using softmax, and these are used to weight the values of each item at each query context 
However, for complex tasks, different parts of a sequence may be relevant in different ways, motivating multi-head attention in Transformers  This is simply the application of Equation DISPLAY_FORM7 in parallel $H$ times, each with a different, learned linear transformation that allows specialization:
In the Transformer, there are three separate multi-head attention mechanisms for distinct purposes:
Encoder self-attention: builds rich, layered representations of each input word, by attending on the entire input sentence 
Context attention: selects a representative weighted average of the encodings of the input words, at each time step of the decoder 
Decoder self-attention: attends over the partial output sentence fragment produced so far 
Together, these mechanisms enable the contextualized flow of information between the input sentence and the sequential decoder 
Background ::: Sparse Attention
The softmax mapping (Equation DISPLAY_FORM8) is elementwise proportional to $\exp $, therefore it can never assign a weight of exactly zero  Thus, unnecessary items are still taken into consideration to some extent  Since its output sums to one, this invariably means less weight is assigned to the relevant items, potentially harming performance and interpretability BIBREF18  This has motivated a line of research on learning networks with sparse mappings BIBREF19, BIBREF20, BIBREF21, BIBREF22  We focus on a recently-introduced flexible family of transformations, $\alpha $-entmax BIBREF23, BIBREF14, defined as:
where $\triangle ^d \lbrace \mathbf {p}\in \mathbb {R}^d:\sum _{i} p_i = 1\rbrace $ is the probability simplex, and, for $\alpha \ge 1$, $\mathsf {H}^{\textsc {T}}_\alpha $ is the Tsallis continuous family of entropies BIBREF24:
This family contains the well-known Shannon and Gini entropies, corresponding to the cases $\alpha =1$ and $\alpha =2$, respectively 
Equation DISPLAY_FORM14 involves a convex optimization subproblem  Using the definition of $\mathsf {H}^{\textsc {T}}_\alpha $, the optimality conditions may be used to derive the following form for the solution (Appendix SECREF83):
where $[\cdot ]_+$ is the positive part (ReLU) function, $\mathbf {1}$ denotes the vector of all ones, and $\tau $ – which acts like a threshold – is the Lagrange multiplier corresponding to the $\sum _i p_i=1$ constraint 
Background ::: Sparse Attention ::: Properties of @!START@$\alpha $@!END@-entmax 
The appeal of $\alpha $-entmax for attention rests on the following properties  For $\alpha =1$ (i e , when $\mathsf {H}^{\textsc {T}}_\alpha $ becomes the Shannon entropy), it exactly recovers the softmax mapping (We provide a short derivation in Appendix SECREF89 )  For all $\alpha >1$ it permits sparse solutions, in stark contrast to softmax  In particular, for $\alpha =2$, it recovers the sparsemax mapping BIBREF19, which is piecewise linear  In-between, as $\alpha $ increases, the mapping continuously gets sparser as its curvature changes 
To compute the value of $\alpha $-entmax, one must find the threshold $\tau $ such that the r h s  in Equation DISPLAY_FORM16 sums to one  BIBREF23 propose a general bisection algorithm  BIBREF14 introduce a faster, exact algorithm for $\alpha =1 5$, and enable using $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with fixed $\alpha $ within a neural network by showing that the $\alpha $-entmax Jacobian w r t  $\mathbf {z}$ for $\mathbf {p}^\star = \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ is
Our work furthers the study of $\alpha $-entmax by providing a derivation of the Jacobian w r t  the hyper-parameter $\alpha $ (Section SECREF3), thereby allowing the shape and sparsity of the mapping to be learned automatically  This is particularly appealing in the context of multi-head attention mechanisms, where we shall show in Section SECREF35 that different heads tend to learn different sparsity behaviors 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax
We now propose a novel Transformer architecture wherein we simply replace softmax with $\alpha $-entmax in the attention heads  Concretely, we replace the row normalization $\mathbf {\pi }$ in Equation DISPLAY_FORM7 by
This change leads to sparse attention weights, as long as $\alpha >1$; in particular, $\alpha =1 5$ is a sensible starting point BIBREF14 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Different @!START@$\alpha $@!END@ per head 
Unlike LSTM-based seq2seq models, where $\alpha $ can be more easily tuned by grid search, in a Transformer, there are many attention heads in multiple layers  Crucial to the power of such models, the different heads capture different linguistic phenomena, some of them isolating important words, others spreading out attention across phrases BIBREF0  This motivates using different, adaptive $\alpha $ values for each attention head, such that some heads may learn to be sparser, and others may become closer to softmax  We propose doing so by treating the $\alpha $ values as neural network parameters, optimized via stochastic gradients along with the other weights 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Derivatives w r t  @!START@$\alpha $@!END@ 
In order to optimize $\alpha $ automatically via gradient methods, we must compute the Jacobian of the entmax output w r t  $\alpha $  Since entmax is defined through an optimization problem, this is non-trivial and cannot be simply handled through automatic differentiation; it falls within the domain of argmin differentiation, an active research topic in optimization BIBREF25, BIBREF26 
One of our key contributions is the derivation of a closed-form expression for this Jacobian  The next proposition provides such an expression, enabling entmax layers with adaptive $\alpha $  To the best of our knowledge, ours is the first neural network module that can automatically, continuously vary in shape away from softmax and toward sparse mappings like sparsemax 
Proposition 1 Let $\mathbf {p}^\star \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ be the solution of Equation DISPLAY_FORM14  Denote the distribution $\tilde{p}_i {(p_i^\star )^{2 - \alpha }}{ \sum _j(p_j^\star )^{2-\alpha }}$ and let $h_i -p^\star _i \log p^\star _i$  The $i$th component of the Jacobian $\mathbf {g} \frac{\partial \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})}{\partial \alpha }$ is
proof uses implicit function differentiation and is given in Appendix SECREF10 
Proposition UNKREF22 provides the remaining missing piece needed for training adaptively sparse Transformers  In the following section, we evaluate this strategy on neural machine translation, and analyze the behavior of the learned attention heads 
Experiments
We apply our adaptively sparse Transformers on four machine translation tasks  For comparison, a natural baseline is the standard Transformer architecture using the softmax transform in its multi-head attention mechanisms  We consider two other model variants in our experiments that make use of different normalizing transformations:
1 5-entmax: a Transformer with sparse entmax attention with fixed $\alpha =1 5$ for all heads  This is a novel model, since 1 5-entmax had only been proposed for RNN-based NMT models BIBREF14, but never in Transformers, where attention modules are not just one single component of the seq2seq model but rather an integral part of all of the model components 
$\alpha $-entmax: an adaptive Transformer with sparse entmax attention with a different, learned $\alpha _{i,j}^t$ for each head 
The adaptive model has an additional scalar parameter per attention head per layer for each of the three attention mechanisms (encoder self-attention, context attention, and decoder self-attention), i e ,
and we set $\alpha _{i,j}^t = 1 + \operatornamewithlimits{\mathsf {sigmoid}}(a_{i,j}^t) \in ]1, 2[$  All or some of the $\alpha $ values can be tied if desired, but we keep them independent for analysis purposes 
Experiments ::: Datasets 
Our models were trained on 4 machine translation datasets of different training sizes:
[itemsep= 5ex,leftmargin=2ex]
IWSLT 2017 German $\rightarrow $ English BIBREF27: 200K sentence pairs 
KFTT Japanese $\rightarrow $ English BIBREF28: 300K sentence pairs 
WMT 2016 Romanian $\rightarrow $ English BIBREF29: 600K sentence pairs 
WMT 2014 English $\rightarrow $ German BIBREF30: 4 5M sentence pairs 
All of these datasets were preprocessed with byte-pair encoding BIBREF31, using joint segmentations of 32k merge operations 
Experiments ::: Training 
We follow the dimensions of the Transformer-Base model of BIBREF0: The number of layers is $L=6$ and number of heads is $H=8$ in the encoder self-attention, the context attention, and the decoder self-attention  We use a mini-batch size of 8192 tokens and warm up the learning rate linearly until 20k steps, after which it decays according to an inverse square root schedule  All models were trained until convergence of validation accuracy, and evaluation was done at each 10k steps for ro$\rightarrow $en and en$\rightarrow $de and at each 5k steps for de$\rightarrow $en and ja$\rightarrow $en  The end-to-end computational overhead of our methods, when compared to standard softmax, is relatively small; in training tokens per second, the models using $\alpha $-entmax and $1 5$-entmax are, respectively, $75\%$ and $90\%$ the speed of the softmax model 
Experiments ::: Results 
We report test set tokenized BLEU BIBREF32 results in Table TABREF27  We can see that replacing softmax by entmax does not hurt performance in any of the datasets; indeed, sparse attention Transformers tend to have slightly higher BLEU, but their sparsity leads to a better potential for analysis  In the next section, we make use of this potential by exploring the learned internal mechanics of the self-attention heads 
Analysis
We conduct an analysis for the higher-resource dataset WMT 2014 English $\rightarrow $ German of the attention in the sparse adaptive Transformer model ($\alpha $-entmax) at multiple levels: we analyze high-level statistics as well as individual head behavior  Moreover, we make a qualitative analysis of the interpretability capabilities of our models 
Analysis ::: High-Level Statistics ::: What kind of @!START@$\alpha $@!END@ values are learned?
Figure FIGREF37 shows the learning trajectories of the $\alpha $ parameters of a selected subset of heads  We generally observe a tendency for the randomly-initialized $\alpha $ parameters to decrease initially, suggesting that softmax-like behavior may be preferable while the model is still very uncertain  After around one thousand steps, some heads change direction and become sparser, perhaps as they become more confident and specialized  This shows that the initialization of $\alpha $ does not predetermine its sparsity level or the role the head will have throughout  In particular, head 8 in the encoder self-attention layer 2 first drops to around $\alpha =1 3$ before becoming one of the sparsest heads, with $\alpha \approx 2$ 
The overall distribution of $\alpha $ values at convergence can be seen in Figure FIGREF38  We can observe that the encoder self-attention blocks learn to concentrate the $\alpha $ values in two modes: a very sparse one around $\alpha \rightarrow 2$, and a dense one between softmax and 1 5-entmax   However, the decoder self and context attention only learn to distribute these parameters in a single mode  We show next that this is reflected in the average density of attention weight vectors as well 
Analysis ::: High-Level Statistics ::: Attention weight density when translating 
For any $\alpha >1$, it would still be possible for the weight matrices in Equation DISPLAY_FORM9 to learn re-scalings so as to make attention sparser or denser  To visualize the impact of adaptive $\alpha $ values, we compare the empirical attention weight density (the average number of tokens receiving non-zero attention) within each module, against sparse Transformers with fixed $\alpha =1 5$ 
Figure FIGREF40 shows that, with fixed $\alpha =1 5$, heads tend to be sparse and similarly-distributed in all three attention modules  With learned $\alpha $, there are two notable changes: (i) a prominent mode corresponding to fully dense probabilities, showing that our models learn to combine sparse and dense attention, and (ii) a distinction between the encoder self-attention – whose background distribution tends toward extreme sparsity – and the other two modules, who exhibit more uniform background distributions  This suggests that perhaps entirely sparse Transformers are suboptimal 
The fact that the decoder seems to prefer denser attention distributions might be attributed to it being auto-regressive, only having access to past tokens and not the full sentence  We speculate that it might lose too much information if it assigned weights of zero to too many tokens in the self-attention, since there are fewer tokens to attend to in the first place 
Teasing this down into separate layers, Figure FIGREF41 shows the average (sorted) density of each head for each layer  We observe that $\alpha $-entmax is able to learn different sparsity patterns at each layer, leading to more variance in individual head behavior, to clearly-identified dense and sparse heads, and overall to different tendencies compared to the fixed case of $\alpha =1 5$ 
Analysis ::: High-Level Statistics ::: Head diversity 
To measure the overall disagreement between attention heads, as a measure of head diversity, we use the following generalization of the Jensen-Shannon divergence:
where $\mathbf {p}_j$ is the vector of attention weights assigned by head $j$ to each word in the sequence, and $\mathsf {H}^\textsc {S}$ is the Shannon entropy, base-adjusted based on the dimension of $\mathbf {p}$ such that $JS \le 1$  We average this measure over the entire validation set  The higher this metric is, the more the heads are taking different roles in the model 
Figure FIGREF44 shows that both sparse Transformer variants show more diversity than the traditional softmax one  Interestingly, diversity seems to peak in the middle layers of the encoder self-attention and context attention, while this is not the case for the decoder self-attention 
The statistics shown in this section can be found for the other language pairs in Appendix SECREF8 
Analysis ::: Identifying Head Specializations
Previous work pointed out some specific roles played by different heads in the softmax Transformer model BIBREF33, BIBREF5, BIBREF9  Identifying the specialization of a head can be done by observing the type of tokens or sequences that the head often assigns most of its attention weight; this is facilitated by sparsity 
Analysis ::: Identifying Head Specializations ::: Positional heads 
One particular type of head, as noted by BIBREF9, is the positional head  These heads tend to focus their attention on either the previous or next token in the sequence, thus obtaining representations of the neighborhood of the current time step  In Figure FIGREF47, we show attention plots for such heads, found for each of the studied models  The sparsity of our models allows these heads to be more confident in their representations, by assigning the whole probability distribution to a single token in the sequence  Concretely, we may measure a positional head's confidence as the average attention weight assigned to the previous token  The softmax model has three heads for position $-1$, with median confidence $93 5\%$  The $1 5$-entmax model also has three heads for this position, with median confidence $94 4\%$  The adaptive model has four heads, with median confidences $95 9\%$, the lowest-confidence head being dense with $\alpha =1 18$, while the highest-confidence head being sparse ($\alpha =1 91$) 
For position $+1$, the models each dedicate one head, with confidence around $95\%$, slightly higher for entmax  The adaptive model sets $\alpha =1 96$ for this head 
Analysis ::: Identifying Head Specializations ::: BPE-merging head 
Due to the sparsity of our models, we are able to identify other head specializations, easily identifying which heads should be further analysed  In Figure FIGREF51 we show one such head where the $\alpha $ value is particularly high (in the encoder, layer 1, head 4 depicted in Figure FIGREF37)  We found that this head most often looks at the current time step with high confidence, making it a positional head with offset 0  However, this head often spreads weight sparsely over 2-3 neighboring tokens, when the tokens are part of the same BPE cluster or hyphenated words  As this head is in the first layer, it provides a useful service to the higher layers by combining information evenly within some BPE clusters 
For each BPE cluster or cluster of hyphenated words, we computed a score between 0 and 1 that corresponds to the maximum attention mass assigned by any token to the rest of the tokens inside the cluster in order to quantify the BPE-merging capabilities of these heads  There are not any attention heads in the softmax model that are able to obtain a score over $80\%$, while for $1 5$-entmax and $\alpha $-entmax there are two heads in each ($83 3\%$ and $85 6\%$ for $1 5$-entmax and $88 5\%$ and $89 8\%$ for $\alpha $-entmax) 
Analysis ::: Identifying Head Specializations ::: Interrogation head 
On the other hand, in Figure FIGREF52 we show a head for which our adaptively sparse model chose an $\alpha $ close to 1, making it closer to softmax (also shown in encoder, layer 1, head 3 depicted in Figure FIGREF37)  We observe that this head assigns a high probability to question marks at the end of the sentence in time steps where the current token is interrogative, thus making it an interrogation-detecting head  We also observe this type of heads in the other models, which we also depict in Figure FIGREF52  The average attention weight placed on the question mark when the current token is an interrogative word is $98 5\%$ for softmax, $97 0\%$ for $1 5$-entmax, and $99 5\%$ for $\alpha $-entmax 
Furthermore, we can examine sentences where some tendentially sparse heads become less so, thus identifying sources of ambiguity where the head is less confident in its prediction  An example is shown in Figure FIGREF55 where sparsity in the same head differs for sentences of similar length 
Related Work ::: Sparse attention 
Prior work has developed sparse attention mechanisms, including applications to NMT BIBREF19, BIBREF12, BIBREF20, BIBREF22, BIBREF34  BIBREF14 introduced the entmax function this work builds upon  In their work, there is a single attention mechanism which is controlled by a fixed $\alpha $  In contrast, this is the first work to allow such attention mappings to dynamically adapt their curvature and sparsity, by automatically adjusting the continuous $\alpha $ parameter  We also provide the first results using sparse attention in a Transformer model 
Related Work ::: Fixed sparsity patterns 
Recent research improves the scalability of Transformer-like networks through static, fixed sparsity patterns BIBREF10, BIBREF35  Our adaptively-sparse Transformer can dynamically select a sparsity pattern that finds relevant words regardless of their position (e g , Figure FIGREF52)  Moreover, the two strategies could be combined  In a concurrent line of research, BIBREF11 propose an adaptive attention span for Transformer language models  While their work has each head learn a different contiguous span of context tokens to attend to, our work finds different sparsity patterns in the same span  Interestingly, some of their findings mirror ours – we found that attention heads in the last layers tend to be denser on average when compared to the ones in the first layers, while their work has found that lower layers tend to have a shorter attention span compared to higher layers 
Related Work ::: Transformer interpretability 
The original Transformer paper BIBREF0 shows attention visualizations, from which some speculation can be made of the roles the several attention heads have  BIBREF7 study the syntactic abilities of the Transformer self-attention, while BIBREF6 extract dependency relations from the attention weights  BIBREF8 find that the self-attentions in BERT BIBREF3 follow a sequence of processes that resembles a classical NLP pipeline  Regarding redundancy of heads, BIBREF9 develop a method that is able to prune heads of the multi-head attention module and make an empirical study of the role that each head has in self-attention (positional, syntactic and rare words)  BIBREF36 also aim to reduce head redundancy by adding a regularization term to the loss that maximizes head disagreement and obtain improved results  While not considering Transformer attentions, BIBREF18 show that traditional attention mechanisms do not necessarily improve interpretability since softmax attention is vulnerable to an adversarial attack leading to wildly different model predictions for the same attention weights  Sparse attention may mitigate these issues; however, our work focuses mostly on a more mechanical aspect of interpretation by analyzing head behavior, rather than on explanations for predictions 
Conclusion and Future Work
We contribute a novel strategy for adaptively sparse attention, and, in particular, for adaptively sparse Transformers  We present the first empirical analysis of Transformers with sparse attention mappings (i e , entmax), showing potential in both translation accuracy as well as in model interpretability 
In particular, we analyzed how the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence  Our adaptivity strategy relies only on gradient-based optimization, side-stepping costly per-head hyper-parameter searches  Further speed-ups are possible by leveraging more parallelism in the bisection algorithm for computing $\alpha $-entmax 
Finally, some of the automatically-learned behaviors of our adaptively sparse Transformers – for instance, the near-deterministic positional heads or the subword joining head – may provide new ideas for designing static variations of the Transformer 
Acknowledgments
This work was supported by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contracts UID/EEA/50008/2019 and CMUPERI/TIC/0046/2014 (GoLocal)  We are grateful to Ben Peters for the $\alpha $-entmax code and Erick Fonseca, Marcos Treviso, Pedro Martins, and Tsvetomila Mihaylova for insightful group discussion  We thank Mathieu Blondel for the idea to learn $\alpha $  We would also like to thank the anonymous reviewers for their helpful feedback 
Supplementary Material
Background ::: Regularized Fenchel-Young prediction functions
Definition 1 (BIBREF23)
Let $\Omega \colon \triangle ^d \rightarrow {\mathbb {R}}\cup \lbrace \infty \rbrace $ be a strictly convex regularization function  We define the prediction function $\mathbf {\pi }_{\Omega }$ as
Background ::: Characterizing the @!START@$\alpha $@!END@-entmax mapping
Lemma 1 (BIBREF14) For any $\mathbf {z}$, there exists a unique $\tau ^\star $ such that
Proof: From the definition of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$,
we may easily identify it with a regularized prediction function (Def  UNKREF81):
We first note that for all $\mathbf {p}\in \triangle ^d$,
From the constant invariance and scaling properties of $\mathbf {\pi }_{\Omega }$ BIBREF23,
Using BIBREF23, noting that $g^{\prime }(t) = t^{\alpha - 1}$ and $(g^{\prime })^{-1}(u) = u^{{1}{\alpha -1}}$, yields
Since $\mathsf {H}^{\textsc {T}}_\alpha $ is strictly convex on the simplex, $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ has a unique solution $\mathbf {p}^\star $  Equation DISPLAY_FORM88 implicitly defines a one-to-one mapping between $\mathbf {p}^\star $ and $\tau ^\star $ as long as $\mathbf {p}^\star \in \triangle $, therefore $\tau ^\star $ is also unique 
Background ::: Connections to softmax and sparsemax
The Euclidean projection onto the simplex, sometimes referred to, in the context of neural attention, as sparsemax BIBREF19, is defined as
The solution can be characterized through the unique threshold $\tau $ such that $\sum _i \operatornamewithlimits{\mathsf {sparsemax}}(\mathbf {z})_i = 1$ and BIBREF38
Thus, each coordinate of the sparsemax solution is a piecewise-linear function  Visibly, this expression is recovered when setting $\alpha =2$ in the $\alpha $-entmax expression (Equation DISPLAY_FORM85); for other values of $\alpha $, the exponent induces curvature 
On the other hand, the well-known softmax is usually defined through the expression
which can be shown to be the unique solution of the optimization problem
where $\mathsf {H}^\textsc {S}(\mathbf {p}) -\sum _i p_i \log p_i$ is the Shannon entropy  Indeed, setting the gradient to 0 yields the condition $\log p_i = z_j - \nu _i - \tau - 1$, where $\tau $ and $\nu > 0$ are Lagrange multipliers for the simplex constraints $\sum _i p_i = 1$ and $p_i \ge 0$, respectively  Since the l h s  is only finite for $p_i>0$, we must have $\nu _i=0$ for all $i$, by complementary slackness  Thus, the solution must have the form $p_i = {\exp (z_i)}{Z}$, yielding Equation DISPLAY_FORM92 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@
Recall that the entmax transformation is defined as:
where $\alpha \ge 1$ and $\mathsf {H}^{\textsc {T}}_{\alpha }$ is the Tsallis entropy,
and $\mathsf {H}^\textsc {S}(\mathbf {p}):= -\sum _j p_j \log p_j$ is the Shannon entropy 
In this section, we derive the Jacobian of $\operatornamewithlimits{\mathsf {entmax }}$ with respect to the scalar parameter $\alpha $ 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: General case of @!START@$\alpha >1$@!END@
From the KKT conditions associated with the optimization problem in Eq  DISPLAY_FORM85, we have that the solution $\mathbf {p}^{\star }$ has the following form, coordinate-wise:
where $\tau ^{\star }$ is a scalar Lagrange multiplier that ensures that $\mathbf {p}^{\star }$ normalizes to 1, i e , it is defined implicitly by the condition:
For general values of $\alpha $, Eq  DISPLAY_FORM98 lacks a closed form solution  This makes the computation of the Jacobian
non-trivial  Fortunately, we can use the technique of implicit differentiation to obtain this Jacobian 
The Jacobian exists almost everywhere, and the expressions we derive expressions yield a generalized Jacobian BIBREF37 at any non-differentiable points that may occur for certain ($\alpha $, $\mathbf {z}$) pairs  We begin by noting that $\frac{\partial p_i^{\star }}{\partial \alpha } = 0$ if $p_i^{\star } = 0$, because increasing $\alpha $ keeps sparse coordinates sparse  Therefore we need to worry only about coordinates that are in the support of $\mathbf {p}^\star $  We will assume hereafter that the $i$th coordinate of $\mathbf {p}^\star $ is non-zero  We have:
We can see that this Jacobian depends on $\frac{\partial \tau ^{\star }}{\partial \alpha }$, which we now compute using implicit differentiation 
Let $\mathcal {S} = \lbrace i: p^\star _i > 0 \rbrace $)  By differentiating both sides of Eq  DISPLAY_FORM98, re-using some of the steps in Eq  DISPLAY_FORM101, and recalling Eq  DISPLAY_FORM97, we get
from which we obtain:
Finally, plugging Eq  DISPLAY_FORM103 into Eq  DISPLAY_FORM101, we get:
where we denote by
The distribution $\tilde{\mathbf {p}}(\alpha )$ can be interpreted as a “skewed” distribution obtained from $\mathbf {p}^{\star }$, which appears in the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ w r t  $\mathbf {z}$ as well BIBREF14 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Solving the indetermination for @!START@$\alpha =1$@!END@
We can write Eq  DISPLAY_FORM104 as
When $\alpha \rightarrow 1^+$, we have $\tilde{\mathbf {p}}(\alpha ) \rightarrow \mathbf {p}^{\star }$, which leads to a $\frac{0}{0}$ indetermination 
To solve this indetermination, we will need to apply L'Hôpital's rule twice  Let us first compute the derivative of $\tilde{p}_i(\alpha )$ with respect to $\alpha $  We have
therefore
Differentiating the numerator and denominator in Eq  DISPLAY_FORM107, we get:
with
and
When $\alpha \rightarrow 1^+$, $B$ becomes again a $\frac{0}{0}$ indetermination, which we can solve by applying again L'Hôpital's rule  Differentiating the numerator and denominator in Eq  DISPLAY_FORM112:
Finally, summing Eq  DISPLAY_FORM111 and Eq  DISPLAY_FORM113, we get
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Summary
To sum up, we have the following expression for the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with respect to $\alpha $:
Context 3: Introduction
Natural languages evolve and words have always been subject to semantic change over time BIBREF1  With the rise of large digitized text resources recent NLP technologies have made it possible to capture such change with vector space models BIBREF2, BIBREF3, BIBREF4, BIBREF5, topic models BIBREF6, BIBREF7, BIBREF8, and sense clustering models BIBREF9  However, many approaches for detecting LSC differ profoundly from each other and therefore drawing comparisons between them can be challenging BIBREF10  Not only do architectures for detecting LSC vary, their performance is also often evaluated without access to evaluation data or too sparse data sets  In cases where evaluation data is available, oftentimes LSCD systems are not evaluated on the same data set which hinders the research community to draw comparisons 
For this reason we report the results of the first shared task on unsupervised lexical semantic change detection in German that is based on an annotated data set to guarantee objective reasoning throughout different approaches  The task was organized as part of the seminar 'Lexical Semantic Change Detection' at the IMS Stuttgart in the summer term of 2019 
Task
The goal of the shared task was to create an architecture to detect semantic change and to rank words according to their degree of change between two different time periods  Given two corpora Ca and Cb, the target words had to be ranked according to their degree of lexical semantic change between Ca and Cb as annotated by human judges  A competition was set up on Codalab and teams mostly consisting of 2 people were formed to take part in the task  There was one group consisting of 3 team members and two individuals who entered the task on their own  In total there were 12 LSCD systems participating in the shared task 
The shared task was divided into three phases, i e , development, testing and analysis phase  In the development phase each team implemented a first version of their model based on a trial data set and submitted it subsequently  In the testing phase the testing data was made public and participants applied their models to the test data with a restriction of possible result uploads to 30  The leaderboard was public at all times  Eventually, the analysis phase was entered and the models of the testing phase were evaluated in terms of the predictions they made and parameters could be tuned further  The models and results will be discussed in detail in sections 7 and 8 
Corpora
The task, as framed above, requires to detect the semantic change between two corpora  The two corpora used in the shared task correspond to the diachronic corpus pair from BIBREF0: DTA18 and DTA19  They consist of subparts of DTA corpus BIBREF11 which is a freely available lemmatized, POS-tagged and spelling-normalized diachronic corpus of German containing texts from the 16th to the 20th century  DTA18 contains 26 million sentences published between 1750-1799 and DTA19 40 million between 1850-1899  The corpus version used in the task has the following format: "year [tab] lemma1 lemma2 lemma3    " 
Evaluation
The Diachronic Usage Relatedness (DURel) gold standard data set includes 22 target words and their varying degrees of semantic change BIBREF12  For each of these target words a random sample of use pairs from the DTA corpus was retrieved and annotated  The annotators were required to rate the pairs according to their semantic relatedness on a scale from 1 to 4 (unrelated - identical meanings) for two time periods  The average Spearman's $\rho $ between the five annotators was 0 66 for 1,320 use paris  The resulting word ranking of the DURel data set is determined by the mean usage relatedness across two time periods and is used as the benchmark to compare the models’ performances in the shared task 
Evaluation ::: Metric
The output of a system with the target words in the predicted order is compared to the gold ranking of the DURel data set  As the metric to assess how well the model's output fits the gold ranking Spearman's $\rho $ was used  The higher Spearman's rank-order correlation the better the system's performance 
Evaluation ::: Baselines
Models were compared to two baselines for the shared task:
log-transformed normalized frequency difference (FD)
count vectors with column intersection and cosine distance (CNT + CI + CD)
The window size for CNT + CI + CD was 10  Find more information on these models in BIBREF0 
Participating Systems
Participants mostly rely on the models compared in BIBREF0 and apply modifications to improve them  In particular, most teams make use of skip-gram with negative sampling (SGNS) based on BIBREF13 to learn the semantic spaces of the two time periods and orthogonal procrustes (OP) to align these vector spaces, similar to the approach by BIBREF14  Different meaning representations such as sense clusters are used as well  As measure to detect the degree of LSC all teams except one choose cosine distance (CD)  This team uses Jensen-Shannon distance (JSD) instead, which computes the distance between probability distributions BIBREF15 
The models of each team will be briefly introduced in this section 
Participating Systems ::: sorensbn
Team sorensbn makes use of SGNS + OP + CD to detect LSC  They use similar hyperparameters as in BIBREF0 to tune the SGNS model  They use an open-sourced noise-aware implementation to improve the OP alignment BIBREF16 
Participating Systems ::: tidoe
Team tidoe builds on SGNS + OP + CD, but they add a transformation step to receive binarized representations of matrices BIBREF17  This step is taken to counter the bias that can occur in vector-space models based on frequencies BIBREF18

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does their model improve interpretability compared to softmax transformers?
Context 3: 
Participating Systems ::: in vain
The team applies a model based on SGNS with vector initialization alignment and cosine distance (SGNS + VI + CD)  Vector initialization is an alignment strategy where the vector space learning model for $t_2$ is initialized with the vectors from $t_1$ BIBREF19  Since SGNS + VI + OP does not perform as well as other models in BIBREF0, they alter the vector initialization process by initializing on the complete model instead of only the word matrix of $t_1$ to receive improved results 
Participating Systems ::: Evilly
In line with previous approaches, team Evilly builds upon SGNS + OP + CD  They alter the OP step by using only high-frequency words for alignment 
Participating Systems ::: DAF
Team DAF uses an architecture based on learning vectors with fastText, alignment with unsupervised and supervised variations of OP, and CD, using the MUSE package BIBREF20, BIBREF21  For the supervised alignment stop words are used  The underlying assumption is that stop words serve as functional units of language and their usage should be consistent over time 
Participating Systems ::: SnakesOnAPlane
The team learns vector spaces with count vectors, positive pointwise mutual information (PPMI), SGNS and uses column intersection (CI) and OP as alignment techniques where applicable  Then they compare two distance measures (CD and JSD) for the different models CNT + CI, PPMI + CI and SGNS + OP to identify which measure performs better for these models  They also experiment with different ways to remove negative values from SGNS vectors, which is needed for JSD 
Participating Systems ::: TeamKulkarni15
TeamKulkarni15 uses SGNS + OP + CD with the modification of local alignment with k nearest neighbors, since other models often use global alignment that can be prone to noise BIBREF22 
Participating Systems ::: Bashmaistori
They use word injection (WI) alignment on PPMI vectors with CD  This approach avoids the complex alignment procedure for embeddings and is applicable to embeddings and count-based methods  They compare two implementations of word injection BIBREF23, BIBREF0 as these showed different results on different data sets 
Participating Systems ::: giki
Team giki uses PPMI + CI + CD to detect LSC  They state that a word sense is determined by its context, but relevant context words can also be found outside a predefined window  Therefore, they use tf-idf to select relevant context BIBREF24 
Participating Systems ::: Edu-Phil
Similar to team DAF they also use fastText + OP + CD  Their hypothesis is that fastText may increase the performance for less frequent words in the corpus since generating word embeddings in fasttext is based on character n-grams 
Participating Systems ::: orangefoxes
They use the model by BIBREF5 which is based on SGNS, but avoids alignment by treating time as a vector that may be combined with word vectors to get time-specific word vectors 
Participating Systems ::: Loud Whisper
Loud Whisper base their approach on BIBREF9 which is a graph-based sense clustering model  They process the data set to receive bigrams, create a co-occurence graph representation and after clustering assess the type of change per word by comparing the results against an intersection table  Their motivation is not only to use a graph-based approach, but to extend the approach by enabling change detection for all parts of speech as opposed to the original model 
Results and Discussion
Table TABREF8 shows the results of the shared task  All teams receive better results than baseline 1 (FD), of which a total of 8 teams outperform baseline 2 (CNT + CI + CD)  The 4 top scores with $\rho $ $>$ 0 7 are either modified versions of SGNS + OP + CD or use SGNS + VI + CD  The following 4 scores in the range of 0 5 $<$ $\rho $ $<$ 0 6 are generated by the models fastText + OP + CD, SGNS + OP + CD/JSD, and PPMI + WI + CD 
Contrary to the results by BIBREF0 the modified version of vector initialization shows high performance similar to OP alignment, as previously reported by BIBREF14  Some modifications to the SGNS + OP + CD approach are able to yield better results than others, e g  noise-aware alignment and binarized matrices as compared to frequency-driven OP alignment or local alignment with KNN  Team SnakesOnAPlane compare two distance measures and their results show that JSD ($\rho $ $=$  561) performs minimally worse than CD ($\rho $ $=$  565) as the semantic change measure for their model 
The overall best-performing model is Skip-Gram with orthogonal alignment and cosine distance (SGNS + OP + CD) with similar hyperparameters as in the model architecture described previously BIBREF0  Said architecture was used as the basis for the two best performing models  Team tidoe reports that binarizing matrices leads to a generally worse performance ($\rho $ $=$  811) compared to the unmodified version of SGNS + OP + CD ($\rho $ $=$ 0 9)  The noise aware alignment approach applied by team sorensbn obtains a higher score ($\rho $ $=$  854) compared to the result reported by tidoe, but is unable to exceed the performance of the unmodified SNGS + OP + CD for the same set of hyperparameters (window size = 10, negative sampling = 1; subsampling = None)  Of the 8 scores above the second baseline, 5 use an architecture that builds upon SGNS + OP + CD  Whereas in the lower score segment $\rho $ $<$ 0 5 none of the models use SGNS + OP + CD  These findings are in line with the results reported by BIBREF0, however the overall best results are lower in this shared task, which is expected from the smaller number of parameter combinations explored  Additionally, in the shared task the objective was to report the best score and not to calculate the mean which makes it more difficult to compare the robustness of the models presented here 

ground_truth: the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence

We introduce sparse attention into the Transformer architecture

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the corpus used for the task?
Context 4: 
Ultimate Post Attention (UPA) Model  In this model we attend to the context represented by hidden state of the INLINEFORM0   We use the post prior to the instructor's reply as a query over the contexts INLINEFORM1 to compute attention weights INLINEFORM2 , which are then used to compute the attended context representation INLINEFORM3 (recall again that the intervention text itself is not available for this purpose)  This attention formulation makes an equivalence between the final INLINEFORM4 post and the prospective intervention, using Post INLINEFORM5 as the query for finding the appropriate context INLINEFORM6 , inclusive of itself INLINEFORM7   Said in another way, UPA uses the most recent content in the thread as the attentional query for context 
For example, if post INLINEFORM0 is the instructor's reply, post INLINEFORM1 will query over the contexts INLINEFORM2 and INLINEFORM3   The model schematic is shown in Figure FIGREF12  
The attended context representations are computed as: DISPLAYFORM0
The INLINEFORM0 representation is then passed through a fully connected softmax layer to yield the binary prediction 
Penultimate Post Attention (PPA) Model  While the UPA model uses the most recent text and makes the ultimate post itself available as potential context, our the ultimate post may be better modeled as having any of its prior posts as potential context  Penultimate Post Attention (PPA) variant does this  The schematic and the equations for the PPA model are obtained by summing over contexts INLINEFORM0 in Equation EQREF10 and Figure FIGREF12   While we could properly model such a context inference decision with any post INLINEFORM1 and prospective contexts INLINEFORM2 (where INLINEFORM3 is a random post), it makes sense to use the penultimate post, as we can make the most information available to the model for the context inference 
The attended context representations are computed as: DISPLAYFORM0
Any Post Attention (APA) Model  APA further relaxes both UPA and PPA, allowing APA to generalize and hypothesize that the prospective instructor intervention is based on the context that any previous post INLINEFORM0 replied to  In this model, each post INLINEFORM1 is set as a query to attend to its previous context INLINEFORM2   For example, INLINEFORM3 will attend to INLINEFORM4   Different from standard attention mechanisms, APA attention weights INLINEFORM5 are obtained by normalising interaction matrix over the different queries 
In APA, the attention context INLINEFORM0 is computed via: DISPLAYFORM0
Evaluation
The baseline and the models are evaluated on a corpus of 12 MOOC discussion forums  We train on 80% of the training data and report evaluation results on the held-out 20% of test data  We report INLINEFORM0 scores on the positive class (interventions), in line with prior work  We also argue that recall of the positive class is more important than precision, since it is costlier for instructors to miss intervening on a thread than spending irrelevant time intervening on a less critical threads due to false positives 
Model hyperpameter settings  All proposed and baseline neural models are trained using Adam optimizer with a learning rate of 0 001  We used cross-entropy as loss function  Importantly we updated the model parameters during training after each instance as in vanilla stochastic gradient descent; this setting was practical since data on most courses had only a few hundred instances enabling convergence within a reasonable training time of a few hours (see Table TABREF15 , column 2)  Models were trained for a single epoch as most of our courses with a few hundred thread converged after a single epoch  We used 300-dimensional GloVe vectors and permitted the embeddings to be updated during the model's end-to-end training  The hidden dimension size of both INLINEFORM0 and INLINEFORM1 are set to 128 for all the models 
Baselines  We compare our models against a neural baseline models, hierarchical LSTM (hLSTM), with the attention ablated but with access to the complete context, and a strong, open-sourced feature-rich baseline BIBREF7   We choose BIBREF7 over other prior works such as BIBREF0 since we do not have access to the dataset or the system used in their papers for replication  BIBREF7 is a logistic regression classifier with features inclusive of bag-of-words representation of the unigrams and thread length, normalised counts of agreements to previous posts, counts of non-lexical reference items such as URLs, and the Coursera forum type in which a thread appeared  We also report aggregated results from a hLSTM model with access only to the last post as context for comparison  Table TABREF17 compares the performance of these baselines against our proposed methods 
Results
Table TABREF15 shows performance of all our proposed models and the neural baseline over our 12 MOOC dataset  Our models of UPA, PPA individually better the baseline by 5 and 2% on INLINEFORM0 and 3 and 6% on recall respectively  UPA performs the best in terms of INLINEFORM1 on average while PPA performs the best in terms of recall on average  At the individual course level, however, the results are mixed  UPA performs the best on INLINEFORM2 on 5 out of 12 courses, PPA on 3 out 12 courses, APA 1 out of 12 courses and the baseline hLSTM on 1  PPA performs the best on recall on 7 out of the 12 courses  We also note that course level performance differences correlate with the course size and intervention ratio (hereafter, i ratio), which is the ratio of intervened to non-intervened threads  UPA performs better than PPA and APA on low intervention courses (i ratio INLINEFORM3 0 25) mainly because PPA and APA's performance drops steeply when i ratio drops (see col 2 parenthesis and INLINEFORM4 of PPA and APA)  While all the proposed models beat the baseline on every course except casebased-2  On medicalneuro-2 and compilers-4 which have the lowest i ratio among the 12 courses none of the neural models better the reported baseline BIBREF7 (course level not scores not shown in this paper)  The effect is pronounced in compilers-4 course where none of the neural models were able to predict any intervened threads  This is due to the inherent weakness of standard neural models, which are unable to learn features well enough when faced with sparse data 
The best performance of UPA indicates that the reply context of the instructor's post INLINEFORM0 correlates strongly with that of the previous post INLINEFORM1   This is not surprising since normal conversations are typically structured that way 
Discussion
In order to further understand the models' ability to infer the context and its effect on intervention prediction, we further investigate the following research questions 
RQ1  Does context inference help intervention prediction?
In order to understand if context inference is useful to intervention prediction, we ablate the attention components and experiment with the vanilla hierarchical LSTM model  Row 3 of Table TABREF17 shows the macro averaged result from this experiment  The UPA and PPA attention models better the vanilla hLSTM by 5% and 2% on average in INLINEFORM0 respectively  Recall that the vanilla hLSTM already has access to a context consisting of all posts (from INLINEFORM1 through INLINEFORM2 )  In contrast, the UPA and PPA models selectively infers a context for INLINEFORM3 and INLINEFORM4 posts, respectively, and use it to predict intervention  The improved performance of our attention models that actively select their optimal context, over a model with the complete thread as context, hLSTM, shows that the context inference improves intervention prediction over using the default full context 
RQ2  How well do the models perform across threads of different lengths? To understand the models' prediction performance across threads of different lengths, we bin threads by length and study the models' recall  We choose three courses, ml-5, rprog-3 and calc-1, from our corpus of 12 with the highest number of positive instances ( INLINEFORM0 100 threads)  We limit our analysis to these since binning renders courses with fewer positive instances sparse  Figure FIGREF18 shows performance across thread lengths from 1 through 7 posts and INLINEFORM1 posts  Clearly, the UPA model performs much better on shorter threads than on longer threads while PPA and APA works better on longer threads  Although, UPA is the best performing model in terms of overall INLINEFORM2 its performance drops steeply on threads of length INLINEFORM3   UPA's overall best performance is because most of the interventions in the corpus happen after one post  To highlight the performance of APA we show an example from smac-1 in Figure FIGREF22 with nine posts which was predicted correctly as intervened by APA but not by other models  Threads shows students confused over a missing figure in a homework  The instructor finally shows up, though late, to resolve the confusion 
RQ3  Do models trained with different context lengths perform better than when trained on a single context length?
We find that context length has a regularising effect on the model's performance at test time  This is not surprising since models trained with threads of single context length will not generalise to infer different context lengths  Row 4 of Table TABREF17 shows a steep performance drop in training by classifier with all threads truncated to a context of just one post, INLINEFORM0 , the post immediately preceding the intervened post  We also conducted an experiment with a multi-objective loss function with an additive cross-entropy term where each term computes loss from a model with context limited to a length of 3  We chose 3 since intervened threads in all the courses had a median length between 3 and 4  We achieved an INLINEFORM1 of 0 45 with a precision of 0 47 and recall of 0 43  This achieves a performance comparable to that of the BIBREF7 with context length set to only to 3  This approach of using infinitely many loss terms for each context length from 1 through the maximum thread length in a course is naive and not practical  We only use this model to show the importance of training the model with loss from threads of different lengths to prevent models overfitting to threads of specific context lengths 
Conclusion
We predict instructor intervention on student discussions by first inferring the optimal size of the context needed to decide on the intervention decision for the intervened post  We first show that a structured representation of the complete thread as the context is better than a bag-of-words, feature-rich representation  We then propose attention-based models to infer and select a context – defined as a contiguous subsequence of student posts – to improve over a model that always takes the complete thread as a context to prediction intervention  Our Any Post Attention (APA) model enables instructors to tune the model to predict intervention early or late  We posit our APA model will enable MOOC instructors employing varying pedagogical styles to use the model equally well  We introspect the attention models' performance across threads of varying lengths and show that APA predicts intervention on longer threads, which possesses more candidate contexts, better 
We note that the recall of the predictive models for longer threads (that is, threads of length greater 2) can still be improved  Models perform differently between shorter and longer length  An ensemble model or a multi-objective loss function is thus planned in our future work to better prediction on such longer threads 

ground_truth: DTA18, DTA19

Diachronic Usage Relatedness (DURel) gold standard data set