prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What tasks are used for evaluation?
Context 1: Introduction
The Transformer architecture BIBREF0 for deep neural networks has quickly risen to prominence in NLP through its efficiency and performance, leading to improvements in the state of the art of Neural Machine Translation BIBREF1, BIBREF2, as well as inspiring other powerful general-purpose models like BERT BIBREF3 and GPT-2 BIBREF4  At the heart of the Transformer lie multi-head attention mechanisms: each word is represented by multiple different weighted averages of its relevant context  As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9 
The attention distribution of each head is predicted typically using the softmax normalizing transform  As a result, all context words have non-zero attention weight  Recent work on single attention architectures suggest that using sparse normalizing transforms in attention mechanisms such as sparsemax – which can yield exactly zero probabilities for irrelevant words – may improve performance and interpretability BIBREF12, BIBREF13, BIBREF14  Qualitative analysis of attention heads BIBREF0 suggests that, depending on what phenomena they capture, heads tend to favor flatter or more peaked distributions 
Recent works have proposed sparse Transformers BIBREF10 and adaptive span Transformers BIBREF11  However, the “sparsity" of those models only limits the attention to a contiguous span of past tokens, while in this work we propose a highly adaptive Transformer model that is capable of attending to a sparse set of words that are not necessarily contiguous  Figure FIGREF1 shows the relationship of these methods with ours 
Our contributions are the following:
We introduce sparse attention into the Transformer architecture, showing that it eases interpretability and leads to slight accuracy gains 
We propose an adaptive version of sparse attention, where the shape of each attention head is learnable and can vary continuously and dynamically between the dense limit case of softmax and the sparse, piecewise-linear sparsemax case 
We make an extensive analysis of the added interpretability of these models, identifying both crisper examples of attention head behavior observed in previous work, as well as novel behaviors unraveled thanks to the sparsity and adaptivity of our proposed model 
Background ::: The Transformer
In NMT, the Transformer BIBREF0 is a sequence-to-sequence (seq2seq) model which maps an input sequence to an output sequence through hierarchical multi-head attention mechanisms, yielding a dynamic, context-dependent strategy for propagating information within and across sentences  It contrasts with previous seq2seq models, which usually rely either on costly gated recurrent operations BIBREF15, BIBREF16 or static convolutions BIBREF17 
Given $n$ query contexts and $m$ sequence items under consideration, attention mechanisms compute, for each query, a weighted representation of the items  The particular attention mechanism used in BIBREF0 is called scaled dot-product attention, and it is computed in the following way:
where $\mathbf {Q} \in \mathbb {R}^{n \times d}$ contains representations of the queries, $\mathbf {K}, \mathbf {V} \in \mathbb {R}^{m \times d}$ are the keys and values of the items attended over, and $d$ is the dimensionality of these representations  The $\mathbf {\pi }$ mapping normalizes row-wise using softmax, $\mathbf {\pi }(\mathbf {Z})_{ij} = \operatornamewithlimits{\mathsf {softmax}}(\mathbf {z}_i)_j$, where
In words, the keys are used to compute a relevance score between each item and query  Then, normalized attention weights are computed using softmax, and these are used to weight the values of each item at each query context 
However, for complex tasks, different parts of a sequence may be relevant in different ways, motivating multi-head attention in Transformers  This is simply the application of Equation DISPLAY_FORM7 in parallel $H$ times, each with a different, learned linear transformation that allows specialization:
In the Transformer, there are three separate multi-head attention mechanisms for distinct purposes:
Encoder self-attention: builds rich, layered representations of each input word, by attending on the entire input sentence 
Context attention: selects a representative weighted average of the encodings of the input words, at each time step of the decoder 
Decoder self-attention: attends over the partial output sentence fragment produced so far 
Together, these mechanisms enable the contextualized flow of information between the input sentence and the sequential decoder 
Background ::: Sparse Attention
The softmax mapping (Equation DISPLAY_FORM8) is elementwise proportional to $\exp $, therefore it can never assign a weight of exactly zero  Thus, unnecessary items are still taken into consideration to some extent  Since its output sums to one, this invariably means less weight is assigned to the relevant items, potentially harming performance and interpretability BIBREF18  This has motivated a line of research on learning networks with sparse mappings BIBREF19, BIBREF20, BIBREF21, BIBREF22  We focus on a recently-introduced flexible family of transformations, $\alpha $-entmax BIBREF23, BIBREF14, defined as:
where $\triangle ^d \lbrace \mathbf {p}\in \mathbb {R}^d:\sum _{i} p_i = 1\rbrace $ is the probability simplex, and, for $\alpha \ge 1$, $\mathsf {H}^{\textsc {T}}_\alpha $ is the Tsallis continuous family of entropies BIBREF24:
This family contains the well-known Shannon and Gini entropies, corresponding to the cases $\alpha =1$ and $\alpha =2$, respectively 
Equation DISPLAY_FORM14 involves a convex optimization subproblem  Using the definition of $\mathsf {H}^{\textsc {T}}_\alpha $, the optimality conditions may be used to derive the following form for the solution (Appendix SECREF83):
where $[\cdot ]_+$ is the positive part (ReLU) function, $\mathbf {1}$ denotes the vector of all ones, and $\tau $ – which acts like a threshold – is the Lagrange multiplier corresponding to the $\sum _i p_i=1$ constraint 
Background ::: Sparse Attention ::: Properties of @!START@$\alpha $@!END@-entmax 
The appeal of $\alpha $-entmax for attention rests on the following properties  For $\alpha =1$ (i e , when $\mathsf {H}^{\textsc {T}}_\alpha $ becomes the Shannon entropy), it exactly recovers the softmax mapping (We provide a short derivation in Appendix SECREF89 )  For all $\alpha >1$ it permits sparse solutions, in stark contrast to softmax  In particular, for $\alpha =2$, it recovers the sparsemax mapping BIBREF19, which is piecewise linear  In-between, as $\alpha $ increases, the mapping continuously gets sparser as its curvature changes 
To compute the value of $\alpha $-entmax, one must find the threshold $\tau $ such that the r h s  in Equation DISPLAY_FORM16 sums to one  BIBREF23 propose a general bisection algorithm  BIBREF14 introduce a faster, exact algorithm for $\alpha =1 5$, and enable using $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with fixed $\alpha $ within a neural network by showing that the $\alpha $-entmax Jacobian w r t  $\mathbf {z}$ for $\mathbf {p}^\star = \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ is
Our work furthers the study of $\alpha $-entmax by providing a derivation of the Jacobian w r t  the hyper-parameter $\alpha $ (Section SECREF3), thereby allowing the shape and sparsity of the mapping to be learned automatically  This is particularly appealing in the context of multi-head attention mechanisms, where we shall show in Section SECREF35 that different heads tend to learn different sparsity behaviors 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax
We now propose a novel Transformer architecture wherein we simply replace softmax with $\alpha $-entmax in the attention heads  Concretely, we replace the row normalization $\mathbf {\pi }$ in Equation DISPLAY_FORM7 by
This change leads to sparse attention weights, as long as $\alpha >1$; in particular, $\alpha =1 5$ is a sensible starting point BIBREF14 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Different @!START@$\alpha $@!END@ per head 
Unlike LSTM-based seq2seq models, where $\alpha $ can be more easily tuned by grid search, in a Transformer, there are many attention heads in multiple layers  Crucial to the power of such models, the different heads capture different linguistic phenomena, some of them isolating important words, others spreading out attention across phrases BIBREF0  This motivates using different, adaptive $\alpha $ values for each attention head, such that some heads may learn to be sparser, and others may become closer to softmax  We propose doing so by treating the $\alpha $ values as neural network parameters, optimized via stochastic gradients along with the other weights 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Derivatives w r t  @!START@$\alpha $@!END@ 
In order to optimize $\alpha $ automatically via gradient methods, we must compute the Jacobian of the entmax output w r t  $\alpha $  Since entmax is defined through an optimization problem, this is non-trivial and cannot be simply handled through automatic differentiation; it falls within the domain of argmin differentiation, an active research topic in optimization BIBREF25, BIBREF26 
One of our key contributions is the derivation of a closed-form expression for this Jacobian  The next proposition provides such an expression, enabling entmax layers with adaptive $\alpha $  To the best of our knowledge, ours is the first neural network module that can automatically, continuously vary in shape away from softmax and toward sparse mappings like sparsemax 
Proposition 1 Let $\mathbf {p}^\star \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ be the solution of Equation DISPLAY_FORM14  Denote the distribution $\tilde{p}_i {(p_i^\star )^{2 - \alpha }}{ \sum _j(p_j^\star )^{2-\alpha }}$ and let $h_i -p^\star _i \log p^\star _i$  The $i$th component of the Jacobian $\mathbf {g} \frac{\partial \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})}{\partial \alpha }$ is
proof uses implicit function differentiation and is given in Appendix SECREF10 
Proposition UNKREF22 provides the remaining missing piece needed for training adaptively sparse Transformers  In the following section, we evaluate this strategy on neural machine translation, and analyze the behavior of the learned attention heads 
Experiments
We apply our adaptively sparse Transformers on four machine translation tasks  For comparison, a natural baseline is the standard Transformer architecture using the softmax transform in its multi-head attention mechanisms  We consider two other model variants in our experiments that make use of different normalizing transformations:
1 5-entmax: a Transformer with sparse entmax attention with fixed $\alpha =1 5$ for all heads  This is a novel model, since 1 5-entmax had only been proposed for RNN-based NMT models BIBREF14, but never in Transformers, where attention modules are not just one single component of the seq2seq model but rather an integral part of all of the model components 
$\alpha $-entmax: an adaptive Transformer with sparse entmax attention with a different, learned $\alpha _{i,j}^t$ for each head 
The adaptive model has an additional scalar parameter per attention head per layer for each of the three attention mechanisms (encoder self-attention, context attention, and decoder self-attention), i e ,
and we set $\alpha _{i,j}^t = 1 + \operatornamewithlimits{\mathsf {sigmoid}}(a_{i,j}^t) \in ]1, 2[$  All or some of the $\alpha $ values can be tied if desired, but we keep them independent for analysis purposes 
Experiments ::: Datasets 
Our models were trained on 4 machine translation datasets of different training sizes:
[itemsep= 5ex,leftmargin=2ex]
IWSLT 2017 German $\rightarrow $ English BIBREF27: 200K sentence pairs 
KFTT Japanese $\rightarrow $ English BIBREF28: 300K sentence pairs 
WMT 2016 Romanian $\rightarrow $ English BIBREF29: 600K sentence pairs 
WMT 2014 English $\rightarrow $ German BIBREF30: 4 5M sentence pairs 
All of these datasets were preprocessed with byte-pair encoding BIBREF31, using joint segmentations of 32k merge operations 
Experiments ::: Training 
We follow the dimensions of the Transformer-Base model of BIBREF0: The number of layers is $L=6$ and number of heads is $H=8$ in the encoder self-attention, the context attention, and the decoder self-attention  We use a mini-batch size of 8192 tokens and warm up the learning rate linearly until 20k steps, after which it decays according to an inverse square root schedule  All models were trained until convergence of validation accuracy, and evaluation was done at each 10k steps for ro$\rightarrow $en and en$\rightarrow $de and at each 5k steps for de$\rightarrow $en and ja$\rightarrow $en  The end-to-end computational overhead of our methods, when compared to standard softmax, is relatively small; in training tokens per second, the models using $\alpha $-entmax and $1 5$-entmax are, respectively, $75\%$ and $90\%$ the speed of the softmax model 
Experiments ::: Results 
We report test set tokenized BLEU BIBREF32 results in Table TABREF27  We can see that replacing softmax by entmax does not hurt performance in any of the datasets; indeed, sparse attention Transformers tend to have slightly higher BLEU, but their sparsity leads to a better potential for analysis  In the next section, we make use of this potential by exploring the learned internal mechanics of the self-attention heads 
Analysis
We conduct an analysis for the higher-resource dataset WMT 2014 English $\rightarrow $ German of the attention in the sparse adaptive Transformer model ($\alpha $-entmax) at multiple levels: we analyze high-level statistics as well as individual head behavior  Moreover, we make a qualitative analysis of the interpretability capabilities of our models 
Analysis ::: High-Level Statistics ::: What kind of @!START@$\alpha $@!END@ values are learned?
Figure FIGREF37 shows the learning trajectories of the $\alpha $ parameters of a selected subset of heads  We generally observe a tendency for the randomly-initialized $\alpha $ parameters to decrease initially, suggesting that softmax-like behavior may be preferable while the model is still very uncertain  After around one thousand steps, some heads change direction and become sparser, perhaps as they become more confident and specialized  This shows that the initialization of $\alpha $ does not predetermine its sparsity level or the role the head will have throughout  In particular, head 8 in the encoder self-attention layer 2 first drops to around $\alpha =1 3$ before becoming one of the sparsest heads, with $\alpha \approx 2$ 
The overall distribution of $\alpha $ values at convergence can be seen in Figure FIGREF38  We can observe that the encoder self-attention blocks learn to concentrate the $\alpha $ values in two modes: a very sparse one around $\alpha \rightarrow 2$, and a dense one between softmax and 1 5-entmax   However, the decoder self and context attention only learn to distribute these parameters in a single mode  We show next that this is reflected in the average density of attention weight vectors as well 
Analysis ::: High-Level Statistics ::: Attention weight density when translating 
For any $\alpha >1$, it would still be possible for the weight matrices in Equation DISPLAY_FORM9 to learn re-scalings so as to make attention sparser or denser  To visualize the impact of adaptive $\alpha $ values, we compare the empirical attention weight density (the average number of tokens receiving non-zero attention) within each module, against sparse Transformers with fixed $\alpha =1 5$ 
Figure FIGREF40 shows that, with fixed $\alpha =1 5$, heads tend to be sparse and similarly-distributed in all three attention modules  With learned $\alpha $, there are two notable changes: (i) a prominent mode corresponding to fully dense probabilities, showing that our models learn to combine sparse and dense attention, and (ii) a distinction between the encoder self-attention – whose background distribution tends toward extreme sparsity – and the other two modules, who exhibit more uniform background distributions  This suggests that perhaps entirely sparse Transformers are suboptimal 
The fact that the decoder seems to prefer denser attention distributions might be attributed to it being auto-regressive, only having access to past tokens and not the full sentence  We speculate that it might lose too much information if it assigned weights of zero to too many tokens in the self-attention, since there are fewer tokens to attend to in the first place 
Teasing this down into separate layers, Figure FIGREF41 shows the average (sorted) density of each head for each layer  We observe that $\alpha $-entmax is able to learn different sparsity patterns at each layer, leading to more variance in individual head behavior, to clearly-identified dense and sparse heads, and overall to different tendencies compared to the fixed case of $\alpha =1 5$ 
Analysis ::: High-Level Statistics ::: Head diversity 
To measure the overall disagreement between attention heads, as a measure of head diversity, we use the following generalization of the Jensen-Shannon divergence:
where $\mathbf {p}_j$ is the vector of attention weights assigned by head $j$ to each word in the sequence, and $\mathsf {H}^\textsc {S}$ is the Shannon entropy, base-adjusted based on the dimension of $\mathbf {p}$ such that $JS \le 1$  We average this measure over the entire validation set  The higher this metric is, the more the heads are taking different roles in the model 
Figure FIGREF44 shows that both sparse Transformer variants show more diversity than the traditional softmax one  Interestingly, diversity seems to peak in the middle layers of the encoder self-attention and context attention, while this is not the case for the decoder self-attention 
The statistics shown in this section can be found for the other language pairs in Appendix SECREF8 
Analysis ::: Identifying Head Specializations
Previous work pointed out some specific roles played by different heads in the softmax Transformer model BIBREF33, BIBREF5, BIBREF9  Identifying the specialization of a head can be done by observing the type of tokens or sequences that the head often assigns most of its attention weight; this is facilitated by sparsity 
Analysis ::: Identifying Head Specializations ::: Positional heads 
One particular type of head, as noted by BIBREF9, is the positional head  These heads tend to focus their attention on either the previous or next token in the sequence, thus obtaining representations of the neighborhood of the current time step  In Figure FIGREF47, we show attention plots for such heads, found for each of the studied models  The sparsity of our models allows these heads to be more confident in their representations, by assigning the whole probability distribution to a single token in the sequence  Concretely, we may measure a positional head's confidence as the average attention weight assigned to the previous token  The softmax model has three heads for position $-1$, with median confidence $93 5\%$  The $1 5$-entmax model also has three heads for this position, with median confidence $94 4\%$  The adaptive model has four heads, with median confidences $95 9\%$, the lowest-confidence head being dense with $\alpha =1 18$, while the highest-confidence head being sparse ($\alpha =1 91$) 
For position $+1$, the models each dedicate one head, with confidence around $95\%$, slightly higher for entmax  The adaptive model sets $\alpha =1 96$ for this head 
Analysis ::: Identifying Head Specializations ::: BPE-merging head 
Due to the sparsity of our models, we are able to identify other head specializations, easily identifying which heads should be further analysed  In Figure FIGREF51 we show one such head where the $\alpha $ value is particularly high (in the encoder, layer 1, head 4 depicted in Figure FIGREF37)  We found that this head most often looks at the current time step with high confidence, making it a positional head with offset 0  However, this head often spreads weight sparsely over 2-3 neighboring tokens, when the tokens are part of the same BPE cluster or hyphenated words  As this head is in the first layer, it provides a useful service to the higher layers by combining information evenly within some BPE clusters 
For each BPE cluster or cluster of hyphenated words, we computed a score between 0 and 1 that corresponds to the maximum attention mass assigned by any token to the rest of the tokens inside the cluster in order to quantify the BPE-merging capabilities of these heads  There are not any attention heads in the softmax model that are able to obtain a score over $80\%$, while for $1 5$-entmax and $\alpha $-entmax there are two heads in each ($83 3\%$ and $85 6\%$ for $1 5$-entmax and $88 5\%$ and $89 8\%$ for $\alpha $-entmax) 
Analysis ::: Identifying Head Specializations ::: Interrogation head 
On the other hand, in Figure FIGREF52 we show a head for which our adaptively sparse model chose an $\alpha $ close to 1, making it closer to softmax (also shown in encoder, layer 1, head 3 depicted in Figure FIGREF37)  We observe that this head assigns a high probability to question marks at the end of the sentence in time steps where the current token is interrogative, thus making it an interrogation-detecting head  We also observe this type of heads in the other models, which we also depict in Figure FIGREF52  The average attention weight placed on the question mark when the current token is an interrogative word is $98 5\%$ for softmax, $97 0\%$ for $1 5$-entmax, and $99 5\%$ for $\alpha $-entmax 
Furthermore, we can examine sentences where some tendentially sparse heads become less so, thus identifying sources of ambiguity where the head is less confident in its prediction  An example is shown in Figure FIGREF55 where sparsity in the same head differs for sentences of similar length 
Related Work ::: Sparse attention 
Prior work has developed sparse attention mechanisms, including applications to NMT BIBREF19, BIBREF12, BIBREF20, BIBREF22, BIBREF34  BIBREF14 introduced the entmax function this work builds upon  In their work, there is a single attention mechanism which is controlled by a fixed $\alpha $  In contrast, this is the first work to allow such attention mappings to dynamically adapt their curvature and sparsity, by automatically adjusting the continuous $\alpha $ parameter  We also provide the first results using sparse attention in a Transformer model 
Related Work ::: Fixed sparsity patterns 
Recent research improves the scalability of Transformer-like networks through static, fixed sparsity patterns BIBREF10, BIBREF35  Our adaptively-sparse Transformer can dynamically select a sparsity pattern that finds relevant words regardless of their position (e g , Figure FIGREF52)  Moreover, the two strategies could be combined  In a concurrent line of research, BIBREF11 propose an adaptive attention span for Transformer language models  While their work has each head learn a different contiguous span of context tokens to attend to, our work finds different sparsity patterns in the same span  Interestingly, some of their findings mirror ours – we found that attention heads in the last layers tend to be denser on average when compared to the ones in the first layers, while their work has found that lower layers tend to have a shorter attention span compared to higher layers 
Related Work ::: Transformer interpretability 
The original Transformer paper BIBREF0 shows attention visualizations, from which some speculation can be made of the roles the several attention heads have  BIBREF7 study the syntactic abilities of the Transformer self-attention, while BIBREF6 extract dependency relations from the attention weights  BIBREF8 find that the self-attentions in BERT BIBREF3 follow a sequence of processes that resembles a classical NLP pipeline  Regarding redundancy of heads, BIBREF9 develop a method that is able to prune heads of the multi-head attention module and make an empirical study of the role that each head has in self-attention (positional, syntactic and rare words)  BIBREF36 also aim to reduce head redundancy by adding a regularization term to the loss that maximizes head disagreement and obtain improved results  While not considering Transformer attentions, BIBREF18 show that traditional attention mechanisms do not necessarily improve interpretability since softmax attention is vulnerable to an adversarial attack leading to wildly different model predictions for the same attention weights  Sparse attention may mitigate these issues; however, our work focuses mostly on a more mechanical aspect of interpretation by analyzing head behavior, rather than on explanations for predictions 
Conclusion and Future Work
We contribute a novel strategy for adaptively sparse attention, and, in particular, for adaptively sparse Transformers  We present the first empirical analysis of Transformers with sparse attention mappings (i e , entmax), showing potential in both translation accuracy as well as in model interpretability 
In particular, we analyzed how the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence  Our adaptivity strategy relies only on gradient-based optimization, side-stepping costly per-head hyper-parameter searches  Further speed-ups are possible by leveraging more parallelism in the bisection algorithm for computing $\alpha $-entmax 
Finally, some of the automatically-learned behaviors of our adaptively sparse Transformers – for instance, the near-deterministic positional heads or the subword joining head – may provide new ideas for designing static variations of the Transformer 
Acknowledgments
This work was supported by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contracts UID/EEA/50008/2019 and CMUPERI/TIC/0046/2014 (GoLocal)  We are grateful to Ben Peters for the $\alpha $-entmax code and Erick Fonseca, Marcos Treviso, Pedro Martins, and Tsvetomila Mihaylova for insightful group discussion  We thank Mathieu Blondel for the idea to learn $\alpha $  We would also like to thank the anonymous reviewers for their helpful feedback 
Supplementary Material
Background ::: Regularized Fenchel-Young prediction functions
Definition 1 (BIBREF23)
Let $\Omega \colon \triangle ^d \rightarrow {\mathbb {R}}\cup \lbrace \infty \rbrace $ be a strictly convex regularization function  We define the prediction function $\mathbf {\pi }_{\Omega }$ as
Background ::: Characterizing the @!START@$\alpha $@!END@-entmax mapping
Lemma 1 (BIBREF14) For any $\mathbf {z}$, there exists a unique $\tau ^\star $ such that
Proof: From the definition of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$,
we may easily identify it with a regularized prediction function (Def  UNKREF81):
We first note that for all $\mathbf {p}\in \triangle ^d$,
From the constant invariance and scaling properties of $\mathbf {\pi }_{\Omega }$ BIBREF23,
Using BIBREF23, noting that $g^{\prime }(t) = t^{\alpha - 1}$ and $(g^{\prime })^{-1}(u) = u^{{1}{\alpha -1}}$, yields
Since $\mathsf {H}^{\textsc {T}}_\alpha $ is strictly convex on the simplex, $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ has a unique solution $\mathbf {p}^\star $  Equation DISPLAY_FORM88 implicitly defines a one-to-one mapping between $\mathbf {p}^\star $ and $\tau ^\star $ as long as $\mathbf {p}^\star \in \triangle $, therefore $\tau ^\star $ is also unique 
Background ::: Connections to softmax and sparsemax
The Euclidean projection onto the simplex, sometimes referred to, in the context of neural attention, as sparsemax BIBREF19, is defined as
The solution can be characterized through the unique threshold $\tau $ such that $\sum _i \operatornamewithlimits{\mathsf {sparsemax}}(\mathbf {z})_i = 1$ and BIBREF38
Thus, each coordinate of the sparsemax solution is a piecewise-linear function  Visibly, this expression is recovered when setting $\alpha =2$ in the $\alpha $-entmax expression (Equation DISPLAY_FORM85); for other values of $\alpha $, the exponent induces curvature 
On the other hand, the well-known softmax is usually defined through the expression
which can be shown to be the unique solution of the optimization problem
where $\mathsf {H}^\textsc {S}(\mathbf {p}) -\sum _i p_i \log p_i$ is the Shannon entropy  Indeed, setting the gradient to 0 yields the condition $\log p_i = z_j - \nu _i - \tau - 1$, where $\tau $ and $\nu > 0$ are Lagrange multipliers for the simplex constraints $\sum _i p_i = 1$ and $p_i \ge 0$, respectively  Since the l h s  is only finite for $p_i>0$, we must have $\nu _i=0$ for all $i$, by complementary slackness  Thus, the solution must have the form $p_i = {\exp (z_i)}{Z}$, yielding Equation DISPLAY_FORM92 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@
Recall that the entmax transformation is defined as:
where $\alpha \ge 1$ and $\mathsf {H}^{\textsc {T}}_{\alpha }$ is the Tsallis entropy,
and $\mathsf {H}^\textsc {S}(\mathbf {p}):= -\sum _j p_j \log p_j$ is the Shannon entropy 
In this section, we derive the Jacobian of $\operatornamewithlimits{\mathsf {entmax }}$ with respect to the scalar parameter $\alpha $ 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: General case of @!START@$\alpha >1$@!END@
From the KKT conditions associated with the optimization problem in Eq  DISPLAY_FORM85, we have that the solution $\mathbf {p}^{\star }$ has the following form, coordinate-wise:
where $\tau ^{\star }$ is a scalar Lagrange multiplier that ensures that $\mathbf {p}^{\star }$ normalizes to 1, i e , it is defined implicitly by the condition:
For general values of $\alpha $, Eq  DISPLAY_FORM98 lacks a closed form solution  This makes the computation of the Jacobian
non-trivial  Fortunately, we can use the technique of implicit differentiation to obtain this Jacobian 
The Jacobian exists almost everywhere, and the expressions we derive expressions yield a generalized Jacobian BIBREF37 at any non-differentiable points that may occur for certain ($\alpha $, $\mathbf {z}$) pairs  We begin by noting that $\frac{\partial p_i^{\star }}{\partial \alpha } = 0$ if $p_i^{\star } = 0$, because increasing $\alpha $ keeps sparse coordinates sparse  Therefore we need to worry only about coordinates that are in the support of $\mathbf {p}^\star $  We will assume hereafter that the $i$th coordinate of $\mathbf {p}^\star $ is non-zero  We have:
We can see that this Jacobian depends on $\frac{\partial \tau ^{\star }}{\partial \alpha }$, which we now compute using implicit differentiation 
Let $\mathcal {S} = \lbrace i: p^\star _i > 0 \rbrace $)  By differentiating both sides of Eq  DISPLAY_FORM98, re-using some of the steps in Eq  DISPLAY_FORM101, and recalling Eq  DISPLAY_FORM97, we get
from which we obtain:
Finally, plugging Eq  DISPLAY_FORM103 into Eq  DISPLAY_FORM101, we get:
where we denote by
The distribution $\tilde{\mathbf {p}}(\alpha )$ can be interpreted as a “skewed” distribution obtained from $\mathbf {p}^{\star }$, which appears in the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ w r t  $\mathbf {z}$ as well BIBREF14 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Solving the indetermination for @!START@$\alpha =1$@!END@
We can write Eq  DISPLAY_FORM104 as
When $\alpha \rightarrow 1^+$, we have $\tilde{\mathbf {p}}(\alpha ) \rightarrow \mathbf {p}^{\star }$, which leads to a $\frac{0}{0}$ indetermination 
To solve this indetermination, we will need to apply L'Hôpital's rule twice  Let us first compute the derivative of $\tilde{p}_i(\alpha )$ with respect to $\alpha $  We have
therefore
Differentiating the numerator and denominator in Eq  DISPLAY_FORM107, we get:
with
and
When $\alpha \rightarrow 1^+$, $B$ becomes again a $\frac{0}{0}$ indetermination, which we can solve by applying again L'Hôpital's rule  Differentiating the numerator and denominator in Eq  DISPLAY_FORM112:
Finally, summing Eq  DISPLAY_FORM111 and Eq  DISPLAY_FORM113, we get
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Summary
To sum up, we have the following expression for the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with respect to $\alpha $:
Context 2: Introduction
There exists a class of language construction known as pun in natural language texts and utterances, where a certain word or other lexical items are used to exploit two or more separate meanings  It has been shown that understanding of puns is an important research question with various real-world applications, such as human-computer interaction BIBREF0 , BIBREF1 and machine translation BIBREF2   Recently, many researchers show their interests in studying puns, like detecting pun sentences BIBREF3 , locating puns in the text BIBREF4 , interpreting pun sentences BIBREF5 and generating sentences containing puns BIBREF6 , BIBREF7 , BIBREF8   A pun is a wordplay in which a certain word suggests two or more meanings by exploiting polysemy, homonymy, or phonological similarity to another sign, for an intended humorous or rhetorical effect  Puns can be generally categorized into two groups, namely heterographic puns (where the pun and its latent target are phonologically similar) and homographic puns (where the two meanings of the pun reflect its two distinct senses) BIBREF9   Consider the following two examples:
The first punning joke exploits the sound similarity between the word “propane" and the latent target “profane", which can be categorized into the group of heterographic puns  Another categorization of English puns is homographic pun, exemplified by the second instance leveraging distinct senses of the word “gut" 
Pun detection is the task of detecting whether there is a pun residing in the given text  The goal of pun location is to find the exact word appearing in the text that implies more than one meanings  Most previous work addresses such two tasks separately and develop separate systems BIBREF10 , BIBREF5   Typically, a system for pun detection is built to make a binary prediction on whether a sentence contains a pun or not, where all instances (with or without puns) are taken into account during training  For the task of pun location, a separate system is used to make a single prediction as to which word in the given sentence in the text that trigger more than one semantic interpretations of the text, where the training data involves only sentences that contain a pun  Therefore, if one is interested in solving both problems at the same time, a pipeline approach that performs pun detection followed by pun location can be used 
Compared to the pipeline methods, joint learning has been shown effective BIBREF11 , BIBREF12 since it is able to reduce error propagation and allows information exchange between tasks which is potentially beneficial to all the tasks  In this work, we demonstrate that the detection and location of puns can be jointly addressed by a single model  The pun detection and location tasks can be combined as a sequence labeling problem, which allows us to jointly detect and locate a pun in a sentence by assigning each word a tag  Since each context contains a maximum of one pun BIBREF9 , we design a novel tagging scheme to capture this structural constraint  Statistics on the corpora also show that a pun tends to appear in the second half of a context  To capture such a structural property, we also incorporate word position knowledge into our structured prediction model  Experiments on the benchmark datasets show that detection and location tasks can reinforce each other, leading to new state-of-the-art performance on these two tasks  To the best of our knowledge, this is the first work that performs joint detection and location of English puns by using a sequence labeling approach 
Problem Definition
We first design a simple tagging scheme consisting of two tags { INLINEFORM0 }:
INLINEFORM0 tag means the current word is not a pun 
INLINEFORM0 tag means the current word is a pun 
If the tag sequence of a sentence contains a INLINEFORM0 tag, then the text contains a pun and the word corresponding to INLINEFORM1 is the pun 
The contexts have the characteristic that each context contains a maximum of one pun BIBREF9   In other words, there exists only one pun if the given sentence is detected as the one containing a pun  Otherwise, there is no pun residing in the text  To capture this interesting property, we propose a new tagging scheme consisting of three tags, namely { INLINEFORM0 } 
INLINEFORM0 tag indicates that the current word appears before the pun in the given context 
INLINEFORM0 tag highlights the current word is a pun 
INLINEFORM0 tag indicates that the current word appears after the pun 
We empirically show that the INLINEFORM0 scheme can guarantee the context property that there exists a maximum of one pun residing in the text 
Given a context from the training set, we will be able to generate its corresponding gold tag sequence using a deterministic procedure  Under the two schemes, if a sentence does not contain any puns, all words will be tagged with INLINEFORM0 or INLINEFORM1 , respectively  Exemplified by the second sentence “Some diets cause a gut reaction," the pun is given as “gut " Thus, under the INLINEFORM2 scheme, it should be tagged with INLINEFORM3 , while the words before it are assigned with the tag INLINEFORM4 and words after it are with INLINEFORM5 , as illustrated in Figure FIGREF8   Likewise, the INLINEFORM6 scheme tags the word “gut" with INLINEFORM7 , while other words are tagged with INLINEFORM8   Therefore, we can combine the pun detection and location tasks into one problem which can be solved by the sequence labeling approach 
Model
Neural models have shown their effectiveness on sequence labeling tasks BIBREF13 , BIBREF14 , BIBREF15   In this work, we adopt the bidirectional Long Short Term Memory (BiLSTM) BIBREF16 networks on top of the Conditional Random Fields BIBREF17 (CRF) architecture to make labeling decisions, which is one of the classical models for sequence labeling  Our model architecture is illustrated in Figure FIGREF8 with a running example  Given a context/sentence INLINEFORM0 where INLINEFORM1 is the length of the context, we generate the corresponding tag sequence INLINEFORM2 based on our designed tagging schemes and the original annotations for pun detection and location provided by the corpora  Our model is then trained on pairs of INLINEFORM3  
Input  The contexts in the pun corpus hold the property that each pun contains exactly one content word, which can be either a noun, a verb, an adjective, or an adverb  To capture this characteristic, we consider lexical features at the character level  Similar to the work of BIBREF15 , the character embeddings are trained by the character-level LSTM networks on the unannotated input sequences  Nonlinear transformations are then applied to the character embeddings by highway networks BIBREF18 , which map the character-level features into different semantic spaces 
We also observe that a pun tends to appear at the end of a sentence  Specifically, based on the statistics, we found that sentences with a pun that locate at the second half of the text account for around 88% and 92% in homographic and heterographic datasets, respectively  We thus introduce a binary feature that indicates if a word is located at the first or the second half of an input sentence to capture such positional information  A binary indicator can be mapped to a vector representation using a randomly initialized embedding table BIBREF19 , BIBREF20   In this work, we directly adopt the value of the binary indicator as part of the input 
The concatenation of the transformed character embeddings, the pre-trained word embeddings BIBREF21 , and the position indicators are taken as input of our model 
Tagging  The input is then fed into a BiLSTM network, which will be able to capture contextual information  For a training instance INLINEFORM0 , we suppose the output by the word-level BiLSTM is INLINEFORM1   The CRF layer is adopted to capture label dependencies and make final tagging decisions at each position, which has been included in many state-of-the-art sequence labeling models BIBREF14 , BIBREF15   The conditional probability is defined as:
where INLINEFORM0 is a set of all possible label sequences consisting of tags from INLINEFORM1 (or INLINEFORM2 ), INLINEFORM3 and INLINEFORM4 are weight and bias parameters corresponding to the label pair INLINEFORM5   During training, we minimize the negative log-likelihood summed over all training instances:
where INLINEFORM0 refers to the INLINEFORM1 -th instance in the training set  During testing, we aim to find the optimal label sequence for a new input INLINEFORM2 :
This search process can be done efficiently using the Viterbi algorithm 
Datasets and Settings
We evaluate our model on two benchmark datasets BIBREF9   The homographic dataset contains 2,250 contexts, 1,607 of which contain a pun  The heterographic dataset consists of 1,780 contexts with 1,271 containing a pun  We notice there is no standard splitting information provided for both datasets  Thus we apply 10-fold cross validation  To make direct comparisons with prior studies, following BIBREF4 , we accumulated the predictions for all ten folds and calculate the scores in the end 
For each fold, we randomly select 10% of the instances from the training set for development  Word embeddings are initialized with the 100-dimensional Glove BIBREF21   The dimension of character embeddings is 30 and they are randomly initialized, which can be fine tuned during training  The pre-trained word embeddings are not updated during training  The dimensions of hidden vectors for both char-level and word-level LSTM units are set to 300  We adopt stochastic gradient descent (SGD) BIBREF26 with a learning rate of 0
Context 3: Introduction
Event detection on microblogging platforms such as Twitter aims to detect events preemptively  A main task in event detection is detecting events of predetermined types BIBREF0, such as concerts or controversial events based on microposts matching specific event descriptions  This task has extensive applications ranging from cyber security BIBREF1, BIBREF2 to political elections BIBREF3 or public health BIBREF4, BIBREF5  Due to the high ambiguity and inconsistency of the terms used in microposts, event detection is generally performed though statistical machine learning models, which require a labeled dataset for model training  Data labeling is, however, a long, laborious, and usually costly process  For the case of micropost classification, though positive labels can be collected (e g , using specific hashtags, or event-related date-time information), there is no straightforward way to generate negative labels useful for model training  To tackle this lack of negative labels and the significant manual efforts in data labeling, BIBREF1 (BIBREF1, BIBREF3) introduced a weak supervision based learning approach, which uses only positively labeled data, accompanied by unlabeled examples by filtering microposts that contain a certain keyword indicative of the event type under consideration (e g , `hack' for cyber security)  Another key technique in this context is expectation regularization BIBREF6, BIBREF7, BIBREF1  Here, the estimated proportion of relevant microposts in an unlabeled dataset containing a keyword is given as a keyword-specific expectation  This expectation is used in the regularization term of the model's objective function to constrain the posterior distribution of the model predictions  By doing so, the model is trained with an expectation on its prediction for microposts that contain the keyword  Such a method, however, suffers from two key problems:
Due to the unpredictability of event occurrences and the constantly changing dynamics of users' posting frequency BIBREF8, estimating the expectation associated with a keyword is a challenging task, even for domain experts;
The performance of the event detection model is constrained by the informativeness of the keyword used for model training  As of now, we lack a principled method for discovering new keywords and improve the model performance 
To address the above issues, we advocate a human-AI loop approach for discovering informative keywords and estimating their expectations reliably  Our approach iteratively leverages 1) crowd workers for estimating keyword-specific expectations, and 2) the disagreement between the model and the crowd for discovering new informative keywords  More specifically, at each iteration after we obtain a keyword-specific expectation from the crowd, we train the model using expectation regularization and select those keyword-related microposts for which the model's prediction disagrees the most with the crowd's expectation; such microposts are then presented to the crowd to identify new keywords that best explain the disagreement  By doing so, our approach identifies new keywords which convey more relevant information with respect to existing ones, thus effectively boosting model performance  By exploiting the disagreement between the model and the crowd, our approach can make efficient use of the crowd, which is of critical importance in a human-in-the-loop context BIBREF9, BIBREF10  An additional advantage of our approach is that by obtaining new keywords that improve model performance over time, we are able to gain insight into how the model learns for specific event detection tasks  Such an advantage is particularly useful for event detection using complex models, e g , deep neural networks, which are intrinsically hard to understand BIBREF11, BIBREF12  An additional challenge in involving crowd workers is that their contributions are not fully reliable BIBREF13  In the crowdsourcing literature, this problem is usually tackled with probabilistic latent variable models BIBREF14, BIBREF15, BIBREF16, which are used to perform truth inference by aggregating a redundant set of crowd contributions  Our human-AI loop approach improves the inference of keyword expectation by aggregating contributions not only from the crowd but also from the model  This, however, comes with its own challenge as the model's predictions are further dependent on the results of expectation inference, which is used for model training  To address this problem, we introduce a unified probabilistic model that seamlessly integrates expectation inference and model training, thereby allowing the former to benefit from the latter while resolving the inter-dependency between the two 
To the best of our knowledge, we are the first to propose a human-AI loop approach that iteratively improves machine learning models for event detection  In summary, our work makes the following key contributions:
A novel human-AI loop approach for micropost event detection that jointly discovers informative keywords and estimates their expectation;
A unified probabilistic model that infers keyword expectation and simultaneously performs model training;
An extensive empirical evaluation of our approach on multiple real-world datasets demonstrating that our approach significantly improves the state of the art by an average of 24 3% AUC 
The rest of this paper is organized as follows  First, we present our human-AI loop approach in Section SECREF2  Subsequently, we introduce our proposed probabilistic model in Section SECREF3  The experimental setup and results are presented in Section SECREF4  Finally, we briefly cover related work in Section SECREF5 before concluding our work in Section SECREF6 
The Human-AI Loop Approach
Given a set of labeled and unlabeled microposts, our goal is to extract informative keywords and estimate their expectations in order to train a machine learning model  To achieve this goal, our proposed human-AI loop approach comprises two crowdsourcing tasks, i e , micropost classification followed by keyword discovery, and a unified probabilistic model for expectation inference and model training  Figure FIGREF6 presents an overview of our approach  Next, we describe our approach from a process-centric perspective 
Following previous studies BIBREF1, BIBREF17, BIBREF2, we collect a set of unlabeled microposts $\mathcal {U}$ from a microblogging platform and post-filter, using an initial (set of) keyword(s), those microposts that are potentially relevant to an event category  Then, we collect a set of event-related microposts (i e , positively labeled microposts) $\mathcal {L}$, post-filtering with a list of seed events  $\mathcal {U}$ and $\mathcal {L}$ are used together to train a discriminative model (e g , a deep neural network) for classifying the relevance of microposts to an event  We denote the target model as $p_\theta (y|x)$, where $\theta $ is the model parameter to be learned and $y$ is the label of an arbitrary micropost, represented by a bag-of-words vector $x$  Our approach iterates several times $t=\lbrace 1, 2, \ldots \rbrace $ until the performance of the target model converges  Each iteration starts from the initial keyword(s) or the new keyword(s) discovered in the previous iteration  Given such a keyword, denoted by $w^{(t)}$, the iteration starts by sampling microposts containing the keyword from $\mathcal {U}$, followed by dynamically creating micropost classification tasks and publishing them on a crowdsourcing platform 
Micropost Classification  The micropost classification task requires crowd workers to label the selected microposts into two classes: event-related and non event-related  In particular, workers are given instructions and examples to differentiate event-instance related microposts and general event-category related microposts  Consider, for example, the following microposts in the context of Cyber attack events, both containing the keyword `hack':
Credit firm Equifax says 143m Americans' social security numbers exposed in hack
This micropost describes an instance of a cyber attack event that the target model should identify  This is, therefore, an event-instance related micropost and should be considered as a positive example  Contrast this with the following example:
Companies need to step their cyber security up
This micropost, though related to cyber security in general, does not mention an instance of a cyber attack event, and is of no interest to us for event detection  This is an example of a general event-category related micropost and should be considered as a negative example 
In this task, each selected micropost is labeled by multiple crowd workers  The annotations are passed to our probabilistic model for expectation inference and model training 
Expectation Inference & Model Training  Our probabilistic model takes crowd-contributed labels and the model trained in the previous iteration as input  As output, it generates a keyword-specific expectation, denoted as $e^{(t)}$, and an improved version of the micropost classification model, denoted as $p_{\theta ^{(t)}}(y|x)$  The details of our probabilistic model are given in Section SECREF3 
Keyword Discovery  The keyword discovery task aims at discovering a new keyword (or a set of keywords) that is most informative for model training with respect to existing keywords  To this end, we first apply the current model $p_{\theta ^{(t)}}(y|x)$ on the unlabeled microposts $\mathcal {U}$  For those that contain the keyword $w^{(t)}$, we calculate the disagreement between the model predictions and the keyword-specific expectation $e^{(t)}$:
and select the ones with the highest disagreement for keyword discovery  These selected microposts are supposed to contain information that can explain the disagreement between the model prediction and keyword-specific expectation, and can thus provide information that is most different from the existing set of keywords for model training 
For instance, our study shows that the expectation for the keyword `hack' is 0 20, which means only 20% of the initial set of microposts retrieved with the keyword are event-related  A micropost selected with the highest disagreement (Eq  DISPLAY_FORM7), whose likelihood of being event-related as predicted by the model is $99 9\%$, is shown as an example below:
RT @xxx: Hong Kong securities brokers hit by cyber attacks, may face more: regulator #cyber #security #hacking https://t co/rC1s9CB
This micropost contains keywords that can better indicate the relevance to a cyber security event than the initial keyword `hack', e g , `securities', `hit', and `attack' 
Note that when the keyword-specific expectation $e^{(t)}$ in Equation DISPLAY_FORM7 is high, the selected microposts will be the ones that contain keywords indicating the irrelevance of the microposts to an event category  Such keywords are also useful for model training as they help improve the model's ability to identify irrelevant microposts 
To identify new keywords in the selected microposts, we again leverage crowdsourcing, as humans are typically better than machines at providing specific explanations BIBREF18, BIBREF19  In the crowdsourcing task, workers are first asked to find those microposts where the model predictions are deemed correct  Then, from those microposts, workers are asked to find the keyword that best indicates the class of the microposts as predicted by the model  The keyword most frequently identified by the workers is then used as the initial keyword for the following iteration  In case multiple keywords are selected, e g , the top-$N$ frequent ones, workers will be asked to perform $N$ micropost classification tasks for each keyword in the next iteration, and the model training will be performed on multiple keyword-specific expectations 
Unified Probabilistic Model
This section introduces our probabilistic model that infers keyword expectation and trains the target model simultaneously  We start by formalizing the problem and introducing our model, before describing the model learning method 
Problem Formalization  We consider the problem at iteration $t$ where the corresponding keyword is $w^{(t)}$  In the current iteration, let $\mathcal {U}^{(t)} \subset \mathcal {U}$ denote the set of all microposts containing the keyword and $\mathcal {M}^{(t)}= \lbrace x_{m}\rbrace _{m=1}^M\subset \mathcal {U}^{(t)}$ be the randomly selected subset of $M$ microposts labeled by $N$ crowd workers $\mathcal {C} = \lbrace c_n\rbrace _{n=1}^N$  The annotations form a matrix $\mathbf {A}\in \mathbb {R}^{M\times N}$ where $\mathbf {A}_{mn}$ is the label for the micropost $x_m$ contributed by crowd worker $c_n$  Our goal is to infer the keyword-specific expectation $e^{(t)}$ and train the target model by learning the model parameter $\theta ^{(t)}$  An additional parameter of our probabilistic model is the reliability of crowd workers, which is essential when involving crowdsourcing  Following Dawid and Skene BIBREF14, BIBREF16, we represent the annotation reliability of worker $c_n$ by a latent confusion matrix $\pi ^{(n)}$, where the $rs$-th element $\pi _{rs}^{(n)}$ denotes the probability of $c_n$ labeling a micropost as class $r$ given the true class $s$ 
Unified Probabilistic Model ::: Expectation as Model Posterior
First, we introduce an expectation regularization technique for the weakly supervised learning of the target model $p_{\theta ^{(t)}}(y|x)$  In this setting, the objective function of the target model is composed of two parts, corresponding to the labeled microposts $\mathcal {L}$ and the unlabeled ones $\mathcal {U}$ 
The former part aims at maximizing the likelihood of the labeled microposts:
where we assume that $\theta $ is generated from a prior distribution (e g , Laplacian or Gaussian) parameterized by $\sigma $ 
To leverage unlabeled data for model training, we make use of the expectations of existing keywords, i e , {($w^{(1)}$, $e^{(1)}$),    , ($w^{(t-1)}$, $e^{(t-1)}$), ($w^{(t)}$, $e^{(t)}$)} (Note that $e^{(t)}$ is inferred), as a regularization term to constrain model training  To do so, we first give the model's expectation for each keyword $w^{(k)}$ ($1\le k\le t$) as follows:
which denotes the empirical expectation of the model’s posterior predictions on the unlabeled microposts $\mathcal {U}^{(k)}$ containing keyword $w^{(k)}$  Expectation regularization can then be formulated as the regularization of the distance between the Bernoulli distribution parameterized by the model's expectation and the expectation of the existing keyword:
where $D_{KL}[\cdot \Vert \cdot ]$ denotes the KL-divergence between the Bernoulli distributions $Ber(e^{(k)})$ and $Ber(\mathbb {E}_{x\sim \mathcal {U}^{(k)}}(y))$, and $\lambda $ controls the strength of expectation regularization 
Unified Probabilistic Model ::: Expectation as Class Prior
To learn the keyword-specific expectation $e^{(t)}$ and the crowd worker reliability $\pi ^{(n)}$ ($1\le n\le N$), we model the likelihood of the crowd-contributed labels $\mathbf {A}$ as a function of these parameters  In this context, we view the expectation as the class prior, thus performing expectation inference as the learning of the class prior  By doing so, we connect expectation inference with model training 
Specifically, we model the likelihood of an arbitrary crowd-contributed label $\mathbf {A}_{mn}$ as a mixture of multinomials where the prior is the keyword-specific expectation $e^{(t)}$:
where $e_s^{(t)}$ is the probability of the ground truth label being $s$ given the keyword-specific expectation as the class prior; $K$ is the set of possible ground truth labels (binary in our context); and $r=\mathbf {A}_{mn}$ is the crowd-contributed label  Then, for an individual micropost $x_m$, the likelihood of crowd-contributed labels $\mathbf {A}_{m:}$ is given by:
Therefore, the objective function for maximizing the likelihood of the entire annotation matrix $\mathbf {A}$ can be described as:
Unified Probabilistic Model ::: Unified Probabilistic Model
Integrating model training with expectation inference, the overall objective function of our proposed model is given by:
Figure FIGREF18 depicts a graphical representation of our model, which combines the target model for training (on the left) with the generative model for crowd-contributed labels (on the right) through a keyword-specific expectation 
Model Learning  Due to the unknown ground truth labels of crowd-annotated microposts ($y_m$ in Figure FIGREF18), we resort to expectation maximization for model learning  The learning algorithm iteratively takes two steps: the E-step and the M-step  The E-step infers the ground truth labels given the current model parameters  The M-step updates the model parameters, including the crowd reliability parameters $\pi ^{(n)}$ ($1\le n\le N$), the keyword-specific expectation $e^{(t)}$, and the parameter of the target model $\theta ^{(t)}$  The E-step and the crowd parameter update in the M-step are similar to the Dawid-Skene model BIBREF14  The keyword expectation is inferred by taking into account both the crowd-contributed labels and the model prediction:
The parameter of the target model is updated by gradient descent  For example, when the target model to be trained is a deep neural network, we use back-propagation with gradient descent to update the weight matrices 
Experiments and Results
This section presents our experimental setup and results for evaluating our approach  We aim at answering the following questions:
[noitemsep,leftmargin=*]
Q1: How effectively does our proposed human-AI loop approach enhance the state-of-the-art machine learning models for event detection?
Q2: How well does our keyword discovery method work compare to existing keyword expansion methods?
Q3: How effective is our approach using crowdsourcing at obtaining new keywords compared with an approach labelling microposts for model training under the same cost?
Q4: How much benefit does our unified probabilistic model bring compared to methods that do not take crowd reliability into account?
Experiments and Results ::: Experimental Setup
Datasets  We perform our experiments with two predetermined event categories: cyber security (CyberAttack) and death of politicians (PoliticianDeath)  These event categories are chosen as they are representative of important event types that are of interest to many governments and companies  The need to create our own dataset was motivated by the lack of public datasets for event detection on microposts  The few available datasets do not suit our requirements  For example, the publicly available Events-2012 Twitter dataset BIBREF20 contains generic event descriptions such as Politics, Sports, Culture etc  Our work targets more specific event categories BIBREF21  Following previous studies BIBREF1, we collect event-related microposts from Twitter using 11 and 8 seed events (see Section SECREF2) for CyberAttack and PoliticianDeath, respectively  Unlabeled microposts are collected by using the keyword `hack' for CyberAttack, while for PoliticianDeath, we use a set of keywords related to `politician' and `death' (such as `bureaucrat', `dead' etc ) For each dataset, we randomly select 500 tweets from the unlabeled subset and manually label them for evaluation  Table TABREF25 shows key statistics from our two datasets 
Comparison Methods  To demonstrate the generality of our approach on different event detection models, we consider Logistic Regression (LR) BIBREF1 and Multilayer Perceptron (MLP) BIBREF2 as the target models  As the goal of our experiments is to demonstrate the effectiveness of our approach as a new model training technique, we use these widely used models  Also, we note that in our case other neural network models with more complex network architectures for event detection, such as the bi-directional LSTM BIBREF17, turn out to be less effective than a simple feedforward network  For both LR and MLP, we evaluate our proposed human-AI loop approach for keyword discovery and expectation estimation by comparing against the weakly supervised learning method proposed by BIBREF1 (BIBREF1) and BIBREF17 (BIBREF17) where only one initial keyword is used with an expectation estimated by an individual expert 
Parameter Settings  We empirically set optimal parameters based on a held-out validation set that contains 20% of the test data  These include the hyperparamters of the target model, those of our proposed probabilistic model, and the parameters used for training the target model  We explore MLP with 1, 2 and 3 hidden layers and apply a grid search in 32, 64, 128, 256, 512 for the dimension of the embeddings and that of the hidden layers  For the coefficient of expectation regularization, we follow BIBREF6 (BIBREF6) and set it to $\lambda =10 \times $ #labeled examples  For model training, we use the Adam BIBREF22 optimization algorithm for both models 
Evaluation  Following BIBREF1 (BIBREF1) and BIBREF3 (BIBREF3), we use accuracy and area under the precision-recall curve (AUC) metrics to measure the performance of our proposed approach  We note that due to the imbalance in our datasets (20% positive microposts in CyberAttack and 27% in PoliticianDeath), accuracy is dominated by negative examples; AUC, in comparison, better characterizes the discriminative power of the model 
Crowdsourcing  We chose Level 3 workers on the Figure-Eight crowdsourcing platform for our experiments  The inter-annotator agreement in micropost classification is taken into account through the EM algorithm  For keyword discovery, we filter keywords based on the frequency of the keyword being selected by the crowd  In terms of cost-effectiveness, our approach is motivated from the fact that crowdsourced data annotation can be expensive, and is thus designed with minimal crowd involvement  For each iteration, we selected 50 tweets for keyword discovery and 50 tweets for micropost classification per keyword  For a dataset with 80k tweets (e g , CyberAttack), our approach only requires to manually inspect 800 tweets (for 8 keywords), which is only 1% of the entire dataset 
Experiments and Results ::: Results of our Human-AI Loop (Q1)
Table TABREF26 reports the evaluation of our approach on both the CyberAttack and PoliticianDeath event categories  Our approach is configured such that each iteration starts with 1 new keyword discovered in the previous iteration 
Our approach improves LR by 5 17% (Accuracy) and 18 38% (AUC), and MLP by 10 71% (Accuracy) and 30 27% (AUC) on average  Such significant improvements clearly demonstrate that our approach is effective at improving model performance  We observe that the target models generally converge between the 7th and 9th iteration on both datasets when performance is measured by AUC  The performance can slightly degrade when the models are further trained for more iterations on both datasets  This is likely due to the fact that over time, the newly discovered keywords entail lower novel information for model training  For instance, for the CyberAttack dataset the new keyword in the 9th iteration `election' frequently co-occurs with the keyword `russia' in the 5th iteration (in microposts that connect Russian hackers with US elections), thus bringing limited new information for improving the model performance  As a side remark, we note that the models converge faster when performance is measured by accuracy  Such a comparison result confirms the difference between the metrics and shows the necessity for more keywords to discriminate event-related microposts from non event-related ones 
Experiments and Results ::: Comparative Results on Keyword Discovery (Q2)
Figure FIGREF31 shows the evaluation of our approach when discovering new informative keywords for model training (see Section SECREF2: Keyword Discovery)  We compare our human-AI collaborative way of discovering new keywords against a query expansion (QE) approach BIBREF23, BIBREF24 that leverages word embeddings to find similar words in the latent semantic space  Specifically, we use pre-trained word embeddings based on a large Google News dataset for query expansion  For instance, the top keywords resulting from QE for `politician' are, `deputy',`ministry',`secretary', and `minister'  For each of these keywords, we use the crowd to label a set of tweets and obtain a corresponding expectation 
We observe that our approach consistently outperforms QE by an average of $4 62\%$ and $52 58\%$ AUC on CyberAttack and PoliticianDeath, respectively  The large gap between the performance improvements for the two datasets is mainly due to the fact that microposts that are relevant for PoliticianDeath are semantically more complex than those for CyberAttack, as they encode noun-verb relationship (e g , “the king of     died    ”) rather than a simple verb (e g , “    hacked ”) for the CyberAttack microposts  QE only finds synonyms of existing keywords related to either `politician' or `death', however cannot find a meaningful keyword that fully characterizes the death of a politician  For instance, QE finds the keywords `kill' and `murder', which are semantically close to `death' but are not specifically relevant to the death of a politician  Unlike QE, our approach identifies keywords that go beyond mere synonyms and that are more directly related to the end task, i e , discriminating event-related microposts from non related ones  Examples are `demise' and `condolence'  As a remark, we note that in Figure FIGREF31(b), the increase in QE performance on PoliticianDeath is due to the keywords `deputy' and `minister', which happen to be highly indicative of the death of a politician in our dataset; these keywords are also identified by our approach 
Experiments and Results ::: Cost-Effectiveness Results (Q3)
To demonstrate the cost-effectiveness of using crowdsourcing for obtaining new keywords and consequently, their expectations, we compare the performance of our approach with an approach using crowdsourcing to only label microposts for model training at the same cost  Specifically, we conducted an additional crowdsourcing experiment where the same cost used for keyword discovery in our approach is used to label additional microposts for model training  These newly labeled microposts are used with the microposts labeled in the micropost classification task of our approach (see Section SECREF2: Micropost Classification) and the expectation of the initial keyword to train the model for comparison  The model trained in this way increases AUC by 0 87% for CyberAttack, and by 1 06% for PoliticianDeath; in comparison, our proposed approach increases AUC by 33 42% for PoliticianDeath and by 15 23% for CyberAttack over the baseline presented by BIBREF1)  These results show that using crowdsourcing for keyword discovery is significantly more cost-effective than simply using crowdsourcing to get additional labels when training the model 
Experiments and Results ::: Expectation Inference Results (Q4)
To investigate the effectiveness of our expectation inference method, we compare it against a majority voting approach, a strong baseline in truth inference BIBREF16  Figure FIGREF36 shows the result of this evaluation  We observe that our approach results in better models for both CyberAttack and PoliticianDeath  Our manual investigation reveals that workers' annotations are of high reliability, which explains the relatively good performance of majority voting  Despite limited margin for improvement, our method of expectation inference improves the performance of majority voting by $0 4\%$ and $1 19\%$ AUC on CyberAttack and PoliticianDeath, respectively 
Related Work
Event Detection  The techniques for event extraction from microblogging platforms can be classified according to their domain specificity and their detection method BIBREF0  Early works mainly focus on open domain event detection BIBREF25, BIBREF26, BIBREF27  Our work falls into the category of domain-specific event detection BIBREF21, which has drawn increasing attention due to its relevance for various applications such as cyber security BIBREF1, BIBREF2 and public health BIBREF4, BIBREF5  In terms of technique, our proposed detection method is related to the recently proposed weakly supervised learning methods BIBREF1, BIBREF17, BIBREF3  This comes in contrast with fully-supervised learning methods, which are often limited by the size of the training data (e g , a few hundred examples) BIBREF28, BIBREF29 
Human-in-the-Loop Approaches  Our work extends weakly supervised learning methods by involving humans in the loop BIBREF13  Existing human-in-the-loop approaches mainly leverage crowds to label individual data instances BIBREF9, BIBREF10 or to debug the training data BIBREF30, BIBREF31 or components BIBREF32, BIBREF33, BIBREF34 of a machine learning system  Unlike these works, we leverage crowd workers to label sampled microposts in order to obtain keyword-specific expectations, which can then be generalized to help classify microposts containing the same keyword, thus amplifying the utility of the crowd  Our work is further connected to the topic of interpretability and transparency of machine learning models BIBREF11, BIBREF35, BIBREF12, for which humans are increasingly involved, for instance for post-hoc evaluations of the model's interpretability  In contrast, our approach directly solicits informative keywords from the crowd for model training, thereby providing human-understandable explanations for the improved model 
Conclusion
In this paper, we presented a new human-AI loop approach for keyword discovery and expectation estimation to better train event detection models  Our approach takes advantage of the disagreement between the crowd and the model to discover informative keywords and leverages the joint power of the crowd and the model in expectation inference  We evaluated our approach on real-world datasets and showed that it significantly outperforms the state of the art and that it is particularly useful for detecting events where relevant microposts are semantically complex, e g , the death of a politician  As future work, we plan to parallelize the crowdsourcing tasks and optimize our pipeline in order to use our event detection approach in real-time 
Acknowledgements
This project has received funding from the Swiss National Science Foundation (grant #407540_167320 Tighten-it-All) and from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement 683253/GraphInt) 

ground_truth: four machine translation tasks: German -> English, Japanese -> English, Romanian -> English, English -> German

 four machine translation tasks, IWSLT 2017 German $\rightarrow $ English BIBREF27, KFTT Japanese $\rightarrow $ English BIBREF28, WMT 2016 Romanian $\rightarrow $ English BIBREF29, WMT 2014 English $\rightarrow $ German BIBREF30

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What is the tagging scheme employed?
Context 2: 015 
For the pun detection task, if the predicted tag sequence contains at least one INLINEFORM0 tag, we regard the output (i e , the prediction of our pun detection model) for this task as true, otherwise false  For the pun location task, a predicted pun is regarded as correct if and only if it is labeled as the gold pun in the dataset  As to pun location, to make fair comparisons with prior studies, we only consider the instances that are labeled as the ones containing a pun  We report precision, recall and INLINEFORM1 score in Table TABREF11   A list of prior works that did not employ joint learning are also shown in the first block of Table TABREF11  
Results
We also implemented a baseline model based on conditional random fields (CRF), where features like POS tags produced by the Stanford POS tagger BIBREF27 , n-grams, label transitions, word suffixes and relative position to the end of the text are considered  We can see that our model with the INLINEFORM0 tagging scheme yields new state-of-the-art INLINEFORM1 scores on pun detection and competitive results on pun location, compared to baselines that do not adopt joint learning in the first block  For location on heterographic puns, our model's performance is slightly lower than the system of BIBREF25 , which is a rule-based locator  Compared to CRF, we can see that our model, either with the INLINEFORM2 or the INLINEFORM3 scheme, yields significantly higher recall on both detection and location tasks, while the precisions are relatively close  This demonstrates the effectiveness of BiLSTM, which learns the contextual features of given texts – such information appears to be helpful in recalling more puns 
Compared to the INLINEFORM0 scheme, the INLINEFORM1 tagging scheme is able to yield better performance on these two tasks  After studying outputs from these two approaches, we found that one leading source of error for the INLINEFORM2 approach is that there exist more than one words in a single instance that are assigned with the INLINEFORM3 tag  However, according to the description of pun in BIBREF9 , each context contains a maximum of one pun  Thus, such a useful structural constraint is not well captured by the simple approach based on the INLINEFORM4 tagging scheme  On the other hand, by applying the INLINEFORM5 tagging scheme, such a constraint is properly captured in the model  As a result, the results for such a approach are significantly better than the approach based on the INLINEFORM6 tagging scheme, as we can observe from the table  Under the same experimental setup, we also attempted to exclude word position features  Results are given by INLINEFORM7 - INLINEFORM8   It is expected that the performance of pun location drops, since such position features are able to capture the interesting property that a pun tends to appear in the second half of a sentence  While such knowledge is helpful for the location task, interestingly, a model without position knowledge yields improved performance on the pun detection task  One possible reason is that detecting whether a sentence contains a pun is not concerned with such word position information 
Additionally, we conduct experiments over sentences containing a pun only, namely 1,607 and 1,271 instances from homographic and heterographic pun corpora separately  It can be regarded as a “pipeline” method where the classifier for pun detection is regarded as perfect  Following the prior work of BIBREF4 , we apply 10-fold cross validation  Since we are given that all input sentences contain a pun, we only report accumulated results on pun location, denoted as Pipeline in Table TABREF11   Compared with our approaches, the performance of such an approach drops significantly  On the other hand, such a fact demonstrates that the two task, detection and location of puns, can reinforce each other  These figures demonstrate the effectiveness of our sequence labeling method to detect and locate English puns in a joint manner 
Error Analysis
We studied the outputs from our system and make some error analysis  We found the errors can be broadly categorized into several types, and we elaborate them here  1) Low word coverage: since the corpora are relatively small, there exist many unseen words in the test set  Learning the representations of such unseen words is challenging, which affects the model's performance  Such errors contribute around 40% of the total errors made by our system  2) Detection errors: we found many errors are due to the model's inability to make correct pun detection  Such inability harms both pun detection and pun location  Although our approach based on the INLINEFORM0 tagging scheme yields relatively higher scores on the detection task, we still found that 40% of the incorrectly predicted instances fall into this group  3) Short sentences: we found it was challenging for our model to make correct predictions when the given text is short  Consider the example “Superglue! Tom rejoined," here the word rejoined is the corresponding pun  However, it would be challenging to figure out the pun with such limited contextual information 
Related Work
Most existing systems address pun detection and location separately  BIBREF22 applied word sense knowledge to conduct pun detection  BIBREF24 trained a bidirectional RNN classifier for detecting homographic puns  Next, a knowledge-based approach is adopted to find the exact pun  Such a system is not applicable to heterographic puns  BIBREF28 applied Google n-gram and word2vec to make decisions  The phonetic distance via the CMU Pronouncing Dictionary is computed to detect heterographic puns  BIBREF10 used the hidden Markov model and a cyclic dependency network with rich features to detect and locate puns  BIBREF23 used a supervised approach to pun detection and a weakly supervised approach to pun location based on the position within the context and part of speech features  BIBREF25 proposed a rule-based system for pun location that scores candidate words according to eleven simple heuristics  Two systems are developed to conduct detection and location separately in the system known as UWAV BIBREF3   The pun detector combines predictions from three classifiers  The pun locator considers word2vec similarity between every pair of words in the context and position to pinpoint the pun  The state-of-the-art system for homographic pun location is a neural method BIBREF4 , where the word senses are incorporated into a bidirectional LSTM model  This method only supports the pun location task on homographic puns  Another line of research efforts related to this work is sequence labeling, such as POS tagging, chunking, word segmentation and NER  The neural methods have shown their effectiveness in this task, such as BiLSTM-CNN BIBREF13 , GRNN BIBREF29 , LSTM-CRF BIBREF30 , LSTM-CNN-CRF BIBREF14 , LM-LSTM-CRF BIBREF15  
In this work, we combine pun detection and location tasks as a single sequence labeling problem  Inspired by the work of BIBREF15 , we also adopt a LSTM-CRF with character embeddings to make labeling decisions 
Conclusion
In this paper, we propose to perform pun detection and location tasks in a joint manner from a sequence labeling perspective  We observe that each text in our corpora contains a maximum of one pun  Hence, we design a novel tagging scheme to incorporate such a constraint  Such a scheme guarantees that there is a maximum of one word that will be tagged as a pun during the testing phase  We also found the interesting structural property such as the fact that most puns tend to appear at the second half of the sentences can be helpful for such a task, but was not explored in previous works  Furthermore, unlike many previous approaches, our approach, though simple, is generally applicable to both heterographic and homographic puns  Empirical results on the benchmark datasets prove the effectiveness of the proposed approach that the two tasks of pun detection and location can be addressed by a single model from a sequence labeling perspective 
Future research includes the investigations on how to make use of richer semantic and linguistic information for detection and location of puns  Research on puns for other languages such as Chinese is still under-explored, which could also be an interesting direction for our future studies 
Acknowledgments
We would like to thank the three anonymous reviewers for their thoughtful and constructive comments  This work is supported by Singapore Ministry of Education Academic Research Fund (AcRF) Tier 2 Project MOE2017-T2-1-156, and is partially supported by SUTD project PIE-SGP-AI-2018-01 

ground_truth: A new tagging scheme that tags the words before and after the pun as well as the pun words.

a new tagging scheme consisting of three tags, namely { INLINEFORM0 }

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What type of classifiers are used?
Context 4: 
Zero-shot Experiments ::: Estimating translation probabilities
Since pre-trained models operate on subword level, we need to estimate subword translation probabilities  Therefore, we subsample 2M sentence pairs from each parallel corpus and tokenize the data into subwords before running fast-align BIBREF13 
Estimating subword translation probabilities from aligned word vectors requires an additional processing step since the provided vectors from fastText are not at subword level  We use the following approximation to obtain subword vectors: the vector $_s$ of subword $s$ is the weighted average of all the aligned word vectors $_{w_i}$ that have $s$ as an subword
where $p(w_j)$ is the unigram probability of word $w_j$ and $n_s = \sum _{w_j:\, s\in w_j} p(w_j)$  We take the top 50,000 words in each aligned word-vectors to compute subword vectors 
In both cases, not all the words in the foreign vocabulary can be initialized from the English word-embeddings  Those words are initialized randomly from a Gaussian $\mathcal {N}(0, {1}{d^2})$ 
Zero-shot Experiments ::: Hyper-parameters
In all the experiments, we tune RAMEN$_{\textsc {base}}$ for 175,000 updates and RAMEN$_{\textsc {large}}$ for 275,000 updates where the first 25,000 updates are for language specific parameters  The sequence length is set to 256  The mini-batch size are 64 and 24 when tuning language specific parameters using RAMEN$_{\textsc {base}}$ and RAMEN$_{\textsc {large}}$ respectively  For tuning bilingual LMs, we use a mini-batch size of 64 for RAMEN$_{\textsc {base}}$ and 24 for RAMEN$_{\textsc {large}}$ where half of the batch are English sequences and the other half are foreign sequences  This strategy of balancing mini-batch has been used in multilingual neural machine translation BIBREF23, BIBREF24 
We optimize RAMEN$_{\textsc {base}}$ using Lookahead optimizer BIBREF25 wrapped around Adam with the learning rate of $10^{-4}$, the number of fast weight updates $k=5$, and interpolation parameter $\alpha =0 5$  We choose Lookahead optimizer because it has been shown to be robust to the initial parameters of the based optimizer (Adam)  For Adam optimizer, we linearly increase the learning rate from $10^{-7}$ to $10^{-4}$ in the first 4000 updates and then follow an inverse square root decay  All RAMEN$_{\textsc {large}}$ models are optimized with Adam due to memory limit 
When fine-tuning RAMEN on XNLI and UD, we use a mini-batch size of 32, Adam's learning rate of $10^{-5}$  The number of epochs are set to 4 and 50 for XNLI and UD tasks respectively  All experiments are carried out on a single Tesla V100 16GB GPU  Each RAMEN$_{\textsc {base}}$ model is trained within a day and each RAMEN$_{\textsc {large}}$ is trained within two days 
Results
In this section, we present the results of out models for two zero-shot cross lingual transfer tasks: XNLI and dependency parsing 
Results ::: Cross-lingual Natural Language Inference
Table TABREF32 shows the XNLI test accuracy  For reference, we also include the scores from the previous work, notably the state-of-the-art system XLM BIBREF6  Before discussing the results, we spell out that the fairest comparison in this experiment is the comparison between mBERT and RAMEN$_{\textsc {base}}$+BERT trained with monolingual only 
We first discuss the transfer results from BERT  Initialized from fastText vectors, RAMEN$_{\textsc {base}}$ slightly outperforms mBERT by 1 9 points on average and widen the gap of 3 3 points on Arabic  RAMEN$_{\textsc {base}}$ gains extra 0 8 points on average when initialized from parallel data  With triple number of parameters, RAMEN$_{\textsc {large}}$ offers an additional boost in term of accuracy and initialization with parallel data consistently improves the performance  It has been shown that BERT$_{\textsc {large}}$ significantly outperforms BERT$_{\textsc {base}}$ on 11 English NLP tasks BIBREF0, the strength of BERT$_{\textsc {large}}$ also shows up when adapted to foreign languages 
Transferring from RoBERTa leads to better zero-shot accuracies  With the same initializing condition, RAMEN$_{\textsc {base}}$+RoBERTa outperforms RAMEN$_{\textsc {base}}$+BERT on average by 2 9 and 2 3 points when initializing from monolingual and parallel data respectively  This result show that with similar number of parameters, our approach benefits from a better English pre-trained model  When transferring from RoBERTa$_{\textsc {large}}$, we obtain state-of-the-art results for five languages 
Currently, RAMEN only uses parallel data to initialize foreign embeddings  RAMEN can also exploit parallel data through translation objective proposed in XLM  We believe that by utilizing parallel data during the fine-tuning of RAMEN would bring additional benefits for zero-shot tasks  We leave this exploration to future work  In summary, starting from BERT$_{\textsc {base}}$, our approach obtains competitive bilingual LMs with mBERT for zero-shot XNLI  Our approach shows the accuracy gains when adapting from a better pre-trained model 
Results ::: Universal Dependency Parsing
We build on top of RAMEN a graph-based dependency parser BIBREF27  For the purpose of evaluating the contextual representations learned by our model, we do not use part-of-speech tags  Contextualized representations are directly fed into Deep-Biaffine layers to predict arc and label scores  Table TABREF34 presents the Labeled Attachment Scores (LAS) for zero-shot dependency parsing 
We first look at the fairest comparison between mBERT and monolingually initialized RAMEN$_{\textsc {base}}$+BERT  The latter outperforms the former on five languages except Arabic  We observe the largest gain of +5 2 LAS for French  Chinese enjoys +3 1 LAS from our approach  With similar architecture (12 or 24 layers) and initialization (using monolingual or parallel data), RAMEN+RoBERTa performs better than RAMEN+BERT for most of the languages  Arabic and Hindi benefit the most from bigger models  For the other four languages, RAMEN$_{\textsc {large}}$ renders a modest improvement over RAMEN$_{\textsc {base}}$ 
Analysis ::: Impact of initialization
Initializing foreign embeddings is the backbone of our approach  A good initialization leads to better zero-shot transfer results and enables fast adaptation  To verify the importance of a good initialization, we train a RAMEN$_{\textsc {base}}$+RoBERTa with foreign word-embeddings are initialized randomly from $\mathcal {N}(0, {1}{d^2})$  For a fair comparison, we use the same hyper-parameters in §SECREF27  Table TABREF36 shows the results of XNLI and UD parsing of random initialization  In comparison to the initialization using aligned fastText vectors, random initialization decreases the zero-shot performance of RAMEN$_{\textsc {base}}$ by 15 9% for XNLI and 27 8 points for UD parsing on average  We also see that zero-shot parsing of SOV languages (Arabic and Hindi) suffers random initialization 
Analysis ::: Are contextual representations from RAMEN also good for supervised parsing?
All the RAMEN models are built from English and tuned on English for zero-shot cross-lingual tasks  It is reasonable to expect RAMENs do well in those tasks as we have shown in our experiments  But are they also a good feature extractor for supervised tasks? We offer a partial answer to this question by evaluating our model for supervised dependency parsing on UD datasets 
We used train/dev/test splits provided in UD to train and evaluate our RAMEN-based parser  Table TABREF38 summarizes the results (LAS) of our supervised parser  For a fair comparison, we choose mBERT as the baseline and all the RAMEN models are initialized from aligned fastText vectors  With the same architecture of 12 Transformer layers, RAMEN$_{\textsc {base}}$+BERT performs competitive to mBERT and outshines mBERT by +1 2 points for Vietnamese  The best LAS results are obtained by RAMEN$_{\textsc {large}}$+RoBERTa with 24 Transformer layers  Overall, our results indicate the potential of using contextual representations from RAMEN for supervised tasks 
Analysis ::: How does linguistic knowledge transfer happen through each training stages?
We evaluate the performance of RAMEN+RoBERTa$_{\textsc {base}}$ (initialized from monolingual data) at each training steps: initialization of word embeddings (0K update), fine-tuning target embeddings (25K), and fine-tuning the model on both English and target language (at each 25K updates)  The results are presented in Figure FIGREF40 
Without fine-tuning, the average accuracy of XLNI is 39 7% for a three-ways classification task, and the average LAS score is 3 6 for dependency parsing  We see the biggest leap in the performance after 50K updates  While semantic similarity task profits significantly at 25K updates of the target embeddings, syntactic task benefits with further fine-tuning the encoder  This is expected since the target languages might exhibit different syntactic structures than English and fine-tuning encoder helps to capture language specific structures  We observe a substantial gain of 19-30 LAS for all languages except French after 50K updates 
Language similarities have more impact on transferring syntax than semantics  Without tuning the English encoder, French enjoys 50 3 LAS for being closely related to English, whereas Arabic and Hindi, SOV languages, modestly reach 4 2 and 6 4 points using the SVO encoder  Although Chinese has SVO order, it is often seen as head-final while English is strong head-initial  Perhaps, this explains the poor performance for Chinese 
Limitations
While we have successfully adapted autoencoding pre-trained LMs from English to other languages, the question whether our approach can also be applied for autoregressive LM such as XLNet still remains  We leave the investigation to future work 
Conclusions
In this work, we have presented a simple and effective approach for rapidly building a bilingual LM under a limited computational budget  Using BERT as the starting point, we demonstrate that our approach produces better than mBERT on two cross-lingual zero-shot sentence classification and dependency parsing  We find that the performance of our bilingual LM, RAMEN, correlates with the performance of the original pre-trained English models  We also find that RAMEN is also a powerful feature extractor in supervised dependency parsing  Finally, we hope that our work sparks of interest in developing fast and effective methods for transferring pre-trained English models to other languages 

ground_truth: probabilistic model

Logistic Regression, Multilayer Perceptron