prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What tasks are used for evaluation?
Context 1: Introduction
The Transformer architecture BIBREF0 for deep neural networks has quickly risen to prominence in NLP through its efficiency and performance, leading to improvements in the state of the art of Neural Machine Translation BIBREF1, BIBREF2, as well as inspiring other powerful general-purpose models like BERT BIBREF3 and GPT-2 BIBREF4  At the heart of the Transformer lie multi-head attention mechanisms: each word is represented by multiple different weighted averages of its relevant context  As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9 
The attention distribution of each head is predicted typically using the softmax normalizing transform  As a result, all context words have non-zero attention weight  Recent work on single attention architectures suggest that using sparse normalizing transforms in attention mechanisms such as sparsemax – which can yield exactly zero probabilities for irrelevant words – may improve performance and interpretability BIBREF12, BIBREF13, BIBREF14  Qualitative analysis of attention heads BIBREF0 suggests that, depending on what phenomena they capture, heads tend to favor flatter or more peaked distributions 
Recent works have proposed sparse Transformers BIBREF10 and adaptive span Transformers BIBREF11  However, the “sparsity" of those models only limits the attention to a contiguous span of past tokens, while in this work we propose a highly adaptive Transformer model that is capable of attending to a sparse set of words that are not necessarily contiguous  Figure FIGREF1 shows the relationship of these methods with ours 
Our contributions are the following:
We introduce sparse attention into the Transformer architecture, showing that it eases interpretability and leads to slight accuracy gains 
We propose an adaptive version of sparse attention, where the shape of each attention head is learnable and can vary continuously and dynamically between the dense limit case of softmax and the sparse, piecewise-linear sparsemax case 
We make an extensive analysis of the added interpretability of these models, identifying both crisper examples of attention head behavior observed in previous work, as well as novel behaviors unraveled thanks to the sparsity and adaptivity of our proposed model 
Background ::: The Transformer
In NMT, the Transformer BIBREF0 is a sequence-to-sequence (seq2seq) model which maps an input sequence to an output sequence through hierarchical multi-head attention mechanisms, yielding a dynamic, context-dependent strategy for propagating information within and across sentences  It contrasts with previous seq2seq models, which usually rely either on costly gated recurrent operations BIBREF15, BIBREF16 or static convolutions BIBREF17 
Given $n$ query contexts and $m$ sequence items under consideration, attention mechanisms compute, for each query, a weighted representation of the items  The particular attention mechanism used in BIBREF0 is called scaled dot-product attention, and it is computed in the following way:
where $\mathbf {Q} \in \mathbb {R}^{n \times d}$ contains representations of the queries, $\mathbf {K}, \mathbf {V} \in \mathbb {R}^{m \times d}$ are the keys and values of the items attended over, and $d$ is the dimensionality of these representations  The $\mathbf {\pi }$ mapping normalizes row-wise using softmax, $\mathbf {\pi }(\mathbf {Z})_{ij} = \operatornamewithlimits{\mathsf {softmax}}(\mathbf {z}_i)_j$, where
In words, the keys are used to compute a relevance score between each item and query  Then, normalized attention weights are computed using softmax, and these are used to weight the values of each item at each query context 
However, for complex tasks, different parts of a sequence may be relevant in different ways, motivating multi-head attention in Transformers  This is simply the application of Equation DISPLAY_FORM7 in parallel $H$ times, each with a different, learned linear transformation that allows specialization:
In the Transformer, there are three separate multi-head attention mechanisms for distinct purposes:
Encoder self-attention: builds rich, layered representations of each input word, by attending on the entire input sentence 
Context attention: selects a representative weighted average of the encodings of the input words, at each time step of the decoder 
Decoder self-attention: attends over the partial output sentence fragment produced so far 
Together, these mechanisms enable the contextualized flow of information between the input sentence and the sequential decoder 
Background ::: Sparse Attention
The softmax mapping (Equation DISPLAY_FORM8) is elementwise proportional to $\exp $, therefore it can never assign a weight of exactly zero  Thus, unnecessary items are still taken into consideration to some extent  Since its output sums to one, this invariably means less weight is assigned to the relevant items, potentially harming performance and interpretability BIBREF18  This has motivated a line of research on learning networks with sparse mappings BIBREF19, BIBREF20, BIBREF21, BIBREF22  We focus on a recently-introduced flexible family of transformations, $\alpha $-entmax BIBREF23, BIBREF14, defined as:
where $\triangle ^d \lbrace \mathbf {p}\in \mathbb {R}^d:\sum _{i} p_i = 1\rbrace $ is the probability simplex, and, for $\alpha \ge 1$, $\mathsf {H}^{\textsc {T}}_\alpha $ is the Tsallis continuous family of entropies BIBREF24:
This family contains the well-known Shannon and Gini entropies, corresponding to the cases $\alpha =1$ and $\alpha =2$, respectively 
Equation DISPLAY_FORM14 involves a convex optimization subproblem  Using the definition of $\mathsf {H}^{\textsc {T}}_\alpha $, the optimality conditions may be used to derive the following form for the solution (Appendix SECREF83):
where $[\cdot ]_+$ is the positive part (ReLU) function, $\mathbf {1}$ denotes the vector of all ones, and $\tau $ – which acts like a threshold – is the Lagrange multiplier corresponding to the $\sum _i p_i=1$ constraint 
Background ::: Sparse Attention ::: Properties of @!START@$\alpha $@!END@-entmax 
The appeal of $\alpha $-entmax for attention rests on the following properties  For $\alpha =1$ (i e , when $\mathsf {H}^{\textsc {T}}_\alpha $ becomes the Shannon entropy), it exactly recovers the softmax mapping (We provide a short derivation in Appendix SECREF89 )  For all $\alpha >1$ it permits sparse solutions, in stark contrast to softmax  In particular, for $\alpha =2$, it recovers the sparsemax mapping BIBREF19, which is piecewise linear  In-between, as $\alpha $ increases, the mapping continuously gets sparser as its curvature changes 
To compute the value of $\alpha $-entmax, one must find the threshold $\tau $ such that the r h s  in Equation DISPLAY_FORM16 sums to one  BIBREF23 propose a general bisection algorithm  BIBREF14 introduce a faster, exact algorithm for $\alpha =1 5$, and enable using $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with fixed $\alpha $ within a neural network by showing that the $\alpha $-entmax Jacobian w r t  $\mathbf {z}$ for $\mathbf {p}^\star = \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ is
Our work furthers the study of $\alpha $-entmax by providing a derivation of the Jacobian w r t  the hyper-parameter $\alpha $ (Section SECREF3), thereby allowing the shape and sparsity of the mapping to be learned automatically  This is particularly appealing in the context of multi-head attention mechanisms, where we shall show in Section SECREF35 that different heads tend to learn different sparsity behaviors 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax
We now propose a novel Transformer architecture wherein we simply replace softmax with $\alpha $-entmax in the attention heads  Concretely, we replace the row normalization $\mathbf {\pi }$ in Equation DISPLAY_FORM7 by
This change leads to sparse attention weights, as long as $\alpha >1$; in particular, $\alpha =1 5$ is a sensible starting point BIBREF14 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Different @!START@$\alpha $@!END@ per head 
Unlike LSTM-based seq2seq models, where $\alpha $ can be more easily tuned by grid search, in a Transformer, there are many attention heads in multiple layers  Crucial to the power of such models, the different heads capture different linguistic phenomena, some of them isolating important words, others spreading out attention across phrases BIBREF0  This motivates using different, adaptive $\alpha $ values for each attention head, such that some heads may learn to be sparser, and others may become closer to softmax  We propose doing so by treating the $\alpha $ values as neural network parameters, optimized via stochastic gradients along with the other weights 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Derivatives w r t  @!START@$\alpha $@!END@ 
In order to optimize $\alpha $ automatically via gradient methods, we must compute the Jacobian of the entmax output w r t  $\alpha $  Since entmax is defined through an optimization problem, this is non-trivial and cannot be simply handled through automatic differentiation; it falls within the domain of argmin differentiation, an active research topic in optimization BIBREF25, BIBREF26 
One of our key contributions is the derivation of a closed-form expression for this Jacobian  The next proposition provides such an expression, enabling entmax layers with adaptive $\alpha $  To the best of our knowledge, ours is the first neural network module that can automatically, continuously vary in shape away from softmax and toward sparse mappings like sparsemax 
Proposition 1 Let $\mathbf {p}^\star \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ be the solution of Equation DISPLAY_FORM14  Denote the distribution $\tilde{p}_i {(p_i^\star )^{2 - \alpha }}{ \sum _j(p_j^\star )^{2-\alpha }}$ and let $h_i -p^\star _i \log p^\star _i$  The $i$th component of the Jacobian $\mathbf {g} \frac{\partial \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})}{\partial \alpha }$ is
proof uses implicit function differentiation and is given in Appendix SECREF10 
Proposition UNKREF22 provides the remaining missing piece needed for training adaptively sparse Transformers  In the following section, we evaluate this strategy on neural machine translation, and analyze the behavior of the learned attention heads 
Experiments
We apply our adaptively sparse Transformers on four machine translation tasks  For comparison, a natural baseline is the standard Transformer architecture using the softmax transform in its multi-head attention mechanisms  We consider two other model variants in our experiments that make use of different normalizing transformations:
1 5-entmax: a Transformer with sparse entmax attention with fixed $\alpha =1 5$ for all heads  This is a novel model, since 1 5-entmax had only been proposed for RNN-based NMT models BIBREF14, but never in Transformers, where attention modules are not just one single component of the seq2seq model but rather an integral part of all of the model components 
$\alpha $-entmax: an adaptive Transformer with sparse entmax attention with a different, learned $\alpha _{i,j}^t$ for each head 
The adaptive model has an additional scalar parameter per attention head per layer for each of the three attention mechanisms (encoder self-attention, context attention, and decoder self-attention), i e ,
and we set $\alpha _{i,j}^t = 1 + \operatornamewithlimits{\mathsf {sigmoid}}(a_{i,j}^t) \in ]1, 2[$  All or some of the $\alpha $ values can be tied if desired, but we keep them independent for analysis purposes 
Experiments ::: Datasets 
Our models were trained on 4 machine translation datasets of different training sizes:
[itemsep= 5ex,leftmargin=2ex]
IWSLT 2017 German $\rightarrow $ English BIBREF27: 200K sentence pairs 
KFTT Japanese $\rightarrow $ English BIBREF28: 300K sentence pairs 
WMT 2016 Romanian $\rightarrow $ English BIBREF29: 600K sentence pairs 
WMT 2014 English $\rightarrow $ German BIBREF30: 4 5M sentence pairs 
All of these datasets were preprocessed with byte-pair encoding BIBREF31, using joint segmentations of 32k merge operations 
Experiments ::: Training 
We follow the dimensions of the Transformer-Base model of BIBREF0: The number of layers is $L=6$ and number of heads is $H=8$ in the encoder self-attention, the context attention, and the decoder self-attention  We use a mini-batch size of 8192 tokens and warm up the learning rate linearly until 20k steps, after which it decays according to an inverse square root schedule  All models were trained until convergence of validation accuracy, and evaluation was done at each 10k steps for ro$\rightarrow $en and en$\rightarrow $de and at each 5k steps for de$\rightarrow $en and ja$\rightarrow $en  The end-to-end computational overhead of our methods, when compared to standard softmax, is relatively small; in training tokens per second, the models using $\alpha $-entmax and $1 5$-entmax are, respectively, $75\%$ and $90\%$ the speed of the softmax model 
Experiments ::: Results 
We report test set tokenized BLEU BIBREF32 results in Table TABREF27  We can see that replacing softmax by entmax does not hurt performance in any of the datasets; indeed, sparse attention Transformers tend to have slightly higher BLEU, but their sparsity leads to a better potential for analysis  In the next section, we make use of this potential by exploring the learned internal mechanics of the self-attention heads 
Analysis
We conduct an analysis for the higher-resource dataset WMT 2014 English $\rightarrow $ German of the attention in the sparse adaptive Transformer model ($\alpha $-entmax) at multiple levels: we analyze high-level statistics as well as individual head behavior  Moreover, we make a qualitative analysis of the interpretability capabilities of our models 
Analysis ::: High-Level Statistics ::: What kind of @!START@$\alpha $@!END@ values are learned?
Figure FIGREF37 shows the learning trajectories of the $\alpha $ parameters of a selected subset of heads  We generally observe a tendency for the randomly-initialized $\alpha $ parameters to decrease initially, suggesting that softmax-like behavior may be preferable while the model is still very uncertain  After around one thousand steps, some heads change direction and become sparser, perhaps as they become more confident and specialized  This shows that the initialization of $\alpha $ does not predetermine its sparsity level or the role the head will have throughout  In particular, head 8 in the encoder self-attention layer 2 first drops to around $\alpha =1 3$ before becoming one of the sparsest heads, with $\alpha \approx 2$ 
The overall distribution of $\alpha $ values at convergence can be seen in Figure FIGREF38  We can observe that the encoder self-attention blocks learn to concentrate the $\alpha $ values in two modes: a very sparse one around $\alpha \rightarrow 2$, and a dense one between softmax and 1 5-entmax   However, the decoder self and context attention only learn to distribute these parameters in a single mode  We show next that this is reflected in the average density of attention weight vectors as well 
Analysis ::: High-Level Statistics ::: Attention weight density when translating 
For any $\alpha >1$, it would still be possible for the weight matrices in Equation DISPLAY_FORM9 to learn re-scalings so as to make attention sparser or denser  To visualize the impact of adaptive $\alpha $ values, we compare the empirical attention weight density (the average number of tokens receiving non-zero attention) within each module, against sparse Transformers with fixed $\alpha =1 5$ 
Figure FIGREF40 shows that, with fixed $\alpha =1 5$, heads tend to be sparse and similarly-distributed in all three attention modules  With learned $\alpha $, there are two notable changes: (i) a prominent mode corresponding to fully dense probabilities, showing that our models learn to combine sparse and dense attention, and (ii) a distinction between the encoder self-attention – whose background distribution tends toward extreme sparsity – and the other two modules, who exhibit more uniform background distributions  This suggests that perhaps entirely sparse Transformers are suboptimal 
The fact that the decoder seems to prefer denser attention distributions might be attributed to it being auto-regressive, only having access to past tokens and not the full sentence  We speculate that it might lose too much information if it assigned weights of zero to too many tokens in the self-attention, since there are fewer tokens to attend to in the first place 
Teasing this down into separate layers, Figure FIGREF41 shows the average (sorted) density of each head for each layer  We observe that $\alpha $-entmax is able to learn different sparsity patterns at each layer, leading to more variance in individual head behavior, to clearly-identified dense and sparse heads, and overall to different tendencies compared to the fixed case of $\alpha =1 5$ 
Analysis ::: High-Level Statistics ::: Head diversity 
To measure the overall disagreement between attention heads, as a measure of head diversity, we use the following generalization of the Jensen-Shannon divergence:
where $\mathbf {p}_j$ is the vector of attention weights assigned by head $j$ to each word in the sequence, and $\mathsf {H}^\textsc {S}$ is the Shannon entropy, base-adjusted based on the dimension of $\mathbf {p}$ such that $JS \le 1$  We average this measure over the entire validation set  The higher this metric is, the more the heads are taking different roles in the model 
Figure FIGREF44 shows that both sparse Transformer variants show more diversity than the traditional softmax one  Interestingly, diversity seems to peak in the middle layers of the encoder self-attention and context attention, while this is not the case for the decoder self-attention 
The statistics shown in this section can be found for the other language pairs in Appendix SECREF8 
Analysis ::: Identifying Head Specializations
Previous work pointed out some specific roles played by different heads in the softmax Transformer model BIBREF33, BIBREF5, BIBREF9  Identifying the specialization of a head can be done by observing the type of tokens or sequences that the head often assigns most of its attention weight; this is facilitated by sparsity 
Analysis ::: Identifying Head Specializations ::: Positional heads 
One particular type of head, as noted by BIBREF9, is the positional head  These heads tend to focus their attention on either the previous or next token in the sequence, thus obtaining representations of the neighborhood of the current time step  In Figure FIGREF47, we show attention plots for such heads, found for each of the studied models  The sparsity of our models allows these heads to be more confident in their representations, by assigning the whole probability distribution to a single token in the sequence  Concretely, we may measure a positional head's confidence as the average attention weight assigned to the previous token  The softmax model has three heads for position $-1$, with median confidence $93 5\%$  The $1 5$-entmax model also has three heads for this position, with median confidence $94 4\%$  The adaptive model has four heads, with median confidences $95 9\%$, the lowest-confidence head being dense with $\alpha =1 18$, while the highest-confidence head being sparse ($\alpha =1 91$) 
For position $+1$, the models each dedicate one head, with confidence around $95\%$, slightly higher for entmax  The adaptive model sets $\alpha =1 96$ for this head 
Analysis ::: Identifying Head Specializations ::: BPE-merging head 
Due to the sparsity of our models, we are able to identify other head specializations, easily identifying which heads should be further analysed  In Figure FIGREF51 we show one such head where the $\alpha $ value is particularly high (in the encoder, layer 1, head 4 depicted in Figure FIGREF37)  We found that this head most often looks at the current time step with high confidence, making it a positional head with offset 0  However, this head often spreads weight sparsely over 2-3 neighboring tokens, when the tokens are part of the same BPE cluster or hyphenated words  As this head is in the first layer, it provides a useful service to the higher layers by combining information evenly within some BPE clusters 
For each BPE cluster or cluster of hyphenated words, we computed a score between 0 and 1 that corresponds to the maximum attention mass assigned by any token to the rest of the tokens inside the cluster in order to quantify the BPE-merging capabilities of these heads  There are not any attention heads in the softmax model that are able to obtain a score over $80\%$, while for $1 5$-entmax and $\alpha $-entmax there are two heads in each ($83 3\%$ and $85 6\%$ for $1 5$-entmax and $88 5\%$ and $89 8\%$ for $\alpha $-entmax) 
Analysis ::: Identifying Head Specializations ::: Interrogation head 
On the other hand, in Figure FIGREF52 we show a head for which our adaptively sparse model chose an $\alpha $ close to 1, making it closer to softmax (also shown in encoder, layer 1, head 3 depicted in Figure FIGREF37)  We observe that this head assigns a high probability to question marks at the end of the sentence in time steps where the current token is interrogative, thus making it an interrogation-detecting head  We also observe this type of heads in the other models, which we also depict in Figure FIGREF52  The average attention weight placed on the question mark when the current token is an interrogative word is $98 5\%$ for softmax, $97 0\%$ for $1 5$-entmax, and $99 5\%$ for $\alpha $-entmax 
Furthermore, we can examine sentences where some tendentially sparse heads become less so, thus identifying sources of ambiguity where the head is less confident in its prediction  An example is shown in Figure FIGREF55 where sparsity in the same head differs for sentences of similar length 
Related Work ::: Sparse attention 
Prior work has developed sparse attention mechanisms, including applications to NMT BIBREF19, BIBREF12, BIBREF20, BIBREF22, BIBREF34  BIBREF14 introduced the entmax function this work builds upon  In their work, there is a single attention mechanism which is controlled by a fixed $\alpha $  In contrast, this is the first work to allow such attention mappings to dynamically adapt their curvature and sparsity, by automatically adjusting the continuous $\alpha $ parameter  We also provide the first results using sparse attention in a Transformer model 
Related Work ::: Fixed sparsity patterns 
Recent research improves the scalability of Transformer-like networks through static, fixed sparsity patterns BIBREF10, BIBREF35  Our adaptively-sparse Transformer can dynamically select a sparsity pattern that finds relevant words regardless of their position (e g , Figure FIGREF52)  Moreover, the two strategies could be combined  In a concurrent line of research, BIBREF11 propose an adaptive attention span for Transformer language models  While their work has each head learn a different contiguous span of context tokens to attend to, our work finds different sparsity patterns in the same span  Interestingly, some of their findings mirror ours – we found that attention heads in the last layers tend to be denser on average when compared to the ones in the first layers, while their work has found that lower layers tend to have a shorter attention span compared to higher layers 
Related Work ::: Transformer interpretability 
The original Transformer paper BIBREF0 shows attention visualizations, from which some speculation can be made of the roles the several attention heads have  BIBREF7 study the syntactic abilities of the Transformer self-attention, while BIBREF6 extract dependency relations from the attention weights  BIBREF8 find that the self-attentions in BERT BIBREF3 follow a sequence of processes that resembles a classical NLP pipeline  Regarding redundancy of heads, BIBREF9 develop a method that is able to prune heads of the multi-head attention module and make an empirical study of the role that each head has in self-attention (positional, syntactic and rare words)  BIBREF36 also aim to reduce head redundancy by adding a regularization term to the loss that maximizes head disagreement and obtain improved results  While not considering Transformer attentions, BIBREF18 show that traditional attention mechanisms do not necessarily improve interpretability since softmax attention is vulnerable to an adversarial attack leading to wildly different model predictions for the same attention weights  Sparse attention may mitigate these issues; however, our work focuses mostly on a more mechanical aspect of interpretation by analyzing head behavior, rather than on explanations for predictions 
Conclusion and Future Work
We contribute a novel strategy for adaptively sparse attention, and, in particular, for adaptively sparse Transformers  We present the first empirical analysis of Transformers with sparse attention mappings (i e , entmax), showing potential in both translation accuracy as well as in model interpretability 
In particular, we analyzed how the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence  Our adaptivity strategy relies only on gradient-based optimization, side-stepping costly per-head hyper-parameter searches  Further speed-ups are possible by leveraging more parallelism in the bisection algorithm for computing $\alpha $-entmax 
Finally, some of the automatically-learned behaviors of our adaptively sparse Transformers – for instance, the near-deterministic positional heads or the subword joining head – may provide new ideas for designing static variations of the Transformer 
Acknowledgments
This work was supported by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contracts UID/EEA/50008/2019 and CMUPERI/TIC/0046/2014 (GoLocal)  We are grateful to Ben Peters for the $\alpha $-entmax code and Erick Fonseca, Marcos Treviso, Pedro Martins, and Tsvetomila Mihaylova for insightful group discussion  We thank Mathieu Blondel for the idea to learn $\alpha $  We would also like to thank the anonymous reviewers for their helpful feedback 
Supplementary Material
Background ::: Regularized Fenchel-Young prediction functions
Definition 1 (BIBREF23)
Let $\Omega \colon \triangle ^d \rightarrow {\mathbb {R}}\cup \lbrace \infty \rbrace $ be a strictly convex regularization function  We define the prediction function $\mathbf {\pi }_{\Omega }$ as
Background ::: Characterizing the @!START@$\alpha $@!END@-entmax mapping
Lemma 1 (BIBREF14) For any $\mathbf {z}$, there exists a unique $\tau ^\star $ such that
Proof: From the definition of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$,
we may easily identify it with a regularized prediction function (Def  UNKREF81):
We first note that for all $\mathbf {p}\in \triangle ^d$,
From the constant invariance and scaling properties of $\mathbf {\pi }_{\Omega }$ BIBREF23,
Using BIBREF23, noting that $g^{\prime }(t) = t^{\alpha - 1}$ and $(g^{\prime })^{-1}(u) = u^{{1}{\alpha -1}}$, yields
Since $\mathsf {H}^{\textsc {T}}_\alpha $ is strictly convex on the simplex, $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ has a unique solution $\mathbf {p}^\star $  Equation DISPLAY_FORM88 implicitly defines a one-to-one mapping between $\mathbf {p}^\star $ and $\tau ^\star $ as long as $\mathbf {p}^\star \in \triangle $, therefore $\tau ^\star $ is also unique 
Background ::: Connections to softmax and sparsemax
The Euclidean projection onto the simplex, sometimes referred to, in the context of neural attention, as sparsemax BIBREF19, is defined as
The solution can be characterized through the unique threshold $\tau $ such that $\sum _i \operatornamewithlimits{\mathsf {sparsemax}}(\mathbf {z})_i = 1$ and BIBREF38
Thus, each coordinate of the sparsemax solution is a piecewise-linear function  Visibly, this expression is recovered when setting $\alpha =2$ in the $\alpha $-entmax expression (Equation DISPLAY_FORM85); for other values of $\alpha $, the exponent induces curvature 
On the other hand, the well-known softmax is usually defined through the expression
which can be shown to be the unique solution of the optimization problem
where $\mathsf {H}^\textsc {S}(\mathbf {p}) -\sum _i p_i \log p_i$ is the Shannon entropy  Indeed, setting the gradient to 0 yields the condition $\log p_i = z_j - \nu _i - \tau - 1$, where $\tau $ and $\nu > 0$ are Lagrange multipliers for the simplex constraints $\sum _i p_i = 1$ and $p_i \ge 0$, respectively  Since the l h s  is only finite for $p_i>0$, we must have $\nu _i=0$ for all $i$, by complementary slackness  Thus, the solution must have the form $p_i = {\exp (z_i)}{Z}$, yielding Equation DISPLAY_FORM92 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@
Recall that the entmax transformation is defined as:
where $\alpha \ge 1$ and $\mathsf {H}^{\textsc {T}}_{\alpha }$ is the Tsallis entropy,
and $\mathsf {H}^\textsc {S}(\mathbf {p}):= -\sum _j p_j \log p_j$ is the Shannon entropy 
In this section, we derive the Jacobian of $\operatornamewithlimits{\mathsf {entmax }}$ with respect to the scalar parameter $\alpha $ 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: General case of @!START@$\alpha >1$@!END@
From the KKT conditions associated with the optimization problem in Eq  DISPLAY_FORM85, we have that the solution $\mathbf {p}^{\star }$ has the following form, coordinate-wise:
where $\tau ^{\star }$ is a scalar Lagrange multiplier that ensures that $\mathbf {p}^{\star }$ normalizes to 1, i e , it is defined implicitly by the condition:
For general values of $\alpha $, Eq  DISPLAY_FORM98 lacks a closed form solution  This makes the computation of the Jacobian
non-trivial  Fortunately, we can use the technique of implicit differentiation to obtain this Jacobian 
The Jacobian exists almost everywhere, and the expressions we derive expressions yield a generalized Jacobian BIBREF37 at any non-differentiable points that may occur for certain ($\alpha $, $\mathbf {z}$) pairs  We begin by noting that $\frac{\partial p_i^{\star }}{\partial \alpha } = 0$ if $p_i^{\star } = 0$, because increasing $\alpha $ keeps sparse coordinates sparse  Therefore we need to worry only about coordinates that are in the support of $\mathbf {p}^\star $  We will assume hereafter that the $i$th coordinate of $\mathbf {p}^\star $ is non-zero  We have:
We can see that this Jacobian depends on $\frac{\partial \tau ^{\star }}{\partial \alpha }$, which we now compute using implicit differentiation 
Let $\mathcal {S} = \lbrace i: p^\star _i > 0 \rbrace $)  By differentiating both sides of Eq  DISPLAY_FORM98, re-using some of the steps in Eq  DISPLAY_FORM101, and recalling Eq  DISPLAY_FORM97, we get
from which we obtain:
Finally, plugging Eq  DISPLAY_FORM103 into Eq  DISPLAY_FORM101, we get:
where we denote by
The distribution $\tilde{\mathbf {p}}(\alpha )$ can be interpreted as a “skewed” distribution obtained from $\mathbf {p}^{\star }$, which appears in the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ w r t  $\mathbf {z}$ as well BIBREF14 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Solving the indetermination for @!START@$\alpha =1$@!END@
We can write Eq  DISPLAY_FORM104 as
When $\alpha \rightarrow 1^+$, we have $\tilde{\mathbf {p}}(\alpha ) \rightarrow \mathbf {p}^{\star }$, which leads to a $\frac{0}{0}$ indetermination 
To solve this indetermination, we will need to apply L'Hôpital's rule twice  Let us first compute the derivative of $\tilde{p}_i(\alpha )$ with respect to $\alpha $  We have
therefore
Differentiating the numerator and denominator in Eq  DISPLAY_FORM107, we get:
with
and
When $\alpha \rightarrow 1^+$, $B$ becomes again a $\frac{0}{0}$ indetermination, which we can solve by applying again L'Hôpital's rule  Differentiating the numerator and denominator in Eq  DISPLAY_FORM112:
Finally, summing Eq  DISPLAY_FORM111 and Eq  DISPLAY_FORM113, we get
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Summary
To sum up, we have the following expression for the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with respect to $\alpha $:
Context 2: Introduction
As social media, specially Twitter, takes on an influential role in presidential elections in the U S , natural language processing of political tweets BIBREF0 has the potential to help with nowcasting and forecasting of election results as well as identifying the main issues with a candidate – tasks of much interest to journalists, political scientists, and campaign organizers BIBREF1  As a methodology to obtain training data for a machine learning system that analyzes political tweets, BIBREF2 devised a crowdsourcing scheme with variable crowdworker numbers based on the difficulty of the annotation task  They provided a dataset of tweets where the sentiments towards political candidates were labeled both by experts in political communication and by crowdworkers who were likely not domain experts  BIBREF2 revealed that crowdworkers can match expert performance relatively accurately and in a budget-efficient manner  Given this result, the authors envisioned future work in which groundtruth labels would be crowdsourced for a large number of tweets and then used to design an automated NLP tool for political tweet analysis 
The question we address here is: How accurate are existing NLP tools for political tweet analysis? These tools would provide a baseline performance that any new machine learning system for political tweet analysis would compete against  We here explore whether existing NLP systems can answer the questions "What sentiment?" and "Towards whom?" accurately for the dataset of political tweets provided by BIBREF2  In our analysis, we include NLP tools with publicly-available APIs, even if the tools were not specifically designed for short texts like tweets, and, in particular, political tweets 
Our experiments reveal that the task of entity-level sentiment analysis is difficult for existing tools to answer accurately while the recognition of the entity, here, which politician, was easier 
NLP Toolkits
NLP toolkits typically have the following capabilities: tokenization, part-of-speech (PoS) tagging, chunking, named entity recognition and sentiment analysis  In a study by BIBREF3, it is shown that the well-known NLP toolkits NLTK BIBREF4, Stanford CoreNLP BIBREF5, and TwitterNLP BIBREF6 have tokenization, PoS tagging and NER modules in their pipelines  There are two main approaches for NER: (1) rule-based and (2) statistical or machine learning based  The most ubiquitous algorithms for sequence tagging use Hidden Markov Models BIBREF7, Maximum Entropy Markov Models BIBREF7, BIBREF8, or Conditional Random Fields BIBREF9  Recent works BIBREF10, BIBREF11 have used recurrent neural networks with attention modules for NER 
Sentiment detection tools like SentiStrength BIBREF12 and TensiStrength BIBREF13 are rule-based tools, relying on various dictionaries of emoticons, slangs, idioms, and ironic phrases, and set of rules that can detect the sentiment of a sentence overall or a targeted sentiment  Given a list of keywords, TensiStrength (similar to SentiStrength) reports the sentiment towards selected entities in a sentence, based on five levels of relaxation and five levels of stress 
Among commercial NLP toolkits (e g , BIBREF14, BIBREF15, BIBREF16), we selected BIBREF17 and BIBREF18 for our experiments, which, to the best of our knowledge, are the only publicly accessible commercial APIs for the task of entity-level sentiment analysis that is agnostic to the text domain  We also report results of TensiStrength BIBREF13, TwitterNLP BIBREF6, BIBREF19, CogComp-NLP BIBREF20, and Stanford NLP NER BIBREF21 
Dataset and Analysis Methodology
We used the 1,000-tweet dataset by BIBREF2 that contains the named-entities labels and entity-level sentiments for each of the four 2016 presidential primary candidates Bernie Sanders, Donald Trump, Hillary Clinton, and Ted Cruz, provided by crowdworkers, and by experts in political communication, whose labels are considered groundtruth  The crowdworkers were located in the US and hired on the BIBREF22 platform  For the task of entity-level sentiment analysis, a 3-scale rating of "negative," "neutral," and "positive" was used by the annotators 
BIBREF2 proposed a decision tree approach for computing the number of crowdworkers who should analyze a tweet based on the difficulty of the task  Tweets are labeled by 2, 3, 5, or 7 workers based on the difficulty of the task and the level of disagreement between the crowdworkers  The model computes the number of workers based on how long a tweet is, the presence of a link in a tweet, and the number of present sarcasm signals  Sarcasm is often used in political tweets and causes disagreement between the crowdworkers  The tweets that are deemed to be sarcastic by the decision tree model, are expected to be more difficult to annotate, and hence are allocated more crowdworkers to work on 
We conducted two sets of experiments  In the first set, we used BIBREF23, BIBREF17, and BIBREF18, for entity-level sentiment analysis; in the second set, BIBREF17, BIBREF19, BIBREF24, BIBREF25, and BIBREF26, BIBREF18 for named-entity recognition 
In the experiments that we conducted with TwitterNLP for named-entity recognition, we worked with the default values of the model  Furthermore, we selected the 3-class Stanford NER model, which uses the classes “person,” “organization,” and “location” because it resulted in higher accuracy compared to the 7-class model  For CogComp-NLP NER we used Ontonotes 5 0 NER model BIBREF27  For spaCy NER we used the `en_core_web_lg' model 
We report the experimental results for our two tasks in terms of the correct classification rate (CCR)  For sentiment analysis, we have a three-class problem (positive, negative, and neutral), where the classes are mutually exclusive  The CCR, averaged for a set of tweets, is defined to be the number of correctly-predicted sentiments over the number of groundtruth sentiments in these tweets  For NER, we consider that each tweet may reference up to four candidates, i e , targeted entities  The CCR, averaged for a set of tweets, is the number of correctly predicted entities (candidates) over the number of groundtruth entities (candidates) in this set 
Results and Discussion
The dataset of 1,000 randomly selected tweets contains more than twice as many tweets about Trump than about the other candidates
Context 3: Introduction
Propaganda aims at influencing people's mindset with the purpose of advancing a specific agenda  In the Internet era, thanks to the mechanism of sharing in social networks, propaganda campaigns have the potential of reaching very large audiences BIBREF0, BIBREF1, BIBREF2 
Propagandist news articles use specific techniques to convey their message, such as whataboutism, red Herring, and name calling, among many others (cf  Section SECREF3)  Whereas proving intent is not easy, we can analyse the language of a claim/article and look for the use of specific propaganda techniques  Going at this fine-grained level can yield more reliable systems and it also makes it possible to explain to the user why an article was judged as propagandist by an automatic system 
With this in mind, we organised the shared task on fine-grained propaganda detection at the NLP4IF@EMNLP-IJCNLP 2019 workshop  The task is based on a corpus of news articles annotated with an inventory of 18 propagandist techniques at the fragment level  We hope that the corpus would raise interest outside of the community of researchers studying propaganda  For example, the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
Related Work
Propaganda has been tackled mostly at the article level  BIBREF3 created a corpus of news articles labelled as propaganda, trusted, hoax, or satire  BIBREF4 experimented with a binarized version of that corpus: propaganda vs  the other three categories  BIBREF5 annotated a large binary corpus of propagandist vs  non-propagandist articles and proposed a feature-based system for discriminating between them  In all these cases, the labels were obtained using distant supervision, assuming that all articles from a given news outlet share the label of that outlet, which inevitably introduces noise BIBREF6 
A related field is that of computational argumentation which, among others, deals with some logical fallacies related to propaganda  BIBREF7 presented a corpus of Web forum discussions with instances of ad hominem fallacy  BIBREF8, BIBREF9 introduced Argotario, a game to educate people to recognize and create fallacies, a by-product of which is a corpus with $1 3k$ arguments annotated with five fallacies such as ad hominem, red herring and irrelevant authority, which directly relate to propaganda 
Unlike BIBREF8, BIBREF9, BIBREF7, our corpus uses 18 techniques annotated on the same set of news articles  Moreover, our annotations aim at identifying the minimal fragments related to a technique instead of flagging entire arguments 
The most relevant related work is our own, which is published in parallel to this paper at EMNLP-IJCNLP 2019 BIBREF10 and describes a corpus that is a subset of the one used for this shared task 
Propaganda Techniques
Propaganda uses psychological and rhetorical techniques to achieve its objective  Such techniques include the use of logical fallacies and appeal to emotions  For the shared task, we use 18 techniques that can be found in news articles and can be judged intrinsically, without the need to retrieve supporting information from external resources  We refer the reader to BIBREF10 for more details on the propaganda techniques; below we report the list of techniques:
Propaganda Techniques ::: 1  Loaded language 
Using words/phrases with strong emotional implications (positive or negative) to influence an audience BIBREF11 
Propaganda Techniques ::: 2  Name calling or labeling 
Labeling the object of the propaganda as something the target audience fears, hates, finds undesirable or otherwise loves or praises BIBREF12 
Propaganda Techniques ::: 3  Repetition 
Repeating the same message over and over again, so that the audience will eventually accept it BIBREF13, BIBREF12 
Propaganda Techniques ::: 4  Exaggeration or minimization 
Either representing something in an excessive manner: making things larger, better, worse, or making something seem less important or smaller than it actually is BIBREF14, e g , saying that an insult was just a joke 
Propaganda Techniques ::: 5  Doubt 
Questioning the credibility of someone or something 
Propaganda Techniques ::: 6  Appeal to fear/prejudice 
Seeking to build support for an idea by instilling anxiety and/or panic in the population towards an alternative, possibly based on preconceived judgments 
Propaganda Techniques ::: 7  Flag-waving 
Playing on strong national feeling (or with respect to a group, e g , race, gender, political preference) to justify or promote an action or idea BIBREF15 
Propaganda Techniques ::: 8  Causal oversimplification 
Assuming one cause when there are multiple causes behind an issue  We include scapegoating as well: the transfer of the blame to one person or group of people without investigating the complexities of an issue 
Propaganda Techniques ::: 9  Slogans 
A brief and striking phrase that may include labeling and stereotyping  Slogans tend to act as emotional appeals BIBREF16 
Propaganda Techniques ::: 10  Appeal to authority 
Stating that a claim is true simply because a valid authority/expert on the issue supports it, without any other supporting evidence BIBREF17  We include the special case where the reference is not an authority/expert, although it is referred to as testimonial in the literature BIBREF14 
Propaganda Techniques ::: 11  Black-and-white fallacy, dictatorship 
Presenting two alternative options as the only possibilities, when in fact more possibilities exist BIBREF13  As an extreme case, telling the audience exactly what actions to take, eliminating any other possible choice (dictatorship) 
Propaganda Techniques ::: 12  Thought-terminating cliché 
Words or phrases that discourage critical thought and meaningful discussion about a given topic  They are typically short and generic sentences that offer seemingly simple answers to complex questions or that distract attention away from other lines of thought BIBREF18 
Propaganda Techniques ::: 13  Whataboutism 
Discredit an opponent's position by charging them with hypocrisy without directly disproving their argument BIBREF19 
Propaganda Techniques ::: 14  Reductio ad Hitlerum 
Persuading an audience to disapprove an action or idea by suggesting that the idea is popular with groups hated in contempt by the target audience  It can refer to any person or concept with a negative connotation BIBREF20 
Propaganda Techniques ::: 15  Red herring 
Introducing irrelevant material to the issue being discussed, so that everyone's attention is diverted away from the points made BIBREF11  Those subjected to a red herring argument are led away from the issue that had been the focus of the discussion and urged to follow an observation or claim that may be associated with the original claim, but is not highly relevant to the issue in dispute BIBREF20 
Propaganda Techniques ::: 16  Bandwagon 
Attempting to persuade the target audience to join in and take the course of action because “everyone else is taking the same action” BIBREF15 
Propaganda Techniques ::: 17  Obfuscation, intentional vagueness, confusion 
Using deliberately unclear words, to let the audience have its own interpretation BIBREF21, BIBREF11  For instance, when an unclear phrase with multiple possible meanings is used within the argument and, therefore, it does not really support the conclusion 
Propaganda Techniques ::: 18  Straw man 
When an opponent's proposition is substituted with a similar one which is then refuted in place of the original BIBREF22 
Tasks
The shared task features two subtasks:
Tasks ::: Fragment-Level Classification task (FLC) 
Given a news article, detect all spans of the text in which a propaganda technique is used  In addition, for each span the propaganda technique applied must be identified

ground_truth: four machine translation tasks: German -> English, Japanese -> English, Romanian -> English, English -> German

 four machine translation tasks, IWSLT 2017 German $\rightarrow $ English BIBREF27, KFTT Japanese $\rightarrow $ English BIBREF28, WMT 2016 Romanian $\rightarrow $ English BIBREF29, WMT 2014 English $\rightarrow $ German BIBREF30

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Is datasets for sentiment analysis balanced?
Context 2:  In the named-entity recognition experiment, the average CCR of crowdworkers was 98 6%, while the CCR of the automated systems ranged from 77 2% to 96 7%  For four of the automated systems, detecting the entity Trump was more difficult than the other entities (e g , spaCy 72 7% for the entity Trump vs  above 91% for the other entities)  An example of incorrect NER is shown in Figure FIGREF1 top  The difficulties the automated tools had in NER may be explained by the fact that the tools were not trained on tweets, except for TwitterNLP, which was not in active development when the data was created BIBREF1 
In the sentiment analysis experiments, we found that a tweet may contain multiple sentiments  The groundtruth labels contain 210 positive sentiments, 521 neutral sentiments, and 305 negative sentiments to the candidates  We measured the CCR, across all tweets, to be 31 7% for Rosette Text Analytics, 43 2% for Google Cloud, 44 2% for TensiStrength, and 74 7% for the crowdworkers  This means the difference between the performance of the tools and the crowdworkers is significant – more than 30 percent points 
Crowdworkers correctly identified 62% of the neutral, 85% of the positive, and 92% of the negative sentiments  Google Cloud correctly identified 88% of the neutral sentiments, but only 3% of the positive, and 19% of the negative sentiments  TensiStrength correctly identified 87 2% of the neutral sentiments, but 10 5% of the positive, and 8 1% of the negative sentiments  Rosette Text Analytics correctly identified 22 7% of neutral sentiments, 38 1% of negative sentiments and 40 9% of positive sentiments  The lowest and highest CCR pertains to tweets about Trump and Sanders for both Google Cloud and TensiStrength, Trump and Clinton for Rosette Text Analytics, and Clinton and Cruz for crowdworkers  An example of incorrect ELS analysis is shown in Figure FIGREF1 bottom 
Conclusions and Future Work
Our results show that existing NLP systems cannot accurately perform sentiment analysis of political tweets in the dataset we experimented with  Labeling by humans, even non-expert crowdworkers, yields accuracy results that are well above the results of existing automated NLP systems  In future work we will therefore use a crowdworker-labeled dataset to train a new machine-learning based NLP system for tweet analysis  We will ensure that the training data is balanced among classes  Our plan is to use state-of-the-art deep neural networks and compare their performance for entity-level sentiment analysis of political tweets 
Acknowledgments
Partial support of this work by the Hariri Institute for Computing and Computational Science & Engineering at Boston University (to L G ) and a Google Faculty Research Award (to M B  and L G ) is gratefully acknowledged  Additionally, we would like to thank Daniel Khashabi for his help in running the CogComp-NLP Python API and Mike Thelwal for his help with TensiStrength  We are also grateful to the Stanford NLP group for clarifying some of the questions we had with regards to the Stanford NER tool 
Context 3: 
Tasks ::: Sentence-Level Classification task (SLC) 
A sentence is considered propagandist if it contains at least one propagandist fragment  We then define a binary classification task in which, given a sentence, the correct label, either propaganda or non-propaganda, is to be predicted 
Data
The input for both tasks consists of news articles in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets and then annotated by professional annotators  More details about the data collection and the annotation, as well as statistics about the corpus can be found in BIBREF10, where an earlier version of the corpus is described, which includes 450 news articles  We further annotated 47 additional articles for the purpose of the shared task using the same protocol and the same annotators 
The training, the development, and the test partitions of the corpus used for the shared task consist of 350, 61, and 86 articles and of 16,965, 2,235, and 3,526 sentences, respectively  Figure FIGREF15 shows an annotated example, which contains several propaganda techniques  For example, the fragment babies on line 1 is an instance of both Name_Calling and Labeling  Note that the fragment not looking as though Trump killed his grandma on line 4 is an instance of Exaggeration_or_Minimisation and it overlaps with the fragment killed his grandma, which is an instance of Loaded_Language 
Table TABREF23 reports the total number of instances per technique and the percentage with respect to the total number of annotations, for the training and for the development sets 
Setup
The shared task had two phases: In the development phase, the participants were provided labeled training and development datasets; in the testing phase, testing input was further provided 
The participants tried to achieve the best performance on the development set  A live leaderboard kept track of the submissions 
The test set was released and the participants had few days to make final predictions 
In phase 2, no immediate feedback on the submissions was provided  The winner was determined based on the performance on the test set 
Evaluation ::: FLC task 
FLC is a composition of two subtasks: the identification of the propagandist text fragments and the identification of the techniques used (18-way classification task)  While F$_1$ measure is appropriate for a multi-class classification task, we modified it to account for partial matching between the spans; see BIBREF10 for more details  We further computed an F$_1$ value for each propaganda technique (not shown below for the sake of saving space, but available on the leaderboard) 
Evaluation ::: SLC task 
SLC is a binary classification task with imbalanced data  Therefore, the official evaluation measure for the task is the standard F$_1$ measure  We further report Precision and Recall 
Baselines
The baseline system for the SLC task is a very simple logistic regression classifier with default parameters, where we represent the input instances with a single feature: the length of the sentence  The performance of this baseline on the SLC task is shown in Tables TABREF33 and TABREF34 
The baseline for the FLC task generates spans and selects one of the 18 techniques randomly  The inefficacy of such a simple random baseline is illustrated in Tables TABREF36 and TABREF41 
Participants and Approaches
A total of 90 teams registered for the shared task, and 39 of them submitted predictions for a total of 3,065 submissions  For the FLC task, 21 teams made a total of 527 submissions, and for the SLC task, 35 teams made a total of 2,538 submissions 
Below, we give an overview of the approaches as described in the participants' papers  Tables TABREF28 and TABREF29 offer a high-level summary 
Participants and Approaches ::: Teams Participating in the Fragment-Level Classification Only
Team newspeak BIBREF23 achieved the best results on the test set for the FLC task using 20-way word-level classification based on BERT BIBREF24: a word could belong to one of the 18 propaganda techniques, to none of them, or to an auxiliary (token-derived) class  The team fed one sentence at a time in order to reduce the workload  In addition to experimenting with an out-of-the-box BERT, they also tried unsupervised fine-tuning both on the 1M news dataset and on Wikipedia  Their best model was based on the uncased base model of BERT, with 12 Transformer layers BIBREF25, and 110 million parameters  Moreover, oversampling of the least represented classes proved to be crucial for the final performance  Finally, careful analysis has shown that the model pays special attention to adjectives and adverbs 
Team Stalin BIBREF26 focused on data augmentation to address the relatively small size of the data for fine-tuning contextual embedding representations based on ELMo BIBREF27, BERT, and Grover BIBREF28  The balancing of the embedding space was carried out by means of synthetic minority class over-sampling  Then, the learned representations were fed into an LSTM 
Participants and Approaches ::: Teams Participating in the Sentence-Level Classification Only
Team CAUnLP BIBREF29 used two context-aware representations based on BERT  In the first representation, the target sentence is followed by the title of the article  In the second representation, the previous sentence is also added  They performed subsampling in order to deal with class imbalance, and experimented with BERT$_{BASE}$ and BERT$_{LARGE}$
Team LIACC BIBREF30 used hand-crafted features and pre-trained ELMo embeddings  They also observed a boost in performance when balancing the dataset by dropping some negative examples 
Team JUSTDeep BIBREF31 used a combination of models and features, including word embeddings based on GloVe BIBREF32 concatenated with vectors representing affection and lexical features  These were combined in an ensemble of supervised models: bi-LSTM, XGBoost, and variations of BERT 
Team YMJA BIBREF33 also based their approach on fine-tuned BERT  Inspired by kaggle competitions on sentiment analysis, they created an ensemble of models via cross-validation 
Team jinfen BIBREF34 used a logistic regression model fed with a manifold of representations, including TF IDF and BERT vectors, as well as vocabularies and readability measures 
Team Tha3aroon BIBREF35 implemented an ensemble of three classifiers: two based on BERT and one based on a universal sentence encoder BIBREF36 
Team NSIT BIBREF37 explored three of the most popular transfer learning models: various versions of ELMo, BERT, and RoBERTa BIBREF38 
Team Mindcoders BIBREF39 combined BERT, Bi-LSTM and Capsule networks BIBREF40 into a single deep neural network and pre-trained the resulting network on corpora used for related tasks, e g , emotion classification 
Finally, team ltuorp BIBREF41 used an attention transformer using BERT trained on Wikipedia and BookCorpus 
Participants and Approaches ::: Teams Participating in Both Tasks
Team MIC-CIS BIBREF42 participated in both tasks  For the sentence-level classification, they used a voting ensemble including logistic regression, convolutional neural networks, and BERT, in all cases using FastText embeddings BIBREF43 and pre-trained BERT models  Beside these representations, multiple features of readability, sentiment and emotions were considered  For the fragment-level task, they used a multi-task neural sequence tagger, based on LSTM-CRF BIBREF44, in conjunction with linguistic features  Finally, they applied sentence- and fragment-level models jointly 
Team CUNLP BIBREF45 considered two approaches for the sentence-level task  The first approach was based on fine-tuning BERT  The second approach complemented the fine-tuned BERT approach by feeding its decision into a logistic regressor, together with features from the Linguistic Inquiry and Word Count (LIWC) lexicon and punctuation-derived features  Similarly to BIBREF42, for the fragment-level problem they used a Bi-LSTM-CRF architecture, combining both character- and word-level embeddings 
Team ProperGander BIBREF46 also used BERT, but they paid special attention to the imbalance of the data, as well as to the differences between training and testing  They showed that augmenting the training data by oversampling yielded improvements when testing on data that is temporally far from the training (by increasing recall)  In order to deal with the imbalance, they performed cost-sensitive classification, i e , the errors on the smaller positive class were more costly  For the fragment-level classification, inspired by named entity recognition, they used a model based on BERT using Continuous Random Field stacked on top of an LSTM 
Evaluation Results
The results on the test set for the SLC task are shown in Table TABREF33, while Table TABREF34 presents the results on the development set at the end of phase 1 (cf  Section SECREF6)  The general decrease of the F$_1$ values between the development and the test set could indicate that systems tend to overfit on the development set  Indeed, the winning team ltuorp chose the parameters of their system both on the development set and on a subset of the training set in order to improve the robustness of their system 
Tables TABREF36 and TABREF41 report the results on the test and on the development sets for the FLC task  For this task, the results tend to be more stable across the two sets  Indeed, team newspeak managed to almost keep the same difference in performance with respect to team Antiganda  Note that team MIC-CIS managed to reach the third position despite never having submitted a run on the development set 
Conclusion and Further Work
We have described the NLP4IF@EMNLP-IJCNLP 2019 shared task on fine-grained propaganda identification  We received 25 and 12 submissions on the test set for the sentence-level classification and the fragment-level classification tasks, respectively  Overall, the sentence-level task was easier and most submitted systems managed to outperform the baseline  The fragment-level task proved to be much more challenging, with lower absolute scores, but most teams still managed to outperform the baseline 
We plan to make the schema and the dataset publicly available to be used beyond NLP4IF  We hope that the corpus would raise interest outside of the community of researchers studying propaganda: the techniques related to fallacies and the ones relying on emotions might provide a novel setting for researchers interested in Argumentation and Sentiment Analysis 
As a kind of advertisement, Task 11 at SemEval 2020 is a follow up of this shared task  It features two complimentary tasks:
Given a free-text article, identify the propagandist text spans 
Given a text span already flagged as propagandist and its context, identify the specific propaganda technique it contains 
This setting would allow participants to focus their efforts on binary sequence labeling for Task 1 and on multi-class classification for Task 2 
Acknowledgments
This research is part of the Propaganda Analysis Project, which is framed within the Tanbih project  The Tanbih project aims to limit the effect of “fake news”, propaganda, and media bias by making users aware of what they are reading, thus promoting media literacy and critical thinking, which is arguably the best way to address disinformation and “fake news ” The project is developed in collaboration between the Qatar Computing Research Institute (QCRI), HBKU and the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) 
The corpus for the task was annotated by A Data Pro, a company that performs high-quality manual annotations 

ground_truth: No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was the baseline for this task?
Context 4: 
Rebuttal is a statement that attacks the claim, thus playing a role of an opposing view  In reality, the authors often attack the presented rebuttals by another counter-rebuttal in order to keep the whole argument's position consistent  Thus we introduced a new component – refutation – which is used for attacking the rebuttal  Annotation of refutation was conditioned of explicit presence of rebuttal and enforced by the annotation guidelines  The chain rebuttal–refutation is also known as the procatalepsis figure in rhetoric, in which the speaker raises an objection to his own argument and then immediately answers it  By doing so, the speaker hopes to strengthen the argument by dealing with possible counter-arguments before the audience can raise them BIBREF43  
The claim of the argument should always reflect the main standpoint with respect to the discussed controversy  We observed that this standpoint is not always explicitly expressed, but remains implicit and must be inferred by the reader  Therefore, we allow the claim to be implicit  In such a case, the annotators must explicitly write down the (inferred) stance of the author 
By definition, the Toulmin's model is intended to model single argument, with the claim in its center  However, we observed in our data, that some authors elaborate on both sides of the controversy equally and put forward an argument for each side (by argument here we mean the claim and its premises, backings, etc )  Therefore we allow multiple arguments to be annotated in one document  At the same time, we restrained the annotators from creating complex argument hierarchies 
Toulmin's grounds have an equivalent role to a premise in the classical view on an argument BIBREF15 , BIBREF60 in terms that they offer the reasons why one should accept the standpoint expressed by the claim  As this terminology has been used in several related works in the argumentation mining field BIBREF7 , BIBREF61 , BIBREF62 , BIBREF11 , we will keep this convention and denote the grounds as premises 
One of the main critiques of the original Toulmin's model was the vague distinction between grounds, warrant, and backing BIBREF63 , BIBREF64 , BIBREF65   The role of backing is to give additional support to the warrant, but there is no warrant in our model anymore  However, what we observed during the analysis, was a presence of some additional evidence  Such evidence does not play the role of the grounds (premises) as it is not meant as a reason supporting the claim, but it also does not explain the reasoning, thus is not a warrant either  It usually supports the whole argument and is stated by the author as a certain fact  Therefore, we extended the scope of backing as an additional support to the whole argument 
The annotators were instructed to distinguish between premises and backing, so that premises should cover generally applicable reasons for the claim, whereas backing is a single personal experience or statements that give credibility or attribute certain expertise to the author  As a sanity check, the argument should still make sense after removing backing (would be only considered “weaker”) 
We call the model as a modified Toulmin's model  It contains five argument components, namely claim, premise, backing, rebuttal, and refutation  When annotating a document, any arbitrary token span can be labeled with an argument component; the components do not overlap  The spans are not known in advance and the annotator thus chooses the span and the component type at the same time  All components are optional (they do not have to be present in the argument) except the claim, which is either explicit or implicit (see above)  If a token span is not labeled by any argument component, it is not considered as a part of the argument and is later denoted as none (this category is not assigned by the annotators) 
An example analysis of a forum post is shown in Figure FIGREF65   Figure FIGREF66 then shows a diagram of the analysis from that example (the content of the argument components was shortened or rephrased) 
The annotation experiment was split into three phases  All documents were annotated by three independent annotators, who participated in two training sessions  During the first phase, 50 random comments and forum posts were annotated  Problematic cases were resolved after discussion and the guidelines were refined  In the second phase, we wanted to extend the range of annotated registers, so we selected 148 comments and forum posts as well as 41 blog posts  After the second phase, the annotation guidelines were final 
In the final phase, we extended the range of annotated registers and added newswire articles from the raw corpus in order to test whether the annotation guidelines (and inherently the model) is general enough  Therefore we selected 96 comments/forum posts, 8 blog posts, and 8 articles for this phase  A detailed inter-annotator agreement study on documents from this final phase will be reported in section UID75  
The annotations were very time-consuming  In total, each annotator spent 35 hours by annotating in the course of five weeks  Discussions and consolidation of the gold data took another 6 hours  Comments and forum posts required on average of 4 minutes per document to annotate, while blog posts and articles on average of 14 minutes per document  Examples of annotated documents from the gold data are listed in Appendix UID158  
We discarded 11 documents out of the total 351 annotated documents  Five forum posts, although annotated as persuasive in the first annotation study, were at a deeper look a mixture of two or more posts with missing quotations, therefore unsuitable for analyzing argumentation  Three blog posts and two articles were found not to be argumentative (the authors took no stance to the discussed controversy) and one article was an interview, which the current model cannot capture (a dialogical argumentation model would be required) 
For each of the 340 documents, the gold standard annotations were obtained using the majority vote  If simple majority voting was not possible (different boundaries of the argument component together with a different component label), the gold standard was set after discussion among the annotators  We will refer to this corpus as the gold standard Toulmin corpus  The distribution of topics and registers in this corpus in shown in Table TABREF71 , and Table TABREF72 presents some lexical statistics 
Based on pre-studies, we set the minimal unit for annotation as token  The documents were pre-segmented using the Stanford Core NLP sentence splitter BIBREF69 embedded in the DKPro Core framework BIBREF70   Annotators were asked to stick to the sentence level by default and label entire pre-segmented sentences  They should switch to annotations on the token level only if (a) a particular sentence contained more than one argument component, or (b) if the automatic sentence segmentation was wrong  Given the “noise” in user-generated Web data (wrong or missing punctuation, casing, etc ), this was often the case 
Annotators were also asked to rephrase (summarize) each annotated argument component into a simple statement when applicable, as shown in Figure FIGREF66   This was used as a first sanity checking step, as each argument component is expected to be a coherent discourse unit  For example, if a particular occurrence of a premise cannot be summarized/rephrased into one statement, this may require further splitting into two or more premises 
For the actual annotations, we developed a custom-made web-based application that allowed users to switch between different granularity of argument components (tokens or sentences), to annotate the same document in different argument “dimensions” (logos and pathos), and to write summary for each annotated argument component 
As a measure of annotation reliability, we rely on Krippendorff's unitized alpha ( INLINEFORM0 ) BIBREF71   To the best of our knowledge, this is the only agreement measure that is applicable when both labels and boundaries of segments are to be annotated 
Although the measure has been used in related annotation works BIBREF61 , BIBREF7 , BIBREF72 , there is one important detail that has not been properly communicated  The INLINEFORM0 is computed over a continuum of the smallest units, such as tokens  This continuum corresponds to a single document in the original Krippendorff's work  However, there are two possible extensions to multiple documents (a corpus), namely (a) to compute INLINEFORM1 for each document first and then report an average value, or (b) to concatenate all documents into one large continuum and compute INLINEFORM2 over it  The first approach with averaging yielded extremely high the standard deviation of INLINEFORM3 (i e , avg  = 0 253; std  dev  = 0 886; median = 0 476 for the claim)  This says that some documents are easy to annotate while others are harder, but interpretation of such averaged value has no evidence either in BIBREF71 or other papers based upon it  Thus we use the other methodology and treat the whole corpus as a single long continuum (which yields in the example of claim 0 541 INLINEFORM4 ) 
Table TABREF77 shows the inter-annotator agreement as measured on documents from the last annotation phase (see section UID67 )  The overall INLINEFORM0 for all register types, topics, and argument components is 0 48 in the logos dimension (annotated with the modified Toulmin's model)  Such agreement can be considered as moderate by the measures proposed by Landis Koch 1977, however, direct interpretation of the agreement value lacks consensus BIBREF54   Similar inter-annotator agreement numbers were achieved in the relevant works in argumentation mining (refer to Table TABREF33 in section SECREF31 ; although most of the numbers are not directly comparable, as different inter-annotator metrics were used on different tasks) 
There is a huge difference in INLINEFORM0 regarding the registers between comments + forums posts ( INLINEFORM1 0 60, Table TABREF77 a) and articles + blog posts ( INLINEFORM2 0 09, Table TABREF77 b) in the logos dimension  If we break down the value with respect to the individual argument components, the agreement on claim and premise is substantial in the case of comments and forum posts (0 59 and 0 69, respectively)  By contrast, these argument components were annotated only with a fair agreement in articles and blog posts (0 22 and 0 24, respectively) 
As can be also observed from Table TABREF77 , the annotation agreement in the logos dimension varies regarding the document topic  While it is substantial/moderate for prayer in schools (0 68) or private vs  public schools (0 44), for some topics it remains rather slight, such as in the case of redshirting (0 14) or mainstreaming (0 08) 
First, we examine the disagreement in annotations by posing the following research question: are there any measurable properties of the annotated documents that might systematically cause low inter-annotator agreement? We use Pearson's correlation coefficient between INLINEFORM0 on each document and the particular property under investigation  We investigated the following set of measures 
Full sentence coverage ratio represents a ratio of argument component boundaries that are aligned to sentence boundaries  The value is 1 0 if all annotations in the particular document are aligned to sentences and 0 0 if no annotations match the sentence boundaries  Our hypothesis was that automatic segmentation to sentences was often incorrect, therefore annotators had to switch to the token level annotations and this might have increased disagreement on boundaries of the argument components 
Document length, paragraph length and average sentence length  Our hypotheses was that the length of documents, paragraphs, or sentences negatively affects the agreement 
Readability measures  We tested four standard readability measures, namely Ari BIBREF73 , Coleman-Liau BIBREF74 , Flesch BIBREF75 , and Lix BIBREF76 to find out whether readability of the documents plays any role in annotation agreement 
Correlation results are listed in Table TABREF82   We observed the following statistically significant ( INLINEFORM0 ) correlations  First, document length negatively correlates with agreement in comments  The longer the comment was the lower the agreement was  Second, average paragraph length negatively correlates with agreement in blog posts  The longer the paragraphs in blogs were, the lower agreement was reached  Third, all readability scores negatively correlate with agreement in the public vs  private school domain, meaning that the more complicated the text in terms of readability is, the lower agreement was reached  We observed no significant correlation in sentence coverage and average sentence length measures  We cannot draw any general conclusion from these results, but we can state that some registers and topics, given their properties, are more challenging to annotate than others 
Another qualitative analysis of disagreements between annotators was performed by constructing a probabilistic confusion matrix BIBREF77 on the token level  The biggest disagreements, as can be seen in Table TABREF85 , is caused by rebuttal and refutation confused with none (0 27 and 0 40, respectively)  This is another sign that these two argument components were very hard to annotate  As shown in Table TABREF77 , the INLINEFORM5 was also low – 0 08 for rebuttal and 0 17 for refutation 
We analyzed the annotations and found the following phenomena that usually caused disagreements between annotators 
Each argument component (e g , premise or backing) should express one consistent and coherent piece of information, for example a single reason in case of the premise (see Section UID73 )  However, the decision whether a longer text should be kept as a single argument component or segmented into multiple components is subjective and highly text-specific 
While rhetorical questions have been researched extensively in linguistics BIBREF78 , BIBREF79 , BIBREF80 , BIBREF81 , their role in argumentation represents a substantial research question BIBREF82 , BIBREF83 , BIBREF84 , BIBREF85 , BIBREF86   Teninbaum 2011 provides a brief history of rhetorical questions in persuasion  In short, rhetorical questions should provoke the reader  From the perspective of our argumentation model, rhetorical questions might fall both into the logos dimension (and thus be labeled as, e g , claim, premise, etc ) or into the pathos dimension (refer to Section SECREF20 )  Again, the decision is usually not clear-cut 
As introduced in section UID55 , rebuttal attacks the claim by presenting an opponent's view  In most cases, the rebuttal is again attacked by the author using refutation  From the pragmatic perspective, refutation thus supports the author's stance expressed by the claim  Therefore, it can be easily confused with premises, as the function of both is to provide support for the claim  Refutation thus only takes place if it is meant as a reaction to the rebuttal  It follows the discussed matter and contradicts it  Such a discourse is usually expressed as:
[claim: My claim ] [rebuttal: On the other hand, some people claim XXX which makes my claim wrong ] [refutation: But this is not true, because of YYY ]
However, the author might also take the following defensible approach to formulate the argument:
[rebuttal: Some people claim XXX-1 which makes my claim wrong ] [refutation: But this is not true, because of YYY-1 ] [rebuttal: Some people claim XXX-2 which makes my claim wrong ] [refutation: But this is not true, because of YYY-2 ] [claim: Therefore my claim ]
If this argument is formulated without stating the rebuttals, it would be equivalent to the following:
[premise: YYY-1 ] [premise: YYY-2 ] [claim: Therefore my claim ]
This example shows that rebuttal and refutation represent a rhetorical device to produce arguments, but the distinction between refutation and premise is context-dependent and on the functional level both premise and refutation have very similar role – to support the author's standpoint  Although introducing dialogical moves into monological model and its practical consequences, as described above, can be seen as a shortcoming of our model, this rhetoric figure has been identified by argumentation researchers as procatalepsis BIBREF43   A broader view on incorporating opposing views (or lack thereof) is discussed under the term confirmation bias by BIBREF21 who claim that “[   ] people are trying to convince others  They are typically looking for arguments and evidence to confirm their own claim, and ignoring negative arguments and evidence unless they anticipate having to rebut them ” The dialectical attack of possible counter-arguments may thus strengthen one's own argument 
One possible solution would be to refrain from capturing this phenomena completely and to simplify the model to claims and premises, for instance  However, the following example would then miss an important piece of information, as the last two clauses would be left un-annotated  At the same time, annotating the last clause as premise would be misleading, because it does not support the claim (in fact, it supports it only indirectly by attacking the rebuttal; this can be seen as a support is considered as an admissible extension of abstract argument graph by BIBREF87 ) 
Doc#422 (forumpost, homeschooling) [claim: I try not to be anti-homeschooling, but    it's just hard for me ] [premise: I really haven't met any homeschoolers who turned out quite right, including myself ] I apologize if what I'm saying offends any of you - that's not my intention, [rebuttal: I know that there are many homeschooled children who do just fine,] but [refutation: that hasn't been my experience ]
To the best of our knowledge, these context-dependent dialogical properties of argument components using Toulmin's model have not been solved in the literature on argumentation theory and we suggest that these observations should be taken into account in the future research in monological argumentation 
Appeal to emotion, sarcasm, irony, or jokes are common in argumentation in user-generated Web content  We also observed documents in our data that were purely sarcastic (the pathos dimension), therefore logical analysis of the argument (the logos dimension) would make no sense  However, given the structure of such documents, some claims or premises might be also identified  Such an argument is a typical example of fallacious argumentation, which intentionally pretends to present a valid argument, but its persuasion is conveyed purely for example by appealing to emotions of the reader BIBREF88  
We present some statistics of the annotated data that are important from the argumentation research perspective  Regardless of the register, 48% of claims are implicit  This means that the authors assume that their standpoint towards the discussed controversy can be inferred by the reader and give only reasons for that standpoint  Also, explicit claims are mainly written just once, only in 3% of the documents the claim was rephrased and occurred multiple times 
In 6% of the documents, the reasons for an implicit claim are given only in the pathos dimension, making the argument purely persuasive without logical argumentation 
The “myside bias”, defined as a bias against information supporting another side of an argument BIBREF89 , BIBREF90 , can be observed by the presence of rebuttals to the author's claim or by formulating arguments for both sides when the overall stance is neutral  While 85% of the documents do not consider any opposing side, only 8% documents present a rebuttal, which is then attacked by refutation in 4% of the documents  Multiple rebuttals and refutations were found in 3% of the documents  Only 4% of the documents were overall neutral and presented arguments for both sides, mainly in blog posts 
We were also interested whether mitigating linguistic devices are employed in the annotated arguments, namely in their main stance-taking components, the claims  Such devices typically include parenthetical verbs, syntactic constructions, token agreements, hedges, challenge questions, discourse markers, and tag questions, among others BIBREF91   In particular, [p  1]Kaltenbock et al 2010 define hedging as a discourse strategy that reduces the force or truth of an utterance and thus reduces the risk a speaker runs when uttering a strong or firm assertion or other speech act  We manually examined the use of hedging in the annotated claims 
Our main observation is that hedging is used differently across topics  For instance, about 30-35% of claims in homeschooling and mainstreaming signal the lack of a full commitment to the expressed stance, in contrast to prayer in schools (15%) or public vs  private schools (about 10%)  Typical hedging cues include speculations and modality (“If I have kids, I will probably homeschool them ”), statements as neutral observations (“It's not wrong to hold the opinion that in general it's better for kids to go to school than to be homeschooled ”), or weasel phrases BIBREF92 (“In some cases, inclusion can work fantastically well ”, “For the majority of the children in the school, mainstream would not have been a suitable placement ”) 
On the other hand, most claims that are used for instance in the prayer in schools arguments are very direct, without trying to diminish its commitment to the conveyed belief (for example, “NO PRAYER IN SCHOOLS!    period ”, “Get it out of public schools”, “Pray at home ”, or “No organized prayers or services anywhere on public school board property - FOR ANYONE ”)  Moreover, some claims are clearly offensive, persuading by direct imperative clauses towards the opponents/audience (“TAKE YOUR KIDS PRIVATE IF YOU CARE AS I DID”, “Run, don't walk, to the nearest private school ”) or even accuse the opponents for taking a certain stance (“You are a bad person if you send your children to private school ”) 
These observations are consistent with the findings from the first annotation study on persuasion (see section UID48 ), namely that some topics attract heated argumentation where participant take very clear and reserved standpoints (such as prayer in schools or private vs  public schools), while discussions about other topics are rather milder  It has been shown that the choices a speaker makes to express a position are informed by their social and cultural background, as well as their ability to speak the language BIBREF93 , BIBREF94 , BIBREF91   However, given the uncontrolled settings of the user-generated Web content, we cannot infer any similar conclusions in this respect 
We investigated premises across all topics in order to find the type of support used in the argument  We followed the approach of Park Cardie 2014, who distinguished three types of propositions in their study, namely unverifiable, verifiable non-experiential, and verifiable experiential 
Verifiable non-experiential and verifiable experiential propositions, unlike unverifiable propositions, contain an objective assertion, where objective means “expressing or dealing with facts or conditions as perceived without distortion by personal feelings, prejudices, or interpretations ” Such assertions have truth values that can be proved or disproved with objective evidence; the correctness of the assertion or the availability of the objective evidence does not matter BIBREF8   A verifiable proposition can further be distinguished as experiential or not, depending on whether the proposition is about the writer's personal state or experience or something non-experiential  Verifiable experiential propositions are sometimes referred to as anectotal evidence, provide the novel knowledge that readers are seeking BIBREF8  
Table TABREF97 shows the distribution of the premise types with examples for each topic from the annotated corpus  As can be seen in the first row, arguments in prayer in schools contain majority (73%) of unverifiable premises  Closer examination reveals that their content vary from general vague propositions to obvious fallacies, such as a hasty generalization, straw men, or slippery slope  As Nieminen Mustonen 2014 found out, fallacies are very common in argumentation about religion-related issues  On the other side of the spectrum, arguments about redshirting rely mostly on anecdotal evidence (61% of verifiable experiential propositions)  We will discuss the phenomena of narratives in argumentation in more detail later in section UID98   All the topics except private vs  public schools exhibit similar amount of verifiable non-experiential premises (9%–22%), usually referring to expert studies or facts  However, this type of premises has usually the lowest frequency 
Manually analyzing argumentative discourse and reconstructing (annotating) the underlying argument structure and its components is difficult  As [p  267]Reed2006 point out, “the analysis of arguments is often hard, not only for students, but for experts too ” According to [p  81]Harrell 2011b, argumentation is a skill and “even for simple arguments, untrained college students can identify the conclusion but without prompting are poor at both identifying the premises and how the premises support the conclusion ” [p  81]Harrell 2011 further claims that “a wide literature supports the contention that the particular skills of understanding, evaluating, and producing arguments are generally poor in the population of people who have not had specific training and that specific training is what improves these skills ” Some studies, for example, show that students perform significantly better on reasoning tasks when they have learned to identify premises and conclusions BIBREF95 or have learned some standard argumentation norms BIBREF96  
One particular extra challenge in analyzing argumentation in Web user-generated discourse is that the authors produce their texts probably without any existing argumentation theory or model in mind  We assume that argumentation or persuasion is inherent when users discuss controversial topics, but the true reasons why people participate in on-line communities and what drives their behavior is another research question BIBREF97 , BIBREF98 , BIBREF99 , BIBREF100   When the analyzed texts have a clear intention to produce argumentative discourse, such as in argumentative essays BIBREF7 , the argumentation is much more explicit and a substantially higher inter-annotator agreement can be achieved 
The model seems to be suitable for short persuasive documents, such as comments and forum posts  Its applicability to longer documents, such as articles or blog posts, is problematic for several reasons 
The argument components of the (modified) Toulmin's model and their roles are not expressive enough to capture argumentation that not only conveys the logical structure (in terms of reasons put forward to support the claim), but also relies heavily on the rhetorical power  This involves various stylistic devices, pervading narratives, direct and indirect speech, or interviews  While in some cases the argument components are easily recognizable, the vast majority of the discourse in articles and blog posts does not correspond to any distinguishable argumentative function in the logos dimension  As the purpose of such discourse relates more to rhetoric than to argumentation, unambiguous analysis of such phenomena goes beyond capabilities of the current argumentation model  For a discussion about metaphors in Toulmin's model of argumentation see, e g , BIBREF102 , BIBREF103  
Articles without a clear standpoint towards the discussed controversy cannot be easily annotated with the model either  Although the matter is viewed from both sides and there might be reasons presented for either of them, the overall persuasive intention is missing and fitting such data to the argumentation framework causes disagreements  One solution might be to break the document down to paragraphs and annotate each paragraph separately, examining argumentation on a different level of granularity 
As introduced in section SECREF20 , there are several dimensions of an argument  The Toulmin's model focuses solely on the logos dimension  We decided to ignore the ethos dimension, because dealing with the author's credibility remains unclear, given the variety of the source web data  However, exploiting the pathos dimension of an argument is prevalent in the web data, for example as an appeal to emotions  Therefore we experimented with annotating appeal to emotions as a separate category independent of components in the logos dimension  We defined some features for the annotators how to distinguish appeal to emotions  Figurative language such as hyperbole, sarcasm, or obvious exaggerating to “spice up” the argument are the typical signs of pathos  In an extreme case, the whole argument might be purely emotional, as in the following example 
Doc#1698 (comment, prayer in schools) [app-to-emot: Prayer being removed from school is just the leading indicator of a nation that is ‘Falling Away’ from Jehovah  [   ] And the disasters we see today are simply God’s finger writing on the wall: Mene, mene, Tekel, Upharsin; that is, God has weighed America in the balances, and we’ve been found wanting  No wonder 50 million babies have been aborted since 1973  [   ]]
We kept annotations on the pathos dimension as simple as possible (with only one appeal to emotions label), but the resulting agreement was unsatisfying ( INLINEFORM0 0 30) even after several annotation iterations  Appeal to emotions is considered as a type of fallacy BIBREF104 , BIBREF18   Given the results, we assume that more carefully designed approach to fallacy annotation should be applied  To the best of our knowledge, there have been very few research works on modeling fallacies similarly to arguments on the discourse level BIBREF105   Therefore the question, in which detail and structure fallacies should be annotated, remains open  For the rest of the paper, we thus focus on the logos dimension solely 
Some of the educational topics under examination relate to young children (e g , redshirting or mainstreaming); therefore we assume that the majority of participants in discussions are their parents  We observed that many documents related to these topics contain narratives  Sometimes the story telling is meant as a support for the argument, but there are documents where the narrative has no intention to persuade and is simply a story sharing 
There is no widely accepted theory of the role of narratives among argumentation scholars  According to Fisher 1987, humans are storytellers by nature, and the “reason” in argumentation is therefore better understood in and through the narratives  He found that good reasons often take the form of narratives  Hoeken Fikkers 2014 investigated how integration of explicit argumentative content into narratives influences issue-relevant thinking and concluded that identifying with the character being in favor of the issue yielded a more positive attitude toward the issue  In a recent research, Bex 2011 proposes an argumentative-narrative model of reasoning with evidence, further elaborated in BIBREF106 ; also Niehaus et al 2012 proposes a computational model of narrative persuasion 
Stemming from another research field, LeytonEscobar2014 found that online community members who use and share narratives have higher participation levels and that narratives are useful tools to build cohesive cultures and increase participation  Betsch et al 2010 examined influencing vaccine intentions among parents and found that narratives carry more weight than statistics 
Summary of annotation studies
This section described two annotation studies that deal with argumentation in user-generated Web content on different levels of detail  In section SECREF44 , we argued for a need of document-level distinction of persuasiveness  We annotated 990 comments and forum posts, reaching moderate inter-annotator agreement (Fleiss' INLINEFORM0 0 59)  Section SECREF51 motivated the selection of a model for micro-level argument annotation, proposed its extension based on pre-study observations, and outlined the annotation set-up  This annotation study resulted into 340 documents annotated with the modified Toulmin's model and reached moderate inter-annotator agreement in the logos dimension (Krippendorff's INLINEFORM1 0 48)  These results make the annotated corpora suitable for training and evaluation computational models and each of these two annotation studies will have their experimental counterparts in the following section 
Experiments
This section presents experiments conducted on the annotated corpora introduced in section SECREF4   We put the main focus on identifying argument components in the discourse  To comply with the machine learning terminology, in this section we will use the term domain as an equivalent to a topic (remember that our dataset includes six different topics; see section SECREF38 ) 
We evaluate three different scenarios  First, we report ten-fold cross validation over a random ordering of the entire data set  Second, we deal with in-domain ten-fold cross validation for each of the six domains  Third, in order to evaluate the domain portability of our approach, we train the system on five domains and test on the remaining one for all six domains (which we report as cross-domain validation) 
Identification of argument components
In the following experiment, we focus on automatic identification of arguments in the discourse  Our approach is based on supervised and semi-supervised machine learning methods on the gold data Toulmin dataset introduced in section SECREF51  
An argument consists of different components (such as premises, backing, etc ) which are implicitly linked to the claim  In principle one document can contain multiple independent arguments  However, only 4% of the documents in our dataset contain arguments for both sides of the issue  Thus we simplify the task and assume there is only one argument per document 
Given the low inter-annotator agreement on the pathos dimension (Table TABREF77 ), we focus solely on recognizing the logical dimension of argument  The pathos dimension of argument remains an open problem for a proper modeling as well as its later recognition 
Since the smallest annotation unit is a token and the argument components do not overlap, we approach identification of argument components as a sequence labeling problem  We use the BIO encoding, so each token belongs to one of the following 11 classes: O (not a part of any argument component), Backing-B, Backing-I, Claim-B, Claim-I, Premise-B, Premise-I, Rebuttal-B, Rebuttal-I, Refutation-B, Refutation-I  This is the minimal encoding that is able to distinguish two adjacent argument components of the same type  In our data, 48% of all adjacent argument components of the same type are direct neighbors (there are no "O" tokens in between) 
We report Macro- INLINEFORM0 score and INLINEFORM1 scores for each of the 11 classes as the main evaluation metric  This evaluation is performed on the token level, and for each token the predicted label must exactly match the gold data label (classification of tokens into 11 classes) 
As instances for the sequence labeling model, we chose sentences rather than tokens  During our initial experiments, we observed that building a sequence labeling model for recognizing argument components as sequences of tokens is too fine-grained, as a single token does not convey enough information that could be encoded as features for a machine learner  However, as discussed in section UID73 , the annotations were performed on data pre-segmented to sentences and annotating tokens was necessary only when the sentence segmentation was wrong or one sentence contained multiple argument components  Our corpus consists of 3899 sentences, from which 2214 sentences (57%) contain no argument component  From the remaining ones, only 50 sentences (1%) have more than one argument component  Although in 19 cases (0 5%) the sentence contains a Claim-Premise pair which is an important distinction from the argumentation perspective, given the overall small number of such occurrences, we simplify the task by treating each sentence as if it has either one argument component or none 
The approximation with sentence-level units is explained in the example in Figure FIGREF112   In order to evaluate the expected performance loss using this approximation, we used an oracle that always predicts the correct label for the unit (sentence) and evaluated it against the true labels (recall that the evaluation against the true gold labels is done always on token level)  We lose only about 10% of macro INLINEFORM0 score (0 906) and only about 2% of accuracy (0 984)  This performance is still acceptable, while allowing to model sequences where the minimal unit is a sentence 
Table TABREF114 shows the distribution of the classes in the gold data Toulmin, where the labeling was already mapped to the sentences  The little presence of rebuttal and refutation (4 classes account only for 3 4% of the data) makes this dataset very unbalanced 
We chose SVMhmm BIBREF111 implementation of Structural Support Vector Machines for sequence labeling  Each sentence ( INLINEFORM0 ) is represented as a vector of real-valued features 
We defined the following feature sets:
FS0: Baseline lexical features
word uni-, bi-, and tri-grams (binary)
FS1: Structural, morphological, and syntactic features
First and last 3 tokens  Motivation: these tokens may contain discourse markers or other indicators for argument components, such as “therefore” and “since” for premises or “think” and “believe” for claims 
Relative position in paragraph and relative position in document  Motivation: We expect that claims are more likely to appear at the beginning or at the end of the document 
Number of POS 1-3 grams, dependency tree depth, constituency tree production rules, and number of sub-clauses  Based on BIBREF113  
FS2: Topic and sentiment features
30 features taken from a vector representation of the sentence obtained by using Gibbs sampling on LDA model BIBREF114 , BIBREF115 with 30 topics trained on unlabeled data from the raw corpus  Motivation: Topic representation of a sentence might be valuable for detecting off-topic sentences, namely non-argument components 
Scores for five sentiment categories (from very negative to very positive) obtained from Stanford sentiment analyzer BIBREF116   Motivation: Claims usually express opinions and carry sentiment 
FS3: Semantic, coreference, and discourse features
Binary features from Clear NLP Semantic Role Labeler BIBREF117   Namely, we extract agent, predicate + agent, predicate + agent + patient + (optional) negation, argument type + argument value, and discourse marker which are based on PropBank semantic role labels  Motivation: Exploit the semantics of Capturing the semantics of the sentences 
Binary features from Stanford Coreference Chain Resolver BIBREF118 , e g , presence of the sentence in a chain, transition type (i e , nominal–pronominal), distance to previous/next sentences in the chain, or number of inter-sentence coreference links  Motivation: Presence of coreference chains indicates links outside the sentence and thus may be informative, for example, for classifying whether the sentence is a part of a larger argument component 
Results of a PTDB-style discourse parser BIBREF119 , namely the type of discourse relation (explicit, implicit), presence of discourse connectives, and attributions  Motivation: It has been claimed that discourse relations play a role in argumentation mining BIBREF120  
FS4: Embedding features
300 features from word embedding vectors using word embeddings trained on part of the Google News dataset BIBREF121   In particular, we sum up embedding vectors (dimensionality 300) of each word, resulting into a single vector for the entire sentence  This vector is then directly used as a feature vector  Motivation: Embeddings helped to achieve state-of-the-art results in various NLP tasks BIBREF116 , BIBREF122  
Except the baseline lexical features, all feature types are extracted not only for the current sentence INLINEFORM0 , but also for INLINEFORM1 preceding and subsequent sentences, namely INLINEFORM2 , INLINEFORM3 , INLINEFORM4 INLINEFORM5 , INLINEFORM6 , where INLINEFORM7 was empirically set to 4  Each feature is then represented with a prefix to determine its relative position to the current sequence unit 
Let us first discuss the upper bounds of the system  Performance of the three human annotators is shown in the first column of Table TABREF139 (results are obtained from a cumulative confusion matrix)  The overall Macro- INLINEFORM0 score is 0 602 (accuracy 0 754)  If we look closer at the different argument components, we observe that humans are good at predicting claims, premises, backing and non-argumentative text (about 0 60-0 80 INLINEFORM1 ), but on rebuttal and refutation they achieve rather low scores  Without these two components, the overall human Macro- INLINEFORM2 would be 0 707  This trend follows the inter-annotator agreement scores, as discussed in section UID75  
In our experiments, the feature sets were combined in the bottom-up manner, starting with the simple lexical features (FS0), adding structural and syntactic features (FS1), then adding topic and sentiment features (FS2), then features reflecting the discourse structure (FS3), and finally enriched with completely unsupervised latent vector space representation (FS4)  In addition, we were gradually removing the simple features (e g , without lexical features, without syntactic features, etc ) to test the system with more “abstract” feature sets (feature ablation)  The results are shown in Table TABREF139  
The overall best performance (Macro- INLINEFORM0 0 251) was achieved using the rich feature sets (01234 and 234) and significantly outperformed the baseline as well as other feature sets  Classification of non-argumentative text (the "O" class) yields about 0 7 INLINEFORM1 score even in the baseline setting  The boundaries of claims (Cla-B), premises (Pre-B), and backing (Bac-B) reach in average lower scores then their respective inside tags (Cla-I, Pre-I, Bac-I)  It can be interpreted such that the system is able to classify that a certain sentence belongs to a certain argument component, but the distinction whether it is a beginning of the argument component is harder  The very low numbers for rebuttal and refutation have two reasons  First, these two argument components caused many disagreements in the annotations, as discussed in section UID86 , and were hard to recognize for the humans too  Second, these four classes have very few instances in the corpus (about 3 4%, see Table TABREF114 ), so the classifier suffers from the lack of training data 
The results for the in-domain cross validation scenario are shown in Table TABREF140   Similarly to the cross-validation scenario, the overall best results were achieved using the largest feature set (01234)  For mainstreaming and red-shirting, the best results were achieved using only the feature set 4 (embeddings)  These two domains contain also fewer documents, compared to other domains (refer to Table TABREF71 )  We suspect that embeddings-based features convey important information when not enough in-domain data are available  This observation will become apparent in the next experiment 
The cross-domain experiments yield rather poor results for most of the feature combinations (Table TABREF141 )  However, using only feature set 4 (embeddings), the system performance increases rapidly, so it is even comparable to numbers achieved in the in-domain scenario  These results indicate that embedding features generalize well across domains in our task of argument component identification  We leave investigating better performing vector representations, such as paragraph vectors BIBREF123 , for future work 
Error analysis based on the probabilistic confusion matrix BIBREF124 shown in Table TABREF142 reveals further details  About a half of the instances for each class are misclassified as non-argumentative (the "O" prediction) 
Backing-B is often confused with Premise-B (12%) and Backing-I with Premise-I (23%)  Similarly, Premise-I is misclassified as Backing-I in 9%  This shows that distinguishing between backing and premises is not easy because these two components are similar such that they support the claim, as discussed in section UID86   We can also see that the misclassification is consistent among *-B and *-I tags 
Rebuttal is often misclassified as Premise (28% for Rebuttal-I and 18% for Rebuttal-B; notice again the consistency in *-B and *-I tags)  This is rather surprising, as one would expect that rebuttal would be confused with a claim, because its role is to provide an opposing view 
Refutation-B and Refutation-I is misclassified as Premise-I in 19% and 27%, respectively  This finding confirms the discussion in section UID86 , because the role of refutation is highly context-dependent  In a pragmatic perspective, it is put forward to indirectly support the claim by attacking the rebuttal, thus having a similar function to the premise 
We manually examined miss-classified examples produced the best-performing system to find out which phenomena pose biggest challenges  Properly detecting boundaries of argument components caused problems, as shown in Figure FIGREF146 (a)  This goes in line with the granularity annotation difficulties discussed in section UID86   The next example in Figure FIGREF146 (b) shows that even if boundaries of components were detected precisely, the distinction between premise and backing fails  The example also shows that in some cases, labeling on clause level is required (left-hand side claim and premise) but the approximation in the system cannot cope with this level of detail (as explained in section UID111 )  Confusing non-argumentative text and argument components by the system is sometimes plausible, as is the case of the last rhetorical question in Figure FIGREF146 (c)  On the other hand, the last example in Figure FIGREF146 (d) shows that some claims using figurative language were hard to be identified  The complete predictions along with the gold data are publicly available 
SVMhmm offers many hyper-parameters with suggested default values, from which three are of importance  Parameter INLINEFORM0 sets the order of dependencies of transitions in HMM, parameter INLINEFORM1 sets the order of dependencies of emissions in HMM, and parameter INLINEFORM2 represents a trading-off slack versus magnitude of the weight-vector  For all experiments, we set all the hyper-parameters to their default values ( INLINEFORM3 , INLINEFORM4 , INLINEFORM5 )  Using the best performing feature set from Table TABREF139 , we experimented with a grid search over different values ( INLINEFORM6 , INLINEFORM7 , INLINEFORM8 ) but the results did not outperform the system trained with default parameter values 
The INLINEFORM0 scores might seem very low at the first glance  One obvious reason is the actual performance of the system, which gives a plenty of room for improvement in the future  But the main cause of low INLINEFORM2 numbers is the evaluation measure — using 11 classes on the token level is very strict, as it penalizes a mismatch in argument component boundaries the same way as a wrongly predicted argument component type  Therefore we also report two another evaluation metrics that help to put our results into a context 
Krippendorff's INLINEFORM0 — It was also used for evaluating inter-annotator agreement (see section UID75 ) 
Boundary similarity BIBREF125 — Using this metric, the problem is treated solely as a segmentation task without recognizing the argument component types 
As shown in Table TABREF157 (the Macro- INLINEFORM0 scores are repeated from Table TABREF139 ), the best-performing system achieves 0 30 score using Krippendorf's INLINEFORM1 , which is in the middle between the baseline and the human performance (0 48) but is considered as poor from the inter-annotator agreement point of view BIBREF54   The boundary similarity metrics is not directly suitable for evaluating argument component classification, but reveals a sub-task of finding the component boundaries  The best system achieved 0 32 on this measure  Vovk2013MT used this measure to annotate argument spans and his annotators achieved 0 36 boundary similarity score  Human annotators in BIBREF125 reached 0 53 boundary similarity score 
The overall performance of the system is also affected by the accuracy of individual NPL tools used for extracting features  One particular problem is that the preprocessing models we rely on (POS, syntax, semantic roles, coreference, discourse; see section UID115 ) were trained on newswire corpora, so one has to expect performance drop when applied on user-generated content  This is however a well-known issue in NLP BIBREF126 , BIBREF127 , BIBREF128  
To get an impression of the actual performance of the system on the data, we also provide the complete output of our best performing system in one PDF document together with the gold annotations in the logos dimension side by side in the accompanying software package  We believe this will help the community to see the strengths of our model as well as possible limitations of our current approaches 
Conclusions
Let us begin with summarizing answers to the research questions stated in the introduction  First, as we showed in section UID55 , existing argumentation theories do offer models for capturing argumentation in user-generated content on the Web  We built upon the Toulmin's model and proposed some extensions 
Second, as compared to the negative experiences with annotating using Walton's schemes (see sections UID52 and SECREF31 ), our modified Toulmin's model offers a trade-off between its expressiveness and annotation reliability  However, we found that the capabilities of the model to capture argumentation depend on the register and topic, the length of the document, and inherently on the literary devices and structures used for expressing argumentation as these properties influenced the agreement among annotators 
Third, there are aspects of online argumentation that lack their established theoretical counterparts, such as rhetorical questions, figurative language, narratives, and fallacies in general  We tried to model some of them in the pathos dimension of argument (section UID103 ), but no satisfying agreement was reached  Furthermore, we dealt with a step that precedes argument analysis by filtering documents given their persuasiveness with respect to the controversy  Finally, we proposed a computational model based on machine learning for identifying argument components (section SECREF108 )  In this identification task, we experimented with a wide range of linguistically motivated features and found that (1) the largest feature set (including n-grams, structural features, syntactic features, topic distribution, sentiment distribution, semantic features, coreference feaures, discourse features, and features based on word embeddings) performs best in both in-domain and all-data cross validation, while (2) features based only on word embeddings yield best results in cross-domain evaluation 
Since there is no one-size-fits-all argumentation theory to be applied to actual data on the Web, the argumentation model and an annotation scheme for argumentation mining is a function of the task requirements and the corpus properties  Its selection should be based on the data at hand and the desired application  Given the proposed use-case scenarios (section SECREF1 ) and the results of our annotation study (section SECREF51 ), we recommend a scheme based on Toulmin's model for short documents, such as comments or forum posts 

ground_truth: The baseline system for the SLC task is a very simple logistic regression classifier with default parameters. The baseline for the FLC task generates spans and selects one of the 18 techniques randomly.

SLC task is a very simple logistic regression classifier, FLC task generates spans and selects one of the 18 techniques randomly