prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. On which tasks do they test their conflict method?
Context 1: Introduction
Modelling the relationship between sequences is extremely significant in most retrieval or classification problems involving two sequences  Traditionally, in Siamese networks, Hadamard product or concatenation have been used to fuse two vector representations of two input sequences to form a final representation for tasks like semantic similarity, passage retrieval  This representation, subsequently, has been used to compute similarity scores which has been used in a variety of training objectives like margin loss for ranking or cross-entropy error in classification 
We have also witnessed word or phrase level similarity to create alignment matrices between two sequences BIBREF0 , BIBREF1   These alignment matrices has proved to be very useful to model the relationship between two word representations as well fuse the relevant information of one sequence into another  Empirical evidences have shown this alignment procedures have significantly performed better then simple concatenation or element-wise multiplication, especially for long sentences or paragraphs 
Attention works on creating neural alignment matrix using learnt weights without pre-computing alignment matrix and using them as features  The main objective of any attentive or alignment process is to look for matching words or phrases between two sequences and assign a high weight to the most similar pairs and vice-versa  The notion of matching or similarity maybe not semantic similarity but based on whatever task we have at hand  For example, for a task that requires capturing semantic similarity between two sequences like "how rich is tom cruise" and "how much wealth does tom cruise have", an attentive model shall discover the high similarity between "rich" and "wealthy" and assign a high weight value to the pair  Likewise, for a different task like question answering, a word "long" in a question like "how long does it take to recover from a mild fever" might be aligned with the phrase "a week" from the candidate answer "it takes almost a week to recover fully from a fever"  Thus, attention significantly aids in better understanding the relevance of a similar user query in a similar measurement task or a candidate answer in a question answering task  The final prediction score is dependent on how well the relationship between two sequences are modeled and established 
The general process of matching one sequence with another through attention includes computing the alignment matrix containing weight value between every pair of word representations belonging to both of the sequences  Subsequently, softmax function is applied on all the elements of one of the two dimensions of the matrix to represent the matching probabilities of all the word of a sequence with respect to one particular word in the other sequence 
Since attention always looks for matching word representations, it operates under the assumption that there is always a match to be found inside the sequences  We provide a theoretical limitation to it and propose another technique called conflict that looks for contrasting relationship between words in two sequences  We empirically verify that our proposed conflict mechanism combined with attention can outperform the performance of attention working solely 
Related Work
Bahdanau et al  BIBREF2 introduced attention first in neural machine translation  It used a feed-forward network over addition of encoder and decoder states to compute alignment score  Our work is very similar to this except we use element wise difference instead of addition to build our conflict function  BIBREF3 came up with a scaled dot-product attention in their Transformer model which is fast and memory-efficient  Due to the scaling factor, it didn't have the issue of gradients zeroing out  On the other hand, BIBREF4 has experimented with global and local attention based on the how many hidden states the attention function takes into account  Their experiments have revolved around three attention functions - dot, concat and general  Their findings include that dot product works best for global attention  Our work also belongs to the global attention family as we consider all the hidden states of the sequence 
Attention has been widely used in pair-classification problems like natural language inference  Wang et al  BIBREF5 introduced BIMPM which matched one sequence with another in four different fashion but one single matching function which they used as cosine  Liu et al  BIBREF6 proposed SAN for language inference which also used dot-product attention between the sequences 
Summarizing, attention has helped in achieving state-of-the-art results in NLI and QA  Prior work in attention has been mostly in similarity based approaches while our work focuses on non-matching sequences 
How attention works
Let us consider that we have two sequences INLINEFORM0 and INLINEFORM1 each with M and N words respectively  The objective of attention is two-fold: compute alignment scores (or weight) between every word representation pairs from INLINEFORM2 and INLINEFORM3 and fuse the matching information of INLINEFORM4 with INLINEFORM5 thus computing a new representation of INLINEFORM6 conditioned on INLINEFORM7  
The word representations that attention operates on can be either embeddings like GloVe or hidden states from any recurrent neural network  We denote these representations as u = INLINEFORM0 and v = INLINEFORM1   We provide a mathematical working of how a general attention mechanism works between two sequences, followed by a explanation in words: DISPLAYFORM0
Explanation: Both are sequences are non-linearly projected into two different spaces (eqn 1) and each word representation in INLINEFORM0 is matched with that in INLINEFORM1 by computing a dot-product (eqn 2)  INLINEFORM2 is a M X N matrix that stores the alignment scores between word INLINEFORM3 and INLINEFORM4 (eqn 2)  Since, the scores are not normalized, a softmax function is applied on each row to convert them to probabilities (eqn  3)  Thus, each row contains relative importance of words in INLINEFORM5 to a particular word INLINEFORM6   Weighted sum of INLINEFORM7 is taken (eqn  4) and fused with the word representation INLINEFORM8 using concatenation (eqn 5) 
Limits of using only Attention
Attention operates by using dot product or sometimes addition followed by linear projection to a scalar which models the similarity between two vectors  Subsequently, softmax is applied which gives high probabilities to most matching word representations  This assumes that there is some highly matched word pairs already existing and high scores will be assigned to them  Given a vector INLINEFORM0 =( INLINEFORM1 ,   , INLINEFORM2 ) on which softmax function is applied, each INLINEFORM3 INLINEFORM4 (0, 1)  It is observable that the average value of INLINEFORM5 is always INLINEFORM6   In other words, it is impossible to produce a vector having all INLINEFORM7 < INLINEFORM8 when two sequences have no matching at all 
In cases, where one or more word pairs from two different sequences are highly dissimilar, it is impossible to assign a very low probability to it without increasing the probability of some other pair somewhere else since INLINEFORM0 = 1 
For example, when we consider two sequences "height of tom cruise" and "age of sun", while computing the attention weights between the word "height" and all the words in the second sequence it can be observed that their no matching word in the latter  In this case, a standard dot-product based attention with softmax won't be able to produce weights which is below 0 33 (=1/3) for all the words in the second sequence with respect to the word "height" in the first sequence 
Conflict model
We propose a different mechanism that does the opposite of what attention does that is computing how much two sequences repel each other  This works very similar to how attention works but inversely 
We demonstrate a general model but we also realize that there can be other variants of it which may be worked out to perform better  Our approach consists of using element wise difference between two vectors followed by a linear transformation to produce a scalar weight  The remaining of the process acts similar to how attention works  Mathematically, we can express it as: DISPLAYFORM0
where INLINEFORM0 INLINEFORM1 INLINEFORM2 is a parameter that we introduce to provide a weight for the pair  The two word representations INLINEFORM3 and INLINEFORM4 are projected to a space where their element wise difference can be used to model their dissimilarity and softmax applied on them can produce high probability to more dissimilar word pairs 
It is good to note that conflict suffers from the same limitation that attention suffers from  This is when a pair of sentences are highly matching especially with multiple associations  But when the two methods work together, each compensates for the other's shortcomings 
Combination of attention and conflict
We used two weighted representations of INLINEFORM0 using weights of attention and conflict as computed in Eqn  (4) and (8) respectively  Our final representation of a word representation INLINEFORM1 conditioned on INLINEFORM2 can be expressed as: DISPLAYFORM0
where A and C denote that they are from attention and conflict models respectively 
Relation to Multi-Head attention
Multi-head attention, as introduced in BIBREF3 , computes multiple identical attention mechanism parallelly on multiple linear projections of same inputs  The parameters of each attention and projections are different in each head  Finally, they concatenate all the attentions which is similar to how we concatenate conflict and attention  However, they use dot-product to compute each of the attention 
Our combined model that contains both attention and conflict can be thought of as a 2-head attention model but both heads are different  Our conflict head explicitly captures difference between the inputs 
Visualizing attention and conflict
We observe how our conflict model learns the dissimilarities between word representations  We achieve that by visualizing the heatmap of the weight matrix INLINEFORM0 for both attention and conflict from eqns  (3) and (8)  While attention successfully learns the alignments, conflict matrix also shows that our approach models the contradicting associations like "animal" and "lake" or "australia" and "world"  These two associations are the unique pairs which are instrumental in determining that the two queries are not similar 
The model
We create two models both of which constitutes of three main parts: encoder, interaction and classifier and take two sequences as input  Except interaction, all the other parts are exactly identical between the two models  The encoder is shared among the sequences simply uses two stacked GRU layers  The interaction part consists of only attention for one model while for the another one it consists of attention and conflict combined as shown in (eqn 11)   The classifier part is simply stacked fully-connected layers  Figure 3 shows a block diagram of how our model looks like 
Task 1: Quora Duplicate Question Pair Detection
The dataset includes pairs of questions labelled as 1 or 0 depending on whether a pair is duplicate or not respectively  This is a popular pair-level classification task on which extensive work has already been done before like BIBREF7 , BIBREF8   For this task, we make the output layer of our model to predict two probabilities for non-duplicate and duplicate  We sample the data from the original dataset so that it contains equal positive and negative classes  Original dataset has some class imbalance but for sake simplicity we don't consider it  The final data that we use has roughly 400,000 question pairs and we split this data into train and test using 8:2 ratio 
We train all our models for roughly 2 epochs with a batch size of 64  We use a hidden dimension of 150 throughout the model  The embedding layer uses ELMO BIBREF9 which has proven to be very useful in various downstream language understanding tasks  Our FC layers consists of four dense layers with INLINEFORM0 activation after each layer  The dropout rate is kept as 0 2 for every recurrent and FC linear layers  We use Adam optimizer in our experiment with epsilon=1e-8, beta=0 9 and learning rate=1e-3 
Task 2: Ranking questions in Bing's People Also Ask
People Also Ask is a feature in Bing search result page where related questions are recommended to the user  User may click on a question to view the answer  Clicking is a positive feedback that shows user's interest in the question  We use this click logs to build a question classifier using the same model in Figure 3  The problem statement is very similar to BIBREF10 where they use logistic regression to predict whether an user would click on ad  Our goal is to classify if a question is potential high-click question or not for a given query  For this, we first create a labelled data set using the click logs where any question having CTR lower than 0 3 is labelled as 0 and a question having CTR more than 0 7 as 1 
Our final data resembles that of a pair-level classifier, as in Task 1, where user query and candidate questions are input  With these data set, we train a binary classifier to detect high-click and low-click questions 
Quantitative Analysis
For both tasks, we compute classification accuracy using three model variants and report the results in Table 1 and Table 2  We observe that model with both attention and conflict combined gives the best results 
We also show the training loss curve for both the models having attention and attention combined with conflict respectively  Figure 4 and 5 shows these curves for Task 1 and Task 2 respectively  The curves are smoothed using moving average having an window size of 8  We notice that the conflict model has much steeper slope and converges to a much better minima in both the tasks  It can also be noticed that in the training procedure for the model which has both attention and conflict, the updates are much smoother 
Qualitative Comparison
We also show qualitative results where we can observe that our model with attention and conflict combined does better on cases where pairs are non-duplicate and has very small difference  We have observed that the conflict model is very sensitive to even minor differences and compensates in such cases where attention poses high bias towards similarities already there in the sequences 
Sequence 1: What are the best ways to learn French ?
Sequence 2: How do I learn french genders ?
Attention only: 1
Attention+Conflict: 0
Ground Truth: 0
Sequence 1: How do I prevent breast cancer ?
Sequence 2: Is breast cancer preventable ?
Attention only: 1
Attention+Conflict: 0
Ground Truth: 0
We provide two examples with predictions from the models with only attention and combination of attention and conflict  Each example is accompanied by the ground truth in our data 
Analyzing the gains
We analyzed the gains in Task 1 which we get from the attention-conflict model in order to ensure that they are not due to randomness in weight initialization or simply additional parameters  We particularly focused on the examples which were incorrectly marked in attention model but correctly in attention-conflict model  We saw that 70% of those cases are the ones where the pair was incorrectly marked as duplicate in the previous model but our combined model correctly marked them as non-duplicate 
Conclusion
In this work, we highlighted the limits of attention especially in cases where two sequences have a contradicting relationship based on the task it performs  To alleviate this problem and further improve the performance, we propose a conflict mechanism that tries to capture how two sequences repel each other  This acts like the inverse of attention and, empirically, we show that how conflict and attention together can improve the performance 
Future research work should be based on alternative design of conflict mechanism using other difference operators other than element wise difference which we use 
Context 2: Introduction
Recently, deep learning algorithms have successfully addressed problems in various fields, such as image classification, machine translation, speech recognition, text-to-speech generation and other machine learning related areas BIBREF0 , BIBREF1 , BIBREF2   Similarly, substantial improvements in performance have been obtained when deep learning algorithms have been applied to statistical speech processing BIBREF3   These fundamental improvements have led researchers to investigate additional topics related to human nature, which have long been objects of study  One such topic involves understanding human emotions and reflecting it through machine intelligence, such as emotional dialogue models BIBREF4 , BIBREF5  
In developing emotionally aware intelligence, the very first step is building robust emotion classifiers that display good performance regardless of the application; this outcome is considered to be one of the fundamental research goals in affective computing BIBREF6   In particular, the speech emotion recognition task is one of the most important problems in the field of paralinguistics  This field has recently broadened its applications, as it is a crucial factor in optimal human-computer interactions, including dialog systems  The goal of speech emotion recognition is to predict the emotional content of speech and to classify speech according to one of several labels (i e , happy, sad, neutral, and angry)  Various types of deep learning methods have been applied to increase the performance of emotion classifiers; however, this task is still considered to be challenging for several reasons  First, insufficient data for training complex neural network-based models are available, due to the costs associated with human involvement  Second, the characteristics of emotions must be learned from low-level speech signals  Feature-based models display limited skills when applied to this problem 
To overcome these limitations, we propose a model that uses high-level text transcription, as well as low-level audio signals, to utilize the information contained within low-resource datasets to a greater degree  Given recent improvements in automatic speech recognition (ASR) technology BIBREF7 , BIBREF2 , BIBREF8 , BIBREF9 , speech transcription can be carried out using audio signals with considerable skill  The emotional content of speech is clearly indicated by the emotion words contained in a sentence BIBREF10 , such as “lovely” and “awesome,” which carry strong emotions compared to generic (non-emotion) words, such as “person” and “day ” Thus, we hypothesize that the speech emotion recognition model will be benefit from the incorporation of high-level textual input 
In this paper, we propose a novel deep dual recurrent encoder model that simultaneously utilizes audio and text data in recognizing emotions from speech  Extensive experiments are conducted to investigate the efficacy and properties of the proposed model  Our proposed model outperforms previous state-of-the-art methods by 68 8% to 71 8% when applied to the IEMOCAP dataset, which is one of the most well-studied datasets  Based on an error analysis of the models, we show that our proposed model accurately identifies emotion classes  Moreover, the neutral class misclassification bias frequently exhibited by previous models, which focus on audio features, is less pronounced in our model 
Related work
Classical machine learning algorithms, such as hidden Markov models (HMMs), support vector machines (SVMs), and decision tree-based methods, have been employed in speech emotion recognition problems BIBREF11 , BIBREF12 , BIBREF13   Recently, researchers have proposed various neural network-based architectures to improve the performance of speech emotion recognition  An initial study utilized deep neural networks (DNNs) to extract high-level features from raw audio data and demonstrated its effectiveness in speech emotion recognition BIBREF14   With the advancement of deep learning methods, more complex neural-based architectures have been proposed  Convolutional neural network (CNN)-based models have been trained on information derived from raw audio signals using spectrograms or audio features such as Mel-frequency cepstral coefficients (MFCCs) and low-level descriptors (LLDs) BIBREF15 , BIBREF16 , BIBREF17   These neural network-based models are combined to produce higher-complexity models BIBREF18 , BIBREF19 , and these models achieved the best-recorded performance when applied to the IEMOCAP dataset 
Another line of research has focused on adopting variant machine learning techniques combined with neural network-based models  One researcher utilized the multiobject learning approach and used gender and naturalness as auxiliary tasks so that the neural network-based model learned more features from a given dataset BIBREF20   Another researcher investigated transfer learning methods, leveraging external data from related domains BIBREF21  
As emotional dialogue is composed of sound and spoken content, researchers have also investigated the combination of acoustic features and language information, built belief network-based methods of identifying emotional key phrases, and assessed the emotional salience of verbal cues from both phoneme sequences and words BIBREF22 , BIBREF23   However, none of these studies have utilized information from speech signals and text sequences simultaneously in an end-to-end learning neural network-based model to classify emotions 
Model
This section describes the methodologies that are applied to the speech emotion recognition task  We start by introducing the recurrent encoder model for the audio and text modalities individually  We then propose a multimodal approach that encodes both audio and textual information simultaneously via a dual recurrent encoder 
Audio Recurrent Encoder (ARE)
Motivated by the architecture used in BIBREF24 , BIBREF25 , we build an audio recurrent encoder (ARE) to predict the class of a given audio signal  Once MFCC features have been extracted from an audio signal, a subset of the sequential features is fed into the RNN (i e , gated recurrent units (GRUs)), which leads to the formation of the network's internal hidden state INLINEFORM0 to model the time series patterns  This internal hidden state is updated at each time step with the input data INLINEFORM1 and the hidden state of the previous time step INLINEFORM2 as follows: DISPLAYFORM0
where INLINEFORM0 is the RNN function with weight parameter INLINEFORM1 , INLINEFORM2 represents the hidden state at t- INLINEFORM3 time step, and INLINEFORM4 represents the t- INLINEFORM5 MFCC features in INLINEFORM6   After encoding the audio signal INLINEFORM7 with the RNN, the last hidden state of the RNN, INLINEFORM8 , is considered to be the representative vector that contains all of the sequential audio data  This vector is then concatenated with another prosodic feature vector, INLINEFORM9 , to generate a more informative vector representation of the signal, INLINEFORM10   The MFCC and the prosodic features are extracted from the audio signal using the openSMILE toolkit BIBREF26 , INLINEFORM11 , respectively  Finally, the emotion class is predicted by applying the softmax function to the vector INLINEFORM12   For a given audio sample INLINEFORM13 , we assume that INLINEFORM14 is the true label vector, which contains all zeros but contains a one at the correct class, and INLINEFORM15 is the predicted probability distribution from the softmax layer  The training objective then takes the following form: DISPLAYFORM0
where INLINEFORM0 is the calculated representative vector of the audio signal with dimensionality INLINEFORM1   The INLINEFORM2 and the bias INLINEFORM3 are learned model parameters  C is the total number of classes, and N is the total number of samples used in training  The upper part of Figure shows the architecture of the ARE model 
Text Recurrent Encoder (TRE)
We assume that speech transcripts can be extracted from audio signals with high accuracy, given the advancement of ASR technologies BIBREF7   We attempt to use the processed textual information as another modality in predicting the emotion class of a given signal  To use textual information, a speech transcript is tokenized and indexed into a sequence of tokens using the Natural Language Toolkit (NLTK) BIBREF27   Each token is then passed through a word-embedding layer that converts a word index to a corresponding 300-dimensional vector that contains additional contextual meaning between words  The sequence of embedded tokens is fed into a text recurrent encoder (TRE) in such a way that the audio MFCC features are encoded using the ARE represented by equation EQREF2   In this case, INLINEFORM0 is the t- INLINEFORM1 embedded token from the text input  Finally, the emotion class is predicted from the last hidden state of the text-RNN using the softmax function 
We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is last hidden state of the text-RNN, INLINEFORM1 , and the INLINEFORM2 and bias INLINEFORM3 are learned model parameters  The lower part of Figure indicates the architecture of the TRE model 
Multimodal Dual Recurrent Encoder (MDRE)
We present a novel architecture called the multimodal dual recurrent encoder (MDRE) to overcome the limitations of existing approaches  In this study, we consider multiple modalities, such as MFCC features, prosodic features and transcripts, which contain sequential audio information, statistical audio information and textual information, respectively  These types of data are the same as those used in the ARE and TRE cases  The MDRE model employs two RNNs to encode data from the audio signal and textual inputs independently  The audio-RNN encodes MFCC features from the audio signal using equation EQREF2   The last hidden state of the audio-RNN is concatenated with the prosodic features to form the final vector representation INLINEFORM0 , and this vector is then passed through a fully connected neural network layer to form the audio encoding vector A  On the other hand, the text-RNN encodes the word sequence of the transcript using equation EQREF2   The final hidden states of the text-RNN are also passed through another fully connected neural network layer to form a textual encoding vector T  Finally, the emotion class is predicted by applying the softmax function to the concatenation of the vectors A and T  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 is the feed-forward neural network with weight parameter INLINEFORM1 , and INLINEFORM2 , INLINEFORM3 are final encoding vectors from the audio-RNN and text-RNN, respectively  INLINEFORM4 and the bias INLINEFORM5 are learned model parameters 
Multimodal Dual Recurrent Encoder with Attention (MDREA)
Inspired by the concept of the attention mechanism used in neural machine translation BIBREF28 , we propose a novel multimodal attention method to focus on the specific parts of a transcript that contain strong emotional information, conditioning on the audio information  Figure shows the architecture of the MDREA model  First, the audio data and text data are encoded with the audio-RNN and text-RNN using equation EQREF2   We then consider the final audio encoding vector INLINEFORM0 as a context vector  As seen in equation EQREF9 , during each time step t, the dot product between the context vector e and the hidden state of the text-RNN at each t-th sequence INLINEFORM1 is evaluated to calculate a similarity score INLINEFORM2   Using this score INLINEFORM3 as a weight parameter, the weighted sum of the sequences of the hidden state of the text-RNN, INLINEFORM4 , is calculated to generate an attention-application vector Z  This attention-application vector is concatenated with the final encoding vector of the audio-RNN INLINEFORM5 (equation EQREF7 ), which will be passed through the softmax function to predict the emotion class  We use the same training objective as the ARE model, and the predicted probability distribution for the target class is as follows: DISPLAYFORM0
where INLINEFORM0 and the bias INLINEFORM1 are learned model parameters 
Dataset
We evaluate our model using the Interactive Emotional Dyadic Motion Capture (IEMOCAP) BIBREF18 dataset  This dataset was collected following theatrical theory in order to simulate natural dyadic interactions between actors

ground_truth: Task 1: Quora Duplicate Question Pair Detection, Task 2: Ranking questions

Quora Duplicate Question Pair Detection, Ranking questions in Bing's People Also Ask

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they combine audio and text sequences in their RNN?
Context 2:  We use categorical evaluations with majority agreement  We use only four emotional categories happy, sad, angry, and neutral to compare the performance of our model with other research using the same categories  The IEMOCAP dataset includes five sessions, and each session contains utterances from two speakers (one male and one female)  This data collection process resulted in 10 unique speakers  For consistent comparison with previous work, we merge the excitement dataset with the happiness dataset  The final dataset contains a total of 5531 utterances (1636 happy, 1084 sad, 1103 angry, 1708 neutral) 
Feature extraction
To extract speech information from audio signals, we use MFCC values, which are widely used in analyzing audio signals  The MFCC feature set contains a total of 39 features, which include 12 MFCC parameters (1-12) from the 26 Mel-frequency bands and log-energy parameters, 13 delta and 13 acceleration coefficients The frame size is set to 25 ms at a rate of 10 ms with the Hamming function  According to the length of each wave file, the sequential step of the MFCC features is varied  To extract additional information from the data, we also use prosodic features, which show effectiveness in affective computing  The prosodic features are composed of 35 features, which include the F0 frequency, the voicing probability, and the loudness contours  All of these MFCC and prosodic features are extracted from the data using the OpenSMILE toolkit BIBREF26  
Implementation details
Among the variants of the RNN function, we use GRUs as they yield comparable performance to that of the LSTM and include a smaller number of weight parameters BIBREF29   We use a max encoder step of 750 for the audio input, based on the implementation choices presented in BIBREF30 and 128 for the text input because it covers the maximum length of the transcripts  The vocabulary size of the dataset is 3,747, including the “_UNK_" token, which represents unknown words, and the “_PAD_" token, which is used to indicate padding information added while preparing mini-batch data  The number of hidden units and the number of layers in the RNN for each model (ARE, TRE, MDRE and MDREA) are selected based on extensive hyperparameter search experiments  The weights of the hidden units are initialized using orthogonal weights BIBREF31 ], and the text embedding layer is initialized from pretrained word-embedding vectors BIBREF32  
In preparing the textual dataset, we first use the released transcripts of the IEMOCAP dataset for simplicity  To investigate the practical performance, we then process all of the IEMOCAP audio data using an ASR system (the Google Cloud Speech API) and retrieve the transcripts  The performance of the Google ASR system is reflected by its word error rate (WER) of 5 53% 
Performance evaluation
As the dataset is not explicitly split beforehand into training, development, and testing sets, we perform 5-fold cross validation to determine the overall performance of the model  The data in each fold are split into training, development, and testing datasets (8:0 5:1 5, respectively)  After training the model, we measure the weighted average precision (WAP) over the 5-fold dataset  We train and evaluate the model 10 times per fold, and the model performance is assessed in terms of the mean score and standard deviation 
We examine the WAP values, which are shown in Table 1  First, our ARE model shows the baseline performance because we use minimal audio features, such as the MFCC and prosodic features with simple architectures  On the other hand, the TRE model shows higher performance gain compared to the ARE  From this result, we note that textual data are informative in emotion prediction tasks, and the recurrent encoder model is effective in understanding these types of sequential data  Second, the newly proposed model, MDRE, shows a substantial performance gain  It thus achieves the state-of-the-art performance with a WAP value of 0 718  This result shows that multimodal information is a key factor in affective computing  Lastly, the attention model, MDREA, also outperforms the best existing research results (WAP 0 690 to 0 688) BIBREF19   However, the MDREA model does not match the performance of the MDRE model, even though it utilizes a more complex architecture  We believe that this result arises because insufficient data are available to properly determine the complex model parameters in the MDREA model  Moreover, we presume that this model will show better performance when the audio signals are aligned with the textual sequence while applying the attention mechanism  We leave the implementation of this point as a future research direction 
To investigate the practical performance of the proposed models, we conduct further experiments with the ASR-processed transcript data (see “-ASR” models in Table )  The label accuracy of the processed transcripts is 5 53% WER  The TRE-ASR, MDRE-ASR and MDREA-ASR models reflect degraded performance compared to that of the TRE, MDRE and MDREA models  However, the performance of these models is still competitive; in particular, the MDRE-ASR model outperforms the previous best-performing model, 3CNN-LSTM10H (WAP 0 691 to 0 688) 
Error analysis
We analyze the predictions of the ARE, TRE, and MDRE models  Figure shows the confusion matrix of each model  The ARE model (Fig  ) incorrectly classifies most instances of happy as neutral (43 51%); thus, it shows reduced accuracy (35 15%) in predicting the the happy class  Overall, most of the emotion classes are frequently confused with the neutral class  This observation is in line with the findings of BIBREF30 , who noted that the neutral class is located in the center of the activation-valence space, complicating its discrimination from the other classes 
Interestingly, the TRE model (Fig  ) shows greater prediction gains in predicting the happy class when compared to the ARE model (35 15% to 75 73%)  This result seems plausible because the model can benefit from the differences among the distributions of words in happy and neutral expressions, which gives more emotional information to the model than that of the audio signal data  On the other hand, it is striking that the TRE model incorrectly predicts instances of the sad class as the happy class 16 20% of the time, even though these emotional states are opposites of one another 
The MDRE model (Fig  ) compensates for the weaknesses of the previous two models (ARE and TRE) and benefits from their strengths to a surprising degree  The values arranged along the diagonal axis show that all of the accuracies of the correctly predicted class have increased  Furthermore, the occurrence of the incorrect “sad-to-happy" cases in the TRE model is reduced from 16 20% to 9 15% 
Conclusions
In this paper, we propose a novel multimodal dual recurrent encoder model that simultaneously utilizes text data, as well as audio signals, to permit the better understanding of speech data  Our model encodes the information from audio and text sequences using dual RNNs and then combines the information from these sources using a feed-forward neural model to predict the emotion class  Extensive experiments show that our proposed model outperforms other state-of-the-art methods in classifying the four emotion categories, and accuracies ranging from 68 8% to 71 8% are obtained when the model is applied to the IEMOCAP dataset  In particular, it resolves the issue in which predictions frequently incorrectly yield the neutral class, as occurs in previous models that focus on audio features 
In the future work, we aim to extend the modalities to audio, text and video inputs  Furthermore, we plan to investigate the application of the attention mechanism to data derived from multiple modalities  This approach seems likely to uncover enhanced learning schemes that will increase performance in both speech emotion recognition and other multimodal classification tasks 
Acknowledgments
K  Jung is with the Department of Electrical and Computer Engineering, ASRI, Seoul National University, Seoul, Korea  This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under Industrial Technology Innovation Program (No 10073144) 

ground_truth: combines the information from these sources using a feed-forward neural model

encodes the information from audio and text sequences using dual RNNs and then combines the information from these sources using a feed-forward neural model

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does their model improve interpretability compared to softmax transformers?
Context 3: Introduction
The Transformer architecture BIBREF0 for deep neural networks has quickly risen to prominence in NLP through its efficiency and performance, leading to improvements in the state of the art of Neural Machine Translation BIBREF1, BIBREF2, as well as inspiring other powerful general-purpose models like BERT BIBREF3 and GPT-2 BIBREF4  At the heart of the Transformer lie multi-head attention mechanisms: each word is represented by multiple different weighted averages of its relevant context  As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9 
The attention distribution of each head is predicted typically using the softmax normalizing transform  As a result, all context words have non-zero attention weight  Recent work on single attention architectures suggest that using sparse normalizing transforms in attention mechanisms such as sparsemax – which can yield exactly zero probabilities for irrelevant words – may improve performance and interpretability BIBREF12, BIBREF13, BIBREF14  Qualitative analysis of attention heads BIBREF0 suggests that, depending on what phenomena they capture, heads tend to favor flatter or more peaked distributions 
Recent works have proposed sparse Transformers BIBREF10 and adaptive span Transformers BIBREF11  However, the “sparsity" of those models only limits the attention to a contiguous span of past tokens, while in this work we propose a highly adaptive Transformer model that is capable of attending to a sparse set of words that are not necessarily contiguous  Figure FIGREF1 shows the relationship of these methods with ours 
Our contributions are the following:
We introduce sparse attention into the Transformer architecture, showing that it eases interpretability and leads to slight accuracy gains 
We propose an adaptive version of sparse attention, where the shape of each attention head is learnable and can vary continuously and dynamically between the dense limit case of softmax and the sparse, piecewise-linear sparsemax case 
We make an extensive analysis of the added interpretability of these models, identifying both crisper examples of attention head behavior observed in previous work, as well as novel behaviors unraveled thanks to the sparsity and adaptivity of our proposed model 
Background ::: The Transformer
In NMT, the Transformer BIBREF0 is a sequence-to-sequence (seq2seq) model which maps an input sequence to an output sequence through hierarchical multi-head attention mechanisms, yielding a dynamic, context-dependent strategy for propagating information within and across sentences  It contrasts with previous seq2seq models, which usually rely either on costly gated recurrent operations BIBREF15, BIBREF16 or static convolutions BIBREF17 
Given $n$ query contexts and $m$ sequence items under consideration, attention mechanisms compute, for each query, a weighted representation of the items  The particular attention mechanism used in BIBREF0 is called scaled dot-product attention, and it is computed in the following way:
where $\mathbf {Q} \in \mathbb {R}^{n \times d}$ contains representations of the queries, $\mathbf {K}, \mathbf {V} \in \mathbb {R}^{m \times d}$ are the keys and values of the items attended over, and $d$ is the dimensionality of these representations  The $\mathbf {\pi }$ mapping normalizes row-wise using softmax, $\mathbf {\pi }(\mathbf {Z})_{ij} = \operatornamewithlimits{\mathsf {softmax}}(\mathbf {z}_i)_j$, where
In words, the keys are used to compute a relevance score between each item and query  Then, normalized attention weights are computed using softmax, and these are used to weight the values of each item at each query context 
However, for complex tasks, different parts of a sequence may be relevant in different ways, motivating multi-head attention in Transformers  This is simply the application of Equation DISPLAY_FORM7 in parallel $H$ times, each with a different, learned linear transformation that allows specialization:
In the Transformer, there are three separate multi-head attention mechanisms for distinct purposes:
Encoder self-attention: builds rich, layered representations of each input word, by attending on the entire input sentence 
Context attention: selects a representative weighted average of the encodings of the input words, at each time step of the decoder 
Decoder self-attention: attends over the partial output sentence fragment produced so far 
Together, these mechanisms enable the contextualized flow of information between the input sentence and the sequential decoder 
Background ::: Sparse Attention
The softmax mapping (Equation DISPLAY_FORM8) is elementwise proportional to $\exp $, therefore it can never assign a weight of exactly zero  Thus, unnecessary items are still taken into consideration to some extent  Since its output sums to one, this invariably means less weight is assigned to the relevant items, potentially harming performance and interpretability BIBREF18  This has motivated a line of research on learning networks with sparse mappings BIBREF19, BIBREF20, BIBREF21, BIBREF22  We focus on a recently-introduced flexible family of transformations, $\alpha $-entmax BIBREF23, BIBREF14, defined as:
where $\triangle ^d \lbrace \mathbf {p}\in \mathbb {R}^d:\sum _{i} p_i = 1\rbrace $ is the probability simplex, and, for $\alpha \ge 1$, $\mathsf {H}^{\textsc {T}}_\alpha $ is the Tsallis continuous family of entropies BIBREF24:
This family contains the well-known Shannon and Gini entropies, corresponding to the cases $\alpha =1$ and $\alpha =2$, respectively 
Equation DISPLAY_FORM14 involves a convex optimization subproblem  Using the definition of $\mathsf {H}^{\textsc {T}}_\alpha $, the optimality conditions may be used to derive the following form for the solution (Appendix SECREF83):
where $[\cdot ]_+$ is the positive part (ReLU) function, $\mathbf {1}$ denotes the vector of all ones, and $\tau $ – which acts like a threshold – is the Lagrange multiplier corresponding to the $\sum _i p_i=1$ constraint 
Background ::: Sparse Attention ::: Properties of @!START@$\alpha $@!END@-entmax 
The appeal of $\alpha $-entmax for attention rests on the following properties  For $\alpha =1$ (i e , when $\mathsf {H}^{\textsc {T}}_\alpha $ becomes the Shannon entropy), it exactly recovers the softmax mapping (We provide a short derivation in Appendix SECREF89 )  For all $\alpha >1$ it permits sparse solutions, in stark contrast to softmax  In particular, for $\alpha =2$, it recovers the sparsemax mapping BIBREF19, which is piecewise linear  In-between, as $\alpha $ increases, the mapping continuously gets sparser as its curvature changes 
To compute the value of $\alpha $-entmax, one must find the threshold $\tau $ such that the r h s  in Equation DISPLAY_FORM16 sums to one  BIBREF23 propose a general bisection algorithm  BIBREF14 introduce a faster, exact algorithm for $\alpha =1 5$, and enable using $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with fixed $\alpha $ within a neural network by showing that the $\alpha $-entmax Jacobian w r t  $\mathbf {z}$ for $\mathbf {p}^\star = \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ is
Our work furthers the study of $\alpha $-entmax by providing a derivation of the Jacobian w r t  the hyper-parameter $\alpha $ (Section SECREF3), thereby allowing the shape and sparsity of the mapping to be learned automatically  This is particularly appealing in the context of multi-head attention mechanisms, where we shall show in Section SECREF35 that different heads tend to learn different sparsity behaviors 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax
We now propose a novel Transformer architecture wherein we simply replace softmax with $\alpha $-entmax in the attention heads  Concretely, we replace the row normalization $\mathbf {\pi }$ in Equation DISPLAY_FORM7 by
This change leads to sparse attention weights, as long as $\alpha >1$; in particular, $\alpha =1 5$ is a sensible starting point BIBREF14 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Different @!START@$\alpha $@!END@ per head 
Unlike LSTM-based seq2seq models, where $\alpha $ can be more easily tuned by grid search, in a Transformer, there are many attention heads in multiple layers  Crucial to the power of such models, the different heads capture different linguistic phenomena, some of them isolating important words, others spreading out attention across phrases BIBREF0  This motivates using different, adaptive $\alpha $ values for each attention head, such that some heads may learn to be sparser, and others may become closer to softmax  We propose doing so by treating the $\alpha $ values as neural network parameters, optimized via stochastic gradients along with the other weights 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Derivatives w r t  @!START@$\alpha $@!END@ 
In order to optimize $\alpha $ automatically via gradient methods, we must compute the Jacobian of the entmax output w r t  $\alpha $  Since entmax is defined through an optimization problem, this is non-trivial and cannot be simply handled through automatic differentiation; it falls within the domain of argmin differentiation, an active research topic in optimization BIBREF25, BIBREF26 
One of our key contributions is the derivation of a closed-form expression for this Jacobian  The next proposition provides such an expression, enabling entmax layers with adaptive $\alpha $  To the best of our knowledge, ours is the first neural network module that can automatically, continuously vary in shape away from softmax and toward sparse mappings like sparsemax 
Proposition 1 Let $\mathbf {p}^\star \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ be the solution of Equation DISPLAY_FORM14  Denote the distribution $\tilde{p}_i {(p_i^\star )^{2 - \alpha }}{ \sum _j(p_j^\star )^{2-\alpha }}$ and let $h_i -p^\star _i \log p^\star _i$  The $i$th component of the Jacobian $\mathbf {g} \frac{\partial \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})}{\partial \alpha }$ is
proof uses implicit function differentiation and is given in Appendix SECREF10 
Proposition UNKREF22 provides the remaining missing piece needed for training adaptively sparse Transformers  In the following section, we evaluate this strategy on neural machine translation, and analyze the behavior of the learned attention heads 
Experiments
We apply our adaptively sparse Transformers on four machine translation tasks  For comparison, a natural baseline is the standard Transformer architecture using the softmax transform in its multi-head attention mechanisms  We consider two other model variants in our experiments that make use of different normalizing transformations:
1 5-entmax: a Transformer with sparse entmax attention with fixed $\alpha =1 5$ for all heads  This is a novel model, since 1 5-entmax had only been proposed for RNN-based NMT models BIBREF14, but never in Transformers, where attention modules are not just one single component of the seq2seq model but rather an integral part of all of the model components 
$\alpha $-entmax: an adaptive Transformer with sparse entmax attention with a different, learned $\alpha _{i,j}^t$ for each head 
The adaptive model has an additional scalar parameter per attention head per layer for each of the three attention mechanisms (encoder self-attention, context attention, and decoder self-attention), i e ,
and we set $\alpha _{i,j}^t = 1 + \operatornamewithlimits{\mathsf {sigmoid}}(a_{i,j}^t) \in ]1, 2[$  All or some of the $\alpha $ values can be tied if desired, but we keep them independent for analysis purposes 
Experiments ::: Datasets 
Our models were trained on 4 machine translation datasets of different training sizes:
[itemsep= 5ex,leftmargin=2ex]
IWSLT 2017 German $\rightarrow $ English BIBREF27: 200K sentence pairs 
KFTT Japanese $\rightarrow $ English BIBREF28: 300K sentence pairs 
WMT 2016 Romanian $\rightarrow $ English BIBREF29: 600K sentence pairs 
WMT 2014 English $\rightarrow $ German BIBREF30: 4 5M sentence pairs 
All of these datasets were preprocessed with byte-pair encoding BIBREF31, using joint segmentations of 32k merge operations 
Experiments ::: Training 
We follow the dimensions of the Transformer-Base model of BIBREF0: The number of layers is $L=6$ and number of heads is $H=8$ in the encoder self-attention, the context attention, and the decoder self-attention  We use a mini-batch size of 8192 tokens and warm up the learning rate linearly until 20k steps, after which it decays according to an inverse square root schedule  All models were trained until convergence of validation accuracy, and evaluation was done at each 10k steps for ro$\rightarrow $en and en$\rightarrow $de and at each 5k steps for de$\rightarrow $en and ja$\rightarrow $en  The end-to-end computational overhead of our methods, when compared to standard softmax, is relatively small; in training tokens per second, the models using $\alpha $-entmax and $1 5$-entmax are, respectively, $75\%$ and $90\%$ the speed of the softmax model 
Experiments ::: Results 
We report test set tokenized BLEU BIBREF32 results in Table TABREF27  We can see that replacing softmax by entmax does not hurt performance in any of the datasets; indeed, sparse attention Transformers tend to have slightly higher BLEU, but their sparsity leads to a better potential for analysis  In the next section, we make use of this potential by exploring the learned internal mechanics of the self-attention heads 
Analysis
We conduct an analysis for the higher-resource dataset WMT 2014 English $\rightarrow $ German of the attention in the sparse adaptive Transformer model ($\alpha $-entmax) at multiple levels: we analyze high-level statistics as well as individual head behavior  Moreover, we make a qualitative analysis of the interpretability capabilities of our models 
Analysis ::: High-Level Statistics ::: What kind of @!START@$\alpha $@!END@ values are learned?
Figure FIGREF37 shows the learning trajectories of the $\alpha $ parameters of a selected subset of heads  We generally observe a tendency for the randomly-initialized $\alpha $ parameters to decrease initially, suggesting that softmax-like behavior may be preferable while the model is still very uncertain  After around one thousand steps, some heads change direction and become sparser, perhaps as they become more confident and specialized  This shows that the initialization of $\alpha $ does not predetermine its sparsity level or the role the head will have throughout  In particular, head 8 in the encoder self-attention layer 2 first drops to around $\alpha =1 3$ before becoming one of the sparsest heads, with $\alpha \approx 2$ 
The overall distribution of $\alpha $ values at convergence can be seen in Figure FIGREF38  We can observe that the encoder self-attention blocks learn to concentrate the $\alpha $ values in two modes: a very sparse one around $\alpha \rightarrow 2$, and a dense one between softmax and 1 5-entmax   However, the decoder self and context attention only learn to distribute these parameters in a single mode  We show next that this is reflected in the average density of attention weight vectors as well 
Analysis ::: High-Level Statistics ::: Attention weight density when translating 
For any $\alpha >1$, it would still be possible for the weight matrices in Equation DISPLAY_FORM9 to learn re-scalings so as to make attention sparser or denser  To visualize the impact of adaptive $\alpha $ values, we compare the empirical attention weight density (the average number of tokens receiving non-zero attention) within each module, against sparse Transformers with fixed $\alpha =1 5$ 
Figure FIGREF40 shows that, with fixed $\alpha =1 5$, heads tend to be sparse and similarly-distributed in all three attention modules  With learned $\alpha $, there are two notable changes: (i) a prominent mode corresponding to fully dense probabilities, showing that our models learn to combine sparse and dense attention, and (ii) a distinction between the encoder self-attention – whose background distribution tends toward extreme sparsity – and the other two modules, who exhibit more uniform background distributions  This suggests that perhaps entirely sparse Transformers are suboptimal 
The fact that the decoder seems to prefer denser attention distributions might be attributed to it being auto-regressive, only having access to past tokens and not the full sentence  We speculate that it might lose too much information if it assigned weights of zero to too many tokens in the self-attention, since there are fewer tokens to attend to in the first place 
Teasing this down into separate layers, Figure FIGREF41 shows the average (sorted) density of each head for each layer  We observe that $\alpha $-entmax is able to learn different sparsity patterns at each layer, leading to more variance in individual head behavior, to clearly-identified dense and sparse heads, and overall to different tendencies compared to the fixed case of $\alpha =1 5$ 
Analysis ::: High-Level Statistics ::: Head diversity 
To measure the overall disagreement between attention heads, as a measure of head diversity, we use the following generalization of the Jensen-Shannon divergence:
where $\mathbf {p}_j$ is the vector of attention weights assigned by head $j$ to each word in the sequence, and $\mathsf {H}^\textsc {S}$ is the Shannon entropy, base-adjusted based on the dimension of $\mathbf {p}$ such that $JS \le 1$  We average this measure over the entire validation set  The higher this metric is, the more the heads are taking different roles in the model 
Figure FIGREF44 shows that both sparse Transformer variants show more diversity than the traditional softmax one  Interestingly, diversity seems to peak in the middle layers of the encoder self-attention and context attention, while this is not the case for the decoder self-attention 
The statistics shown in this section can be found for the other language pairs in Appendix SECREF8 
Analysis ::: Identifying Head Specializations
Previous work pointed out some specific roles played by different heads in the softmax Transformer model BIBREF33, BIBREF5, BIBREF9  Identifying the specialization of a head can be done by observing the type of tokens or sequences that the head often assigns most of its attention weight; this is facilitated by sparsity 
Analysis ::: Identifying Head Specializations ::: Positional heads 
One particular type of head, as noted by BIBREF9, is the positional head  These heads tend to focus their attention on either the previous or next token in the sequence, thus obtaining representations of the neighborhood of the current time step  In Figure FIGREF47, we show attention plots for such heads, found for each of the studied models  The sparsity of our models allows these heads to be more confident in their representations, by assigning the whole probability distribution to a single token in the sequence  Concretely, we may measure a positional head's confidence as the average attention weight assigned to the previous token  The softmax model has three heads for position $-1$, with median confidence $93 5\%$  The $1 5$-entmax model also has three heads for this position, with median confidence $94 4\%$  The adaptive model has four heads, with median confidences $95 9\%$, the lowest-confidence head being dense with $\alpha =1 18$, while the highest-confidence head being sparse ($\alpha =1 91$) 
For position $+1$, the models each dedicate one head, with confidence around $95\%$, slightly higher for entmax  The adaptive model sets $\alpha =1 96$ for this head 
Analysis ::: Identifying Head Specializations ::: BPE-merging head 
Due to the sparsity of our models, we are able to identify other head specializations, easily identifying which heads should be further analysed  In Figure FIGREF51 we show one such head where the $\alpha $ value is particularly high (in the encoder, layer 1, head 4 depicted in Figure FIGREF37)  We found that this head most often looks at the current time step with high confidence, making it a positional head with offset 0  However, this head often spreads weight sparsely over 2-3 neighboring tokens, when the tokens are part of the same BPE cluster or hyphenated words  As this head is in the first layer, it provides a useful service to the higher layers by combining information evenly within some BPE clusters 
For each BPE cluster or cluster of hyphenated words, we computed a score between 0 and 1 that corresponds to the maximum attention mass assigned by any token to the rest of the tokens inside the cluster in order to quantify the BPE-merging capabilities of these heads  There are not any attention heads in the softmax model that are able to obtain a score over $80\%$, while for $1 5$-entmax and $\alpha $-entmax there are two heads in each ($83 3\%$ and $85 6\%$ for $1 5$-entmax and $88 5\%$ and $89 8\%$ for $\alpha $-entmax) 
Analysis ::: Identifying Head Specializations ::: Interrogation head 
On the other hand, in Figure FIGREF52 we show a head for which our adaptively sparse model chose an $\alpha $ close to 1, making it closer to softmax (also shown in encoder, layer 1, head 3 depicted in Figure FIGREF37)  We observe that this head assigns a high probability to question marks at the end of the sentence in time steps where the current token is interrogative, thus making it an interrogation-detecting head  We also observe this type of heads in the other models, which we also depict in Figure FIGREF52  The average attention weight placed on the question mark when the current token is an interrogative word is $98 5\%$ for softmax, $97 0\%$ for $1 5$-entmax, and $99 5\%$ for $\alpha $-entmax 
Furthermore, we can examine sentences where some tendentially sparse heads become less so, thus identifying sources of ambiguity where the head is less confident in its prediction  An example is shown in Figure FIGREF55 where sparsity in the same head differs for sentences of similar length 
Related Work ::: Sparse attention 
Prior work has developed sparse attention mechanisms, including applications to NMT BIBREF19, BIBREF12, BIBREF20, BIBREF22, BIBREF34  BIBREF14 introduced the entmax function this work builds upon  In their work, there is a single attention mechanism which is controlled by a fixed $\alpha $  In contrast, this is the first work to allow such attention mappings to dynamically adapt their curvature and sparsity, by automatically adjusting the continuous $\alpha $ parameter  We also provide the first results using sparse attention in a Transformer model 
Related Work ::: Fixed sparsity patterns 
Recent research improves the scalability of Transformer-like networks through static, fixed sparsity patterns BIBREF10, BIBREF35  Our adaptively-sparse Transformer can dynamically select a sparsity pattern that finds relevant words regardless of their position (e g , Figure FIGREF52)  Moreover, the two strategies could be combined  In a concurrent line of research, BIBREF11 propose an adaptive attention span for Transformer language models  While their work has each head learn a different contiguous span of context tokens to attend to, our work finds different sparsity patterns in the same span  Interestingly, some of their findings mirror ours – we found that attention heads in the last layers tend to be denser on average when compared to the ones in the first layers, while their work has found that lower layers tend to have a shorter attention span compared to higher layers 
Related Work ::: Transformer interpretability 
The original Transformer paper BIBREF0 shows attention visualizations, from which some speculation can be made of the roles the several attention heads have  BIBREF7 study the syntactic abilities of the Transformer self-attention, while BIBREF6 extract dependency relations from the attention weights  BIBREF8 find that the self-attentions in BERT BIBREF3 follow a sequence of processes that resembles a classical NLP pipeline  Regarding redundancy of heads, BIBREF9 develop a method that is able to prune heads of the multi-head attention module and make an empirical study of the role that each head has in self-attention (positional, syntactic and rare words)  BIBREF36 also aim to reduce head redundancy by adding a regularization term to the loss that maximizes head disagreement and obtain improved results  While not considering Transformer attentions, BIBREF18 show that traditional attention mechanisms do not necessarily improve interpretability since softmax attention is vulnerable to an adversarial attack leading to wildly different model predictions for the same attention weights  Sparse attention may mitigate these issues; however, our work focuses mostly on a more mechanical aspect of interpretation by analyzing head behavior, rather than on explanations for predictions 
Conclusion and Future Work
We contribute a novel strategy for adaptively sparse attention, and, in particular, for adaptively sparse Transformers  We present the first empirical analysis of Transformers with sparse attention mappings (i e , entmax), showing potential in both translation accuracy as well as in model interpretability 
In particular, we analyzed how the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence  Our adaptivity strategy relies only on gradient-based optimization, side-stepping costly per-head hyper-parameter searches  Further speed-ups are possible by leveraging more parallelism in the bisection algorithm for computing $\alpha $-entmax 
Finally, some of the automatically-learned behaviors of our adaptively sparse Transformers – for instance, the near-deterministic positional heads or the subword joining head – may provide new ideas for designing static variations of the Transformer 
Acknowledgments
This work was supported by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contracts UID/EEA/50008/2019 and CMUPERI/TIC/0046/2014 (GoLocal)  We are grateful to Ben Peters for the $\alpha $-entmax code and Erick Fonseca, Marcos Treviso, Pedro Martins, and Tsvetomila Mihaylova for insightful group discussion  We thank Mathieu Blondel for the idea to learn $\alpha $  We would also like to thank the anonymous reviewers for their helpful feedback 
Supplementary Material
Background ::: Regularized Fenchel-Young prediction functions
Definition 1 (BIBREF23)
Let $\Omega \colon \triangle ^d \rightarrow {\mathbb {R}}\cup \lbrace \infty \rbrace $ be a strictly convex regularization function  We define the prediction function $\mathbf {\pi }_{\Omega }$ as
Background ::: Characterizing the @!START@$\alpha $@!END@-entmax mapping
Lemma 1 (BIBREF14) For any $\mathbf {z}$, there exists a unique $\tau ^\star $ such that
Proof: From the definition of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$,
we may easily identify it with a regularized prediction function (Def  UNKREF81):
We first note that for all $\mathbf {p}\in \triangle ^d$,
From the constant invariance and scaling properties of $\mathbf {\pi }_{\Omega }$ BIBREF23,
Using BIBREF23, noting that $g^{\prime }(t) = t^{\alpha - 1}$ and $(g^{\prime })^{-1}(u) = u^{{1}{\alpha -1}}$, yields
Since $\mathsf {H}^{\textsc {T}}_\alpha $ is strictly convex on the simplex, $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ has a unique solution $\mathbf {p}^\star $  Equation DISPLAY_FORM88 implicitly defines a one-to-one mapping between $\mathbf {p}^\star $ and $\tau ^\star $ as long as $\mathbf {p}^\star \in \triangle $, therefore $\tau ^\star $ is also unique 
Background ::: Connections to softmax and sparsemax
The Euclidean projection onto the simplex, sometimes referred to, in the context of neural attention, as sparsemax BIBREF19, is defined as
The solution can be characterized through the unique threshold $\tau $ such that $\sum _i \operatornamewithlimits{\mathsf {sparsemax}}(\mathbf {z})_i = 1$ and BIBREF38
Thus, each coordinate of the sparsemax solution is a piecewise-linear function  Visibly, this expression is recovered when setting $\alpha =2$ in the $\alpha $-entmax expression (Equation DISPLAY_FORM85); for other values of $\alpha $, the exponent induces curvature 
On the other hand, the well-known softmax is usually defined through the expression
which can be shown to be the unique solution of the optimization problem
where $\mathsf {H}^\textsc {S}(\mathbf {p}) -\sum _i p_i \log p_i$ is the Shannon entropy  Indeed, setting the gradient to 0 yields the condition $\log p_i = z_j - \nu _i - \tau - 1$, where $\tau $ and $\nu > 0$ are Lagrange multipliers for the simplex constraints $\sum _i p_i = 1$ and $p_i \ge 0$, respectively  Since the l h s  is only finite for $p_i>0$, we must have $\nu _i=0$ for all $i$, by complementary slackness  Thus, the solution must have the form $p_i = {\exp (z_i)}{Z}$, yielding Equation DISPLAY_FORM92 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@
Recall that the entmax transformation is defined as:
where $\alpha \ge 1$ and $\mathsf {H}^{\textsc {T}}_{\alpha }$ is the Tsallis entropy,
and $\mathsf {H}^\textsc {S}(\mathbf {p}):= -\sum _j p_j \log p_j$ is the Shannon entropy 
In this section, we derive the Jacobian of $\operatornamewithlimits{\mathsf {entmax }}$ with respect to the scalar parameter $\alpha $ 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: General case of @!START@$\alpha >1$@!END@
From the KKT conditions associated with the optimization problem in Eq  DISPLAY_FORM85, we have that the solution $\mathbf {p}^{\star }$ has the following form, coordinate-wise:
where $\tau ^{\star }$ is a scalar Lagrange multiplier that ensures that $\mathbf {p}^{\star }$ normalizes to 1, i e , it is defined implicitly by the condition:
For general values of $\alpha $, Eq  DISPLAY_FORM98 lacks a closed form solution  This makes the computation of the Jacobian
non-trivial  Fortunately, we can use the technique of implicit differentiation to obtain this Jacobian 
The Jacobian exists almost everywhere, and the expressions we derive expressions yield a generalized Jacobian BIBREF37 at any non-differentiable points that may occur for certain ($\alpha $, $\mathbf {z}$) pairs  We begin by noting that $\frac{\partial p_i^{\star }}{\partial \alpha } = 0$ if $p_i^{\star } = 0$, because increasing $\alpha $ keeps sparse coordinates sparse  Therefore we need to worry only about coordinates that are in the support of $\mathbf {p}^\star $  We will assume hereafter that the $i$th coordinate of $\mathbf {p}^\star $ is non-zero  We have:
We can see that this Jacobian depends on $\frac{\partial \tau ^{\star }}{\partial \alpha }$, which we now compute using implicit differentiation 
Let $\mathcal {S} = \lbrace i: p^\star _i > 0 \rbrace $)  By differentiating both sides of Eq  DISPLAY_FORM98, re-using some of the steps in Eq  DISPLAY_FORM101, and recalling Eq  DISPLAY_FORM97, we get
from which we obtain:
Finally, plugging Eq  DISPLAY_FORM103 into Eq  DISPLAY_FORM101, we get:
where we denote by
The distribution $\tilde{\mathbf {p}}(\alpha )$ can be interpreted as a “skewed” distribution obtained from $\mathbf {p}^{\star }$, which appears in the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ w r t  $\mathbf {z}$ as well BIBREF14 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Solving the indetermination for @!START@$\alpha =1$@!END@
We can write Eq  DISPLAY_FORM104 as
When $\alpha \rightarrow 1^+$, we have $\tilde{\mathbf {p}}(\alpha ) \rightarrow \mathbf {p}^{\star }$, which leads to a $\frac{0}{0}$ indetermination 
To solve this indetermination, we will need to apply L'Hôpital's rule twice  Let us first compute the derivative of $\tilde{p}_i(\alpha )$ with respect to $\alpha $  We have
therefore
Differentiating the numerator and denominator in Eq  DISPLAY_FORM107, we get:
with
and
When $\alpha \rightarrow 1^+$, $B$ becomes again a $\frac{0}{0}$ indetermination, which we can solve by applying again L'Hôpital's rule  Differentiating the numerator and denominator in Eq  DISPLAY_FORM112:
Finally, summing Eq  DISPLAY_FORM111 and Eq  DISPLAY_FORM113, we get
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Summary
To sum up, we have the following expression for the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with respect to $\alpha $:

ground_truth: the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence

We introduce sparse attention into the Transformer architecture