prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What was their highest recall score?
Context 1: Introduction
BioASQ is a biomedical document classification, document retrieval, and question answering competition, currently in its seventh year  We provide an overview of our submissions to semantic question answering task (7b, Phase B) of BioASQ 7 (except for 'ideal answer' test, in which we did not participate this year)  In this task systems are provided with biomedical questions and are required to submit ideal and exact answers to those questions  We have used BioBERT BIBREF0 based system , see also Bidirectional Encoder Representations from Transformers(BERT) BIBREF1, and we fine tuned it for the biomedical question answering task  Our system scored near the top for factoid questions for all the batches of the challenge  More specifially, in the third test batch set, our system achieved highest ‘MRR’ score for Factoid Question Answering task  Also, for List-type question answering task our system achieved highest recall score in the fourth test batch set  Along with our detailed approach, we present the results for our submissions and also highlight identified downsides for our current approach and ways to improve them in our future experiments  In last test batch results we placed 4th for List-type questions and 3rd for Factoid-type questions )
The QA task is organized in two phases  Phase A deals with retrieval of the relevant document, snippets, concepts, and RDF triples, and phase B deals with exact and ideal answer generations (which is a paragraph size summary of snippets)  Exact answer generation is required for factoid, list, and yes/no type question 
BioASQ organizers provide the training and testing data  The training data consists of questions, gold standard documents, snippets, concepts, and ideal answers (which we did not use in this paper, but we used last year BIBREF2)  The test data is split between phases A and B  The Phase A dataset consists of the questions, unique ids, question types  The Phase B dataset consists of the questions, golden standard documents, snippets, unique ids and question types  Exact answers for factoid type questions are evaluated using strict accuracy (the top answer), lenient accuracy (the top 5 answers), and MRR (Mean Reciprocal Rank) which takes into account the ranks of returned answers  Answers for the list type question are evaluated using precision, recall, and F-measure 
Related Work ::: BioAsq
Sharma et al  BIBREF3 describe a system with two stage process for factoid and list type question answering  Their system extracts relevant entities and then runs supervised classifier to rank the entities  Wiese et al  BIBREF4 propose neural network based model for Factoid and List-type question answering task  The model is based on Fast QA and predicts the answer span in the passage for a given question  The model is trained on SQuAD data set and fine tuned on the BioASQ data  Dimitriadis et al  BIBREF5 proposed two stage process for Factoid question answering task  Their system uses general purpose tools such as Metamap, BeCas to identify candidate sentences  These candidate sentences are represented in the form of features, and are then ranked by the binary classifier  Classifier is trained on candidate sentences extracted from relevant questions, snippets and correct answers from BioASQ challenge  For factoid question answering task highest ‘MRR’ achieved in the 6th edition of BioASQ competition is ‘0 4325’  Our system is a neural network model based on contextual word embeddings BIBREF1 and achieved a ‘MRR’ score ‘0 6103’ in one of the test batches for Factoid Question Answering task 
Related Work ::: A minimum background on BERT
BERT stands for "Bidirectional Encoder Representations from Transformers" BIBREF1 is a contextual word embedding model  Given a sentence as an input, contextual embedding for the words are returned  The BERT model was designed so it can be fine tuned for 11 different tasks BIBREF1, including question answering tasks  For a question answering task, question and paragraph (context) are given as an input  A BERT standard is that question text and paragraph text are separated by a separator [Sep]  BERT question-answering fine tuning involves adding softmax layer  Softmax layer takes contextual word embeddings from BERT as input and learns to identity answer span present in the paragraph (context)  This process is represented in Figure FIGREF4 
BERT was originally trained to perform tasks such as language model creation using masked words and next-sentence-prediction  In other words BERT weights are learned such that context is used in building the representation of the word, not just as a loss function to help learn a context-independent representation  For detailed understanding of BERT Architecture, please refer to the original BERT paper BIBREF1 
Related Work ::: A minimum background on BERT ::: Comparison of Word Embeddings and Contextual Word Embeddings
A ‘word embedding’ is a learned representation  It is represented in the form of vector where words that have the same meaning have a similar vector representation  Consider a word embedding model 'word2vec' BIBREF6 trained on a corpus  Word embeddings generated from the model are context independent that is, word embeddings are returned regardless of where the words appear in a sentence and regardless of e g  the sentiment of the sentence  However, contextual word embedding models like BERT also takes context of the word into consideration 
Related Work ::: Comparison of BERT and Bio-BERT
‘BERT’ and BioBERT are very similar in terms of architecture  Difference is that ‘BERT’ is pretrained on Wikipedia articles, whereas BioBERT version used in our experiments is pretrained on Wikipedia, PMC and PubMed articles  Therefore BioBERT model is expected to perform well with biomedical text, in terms of generating contextual word embeddings 
BioBERT model used in our experiments is based on BERT-Base Architecture; BERT-Base has 12 transformer Layers where as BERT-Large has 24 transformer layers  Moreover contextual word embedding vector size is 768 for BERT-Base and more for BERT-large  According to BIBREF1 Bert-Large, fine-tuned on SQuAD 1 1 question answering data BIBREF7 can achieve F1 Score of 90 9 for Question Answering task where as BERT-Base Fine-tuned on the same SQuAD question answering BIBREF7 data could achieve F1 score of 88 5  One downside of the current version BioBERT is that word-piece vocabulary is the same as that of original BERT Model, as a result word-piece vocabulary does not include biomedical jargon  Lee et al  BIBREF0 created BioBERT, using the same pre-trained BERT released by Google, and hence in the word-piece vocabulary (vocab txt), as a result biomedical jargon is not included in word-piece vocabulary  Modifying word-piece vocabulary (vocab txt) at this stage would loose original compatibility with ‘BERT’, hence it is left unmodified 
In our future work we would like to build pre-trained ‘BERT’ model from scratch  We would pretrain the model with biomedical corpus (PubMed, ‘PMC’) and Wikipedia  Doing so would give us scope to create word piece vocabulary to include biomedical jargon and there are chances of model performing better with biomedical jargon being included in the word piece vocabulary  We will consider this scenario in the future, or wait for the next version of BioBERT 
Experiments: Factoid Question Answering Task
For Factoid Question Answering task, we fine tuned BioBERT BIBREF0 with question answering data and added new features  Fig  FIGREF4 shows the architecture of BioBERT fine tuned for question answering tasks: Input to BioBERT is word tokenized embeddings for question and the paragraph (Context)  As per the ‘BERT’ BIBREF1 standards, tokens ‘[CLS]’ and ‘[SEP]’ are appended to the tokenized input as illustrated in the figure  The resulting model has a softmax layer formed for predicting answer span indices in the given paragraph (Context)  On test data, the fine tuned model generates $n$-best predictions for each question  For a question, $n$-best corresponds that $n$ answers are returned as possible answers in the decreasing order of confidence  Variable $n$ is configurable  In our paper, any further mentions of ‘answer returned by the model’ correspond to the top answer returned by the model 
Experiments: Factoid Question Answering Task ::: Setup
BioASQ provides the training data  This data is based on previous BioASQ competitions  Train data we have considered is aggregate of all train data sets till the 5th version of BioASQ competition  We cleaned the data, that is, question-answering data without answers are removed and left with a total count of ‘530’ question answers  The data is split into train and test data in the ratio of 94 to 6; that is, count of '495' for training and '35' for testing 
The original data format is converted to the BERT/BioBERT format, where BioBERT expects ‘start_index’ of the actual answer  The ‘start_index corresponds to the index of the answer text present in the paragraph/ Context  For finding ‘start_index’ we used built-in python function find()  The function returns the lowest index of the actual answer present in the context(paragraph)  If the answer is not found ‘-1’ is returned as the index  The efficient way of finding start_index is, if the paragraph (Context) has multiple instances of answer text, then ‘start_index’ of the answer should be that instance of answer text whose context actually matches with what’s been asked in the question 
Example (Question, Answer and Paragraph from BIBREF8):
Question: Which drug should be used as an antidote in benzodiazepine overdose?
Answer: 'Flumazenil'
Paragraph(context):
"Flumazenil use in benzodiazepine overdose in the UK: a retrospective survey of NPIS data  OBJECTIVE: Benzodiazepine (BZD) overdose (OD) continues to cause significant morbidity and mortality in the UK  Flumazenil is an effective antidote but there is a risk of seizures, particularly in those who have co-ingested tricyclic antidepressants  A study was undertaken to examine the frequency of use, safety and efficacy of flumazenil in the management of BZD OD in the UK  METHODS: A 2-year retrospective cohort study was performed of all enquiries to the UK National Poisons Information Service involving BZD OD  RESULTS: Flumazenil was administered to 80 patients in 4504 BZD-related enquiries, 68 of whom did not have ventilatory failure or had recognised contraindications to flumazenil  Factors associated with flumazenil use were increased age, severe poisoning and ventilatory failure  Co-ingestion of tricyclic antidepressants and chronic obstructive pulmonary disease did not influence flumazenil administration  Seizure frequency in patients not treated with flumazenil was 0 3%" 
Actual answer is 'Flumazenil', but there are multiple instances of word 'Flu-mazenil'  Efficient way to identify the start-index for 'Flumazenil'(answer) is to find that particular instance of the word 'Flumazenil' which matches the context for the question  In the above example 'Flumazenil' highlighted in bold is the actual instance that matches question's context  Unfortunately, we could not identify readily available tools that can achieve this goal  In our future work, we look forward to handling these scenarios effectively 
Note: The creators of 'SQuAD' BIBREF7 have handled the task of identifying answer's start_index effectively  But 'SQuAD' data set is much more general and does not include biomedical question answering data 
Experiments: Factoid Question Answering Task ::: Training and error analysis
During our training with the BioASQ data, learning rate is set to 3e-5, as mentioned in the BioBERT paper BIBREF0  We started training the model with 495 available train data and 35 test data by setting number of epochs to 50  After training with these hyper-parameters training accuracy(exact match) was 99 3%(overfitting) and testing accuracy is only 4%  In the next iteration we reduced the number of epochs to 25 then training accuracy is reduced to 98 5% and test accuracy moved to 5%  We further reduced number of epochs to 15, and the resulting training accuracy was 70% and test accuracy 15%  In the next iteration set number of epochs to 12 and achieved train accuracy of 57 7% and test accuracy 23 3%  Repeated the experiment with 11 epochs and found training accuracy to be 57 7% and test accuracy to be same 22%  In the next iteration we set number of epochs to '9' and found training accuracy of 48% and test accuracy of 15%  Hence optimum number of epochs is taken as 12 epochs 
During our error analysis we found that on test data, model tends to return text in the beginning of the context(paragraph) as the answer  On analysing train data, we found that there are '120'(out of '495') question answering data instances having start_index:0, meaning 120( 25%) question answering data has first word(s) in the context(paragraph) as the answer  We removed 70% of those instances in order to make train data more balanced  In the new train data set we are left with '411' question answering data instances  This time we got the highest test accuracy of 26% at 11 epochs  We have submitted our results for BioASQ test batch-2, got strict accuracy of 32% and our system stood in 2nd place  Initially, hyper-parameter- 'batch size' is set to ‘400’  Later it is tuned to '32'  Although accuracy(exact answer match) remained at 26%, model generated concise and better answers at batch size ‘32’, that is wrong answers are close to the expected answer in good number of cases 
Example (from BIBREF8)
Question: Which mutated gene causes Chediak Higashi Syndrome?
Exact Answer: ‘lysosomal trafficking regulator gene’ 
The answer returned by a model trained at ‘400’ batch size is ‘Autosomal-recessive complicated spastic paraplegia with a novel lysosomal trafficking regulator’, and from the one trained at ‘32’ batch size is ‘lysosomal trafficking regulator’ 
In further experiments, we have fine tuned the BioBERT model with both ‘SQuAD’ dataset (version 2 0) and BioAsq train data  For training on ‘SQuAD’, hyper parameters- Learning rate and number of epochs are set to ‘3e-3’ and ‘3’ respectively as mentioned in the paper BIBREF1  Test accuracy of the model boosted to 44%  In one more experiment we trained model only on ‘SQuAD’ dataset, this time test accuracy of the model moved to 47%  The reason model did not perform up to the mark when trained with ‘SQuAD’ alongside BioASQ data could be that in formatted BioASQ data, start_index for the answer is not accurate, and affected the overall accuracy 
Our Systems and Their Performance on Factoid Questions
We have experimented with several systems and their variations, e g  created by training with specific additional features (see next subsection)  Here is their list and short descriptions  Unfortunately we did not pay attention to naming, and the systems evolved between test batches, so the overall picture can only be understood by looking at the details 
When we started the experiments our objective was to see whether BioBERT and entailment-based techniques can provide value for in the context of biomedical question answering  The answer to both questions was a yes, qualified by many examples clearly showing the limitations of both methods  Therefore we tried to address some of these limitations using feature engineering with mixed results: some clear errors got corrected and new errors got introduced, without overall improvement but convincing us that in future experiments it might be worth trying feature engineering again especially if more training data were available 
Overall we experimented with several approaches with the following aspects of the systems changing between batches, that is being absent or present:
training on BioAsq data vs  training on SQuAD
using the BioAsq snippets for context vs  using the documents from the provided URLs for context
adding or not the LAT, i e  lexical answer type, feature (see BIBREF9, BIBREF10 and an explanation in the subsection just below) 
For Yes/No questions (only) we experimented with the entailment methods 
We will discuss the performance of these models below and in Section 6  But before we do that, let us discuss a feature engineering experiment which eventually produced mixed results, but where we feel it is potentially useful in future experiments 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative)
During error analysis we found that for some cases, answer being returned by the model is far away from what it is being asked in the Question 
Example: (from BIBREF8)
Question: Hy's law measures failure of which organ?
Actual Answer: ‘Liver’ 
The answer returned by one of our models was ‘alanine aminotransferase’, which is an enzyme  The model returns an enzyme, when the question asked for the organ name  To address this type of errors, we decided to try the concepts of ‘Lexical Answer Type’ (LAT) and Focus Word, which was used in IBM Watson, see BIBREF11 for overview; BIBREF10 for technical details, and BIBREF9 for details on question analysis  In an example given in the last source we read:
POETS & POETRY: He was a bank clerk in the Yukon before he published "Songs of a Sourdough" in 1907 
The focus is the part of the question that is a reference to the answer  In the example above, the focus is "he" 
LATs are terms in the question that indicate what type of entity is being asked for 
The headword of the focus is generally a LAT, but questions often contain additional LATs, and in the Jeopardy! domain, categories are an additional source of LATs 
(   ) In the example, LATs are "he", "clerk", and "poet" 
For example in the question "Which plant does oleuropein originate from?" (BIBREF8)  The LAT here is ‘plant’  For the BioAsq task we did not need to explicitly distinguish between the focus and the LAT concepts  In this example, the expectation is that answer returned by the model is a plant  Thus it is conceivable that the cosine distance between contextual embedding of word 'plant' in the question and contextual embedding for the answer present in the paragraph(context) is comparatively low  As a result model learns to adjust its weights during training phase and returns answers with low cosine distance with the LAT 
We used Stanford CoreNLP BIBREF12 library to write rules for extracting lexical answer type present in the question, both 'parts of speech'(POS) and dependency parsing functionality was used  We incorporated the Lexical Answer Type into one of our systems, UNCC_QA1 in Batch 4  This system underperformed our system FACTOIDS by about 3% in the MRR measure, but corrected errors such as in the example above 
Our Systems and Their Performance on Factoid Questions ::: LAT Feature considered and its impact (slightly negative) ::: Assumptions and rules for deriving lexical answer type 
There are different question types: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  Question words are identified through parts of speech tags: 'WDT', 'WRB' ,'WP'  We assumed that LAT is a ‘Noun’ and follows the question word  Often it was also a subject (nsubj)  This process is illustrated in Fig FIGREF15 
LAT computation was governed by a few simple rules, e g  when a question has multiple words that are 'Subjects’ (and ‘Noun’), a word that is in proximity to the question word is considered as ‘LAT’  These rules are different for each "Wh" word 
Namely, when the word immediately following the question word is a Noun, window size is set to ‘3’  The window size ‘3’ means we iterate through the next ‘3’ words to check if any of the word is both Noun and Subject, If so, such word is considered the ‘LAT’; else the word that is present very next to the question word is considered as the ‘LAT’ 
For questions with words ‘Which’ , ‘What’, ‘When’; a Noun immediately following the question word is very often the LAT, e g  'enzyme' in Which enzyme is targeted by Evolocumab?  When the word immediately following the question word is not a Noun, e g  in What is the function of the protein Magt1? the window size is set to ‘5’, and we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject  If present, the word is considered as the ‘LAT’; else, the Noun in close proximity to the question word and following it is returned as the ‘LAT’ 
For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself  For the word ‘How', e g  in How many selenoproteins are encoded in the human genome?, we look at the adjective and if we find one, we take it to be the LAT, otherwise the word 'How' is considered as the ‘LAT’ 
Perhaps because of using only very simple rules, the accuracy for ‘LAT’ derivation is 75%; that is, in the remaining 25% of the cases the LAT word is identified incorrectly  Worth noting is that the overall performance the system that used LATs was slightly inferior to the system without LATs, but the types of errors changed  The training used BioBERT with the LAT feature as part of the input string  The errors it introduces usually involve finding the wrong element of the correct type e g  wrong enzyme when two similar enzymes are described in the text, or 'neuron' when asked about a type of cell with a certain function, when the answer calls for a different cell category, adipocytes, and both are mentioned in the text  We feel with more data and additional tuning or perhaps using an ensemble model, we might be able to keep the correct answers, and improve the results on the confusing examples like the one mentioned above  Therefore if we improve our ‘LAT’ derivation logic, or have larger datasets, then perhaps the neural network techniques they will yield better results 
Our Systems and Their Performance on Factoid Questions ::: Impact of Training using BioAsq data (slightly negative)
Training on BioAsq data in our entry in Batch 1 and Batch 2 under the name QA1 showed it might lead to overfitting  This happened both with (Batch 2) and without (Batch 1) hyperparameters tuning: abysmal 18% MRR in Batch 1, and slighly better one, 40% in Batch 2 (although in Batch 2 it was overall the second best result in MRR but 16% lower than the highest score) 
In Batch 3 (only), our UNCC_QA3 system was fine tuned on BioAsq and SQuAD 2 0 BIBREF7, and for data preprocessing Context paragraph is generated from relevant snippets provided in the test data  This system underperformed, by about 2% in MRR, our other entry UNCC_QA1, which was also an overall category winner for this batch  The latter was also trained on SQuAD, but not on BioAsq  We suspect that the reason could be the simplistic nature of the find() function described in Section 3 1  So, this could be an area where a better algorithm for finding the best occurrence of an entity could improve performance 
Our Systems and Their Performance on Factoid Questions ::: Impact of Using Context from URLs (negative)
In some experiments, for context in testing, we used documents for which URL pointers are provided in BioAsq  However, our system UNCC_QA3 underperformed our other system tested only on the provided snippets 
In Batch 5 the underperformance was about 6% of MRR, compared to our best system UNCC_QA1, and by 9% to the top performer 
Performance on Yes/No and List questions
Our work focused on Factoid questions  But we also have done experiments on List-type and Yes/No questions 
Performance on Yes/No and List questions ::: Entailment improves Yes/No accuracy
We started by answering always YES (in batch 2 and 3) to get the baseline performance  For batch 4 we used entailment  Our algorithm was very simple: Given a question we iterate through the candidate sentences and try to find any candidate sentence is contradicting the question (with confidence over 50%), if so 'No' is returned as answer, else 'Yes' is returned  In batch 4 this strategy produced better than the BioAsq baseline performance, and compared to our other systems, the use of entailment increased the performance by about 13% (macro F1 score)  We used 'AllenNlp' BIBREF13 entailment library to find entailment of the candidate sentences with question 
Performance on Yes/No and List questions ::: For List-type the URLs have negative impact
Overall, we followed the similar strategy that's been followed for Factoid Question Answering task  We started our experiment with batch 2, where we submitted 20 best answers (with context from snippets)  Starting with batch 3, we performed post processing: once models generate answer predictions (n-best predictions), we do post-processing on the predicted answers  In test batch 4, our system (called FACTOIDS) achieved highest recall score of ‘0 7033’ but low precision of 0 1119, leaving open the question of how could we have better balanced the two measures 
In the post-processing phase, we take the top ‘20’ (batch 3) and top 5 (batch 4 and 5), predicted answers, tokenize them using common separators: 'comma' , 'and', 'also', 'as well as'  Tokens with characters count more than ‘100’ are eliminated and rest of the tokens are added to the list of possible answers  BioASQ evaluation mechanism does not consider snippets with more than ‘100’ characters as a valid answer  Considering lengthy snippets in to the list of answers would reduce the mean precision score  As a final step, duplicate snippets in the answer pool are removed  For example, consider these top 3 answers predicted by the system (before post-processing):
{
"text": "dendritic cells",
"probability": 0 7554540733426441,
"start_logit": 8 466046333312988,
"end_logit": 9 536355018615723
},
{
"text": "neutrophils, macrophages and
distinct subtypes of dendritic cells",
"probability": 0 13806867348304214,
"start_logit": 6 766478538513184,
"end_logit": 9 536355018615723
},
{
"text": "macrophages and distinct subtypes of dendritic",
"probability": 0 013973475271178242,
"start_logit": 6 766478538513184,
"end_logit": 7 24576473236084
},
After execution of post-processing heuristics, the list of answers returned is as follows:
["dendritic cells"],
["neutrophils"],
["macrophages"],
["distinct subtypes of dendritic cells"]
Summary of our results
The tables below summarize all our results  They show that the performance of our systems was mixed  The simple architectures and algorithm we used worked very well only in Batch 3  However, we feel we can built a better system based on this experience  In particular we observed both the value of contextual embeddings and of feature engineering (LAT), however we failed to combine them properly 
Summary of our results ::: Factoid questions ::: Systems used in Batch 5 experiments
System description for ‘UNCC_QA1’: The system was finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘QA1’ : ‘LAT’ feature was added and finetuned with SQuAD 2 0  For data preprocessing Context / paragraph was generated from relevant snippets provided in the test data 
System description for ‘UNCC_QA3’ : Fine tuning process is same as it is done for the system ‘UNCC_QA1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated from the relevant documents for which URLS are included in the test data 
Summary of our results ::: List Questions
For List-type questions, although post processing helped in the later batches, we never managed to obtain competitive precision, although our recall was good 
Summary of our results ::: Yes/No questions
The only thing worth remembering from our performance is that using entailment can have a measurable impact (at least with respect to a weak baseline)  The results (weak) are in Table 3 
Discussion, Future Experiments, and Conclusions ::: Summary:
In contrast to 2018, when we submitted BIBREF2 to BioASQ a system based on extractive summarization (and scored very high in the ideal answer category), this year we mainly targeted factoid question answering task and focused on experimenting with BioBERT  After these experiments we see the promise of BioBERT in QA tasks, but we also see its limitations  The latter we tried to address with mixed results using feature engineering  Overall these experiments allowed us to secure a best and a second best score in different test batches  Along with Factoid-type question, we also tried ‘Yes/No’ and ‘List’-type questions, and did reasonably well with our very simple approach 
For Yes/No the moral worth remembering is that reasoning has a potential to influence results, as evidenced by our adding the AllenNLP entailment BIBREF13 system increased its performance 
All our data and software is available at Github, in the previously referenced URL (end of Section 2) 
Discussion, Future Experiments, and Conclusions ::: Future experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding this word piece embedding directly to the BioBERT (as we did in our above experiments) would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 
We also see potential for incorporating domain specific inference into the task e g  using the MedNLI dataset BIBREF14  For all types of experiments it might be worth exploring clinical BERT embeddings BIBREF15, explicitly incorporating domain knowledge (e g  BIBREF16) and possibly deeper discourse representations (e g  BIBREF17) 
APPENDIX
In this appendix we provide additional details about the implementations 
APPENDIX ::: Systems and their descriptions:
We used several variants of our systems when experimenting with the BioASQ problems  In retrospect, it would be much easier to understand the changes if we adopted some mnemonic conventions in naming the systems  So, we apologize for the names that do not reflect the modifications, and necessitate this list 
APPENDIX ::: Systems and their descriptions: ::: Factoid Type Question Answering:
We preprocessed the test data to convert test data to BioBERT format, We generated Context/paragraph by either aggregating relevant snippets provided or by aggregating documents for which URLS are provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: System description for QA1:
We generated Context/paragraph by aggregating relevant snippets available in the test data and mapped it against the question text and question id  We ignored the content present in the documents (document URLS were provided in the original test data)  The model is finetuned with BioASQ data 
data preprocessing is done in the same way as it is done for test batch-1  Model fine tuned on BioASQ data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA_1:
System is finetuned on the SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
‘LAT’/ Focus word feature added and fine tuned with SQuAD 2 0 [reference]  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA3:
System is finetuned on the SQuAD 2 0 [reference] and BioASQ dataset[] For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
Fine tuning process is same as it is done for the system ‘UNCC_QA_1’ in test batch-5  Difference is during data preprocessing, Context/paragraph is generated form from the relevant documents for which URLS are included in the test data 
APPENDIX ::: Systems and their descriptions: ::: System description for UNCC_QA2:
Fine tuning process is same as for ‘UNCC_QA_1 ’  Difference is Context/paragraph is generated form from the relevant documents for which URLS are included in the test data  System ‘UNCC_QA_1’ got the highest ‘MRR’ score in the 3rd test batch set 
APPENDIX ::: Systems and their descriptions: ::: System description for FACTOIDS:
The System is finetuned on the SQuAD 2 0  For data preprocessing Context / paragraph is generated from relevant snippets provided in the test data 
APPENDIX ::: Systems and their descriptions: ::: List Type Questions:
We attempted List type questions starting from test batch ‘2’  Used similar approach that's been followed for Factoid Question answering task  For all the test batch sets, in the data pre processing phase Context/ paragraph is generated either by aggregating relevant snippets or by aggregating documents(URLS) provided in the BioASQ test data 
For test batch-2, model (System: QA1) is finetuned on BioASQ data and submitted top ‘20’ answers predicted by the model as the list of answers  system ‘QA1’ achieved low F-Measure score:‘0 0786’ in the second test batch  In the further test batches for List type questions, we finetuned the model on Squad data set [reference], implemented post processing techniques (refer section 5 2) and achieved a better F-measure score: ‘0 2862’ in the final test batch set 
In test batch-3 (Systems : ‘QA1’/’’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’) top 20 answers returned by the model is sent for post processing and in test batch 4 and 5 only top 5 answers are sent for post processing  System UNCC_QA2(in batch 3) for List type question answering, Context is generated from documents for which URLS are provided in the BioASQ test data  for the rest of the systems (in test batch-3) for List Type question answering task snippets present in the BioaSQ test data are used to generate context 
In test batch-4 (System : ‘FACTOIDS’/’UNCC_QA_1’/’UNCC_QA3’’) top 5 answers returned by the model is sent for post processing  In case of system ‘FACTOIDS’ snippets in the test data were used to generate context  for systems ’UNCC_QA_1’ and ’UNCC_QA3’ context is generated from the documents for which URLS are provided in the BioASQ test data 
In test batch-5 ( Systems: ‘QA1’/’UNCC_QA_1’/’UNCC_QA3’/’UNCC_QA2’ ) our approach is the same as that of test batch-4 where top 5 answers returned by the model is sent for post processing  for all the systems (in test batch-5) context is generated from the snippets provided in the BioASQ test data 
APPENDIX ::: Systems and their descriptions: ::: Yes/No Type Questions:
For the first 3 test batches, We have submitted answer ‘Yes’ to all the questions  Later, we employed ‘Sentence Entailment’ techniques(refer section 6 0) for the fourth and fifth test batch sets  Our Systems with ‘Sentence Entailment’ approach (for ‘Yes’/ ‘No’ question answering): ‘UNCC_QA_1’(test batch-4), UNCC_QA3(test batch-5) 
APPENDIX ::: Additional details for Yes/No Type Questions
We used Textual Entailment in Batch 4 and 5 for ‘Yes’/‘No’ question type  The algorithm was very simple: Given a question we iterate through the candidate sentences, and look for any candidate sentences contradicting the question  If we find one 'No' is returned as answer, else 'Yes' is returned  (The confidence for contradiction was set at 50%) We used AllenNLP BIBREF13 entailment library to find entailment of the candidate sentences with question 
Flow Chart for Yes/No Question answer processing is shown in Fig FIGREF51
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions
There are different question types, and we distinguished them based on the question words: ‘Which’, ‘What’, ‘When’, ‘How’ etc  Each type of question is being handled differently and there are commonalities among the rules written for different question types  How are question words identified? question words have parts of speech(POS): 'WDT', 'WRB', 'WP' 
Assumptions:
1) Lexical answer type (‘LAT’) or focus word is of type Noun and follows the question word 
2) The LAT word is a Subject  (This clearly not always true, but we used a very simple method)  Note: ‘StanfordNLP’ dependency parsing tag for identifying subject is 'nsubj' or 'nsubjpass' 
3) When a question has multiple words that are of type Subject (and Noun), a word that is in proximity to the question word is considered as ‘LAT’ 
4) For questions with question words: ‘When’, ‘Who’, ‘Why’, the ’LAT’ is a question word itself that is, ‘When’, ‘Who’, ‘Why’ respectively 
Rules and logic flow to traverse a question: The three cases below describe the logic flow of finding LATs  The figures show the grammatical structures used for this purpose 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-1:
Question with question word ‘How’ 
For questions with question word 'How', the adjective that follows the question word is considered as ‘LAT’ (need not follow immediately)  If an adjective is absent, word 'How' is considered as ‘LAT’  When there are multiple words that are adjectives, a word in close proximity to the question word and follows it is returned as ‘LAT’  Note: The part of speech tag to identify adjectives is 'JJ'  For Other possible question words like ‘whose’  ‘LAT’/Focus word is question words itself 
Example Question: How many selenoproteins are encoded in the human genome?
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-2:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; a 'Noun' immediately following the question word 
Example Question: Which enzyme is targeted by Evolocumab?
Here, Focus word/LAT is ‘enzyme’ which is both Noun and Subject and immediately follows the question word 
When the word immediately following the question word is a noun, the window size is set to ‘3’  This size ‘3’ means that we iterate through the next ‘3’ words (if present) to check if any of the word is both 'Noun' and 'Subject', If so, the word is considered as ‘LAT’/Focus Word  Else the word that is present very next to the question word is considered as ‘LAT’ 
APPENDIX ::: Assumptions, rules and logic flow for deriving Lexical Answer Types from questions ::: Case-3:
Questions with question words ‘Which’ , ‘What’ and all other possible question words; word immediately following the question word is not a 'Noun' 
Example Question: What is the function of the protein Magt1?
Here, Focus word/LAT is ‘function ’ which is both Noun and Subject and does not immediately follow the question word 
When the very next word following the question word is not a Noun, window size is set to ‘5’  Window size ‘5’ corresponds that we iterate through the next ‘5’ words (if present) and search for the word that is both Noun and Subject  If present, the word is considered as ‘LAT’  Else, the 'Noun' close proximity to the question word and follows it is returned as ‘LAT’ 
Ad we mentioned earlier, the accuracy for ‘LAT’ derivation is 75 percent  But clearly the simple logic described above can be improved, as shown in BIBREF9, BIBREF10  Whether this in turn produces improvements in this particular task is an open question 
APPENDIX ::: Proposing Future Experiments
In the current model, we have a shallow neural network with a softmax layer for predicting answer span  Shallow networks however are not good at generalizations  In our future experiments we would like to create dense question answering neural network with a softmax layer for predicting answer span  The main idea is to get contextual word embedding for the words present in the question and paragraph (Context) and feed the contextual word embeddings retrieved from the last layer of BioBERT to the dense question answering network  The mentioned dense layered question answering Neural network need to be tuned for finding right hyper parameters  An example of such architecture is shown in Fig FIGREF30 
In another experiment we would like to only feed contextual word embeddings for Focus word/ ‘LAT’, paragraph/ Context as input to the question answering neural network  In this experiment we would neglect all embeddings for the question text except that of Focus word/ ‘LAT’  Our assumption and idea for considering focus word and neglecting remaining words in the question is that during training phase it would make more precise for the model to identify the focus of the question and map answers against the question’s focus  To validate our assumption, we would like to take sample question answering data and find the cosine distance between contextual embedding of Focus word and that of the actual answer and verify if the cosine distance is comparatively low in most of the cases 
In one more experiment, we would like to add a better version of ‘LAT’ contextual word embedding as a feature, along with the actual contextual word embeddings for question text, and Context and feed them as input to the dense question answering neural network  By this experiment, we would like to find if ‘LAT’ feature is improving overall answer prediction accuracy  Adding ‘LAT’ feature this way instead of feeding Focus word’s word piece embedding directly (as we did in our above experiments) to the BioBERT would not downgrade the quality of contextual word embeddings generated form ‘BioBERT'  Quality contextual word embeddings would lead to efficient transfer learning and chances are that it would improve the model's answer prediction accuracy 
Context 2: Introduction
In this publication, we present Sentence-BERT (SBERT), a modification of the BERT network using siamese and triplet networks that is able to derive semantically meaningful sentence embeddings  This enables BERT to be used for certain new tasks, which up-to-now were not applicable for BERT  These tasks include large-scale semantic similarity comparison, clustering, and information retrieval via semantic search 
BERT set new state-of-the-art performance on various sentence classification and sentence-pair regression tasks  BERT uses a cross-encoder: Two sentences are passed to the transformer network and the target value is predicted  However, this setup is unsuitable for various pair regression tasks due to too many possible combinations  Finding in a collection of $n=10\,000$ sentences the pair with the highest similarity requires with BERT $n\cdot (n-1)/2=49\,995\,000$ inference computations  On a modern V100 GPU, this requires about 65 hours  Similar, finding which of the over 40 million existent questions of Quora is the most similar for a new question could be modeled as a pair-wise comparison with BERT, however, answering a single query would require over 50 hours 
A common method to address clustering and semantic search is to map each sentence to a vector space such that semantically similar sentences are close  Researchers have started to input individual sentences into BERT and to derive fixed-size sentence embeddings  The most commonly used approach is to average the BERT output layer (known as BERT embeddings) or by using the output of the first token (the [CLS] token)  As we will show, this common practice yields rather bad sentence embeddings, often worse than averaging GloVe embeddings BIBREF2 
To alleviate this issue, we developed SBERT  The siamese network architecture enables that fixed-sized vectors for input sentences can be derived  Using a similarity measure like cosine-similarity or Manhatten / Euclidean distance, semantically similar sentences can be found  These similarity measures can be performed extremely efficient on modern hardware, allowing SBERT to be used for semantic similarity search as well as for clustering  The complexity for finding the most similar sentence pair in a collection of 10,000 sentences is reduced from 65 hours with BERT to the computation of 10,000 sentence embeddings (5 seconds with SBERT) and computing cosine-similarity (0 01 seconds)  By using optimized index structures, finding the most similar Quora question can be reduced from 50 hours to a few milliseconds BIBREF3 
We fine-tune SBERT on NLI data, which creates sentence embeddings that significantly outperform other state-of-the-art sentence embedding methods like InferSent BIBREF4 and Universal Sentence Encoder BIBREF5  On seven Semantic Textual Similarity (STS) tasks, SBERT achieves an improvement of 11 7 points compared to InferSent and 5 5 points compared to Universal Sentence Encoder  On SentEval BIBREF6, an evaluation toolkit for sentence embeddings, we achieve an improvement of 2 1 and 2 6 points, respectively 
SBERT can be adapted to a specific task  It sets new state-of-the-art performance on a challenging argument similarity dataset BIBREF7 and on a triplet dataset to distinguish sentences from different sections of a Wikipedia article BIBREF8 
The paper is structured in the following way: Section SECREF3 presents SBERT, section SECREF4 evaluates SBERT on common STS tasks and on the challenging Argument Facet Similarity (AFS) corpus BIBREF7  Section SECREF5 evaluates SBERT on SentEval  In section SECREF6, we perform an ablation study to test some design aspect of SBERT  In section SECREF7, we compare the computational efficiency of SBERT sentence embeddings in contrast to other state-of-the-art sentence embedding methods 
Related Work
We first introduce BERT, then, we discuss state-of-the-art sentence embedding methods 
BERT BIBREF0 is a pre-trained transformer network BIBREF9, which set for various NLP tasks new state-of-the-art results, including question answering, sentence classification, and sentence-pair regression  The input for BERT for sentence-pair regression consists of the two sentences, separated by a special [SEP] token  Multi-head attention over 12 (base-model) or 24 layers (large-model) is applied and the output is passed to a simple regression function to derive the final label  Using this setup, BERT set a new state-of-the-art performance on the Semantic Textual Semilarity (STS) benchmark BIBREF10  RoBERTa BIBREF1 showed, that the performance of BERT can further improved by small adaptations to the pre-training process  We also tested XLNet BIBREF11, but it led in general to worse results than BERT 
A large disadvantage of the BERT network structure is that no independent sentence embeddings are computed, which makes it difficult to derive sentence embeddings from BERT  To bypass this limitations, researchers passed single sentences through BERT and then derive a fixed sized vector by either averaging the outputs (similar to average word embeddings) or by using the output of the special CLS token (for example: bertsentenceembeddings1,bertsentenceembeddings2,bertsentenceembeddings3)  These two options are also provided by the popular bert-as-a-service-repository  Up to our knowledge, there is so far no evaluation if these methods lead to useful sentence embeddings 
Sentence embeddings are a well studied area with dozens of proposed methods  Skip-Thought BIBREF12 trains an encoder-decoder architecture to predict the surrounding sentences  InferSent BIBREF4 uses labeled data of the Stanford Natural Language Inference dataset BIBREF13 and the Multi-Genre NLI dataset BIBREF14 to train a siamese BiLSTM network with max-pooling over the output  Conneau et al  showed, that InferSent consistently outperforms unsupervised methods like SkipThought  Universal Sentence Encoder BIBREF5 trains a transformer network and augments unsupervised learning with training on SNLI  hill-etal-2016-learning showed, that the task on which sentence embeddings are trained significantly impacts their quality  Previous work BIBREF4, BIBREF5 found that the SNLI datasets are suitable for training sentence embeddings  yang-2018-learning presented a method to train on conversations from Reddit using siamese DAN and siamese transformer networks, which yielded good results on the STS benchmark dataset 
polyencoders addresses the run-time overhead of the cross-encoder from BERT and present a method (poly-encoders) to compute a score between $m$ context vectors and pre-computed candidate embeddings using attention  This idea works for finding the highest scoring sentence in a larger collection  However, poly-encoders have the drawback that the score function is not symmetric and the computational overhead is too large for use-cases like clustering, which would require $O(n^2)$ score computations 
Previous neural sentence embedding methods started the training from a random initialization  In this publication, we use the pre-trained BERT and RoBERTa network and only fine-tune it to yield useful sentence embeddings  This reduces significantly the needed training time: SBERT can be tuned in less than 20 minutes, while yielding better results than comparable sentence embedding methods 
Model
SBERT adds a pooling operation to the output of BERT / RoBERTa to derive a fixed sized sentence embedding  We experiment with three pooling strategies: Using the output of the CLS-token, computing the mean of all output vectors (MEAN-strategy), and computing a max-over-time of the output vectors (MAX-strategy)  The default configuration is MEAN 
In order to fine-tune BERT / RoBERTa, we create siamese and triplet networks BIBREF15 to update the weights such that the produced sentence embeddings are semantically meaningful and can be compared with cosine-similarity 
The network structure depends on the available training data  We experiment with the following structures and objective functions 
Classification Objective Function  We concatenate the sentence embeddings $u$ and $v$ with the element-wise difference $|u-v|$ and multiply it with the trainable weight $W_t \in \mathbb {R}^{3n \times k}$:
where $n$ is the dimension of the sentence embeddings and $k$ the number of labels  We optimize cross-entropy loss  This structure is depicted in Figure FIGREF4 
Regression Objective Function  The cosine-similarity between the two sentence embeddings $u$ and $v$ is computed (Figure FIGREF5)  We use mean-squared-error loss as the objective function 
Triplet Objective Function  Given an anchor sentence $a$, a positive sentence $p$, and a negative sentence $n$, triplet loss tunes the network such that the distance between $a$ and $p$ is smaller than the distance between $a$ and $n$  Mathematically, we minimize the following loss function:
with $s_x$ the sentence embedding for $a$/$n$/$p$, $||\cdot ||$ a distance metric and margin $\epsilon $  Margin $\epsilon $ ensures that $s_p$ is at least $\epsilon $ closer to $s_a$ than $s_n$  As metric we use Euclidean distance and we set $\epsilon =1$ in our experiments 
Model ::: Training Details
We train SBERT on the combination of the SNLI BIBREF13 and the Multi-Genre NLI BIBREF14 dataset  The SNLI is a collection of 570,000 sentence pairs annotated with the labels contradiction, eintailment, and neutral  MultiNLI contains 430,000 sentence pairs and covers a range of genres of spoken and written text  We fine-tune SBERT with a 3-way softmax-classifier objective function for one epoch  We used a batch-size of 16, Adam optimizer with learning rate $2\mathrm {e}{-5}$, and a linear learning rate warm-up over 10% of the training data  Our default pooling strategy is MEAN 
Evaluation - Semantic Textual Similarity
We evaluate the performance of SBERT for common Semantic Textual Similarity (STS) tasks  State-of-the-art methods often learn a (complex) regression function that maps sentence embeddings to a similarity score  However, these regression functions work pair-wise and due to the combinatorial explosion those are often not scalable if the collection of sentences reaches a certain size  Instead, we always use cosine-similarity to compare the similarity between two sentence embeddings  We ran our experiments also with negative Manhatten and negative Euclidean distances as similarity measures, but the results for all approaches remained roughly the same 
Evaluation - Semantic Textual Similarity ::: Unsupervised STS
We evaluate the performance of SBERT for STS without using any STS specific training data  We use the STS tasks 2012 - 2016 BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF20, the STS benchmark BIBREF10, and the SICK-Relatedness dataset BIBREF21  These datasets provide labels between 0 and 5 on the semantic relatedness of sentence pairs  We showed in BIBREF22 that Pearson correlation is badly suited for STS  Instead, we compute the Spearman's rank correlation between the cosine-similarity of the sentence embeddings and the gold labels  The setup for the other sentence embedding methods is equivalent, the similarity is computed by cosine-similarity  The results are depicted in Table TABREF6 
The results shows that directly using the output of BERT leads to rather poor performances  Averaging the BERT embeddings achieves an average correlation of only 54 81, and using the CLS-token output only achieves an average correlation of 29 19  Both are worse than computing average GloVe embeddings 
Using the described siamese network structure and fine-tuning mechanism substantially improves the correlation, outperforming both InferSent and Universal Sentence Encoder substantially  The only dataset where SBERT performs worse than Universal Sentence Encoder is SICK-R  Universal Sentence Encoder was trained on various datasets, including news, question-answer pages and discussion forums, which appears to be more suitable to the data of SICK-R  In contrast, SBERT was pre-trained only on Wikipedia (via BERT) and on NLI data 
While RoBERTa was able to improve the performance for several supervised tasks, we only observe minor difference between SBERT and SRoBERTa for generating sentence embeddings 
Evaluation - Semantic Textual Similarity ::: Supervised STS
The STS benchmark (STSb) BIBREF10 provides is a popular dataset to evaluate supervised STS systems  The data includes 8,628 sentence pairs from the three categories captions, news, and forums  It is divided into train (5,749), dev (1,500) and test (1,379)  BERT set a new state-of-the-art performance on this dataset by passing both sentences to the network and using a simple regression method for the output 
We use the training set to fine-tune SBERT using the regression objective function  At prediction time, we compute the cosine-similarity between the sentence embeddings  All systems are trained with 10 random seeds to counter variances BIBREF23 
The results are depicted in Table TABREF10  We experimented with two setups: Only training on STSb, and first training on NLI, then training on STSb  We observe that the later strategy leads to a slight improvement of 1-2 points  This two-step approach had an especially large impact for the BERT cross-encoder, which improved the performance by 3-4 points  We do not observe a significant difference between BERT and RoBERTa 
Evaluation - Semantic Textual Similarity ::: Argument Facet Similarity
We evaluate SBERT on the Argument Facet Similarity (AFS) corpus by MisraEW16  The AFS corpus annotated 6,000 sentential argument pairs from social media dialogs on three controversial topics: gun control, gay marriage, and death penalty  The data was annotated on a scale from 0 (“different topic") to 5 (“completely equivalent")  The similarity notion in the AFS corpus is fairly different to the similarity notion in the STS datasets from SemEval  STS data is usually descriptive, while AFS data are argumentative excerpts from dialogs  To be considered similar, arguments must not only make similar claims, but also provide a similar reasoning

ground_truth: 0.7033

0.7033

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What other sentence embeddings methods are evaluated?
Context 2:  Further, the lexical gap between the sentences in AFS is much larger  Hence, simple unsupervised methods as well as state-of-the-art STS systems perform badly on this dataset BIBREF24 
We evaluate SBERT on this dataset in two scenarios: 1) As proposed by Misra et al , we evaluate SBERT using 10-fold cross-validation  A draw-back of this evaluation setup is that it is not clear how well approaches generalize to different topics  Hence, 2) we evaluate SBERT in a cross-topic setup  Two topics serve for training and the approach is evaluated on the left-out topic  We repeat this for all three topics and average the results 
SBERT is fine-tuned using the Regression Objective Function  The similarity score is computed using cosine-similarity based on the sentence embeddings  We also provide the Pearson correlation $r$ to make the results comparable to Misra et al  However, we showed BIBREF22 that Pearson correlation has some serious drawbacks and should be avoided for comparing STS systems  The results are depicted in Table TABREF12 
Unsupervised methods like tf-idf, average GloVe embeddings or InferSent perform rather badly on this dataset with low scores  Training SBERT in the 10-fold cross-validation setup gives a performance that is nearly on-par with BERT 
However, in the cross-topic evaluation, we observe a performance drop of SBERT by about 7 points Spearman correlation  To be considered similar, arguments should address the same claims and provide the same reasoning  BERT is able to use attention to compare directly both sentences (e g  word-by-word comparison), while SBERT must map individual sentences from an unseen topic to a vector space such that arguments with similar claims and reasons are close  This is a much more challenging task, which appears to require more than just two topics for training to work on-par with BERT 
Evaluation - Semantic Textual Similarity ::: Wikipedia Sections Distinction
ein-dor-etal-2018-learning use Wikipedia to create a thematically fine-grained train, dev and test set for sentence embeddings methods  Wikipedia articles are separated into distinct sections focusing on certain aspects  Dor et al  assume that sentences in the same section are thematically closer than sentences in different sections  They use this to create a large dataset of weakly labeled sentence triplets: The anchor and the positive example come from the same section, while the negative example comes from a different section of the same article  For example, from the Alice Arnold article: Anchor: Arnold joined the BBC Radio Drama Company in 1988 , positive: Arnold gained media attention in May 2012 , negative: Balding and Arnold are keen amateur golfers 
We use the dataset from Dor et al  We use the Triplet Objective, train SBERT for one epoch on the about 1 8 Million training triplets and evaluate it on the 222,957 test triplets  Test triplets are from a distinct set of Wikipedia articles  As evaluation metric, we use accuracy: Is the positive example closer to the anchor than the negative example?
Results are presented in Table TABREF14  Dor et al  fine-tuned a BiLSTM architecture with triplet loss to derive sentence embeddings for this dataset  As the table shows, SBERT clearly outperforms the BiLSTM approach by Dor et al 
Evaluation - SentEval
SentEval BIBREF6 is a popular toolkit to evaluate the quality of sentence embeddings  Sentence embeddings are used as features for a logistic regression classifier  The logistic regression classifier is trained on various tasks in a 10-fold cross-validation setup and the prediction accuracy is computed for the test-fold 
The purpose of SBERT sentence embeddings are not to be used for transfer learning for other tasks  Here, we think fine-tuning BERT as described by devlin2018bert for new tasks is the more suitable method, as it updates all layers of the BERT network  However, SentEval can still give an impression on the quality of our sentence embeddings for various tasks 
We compare the SBERT sentence embeddings to other sentence embeddings methods on the following seven SentEval transfer tasks:
MR: Sentiment prediction for movie reviews snippets on a five start scale BIBREF25 
CR: Sentiment prediction of customer product reviews BIBREF26 
SUBJ: Subjectivity prediction of sentences from movie reviews and plot summaries BIBREF27 
MPQA: Phrase level opinion polarity classification from newswire BIBREF28 
SST: Stanford Sentiment Treebank with binary labels BIBREF29 
TREC: Fine grained question-type classification from TREC BIBREF30 
MRPC: Microsoft Research Paraphrase Corpus from parallel news sources BIBREF31 
The results can be found in Table TABREF15  SBERT is able to achieve the best performance in 5 out of 7 tasks  The average performance increases by about 2 percentage points compared to InferSent as well as the Universal Sentence Encoder  Even though transfer learning is not the purpose of SBERT, it outperforms other state-of-the-art sentence embeddings methods on this task 
It appears that the sentence embeddings from SBERT capture well sentiment information: We observe large improvements for all sentiment tasks (MR, CR, and SST) from SentEval in comparison to InferSent and Universal Sentence Encoder 
The only dataset where SBERT is significantly worse than Universal Sentence Encoder is the TREC dataset  Universal Sentence Encoder was pre-trained on question-answering data, which appears to be beneficial for the question-type classification task of the TREC dataset 
Average BERT embeddings or using the CLS-token output from a BERT network achieved bad results for various STS tasks (Table TABREF6), worse than average GloVe embeddings  However, for SentEval, average BERT embeddings and the BERT CLS-token output achieves decent results (Table TABREF15), outperforming average GloVe embeddings  The reason for this are the different setups  For the STS tasks, we used cosine-similarity to estimate the similarities between sentence embeddings  Cosine-similarity treats all dimensions equally  In contrast, SentEval fits a logistic regression classifier to the sentence embeddings  This allows that certain dimensions can have higher or lower impact on the classification result 
We conclude that average BERT embeddings / CLS-token output from BERT return sentence embeddings that are infeasible to be used with cosine-similarity or with Manhatten / Euclidean distance  For transfer learning, they yield slightly worse results than InferSent or Universal Sentence Encoder  However, using the described fine-tuning setup with a siamese network structure on NLI datasets yields sentence embeddings that achieve a new state-of-the-art for the SentEval toolkit 
Ablation Study
We have demonstrated strong empirical results for the quality of SBERT sentence embeddings  In this section, we perform an ablation study of different aspects of SBERT in order to get a better understanding of their relative importance 
We evaluated different pooling strategies (MEAN, MAX, and CLS)  For the classification objective function, we evaluate different concatenation methods  For each possible configuration, we train SBERT with 10 different random seeds and average the performances 
The objective function (classification vs  regression) depends on the annotated dataset  For the classification objective function, we train SBERT-base on the SNLI and the Multi-NLI dataset  For the regression objective function, we train on the training set of the STS benchmark dataset  Performances are measured on the development split of the STS benchmark dataset  Results are shown in Table TABREF23 
When trained with the classification objective function on NLI data, the pooling strategy has a rather minor impact  The impact of the concatenation mode is much larger  InferSent BIBREF4 and Universal Sentence Encoder BIBREF5 both use $(u, v, |u-v|, u*v)$ as input for a softmax classifier  However, in our architecture, adding the element-wise $u*v$ decreased the performance 
The most important component is the element-wise difference $|u-v|$  Note, that the concatenation mode is only relevant for training the softmax classifier  At inference, when predicting similarities for the STS benchmark dataset, only the sentence embeddings $u$ and $v$ are used in combination with cosine-similarity  The element-wise difference measures the distance between the dimensions of the two sentence embeddings, ensuring that similar pairs are closer and dissimilar pairs are further apart 
When trained with the regression objective function, we observe that the pooling strategy has a large impact  There, the MAX strategy perform significantly worse than MEAN or CLS-token strategy  This is in contrast to BIBREF4, who found it beneficial for the BiLSTM-layer of InferSent to use MAX instead of MEAN pooling 
Computational Efficiency
Sentence embeddings need potentially be computed for Millions of sentences, hence, a high computation speed is desired  In this section, we compare SBERT to average GloVe embeddings, InferSent BIBREF4, and Universal Sentence Encoder BIBREF5 
For our comparison we use the sentences from the STS benchmark BIBREF10  We compute average GloVe embeddings using a simple for-loop with python dictionary lookups and NumPy  InferSent is based on PyTorch  For Universal Sentence Encoder, we use the TensorFlow Hub version, which is based on TensorFlow  SBERT is based on PyTorch  For improved computation of sentence embeddings, we implemented a smart batching strategy: Sentences with similar lengths are grouped together and are only padded to the longest element in a mini-batch  This drastically reduces computational overhead from padding tokens 
Performances were measured on a server with Intel i7-5820K CPU @ 3 30GHz, Nvidia Tesla V100 GPU, CUDA 9 2 and cuDNN  The results are depicted in Table TABREF26 
On CPU, InferSent is about 65% faster than SBERT  This is due to the much simpler network architecture  InferSent uses a single BiLSTM layer, while BERT uses 12 stacked transformer layers  However, an advantage of transformer networks is the computational efficiency on GPUs  There, SBERT with smart batching is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder  Smart batching achieves a speed-up of 89% on CPU and 48% on GPU  Average GloVe embeddings is obviously by a large margin the fastest method to compute sentence embeddings 
Conclusion
We showed that BERT out-of-the-box maps sentences to a vector space that is rather unsuitable to be used with common similarity measures like cosine-similarity  The performance for seven STS tasks was below the performance of average GloVe embeddings 
To overcome this shortcoming, we presented Sentence-BERT (SBERT)  SBERT fine-tunes BERT in a siamese / triplet network architecture  We evaluated the quality on various common benchmarks, where it could achieve a significant improvement over state-of-the-art sentence embeddings methods  Replacing BERT with RoBERTa did not yield a significant improvement in our experiments 
SBERT is computationally efficient  On a GPU, it is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder  SBERT can be used for tasks which are computationally not feasible to be modeled with BERT  For example, clustering of 10,000 sentences with hierarchical clustering requires with BERT about 65 hours, as around 50 Million sentence combinations must be computed  With SBERT, we were able to reduce the effort to about 5 seconds 
Acknowledgments
This work has been supported by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1)  It has been co-funded by the German Federal Ministry of Education and Research (BMBF) under the promotional references 03VP02540 (ArgumenText) 

ground_truth: GloVe, BERT, Universal Sentence Encoder, TF-IDF, InferSent

Avg. GloVe embeddings, Avg. fast-text embeddings, Avg. BERT embeddings, BERT CLS-vector, InferSent - GloVe and Universal Sentence Encoder.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How does their model improve interpretability compared to softmax transformers?
Context 3: Introduction
The Transformer architecture BIBREF0 for deep neural networks has quickly risen to prominence in NLP through its efficiency and performance, leading to improvements in the state of the art of Neural Machine Translation BIBREF1, BIBREF2, as well as inspiring other powerful general-purpose models like BERT BIBREF3 and GPT-2 BIBREF4  At the heart of the Transformer lie multi-head attention mechanisms: each word is represented by multiple different weighted averages of its relevant context  As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9 
The attention distribution of each head is predicted typically using the softmax normalizing transform  As a result, all context words have non-zero attention weight  Recent work on single attention architectures suggest that using sparse normalizing transforms in attention mechanisms such as sparsemax – which can yield exactly zero probabilities for irrelevant words – may improve performance and interpretability BIBREF12, BIBREF13, BIBREF14  Qualitative analysis of attention heads BIBREF0 suggests that, depending on what phenomena they capture, heads tend to favor flatter or more peaked distributions 
Recent works have proposed sparse Transformers BIBREF10 and adaptive span Transformers BIBREF11  However, the “sparsity" of those models only limits the attention to a contiguous span of past tokens, while in this work we propose a highly adaptive Transformer model that is capable of attending to a sparse set of words that are not necessarily contiguous  Figure FIGREF1 shows the relationship of these methods with ours 
Our contributions are the following:
We introduce sparse attention into the Transformer architecture, showing that it eases interpretability and leads to slight accuracy gains 
We propose an adaptive version of sparse attention, where the shape of each attention head is learnable and can vary continuously and dynamically between the dense limit case of softmax and the sparse, piecewise-linear sparsemax case 
We make an extensive analysis of the added interpretability of these models, identifying both crisper examples of attention head behavior observed in previous work, as well as novel behaviors unraveled thanks to the sparsity and adaptivity of our proposed model 
Background ::: The Transformer
In NMT, the Transformer BIBREF0 is a sequence-to-sequence (seq2seq) model which maps an input sequence to an output sequence through hierarchical multi-head attention mechanisms, yielding a dynamic, context-dependent strategy for propagating information within and across sentences  It contrasts with previous seq2seq models, which usually rely either on costly gated recurrent operations BIBREF15, BIBREF16 or static convolutions BIBREF17 
Given $n$ query contexts and $m$ sequence items under consideration, attention mechanisms compute, for each query, a weighted representation of the items  The particular attention mechanism used in BIBREF0 is called scaled dot-product attention, and it is computed in the following way:
where $\mathbf {Q} \in \mathbb {R}^{n \times d}$ contains representations of the queries, $\mathbf {K}, \mathbf {V} \in \mathbb {R}^{m \times d}$ are the keys and values of the items attended over, and $d$ is the dimensionality of these representations  The $\mathbf {\pi }$ mapping normalizes row-wise using softmax, $\mathbf {\pi }(\mathbf {Z})_{ij} = \operatornamewithlimits{\mathsf {softmax}}(\mathbf {z}_i)_j$, where
In words, the keys are used to compute a relevance score between each item and query  Then, normalized attention weights are computed using softmax, and these are used to weight the values of each item at each query context 
However, for complex tasks, different parts of a sequence may be relevant in different ways, motivating multi-head attention in Transformers  This is simply the application of Equation DISPLAY_FORM7 in parallel $H$ times, each with a different, learned linear transformation that allows specialization:
In the Transformer, there are three separate multi-head attention mechanisms for distinct purposes:
Encoder self-attention: builds rich, layered representations of each input word, by attending on the entire input sentence 
Context attention: selects a representative weighted average of the encodings of the input words, at each time step of the decoder 
Decoder self-attention: attends over the partial output sentence fragment produced so far 
Together, these mechanisms enable the contextualized flow of information between the input sentence and the sequential decoder 
Background ::: Sparse Attention
The softmax mapping (Equation DISPLAY_FORM8) is elementwise proportional to $\exp $, therefore it can never assign a weight of exactly zero  Thus, unnecessary items are still taken into consideration to some extent  Since its output sums to one, this invariably means less weight is assigned to the relevant items, potentially harming performance and interpretability BIBREF18  This has motivated a line of research on learning networks with sparse mappings BIBREF19, BIBREF20, BIBREF21, BIBREF22  We focus on a recently-introduced flexible family of transformations, $\alpha $-entmax BIBREF23, BIBREF14, defined as:
where $\triangle ^d \lbrace \mathbf {p}\in \mathbb {R}^d:\sum _{i} p_i = 1\rbrace $ is the probability simplex, and, for $\alpha \ge 1$, $\mathsf {H}^{\textsc {T}}_\alpha $ is the Tsallis continuous family of entropies BIBREF24:
This family contains the well-known Shannon and Gini entropies, corresponding to the cases $\alpha =1$ and $\alpha =2$, respectively 
Equation DISPLAY_FORM14 involves a convex optimization subproblem  Using the definition of $\mathsf {H}^{\textsc {T}}_\alpha $, the optimality conditions may be used to derive the following form for the solution (Appendix SECREF83):
where $[\cdot ]_+$ is the positive part (ReLU) function, $\mathbf {1}$ denotes the vector of all ones, and $\tau $ – which acts like a threshold – is the Lagrange multiplier corresponding to the $\sum _i p_i=1$ constraint 
Background ::: Sparse Attention ::: Properties of @!START@$\alpha $@!END@-entmax 
The appeal of $\alpha $-entmax for attention rests on the following properties  For $\alpha =1$ (i e , when $\mathsf {H}^{\textsc {T}}_\alpha $ becomes the Shannon entropy), it exactly recovers the softmax mapping (We provide a short derivation in Appendix SECREF89 )  For all $\alpha >1$ it permits sparse solutions, in stark contrast to softmax  In particular, for $\alpha =2$, it recovers the sparsemax mapping BIBREF19, which is piecewise linear  In-between, as $\alpha $ increases, the mapping continuously gets sparser as its curvature changes 
To compute the value of $\alpha $-entmax, one must find the threshold $\tau $ such that the r h s  in Equation DISPLAY_FORM16 sums to one  BIBREF23 propose a general bisection algorithm  BIBREF14 introduce a faster, exact algorithm for $\alpha =1 5$, and enable using $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with fixed $\alpha $ within a neural network by showing that the $\alpha $-entmax Jacobian w r t  $\mathbf {z}$ for $\mathbf {p}^\star = \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ is
Our work furthers the study of $\alpha $-entmax by providing a derivation of the Jacobian w r t  the hyper-parameter $\alpha $ (Section SECREF3), thereby allowing the shape and sparsity of the mapping to be learned automatically  This is particularly appealing in the context of multi-head attention mechanisms, where we shall show in Section SECREF35 that different heads tend to learn different sparsity behaviors 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax
We now propose a novel Transformer architecture wherein we simply replace softmax with $\alpha $-entmax in the attention heads  Concretely, we replace the row normalization $\mathbf {\pi }$ in Equation DISPLAY_FORM7 by
This change leads to sparse attention weights, as long as $\alpha >1$; in particular, $\alpha =1 5$ is a sensible starting point BIBREF14 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Different @!START@$\alpha $@!END@ per head 
Unlike LSTM-based seq2seq models, where $\alpha $ can be more easily tuned by grid search, in a Transformer, there are many attention heads in multiple layers  Crucial to the power of such models, the different heads capture different linguistic phenomena, some of them isolating important words, others spreading out attention across phrases BIBREF0  This motivates using different, adaptive $\alpha $ values for each attention head, such that some heads may learn to be sparser, and others may become closer to softmax  We propose doing so by treating the $\alpha $ values as neural network parameters, optimized via stochastic gradients along with the other weights 
Adaptively Sparse Transformers with @!START@$\alpha $@!END@-entmax ::: Derivatives w r t  @!START@$\alpha $@!END@ 
In order to optimize $\alpha $ automatically via gradient methods, we must compute the Jacobian of the entmax output w r t  $\alpha $  Since entmax is defined through an optimization problem, this is non-trivial and cannot be simply handled through automatic differentiation; it falls within the domain of argmin differentiation, an active research topic in optimization BIBREF25, BIBREF26 
One of our key contributions is the derivation of a closed-form expression for this Jacobian  The next proposition provides such an expression, enabling entmax layers with adaptive $\alpha $  To the best of our knowledge, ours is the first neural network module that can automatically, continuously vary in shape away from softmax and toward sparse mappings like sparsemax 
Proposition 1 Let $\mathbf {p}^\star \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ be the solution of Equation DISPLAY_FORM14  Denote the distribution $\tilde{p}_i {(p_i^\star )^{2 - \alpha }}{ \sum _j(p_j^\star )^{2-\alpha }}$ and let $h_i -p^\star _i \log p^\star _i$  The $i$th component of the Jacobian $\mathbf {g} \frac{\partial \mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})}{\partial \alpha }$ is
proof uses implicit function differentiation and is given in Appendix SECREF10 
Proposition UNKREF22 provides the remaining missing piece needed for training adaptively sparse Transformers  In the following section, we evaluate this strategy on neural machine translation, and analyze the behavior of the learned attention heads 
Experiments
We apply our adaptively sparse Transformers on four machine translation tasks  For comparison, a natural baseline is the standard Transformer architecture using the softmax transform in its multi-head attention mechanisms  We consider two other model variants in our experiments that make use of different normalizing transformations:
1 5-entmax: a Transformer with sparse entmax attention with fixed $\alpha =1 5$ for all heads  This is a novel model, since 1 5-entmax had only been proposed for RNN-based NMT models BIBREF14, but never in Transformers, where attention modules are not just one single component of the seq2seq model but rather an integral part of all of the model components 
$\alpha $-entmax: an adaptive Transformer with sparse entmax attention with a different, learned $\alpha _{i,j}^t$ for each head 
The adaptive model has an additional scalar parameter per attention head per layer for each of the three attention mechanisms (encoder self-attention, context attention, and decoder self-attention), i e ,
and we set $\alpha _{i,j}^t = 1 + \operatornamewithlimits{\mathsf {sigmoid}}(a_{i,j}^t) \in ]1, 2[$  All or some of the $\alpha $ values can be tied if desired, but we keep them independent for analysis purposes 
Experiments ::: Datasets 
Our models were trained on 4 machine translation datasets of different training sizes:
[itemsep= 5ex,leftmargin=2ex]
IWSLT 2017 German $\rightarrow $ English BIBREF27: 200K sentence pairs 
KFTT Japanese $\rightarrow $ English BIBREF28: 300K sentence pairs 
WMT 2016 Romanian $\rightarrow $ English BIBREF29: 600K sentence pairs 
WMT 2014 English $\rightarrow $ German BIBREF30: 4 5M sentence pairs 
All of these datasets were preprocessed with byte-pair encoding BIBREF31, using joint segmentations of 32k merge operations 
Experiments ::: Training 
We follow the dimensions of the Transformer-Base model of BIBREF0: The number of layers is $L=6$ and number of heads is $H=8$ in the encoder self-attention, the context attention, and the decoder self-attention  We use a mini-batch size of 8192 tokens and warm up the learning rate linearly until 20k steps, after which it decays according to an inverse square root schedule  All models were trained until convergence of validation accuracy, and evaluation was done at each 10k steps for ro$\rightarrow $en and en$\rightarrow $de and at each 5k steps for de$\rightarrow $en and ja$\rightarrow $en  The end-to-end computational overhead of our methods, when compared to standard softmax, is relatively small; in training tokens per second, the models using $\alpha $-entmax and $1 5$-entmax are, respectively, $75\%$ and $90\%$ the speed of the softmax model 
Experiments ::: Results 
We report test set tokenized BLEU BIBREF32 results in Table TABREF27  We can see that replacing softmax by entmax does not hurt performance in any of the datasets; indeed, sparse attention Transformers tend to have slightly higher BLEU, but their sparsity leads to a better potential for analysis  In the next section, we make use of this potential by exploring the learned internal mechanics of the self-attention heads 
Analysis
We conduct an analysis for the higher-resource dataset WMT 2014 English $\rightarrow $ German of the attention in the sparse adaptive Transformer model ($\alpha $-entmax) at multiple levels: we analyze high-level statistics as well as individual head behavior  Moreover, we make a qualitative analysis of the interpretability capabilities of our models 
Analysis ::: High-Level Statistics ::: What kind of @!START@$\alpha $@!END@ values are learned?
Figure FIGREF37 shows the learning trajectories of the $\alpha $ parameters of a selected subset of heads  We generally observe a tendency for the randomly-initialized $\alpha $ parameters to decrease initially, suggesting that softmax-like behavior may be preferable while the model is still very uncertain  After around one thousand steps, some heads change direction and become sparser, perhaps as they become more confident and specialized  This shows that the initialization of $\alpha $ does not predetermine its sparsity level or the role the head will have throughout  In particular, head 8 in the encoder self-attention layer 2 first drops to around $\alpha =1 3$ before becoming one of the sparsest heads, with $\alpha \approx 2$ 
The overall distribution of $\alpha $ values at convergence can be seen in Figure FIGREF38  We can observe that the encoder self-attention blocks learn to concentrate the $\alpha $ values in two modes: a very sparse one around $\alpha \rightarrow 2$, and a dense one between softmax and 1 5-entmax   However, the decoder self and context attention only learn to distribute these parameters in a single mode  We show next that this is reflected in the average density of attention weight vectors as well 
Analysis ::: High-Level Statistics ::: Attention weight density when translating 
For any $\alpha >1$, it would still be possible for the weight matrices in Equation DISPLAY_FORM9 to learn re-scalings so as to make attention sparser or denser  To visualize the impact of adaptive $\alpha $ values, we compare the empirical attention weight density (the average number of tokens receiving non-zero attention) within each module, against sparse Transformers with fixed $\alpha =1 5$ 
Figure FIGREF40 shows that, with fixed $\alpha =1 5$, heads tend to be sparse and similarly-distributed in all three attention modules  With learned $\alpha $, there are two notable changes: (i) a prominent mode corresponding to fully dense probabilities, showing that our models learn to combine sparse and dense attention, and (ii) a distinction between the encoder self-attention – whose background distribution tends toward extreme sparsity – and the other two modules, who exhibit more uniform background distributions  This suggests that perhaps entirely sparse Transformers are suboptimal 
The fact that the decoder seems to prefer denser attention distributions might be attributed to it being auto-regressive, only having access to past tokens and not the full sentence  We speculate that it might lose too much information if it assigned weights of zero to too many tokens in the self-attention, since there are fewer tokens to attend to in the first place 
Teasing this down into separate layers, Figure FIGREF41 shows the average (sorted) density of each head for each layer  We observe that $\alpha $-entmax is able to learn different sparsity patterns at each layer, leading to more variance in individual head behavior, to clearly-identified dense and sparse heads, and overall to different tendencies compared to the fixed case of $\alpha =1 5$ 
Analysis ::: High-Level Statistics ::: Head diversity 
To measure the overall disagreement between attention heads, as a measure of head diversity, we use the following generalization of the Jensen-Shannon divergence:
where $\mathbf {p}_j$ is the vector of attention weights assigned by head $j$ to each word in the sequence, and $\mathsf {H}^\textsc {S}$ is the Shannon entropy, base-adjusted based on the dimension of $\mathbf {p}$ such that $JS \le 1$  We average this measure over the entire validation set  The higher this metric is, the more the heads are taking different roles in the model 
Figure FIGREF44 shows that both sparse Transformer variants show more diversity than the traditional softmax one  Interestingly, diversity seems to peak in the middle layers of the encoder self-attention and context attention, while this is not the case for the decoder self-attention 
The statistics shown in this section can be found for the other language pairs in Appendix SECREF8 
Analysis ::: Identifying Head Specializations
Previous work pointed out some specific roles played by different heads in the softmax Transformer model BIBREF33, BIBREF5, BIBREF9  Identifying the specialization of a head can be done by observing the type of tokens or sequences that the head often assigns most of its attention weight; this is facilitated by sparsity 
Analysis ::: Identifying Head Specializations ::: Positional heads 
One particular type of head, as noted by BIBREF9, is the positional head  These heads tend to focus their attention on either the previous or next token in the sequence, thus obtaining representations of the neighborhood of the current time step  In Figure FIGREF47, we show attention plots for such heads, found for each of the studied models  The sparsity of our models allows these heads to be more confident in their representations, by assigning the whole probability distribution to a single token in the sequence  Concretely, we may measure a positional head's confidence as the average attention weight assigned to the previous token  The softmax model has three heads for position $-1$, with median confidence $93 5\%$  The $1 5$-entmax model also has three heads for this position, with median confidence $94 4\%$  The adaptive model has four heads, with median confidences $95 9\%$, the lowest-confidence head being dense with $\alpha =1 18$, while the highest-confidence head being sparse ($\alpha =1 91$) 
For position $+1$, the models each dedicate one head, with confidence around $95\%$, slightly higher for entmax  The adaptive model sets $\alpha =1 96$ for this head 
Analysis ::: Identifying Head Specializations ::: BPE-merging head 
Due to the sparsity of our models, we are able to identify other head specializations, easily identifying which heads should be further analysed  In Figure FIGREF51 we show one such head where the $\alpha $ value is particularly high (in the encoder, layer 1, head 4 depicted in Figure FIGREF37)  We found that this head most often looks at the current time step with high confidence, making it a positional head with offset 0  However, this head often spreads weight sparsely over 2-3 neighboring tokens, when the tokens are part of the same BPE cluster or hyphenated words  As this head is in the first layer, it provides a useful service to the higher layers by combining information evenly within some BPE clusters 
For each BPE cluster or cluster of hyphenated words, we computed a score between 0 and 1 that corresponds to the maximum attention mass assigned by any token to the rest of the tokens inside the cluster in order to quantify the BPE-merging capabilities of these heads  There are not any attention heads in the softmax model that are able to obtain a score over $80\%$, while for $1 5$-entmax and $\alpha $-entmax there are two heads in each ($83 3\%$ and $85 6\%$ for $1 5$-entmax and $88 5\%$ and $89 8\%$ for $\alpha $-entmax) 
Analysis ::: Identifying Head Specializations ::: Interrogation head 
On the other hand, in Figure FIGREF52 we show a head for which our adaptively sparse model chose an $\alpha $ close to 1, making it closer to softmax (also shown in encoder, layer 1, head 3 depicted in Figure FIGREF37)  We observe that this head assigns a high probability to question marks at the end of the sentence in time steps where the current token is interrogative, thus making it an interrogation-detecting head  We also observe this type of heads in the other models, which we also depict in Figure FIGREF52  The average attention weight placed on the question mark when the current token is an interrogative word is $98 5\%$ for softmax, $97 0\%$ for $1 5$-entmax, and $99 5\%$ for $\alpha $-entmax 
Furthermore, we can examine sentences where some tendentially sparse heads become less so, thus identifying sources of ambiguity where the head is less confident in its prediction  An example is shown in Figure FIGREF55 where sparsity in the same head differs for sentences of similar length 
Related Work ::: Sparse attention 
Prior work has developed sparse attention mechanisms, including applications to NMT BIBREF19, BIBREF12, BIBREF20, BIBREF22, BIBREF34  BIBREF14 introduced the entmax function this work builds upon  In their work, there is a single attention mechanism which is controlled by a fixed $\alpha $  In contrast, this is the first work to allow such attention mappings to dynamically adapt their curvature and sparsity, by automatically adjusting the continuous $\alpha $ parameter  We also provide the first results using sparse attention in a Transformer model 
Related Work ::: Fixed sparsity patterns 
Recent research improves the scalability of Transformer-like networks through static, fixed sparsity patterns BIBREF10, BIBREF35  Our adaptively-sparse Transformer can dynamically select a sparsity pattern that finds relevant words regardless of their position (e g , Figure FIGREF52)  Moreover, the two strategies could be combined  In a concurrent line of research, BIBREF11 propose an adaptive attention span for Transformer language models  While their work has each head learn a different contiguous span of context tokens to attend to, our work finds different sparsity patterns in the same span  Interestingly, some of their findings mirror ours – we found that attention heads in the last layers tend to be denser on average when compared to the ones in the first layers, while their work has found that lower layers tend to have a shorter attention span compared to higher layers 
Related Work ::: Transformer interpretability 
The original Transformer paper BIBREF0 shows attention visualizations, from which some speculation can be made of the roles the several attention heads have  BIBREF7 study the syntactic abilities of the Transformer self-attention, while BIBREF6 extract dependency relations from the attention weights  BIBREF8 find that the self-attentions in BERT BIBREF3 follow a sequence of processes that resembles a classical NLP pipeline  Regarding redundancy of heads, BIBREF9 develop a method that is able to prune heads of the multi-head attention module and make an empirical study of the role that each head has in self-attention (positional, syntactic and rare words)  BIBREF36 also aim to reduce head redundancy by adding a regularization term to the loss that maximizes head disagreement and obtain improved results  While not considering Transformer attentions, BIBREF18 show that traditional attention mechanisms do not necessarily improve interpretability since softmax attention is vulnerable to an adversarial attack leading to wildly different model predictions for the same attention weights  Sparse attention may mitigate these issues; however, our work focuses mostly on a more mechanical aspect of interpretation by analyzing head behavior, rather than on explanations for predictions 
Conclusion and Future Work
We contribute a novel strategy for adaptively sparse attention, and, in particular, for adaptively sparse Transformers  We present the first empirical analysis of Transformers with sparse attention mappings (i e , entmax), showing potential in both translation accuracy as well as in model interpretability 
In particular, we analyzed how the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence  Our adaptivity strategy relies only on gradient-based optimization, side-stepping costly per-head hyper-parameter searches  Further speed-ups are possible by leveraging more parallelism in the bisection algorithm for computing $\alpha $-entmax 
Finally, some of the automatically-learned behaviors of our adaptively sparse Transformers – for instance, the near-deterministic positional heads or the subword joining head – may provide new ideas for designing static variations of the Transformer 
Acknowledgments
This work was supported by the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para a Ciência e Tecnologia through contracts UID/EEA/50008/2019 and CMUPERI/TIC/0046/2014 (GoLocal)  We are grateful to Ben Peters for the $\alpha $-entmax code and Erick Fonseca, Marcos Treviso, Pedro Martins, and Tsvetomila Mihaylova for insightful group discussion  We thank Mathieu Blondel for the idea to learn $\alpha $  We would also like to thank the anonymous reviewers for their helpful feedback 
Supplementary Material
Background ::: Regularized Fenchel-Young prediction functions
Definition 1 (BIBREF23)
Let $\Omega \colon \triangle ^d \rightarrow {\mathbb {R}}\cup \lbrace \infty \rbrace $ be a strictly convex regularization function  We define the prediction function $\mathbf {\pi }_{\Omega }$ as
Background ::: Characterizing the @!START@$\alpha $@!END@-entmax mapping
Lemma 1 (BIBREF14) For any $\mathbf {z}$, there exists a unique $\tau ^\star $ such that
Proof: From the definition of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$,
we may easily identify it with a regularized prediction function (Def  UNKREF81):
We first note that for all $\mathbf {p}\in \triangle ^d$,
From the constant invariance and scaling properties of $\mathbf {\pi }_{\Omega }$ BIBREF23,
Using BIBREF23, noting that $g^{\prime }(t) = t^{\alpha - 1}$ and $(g^{\prime })^{-1}(u) = u^{{1}{\alpha -1}}$, yields
Since $\mathsf {H}^{\textsc {T}}_\alpha $ is strictly convex on the simplex, $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ has a unique solution $\mathbf {p}^\star $  Equation DISPLAY_FORM88 implicitly defines a one-to-one mapping between $\mathbf {p}^\star $ and $\tau ^\star $ as long as $\mathbf {p}^\star \in \triangle $, therefore $\tau ^\star $ is also unique 
Background ::: Connections to softmax and sparsemax
The Euclidean projection onto the simplex, sometimes referred to, in the context of neural attention, as sparsemax BIBREF19, is defined as
The solution can be characterized through the unique threshold $\tau $ such that $\sum _i \operatornamewithlimits{\mathsf {sparsemax}}(\mathbf {z})_i = 1$ and BIBREF38
Thus, each coordinate of the sparsemax solution is a piecewise-linear function  Visibly, this expression is recovered when setting $\alpha =2$ in the $\alpha $-entmax expression (Equation DISPLAY_FORM85); for other values of $\alpha $, the exponent induces curvature 
On the other hand, the well-known softmax is usually defined through the expression
which can be shown to be the unique solution of the optimization problem
where $\mathsf {H}^\textsc {S}(\mathbf {p}) -\sum _i p_i \log p_i$ is the Shannon entropy  Indeed, setting the gradient to 0 yields the condition $\log p_i = z_j - \nu _i - \tau - 1$, where $\tau $ and $\nu > 0$ are Lagrange multipliers for the simplex constraints $\sum _i p_i = 1$ and $p_i \ge 0$, respectively  Since the l h s  is only finite for $p_i>0$, we must have $\nu _i=0$ for all $i$, by complementary slackness  Thus, the solution must have the form $p_i = {\exp (z_i)}{Z}$, yielding Equation DISPLAY_FORM92 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@
Recall that the entmax transformation is defined as:
where $\alpha \ge 1$ and $\mathsf {H}^{\textsc {T}}_{\alpha }$ is the Tsallis entropy,
and $\mathsf {H}^\textsc {S}(\mathbf {p}):= -\sum _j p_j \log p_j$ is the Shannon entropy 
In this section, we derive the Jacobian of $\operatornamewithlimits{\mathsf {entmax }}$ with respect to the scalar parameter $\alpha $ 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: General case of @!START@$\alpha >1$@!END@
From the KKT conditions associated with the optimization problem in Eq  DISPLAY_FORM85, we have that the solution $\mathbf {p}^{\star }$ has the following form, coordinate-wise:
where $\tau ^{\star }$ is a scalar Lagrange multiplier that ensures that $\mathbf {p}^{\star }$ normalizes to 1, i e , it is defined implicitly by the condition:
For general values of $\alpha $, Eq  DISPLAY_FORM98 lacks a closed form solution  This makes the computation of the Jacobian
non-trivial  Fortunately, we can use the technique of implicit differentiation to obtain this Jacobian 
The Jacobian exists almost everywhere, and the expressions we derive expressions yield a generalized Jacobian BIBREF37 at any non-differentiable points that may occur for certain ($\alpha $, $\mathbf {z}$) pairs  We begin by noting that $\frac{\partial p_i^{\star }}{\partial \alpha } = 0$ if $p_i^{\star } = 0$, because increasing $\alpha $ keeps sparse coordinates sparse  Therefore we need to worry only about coordinates that are in the support of $\mathbf {p}^\star $  We will assume hereafter that the $i$th coordinate of $\mathbf {p}^\star $ is non-zero  We have:
We can see that this Jacobian depends on $\frac{\partial \tau ^{\star }}{\partial \alpha }$, which we now compute using implicit differentiation 
Let $\mathcal {S} = \lbrace i: p^\star _i > 0 \rbrace $)  By differentiating both sides of Eq  DISPLAY_FORM98, re-using some of the steps in Eq  DISPLAY_FORM101, and recalling Eq  DISPLAY_FORM97, we get
from which we obtain:
Finally, plugging Eq  DISPLAY_FORM103 into Eq  DISPLAY_FORM101, we get:
where we denote by
The distribution $\tilde{\mathbf {p}}(\alpha )$ can be interpreted as a “skewed” distribution obtained from $\mathbf {p}^{\star }$, which appears in the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}(\mathbf {z})$ w r t  $\mathbf {z}$ as well BIBREF14 
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Solving the indetermination for @!START@$\alpha =1$@!END@
We can write Eq  DISPLAY_FORM104 as
When $\alpha \rightarrow 1^+$, we have $\tilde{\mathbf {p}}(\alpha ) \rightarrow \mathbf {p}^{\star }$, which leads to a $\frac{0}{0}$ indetermination 
To solve this indetermination, we will need to apply L'Hôpital's rule twice  Let us first compute the derivative of $\tilde{p}_i(\alpha )$ with respect to $\alpha $  We have
therefore
Differentiating the numerator and denominator in Eq  DISPLAY_FORM107, we get:
with
and
When $\alpha \rightarrow 1^+$, $B$ becomes again a $\frac{0}{0}$ indetermination, which we can solve by applying again L'Hôpital's rule  Differentiating the numerator and denominator in Eq  DISPLAY_FORM112:
Finally, summing Eq  DISPLAY_FORM111 and Eq  DISPLAY_FORM113, we get
Jacobian of @!START@$\alpha $@!END@-entmax w r t  the shape parameter @!START@$\alpha $@!END@: Proof of Proposition @!START@UID22@!END@ ::: Summary
To sum up, we have the following expression for the Jacobian of $\mathop {\mathsf {\alpha }\textnormal {-}\mathsf {entmax }}$ with respect to $\alpha $:

ground_truth: the attention heads in the proposed adaptively sparse Transformer can specialize more and with higher confidence

We introduce sparse attention into the Transformer architecture