prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What type of classifiers are used?
Context 1: Introduction
Event detection on microblogging platforms such as Twitter aims to detect events preemptively  A main task in event detection is detecting events of predetermined types BIBREF0, such as concerts or controversial events based on microposts matching specific event descriptions  This task has extensive applications ranging from cyber security BIBREF1, BIBREF2 to political elections BIBREF3 or public health BIBREF4, BIBREF5  Due to the high ambiguity and inconsistency of the terms used in microposts, event detection is generally performed though statistical machine learning models, which require a labeled dataset for model training  Data labeling is, however, a long, laborious, and usually costly process  For the case of micropost classification, though positive labels can be collected (e g , using specific hashtags, or event-related date-time information), there is no straightforward way to generate negative labels useful for model training  To tackle this lack of negative labels and the significant manual efforts in data labeling, BIBREF1 (BIBREF1, BIBREF3) introduced a weak supervision based learning approach, which uses only positively labeled data, accompanied by unlabeled examples by filtering microposts that contain a certain keyword indicative of the event type under consideration (e g , `hack' for cyber security)  Another key technique in this context is expectation regularization BIBREF6, BIBREF7, BIBREF1  Here, the estimated proportion of relevant microposts in an unlabeled dataset containing a keyword is given as a keyword-specific expectation  This expectation is used in the regularization term of the model's objective function to constrain the posterior distribution of the model predictions  By doing so, the model is trained with an expectation on its prediction for microposts that contain the keyword  Such a method, however, suffers from two key problems:
Due to the unpredictability of event occurrences and the constantly changing dynamics of users' posting frequency BIBREF8, estimating the expectation associated with a keyword is a challenging task, even for domain experts;
The performance of the event detection model is constrained by the informativeness of the keyword used for model training  As of now, we lack a principled method for discovering new keywords and improve the model performance 
To address the above issues, we advocate a human-AI loop approach for discovering informative keywords and estimating their expectations reliably  Our approach iteratively leverages 1) crowd workers for estimating keyword-specific expectations, and 2) the disagreement between the model and the crowd for discovering new informative keywords  More specifically, at each iteration after we obtain a keyword-specific expectation from the crowd, we train the model using expectation regularization and select those keyword-related microposts for which the model's prediction disagrees the most with the crowd's expectation; such microposts are then presented to the crowd to identify new keywords that best explain the disagreement  By doing so, our approach identifies new keywords which convey more relevant information with respect to existing ones, thus effectively boosting model performance  By exploiting the disagreement between the model and the crowd, our approach can make efficient use of the crowd, which is of critical importance in a human-in-the-loop context BIBREF9, BIBREF10  An additional advantage of our approach is that by obtaining new keywords that improve model performance over time, we are able to gain insight into how the model learns for specific event detection tasks  Such an advantage is particularly useful for event detection using complex models, e g , deep neural networks, which are intrinsically hard to understand BIBREF11, BIBREF12  An additional challenge in involving crowd workers is that their contributions are not fully reliable BIBREF13  In the crowdsourcing literature, this problem is usually tackled with probabilistic latent variable models BIBREF14, BIBREF15, BIBREF16, which are used to perform truth inference by aggregating a redundant set of crowd contributions  Our human-AI loop approach improves the inference of keyword expectation by aggregating contributions not only from the crowd but also from the model  This, however, comes with its own challenge as the model's predictions are further dependent on the results of expectation inference, which is used for model training  To address this problem, we introduce a unified probabilistic model that seamlessly integrates expectation inference and model training, thereby allowing the former to benefit from the latter while resolving the inter-dependency between the two 
To the best of our knowledge, we are the first to propose a human-AI loop approach that iteratively improves machine learning models for event detection  In summary, our work makes the following key contributions:
A novel human-AI loop approach for micropost event detection that jointly discovers informative keywords and estimates their expectation;
A unified probabilistic model that infers keyword expectation and simultaneously performs model training;
An extensive empirical evaluation of our approach on multiple real-world datasets demonstrating that our approach significantly improves the state of the art by an average of 24 3% AUC 
The rest of this paper is organized as follows  First, we present our human-AI loop approach in Section SECREF2  Subsequently, we introduce our proposed probabilistic model in Section SECREF3  The experimental setup and results are presented in Section SECREF4  Finally, we briefly cover related work in Section SECREF5 before concluding our work in Section SECREF6 
The Human-AI Loop Approach
Given a set of labeled and unlabeled microposts, our goal is to extract informative keywords and estimate their expectations in order to train a machine learning model  To achieve this goal, our proposed human-AI loop approach comprises two crowdsourcing tasks, i e , micropost classification followed by keyword discovery, and a unified probabilistic model for expectation inference and model training  Figure FIGREF6 presents an overview of our approach  Next, we describe our approach from a process-centric perspective 
Following previous studies BIBREF1, BIBREF17, BIBREF2, we collect a set of unlabeled microposts $\mathcal {U}$ from a microblogging platform and post-filter, using an initial (set of) keyword(s), those microposts that are potentially relevant to an event category  Then, we collect a set of event-related microposts (i e , positively labeled microposts) $\mathcal {L}$, post-filtering with a list of seed events  $\mathcal {U}$ and $\mathcal {L}$ are used together to train a discriminative model (e g , a deep neural network) for classifying the relevance of microposts to an event  We denote the target model as $p_\theta (y|x)$, where $\theta $ is the model parameter to be learned and $y$ is the label of an arbitrary micropost, represented by a bag-of-words vector $x$  Our approach iterates several times $t=\lbrace 1, 2, \ldots \rbrace $ until the performance of the target model converges  Each iteration starts from the initial keyword(s) or the new keyword(s) discovered in the previous iteration  Given such a keyword, denoted by $w^{(t)}$, the iteration starts by sampling microposts containing the keyword from $\mathcal {U}$, followed by dynamically creating micropost classification tasks and publishing them on a crowdsourcing platform 
Micropost Classification  The micropost classification task requires crowd workers to label the selected microposts into two classes: event-related and non event-related  In particular, workers are given instructions and examples to differentiate event-instance related microposts and general event-category related microposts  Consider, for example, the following microposts in the context of Cyber attack events, both containing the keyword `hack':
Credit firm Equifax says 143m Americans' social security numbers exposed in hack
This micropost describes an instance of a cyber attack event that the target model should identify  This is, therefore, an event-instance related micropost and should be considered as a positive example  Contrast this with the following example:
Companies need to step their cyber security up
This micropost, though related to cyber security in general, does not mention an instance of a cyber attack event, and is of no interest to us for event detection  This is an example of a general event-category related micropost and should be considered as a negative example 
In this task, each selected micropost is labeled by multiple crowd workers  The annotations are passed to our probabilistic model for expectation inference and model training 
Expectation Inference & Model Training  Our probabilistic model takes crowd-contributed labels and the model trained in the previous iteration as input  As output, it generates a keyword-specific expectation, denoted as $e^{(t)}$, and an improved version of the micropost classification model, denoted as $p_{\theta ^{(t)}}(y|x)$  The details of our probabilistic model are given in Section SECREF3 
Keyword Discovery  The keyword discovery task aims at discovering a new keyword (or a set of keywords) that is most informative for model training with respect to existing keywords  To this end, we first apply the current model $p_{\theta ^{(t)}}(y|x)$ on the unlabeled microposts $\mathcal {U}$  For those that contain the keyword $w^{(t)}$, we calculate the disagreement between the model predictions and the keyword-specific expectation $e^{(t)}$:
and select the ones with the highest disagreement for keyword discovery  These selected microposts are supposed to contain information that can explain the disagreement between the model prediction and keyword-specific expectation, and can thus provide information that is most different from the existing set of keywords for model training 
For instance, our study shows that the expectation for the keyword `hack' is 0 20, which means only 20% of the initial set of microposts retrieved with the keyword are event-related  A micropost selected with the highest disagreement (Eq  DISPLAY_FORM7), whose likelihood of being event-related as predicted by the model is $99 9\%$, is shown as an example below:
RT @xxx: Hong Kong securities brokers hit by cyber attacks, may face more: regulator #cyber #security #hacking https://t co/rC1s9CB
This micropost contains keywords that can better indicate the relevance to a cyber security event than the initial keyword `hack', e g , `securities', `hit', and `attack' 
Note that when the keyword-specific expectation $e^{(t)}$ in Equation DISPLAY_FORM7 is high, the selected microposts will be the ones that contain keywords indicating the irrelevance of the microposts to an event category  Such keywords are also useful for model training as they help improve the model's ability to identify irrelevant microposts 
To identify new keywords in the selected microposts, we again leverage crowdsourcing, as humans are typically better than machines at providing specific explanations BIBREF18, BIBREF19  In the crowdsourcing task, workers are first asked to find those microposts where the model predictions are deemed correct  Then, from those microposts, workers are asked to find the keyword that best indicates the class of the microposts as predicted by the model  The keyword most frequently identified by the workers is then used as the initial keyword for the following iteration  In case multiple keywords are selected, e g , the top-$N$ frequent ones, workers will be asked to perform $N$ micropost classification tasks for each keyword in the next iteration, and the model training will be performed on multiple keyword-specific expectations 
Unified Probabilistic Model
This section introduces our probabilistic model that infers keyword expectation and trains the target model simultaneously  We start by formalizing the problem and introducing our model, before describing the model learning method 
Problem Formalization  We consider the problem at iteration $t$ where the corresponding keyword is $w^{(t)}$  In the current iteration, let $\mathcal {U}^{(t)} \subset \mathcal {U}$ denote the set of all microposts containing the keyword and $\mathcal {M}^{(t)}= \lbrace x_{m}\rbrace _{m=1}^M\subset \mathcal {U}^{(t)}$ be the randomly selected subset of $M$ microposts labeled by $N$ crowd workers $\mathcal {C} = \lbrace c_n\rbrace _{n=1}^N$  The annotations form a matrix $\mathbf {A}\in \mathbb {R}^{M\times N}$ where $\mathbf {A}_{mn}$ is the label for the micropost $x_m$ contributed by crowd worker $c_n$  Our goal is to infer the keyword-specific expectation $e^{(t)}$ and train the target model by learning the model parameter $\theta ^{(t)}$  An additional parameter of our probabilistic model is the reliability of crowd workers, which is essential when involving crowdsourcing  Following Dawid and Skene BIBREF14, BIBREF16, we represent the annotation reliability of worker $c_n$ by a latent confusion matrix $\pi ^{(n)}$, where the $rs$-th element $\pi _{rs}^{(n)}$ denotes the probability of $c_n$ labeling a micropost as class $r$ given the true class $s$ 
Unified Probabilistic Model ::: Expectation as Model Posterior
First, we introduce an expectation regularization technique for the weakly supervised learning of the target model $p_{\theta ^{(t)}}(y|x)$  In this setting, the objective function of the target model is composed of two parts, corresponding to the labeled microposts $\mathcal {L}$ and the unlabeled ones $\mathcal {U}$ 
The former part aims at maximizing the likelihood of the labeled microposts:
where we assume that $\theta $ is generated from a prior distribution (e g , Laplacian or Gaussian) parameterized by $\sigma $ 
To leverage unlabeled data for model training, we make use of the expectations of existing keywords, i e , {($w^{(1)}$, $e^{(1)}$),    , ($w^{(t-1)}$, $e^{(t-1)}$), ($w^{(t)}$, $e^{(t)}$)} (Note that $e^{(t)}$ is inferred), as a regularization term to constrain model training  To do so, we first give the model's expectation for each keyword $w^{(k)}$ ($1\le k\le t$) as follows:
which denotes the empirical expectation of the model’s posterior predictions on the unlabeled microposts $\mathcal {U}^{(k)}$ containing keyword $w^{(k)}$  Expectation regularization can then be formulated as the regularization of the distance between the Bernoulli distribution parameterized by the model's expectation and the expectation of the existing keyword:
where $D_{KL}[\cdot \Vert \cdot ]$ denotes the KL-divergence between the Bernoulli distributions $Ber(e^{(k)})$ and $Ber(\mathbb {E}_{x\sim \mathcal {U}^{(k)}}(y))$, and $\lambda $ controls the strength of expectation regularization 
Unified Probabilistic Model ::: Expectation as Class Prior
To learn the keyword-specific expectation $e^{(t)}$ and the crowd worker reliability $\pi ^{(n)}$ ($1\le n\le N$), we model the likelihood of the crowd-contributed labels $\mathbf {A}$ as a function of these parameters  In this context, we view the expectation as the class prior, thus performing expectation inference as the learning of the class prior  By doing so, we connect expectation inference with model training 
Specifically, we model the likelihood of an arbitrary crowd-contributed label $\mathbf {A}_{mn}$ as a mixture of multinomials where the prior is the keyword-specific expectation $e^{(t)}$:
where $e_s^{(t)}$ is the probability of the ground truth label being $s$ given the keyword-specific expectation as the class prior; $K$ is the set of possible ground truth labels (binary in our context); and $r=\mathbf {A}_{mn}$ is the crowd-contributed label  Then, for an individual micropost $x_m$, the likelihood of crowd-contributed labels $\mathbf {A}_{m:}$ is given by:
Therefore, the objective function for maximizing the likelihood of the entire annotation matrix $\mathbf {A}$ can be described as:
Unified Probabilistic Model ::: Unified Probabilistic Model
Integrating model training with expectation inference, the overall objective function of our proposed model is given by:
Figure FIGREF18 depicts a graphical representation of our model, which combines the target model for training (on the left) with the generative model for crowd-contributed labels (on the right) through a keyword-specific expectation 
Model Learning  Due to the unknown ground truth labels of crowd-annotated microposts ($y_m$ in Figure FIGREF18), we resort to expectation maximization for model learning  The learning algorithm iteratively takes two steps: the E-step and the M-step  The E-step infers the ground truth labels given the current model parameters  The M-step updates the model parameters, including the crowd reliability parameters $\pi ^{(n)}$ ($1\le n\le N$), the keyword-specific expectation $e^{(t)}$, and the parameter of the target model $\theta ^{(t)}$  The E-step and the crowd parameter update in the M-step are similar to the Dawid-Skene model BIBREF14  The keyword expectation is inferred by taking into account both the crowd-contributed labels and the model prediction:
The parameter of the target model is updated by gradient descent  For example, when the target model to be trained is a deep neural network, we use back-propagation with gradient descent to update the weight matrices 
Experiments and Results
This section presents our experimental setup and results for evaluating our approach  We aim at answering the following questions:
[noitemsep,leftmargin=*]
Q1: How effectively does our proposed human-AI loop approach enhance the state-of-the-art machine learning models for event detection?
Q2: How well does our keyword discovery method work compare to existing keyword expansion methods?
Q3: How effective is our approach using crowdsourcing at obtaining new keywords compared with an approach labelling microposts for model training under the same cost?
Q4: How much benefit does our unified probabilistic model bring compared to methods that do not take crowd reliability into account?
Experiments and Results ::: Experimental Setup
Datasets  We perform our experiments with two predetermined event categories: cyber security (CyberAttack) and death of politicians (PoliticianDeath)  These event categories are chosen as they are representative of important event types that are of interest to many governments and companies  The need to create our own dataset was motivated by the lack of public datasets for event detection on microposts  The few available datasets do not suit our requirements  For example, the publicly available Events-2012 Twitter dataset BIBREF20 contains generic event descriptions such as Politics, Sports, Culture etc  Our work targets more specific event categories BIBREF21  Following previous studies BIBREF1, we collect event-related microposts from Twitter using 11 and 8 seed events (see Section SECREF2) for CyberAttack and PoliticianDeath, respectively  Unlabeled microposts are collected by using the keyword `hack' for CyberAttack, while for PoliticianDeath, we use a set of keywords related to `politician' and `death' (such as `bureaucrat', `dead' etc ) For each dataset, we randomly select 500 tweets from the unlabeled subset and manually label them for evaluation  Table TABREF25 shows key statistics from our two datasets 
Comparison Methods  To demonstrate the generality of our approach on different event detection models, we consider Logistic Regression (LR) BIBREF1 and Multilayer Perceptron (MLP) BIBREF2 as the target models  As the goal of our experiments is to demonstrate the effectiveness of our approach as a new model training technique, we use these widely used models  Also, we note that in our case other neural network models with more complex network architectures for event detection, such as the bi-directional LSTM BIBREF17, turn out to be less effective than a simple feedforward network  For both LR and MLP, we evaluate our proposed human-AI loop approach for keyword discovery and expectation estimation by comparing against the weakly supervised learning method proposed by BIBREF1 (BIBREF1) and BIBREF17 (BIBREF17) where only one initial keyword is used with an expectation estimated by an individual expert 
Parameter Settings  We empirically set optimal parameters based on a held-out validation set that contains 20% of the test data  These include the hyperparamters of the target model, those of our proposed probabilistic model, and the parameters used for training the target model  We explore MLP with 1, 2 and 3 hidden layers and apply a grid search in 32, 64, 128, 256, 512 for the dimension of the embeddings and that of the hidden layers  For the coefficient of expectation regularization, we follow BIBREF6 (BIBREF6) and set it to $\lambda =10 \times $ #labeled examples  For model training, we use the Adam BIBREF22 optimization algorithm for both models 
Evaluation  Following BIBREF1 (BIBREF1) and BIBREF3 (BIBREF3), we use accuracy and area under the precision-recall curve (AUC) metrics to measure the performance of our proposed approach  We note that due to the imbalance in our datasets (20% positive microposts in CyberAttack and 27% in PoliticianDeath), accuracy is dominated by negative examples; AUC, in comparison, better characterizes the discriminative power of the model 
Crowdsourcing  We chose Level 3 workers on the Figure-Eight crowdsourcing platform for our experiments  The inter-annotator agreement in micropost classification is taken into account through the EM algorithm  For keyword discovery, we filter keywords based on the frequency of the keyword being selected by the crowd  In terms of cost-effectiveness, our approach is motivated from the fact that crowdsourced data annotation can be expensive, and is thus designed with minimal crowd involvement  For each iteration, we selected 50 tweets for keyword discovery and 50 tweets for micropost classification per keyword  For a dataset with 80k tweets (e g , CyberAttack), our approach only requires to manually inspect 800 tweets (for 8 keywords), which is only 1% of the entire dataset 
Experiments and Results ::: Results of our Human-AI Loop (Q1)
Table TABREF26 reports the evaluation of our approach on both the CyberAttack and PoliticianDeath event categories  Our approach is configured such that each iteration starts with 1 new keyword discovered in the previous iteration 
Our approach improves LR by 5 17% (Accuracy) and 18 38% (AUC), and MLP by 10 71% (Accuracy) and 30 27% (AUC) on average  Such significant improvements clearly demonstrate that our approach is effective at improving model performance  We observe that the target models generally converge between the 7th and 9th iteration on both datasets when performance is measured by AUC  The performance can slightly degrade when the models are further trained for more iterations on both datasets  This is likely due to the fact that over time, the newly discovered keywords entail lower novel information for model training  For instance, for the CyberAttack dataset the new keyword in the 9th iteration `election' frequently co-occurs with the keyword `russia' in the 5th iteration (in microposts that connect Russian hackers with US elections), thus bringing limited new information for improving the model performance  As a side remark, we note that the models converge faster when performance is measured by accuracy  Such a comparison result confirms the difference between the metrics and shows the necessity for more keywords to discriminate event-related microposts from non event-related ones 
Experiments and Results ::: Comparative Results on Keyword Discovery (Q2)
Figure FIGREF31 shows the evaluation of our approach when discovering new informative keywords for model training (see Section SECREF2: Keyword Discovery)  We compare our human-AI collaborative way of discovering new keywords against a query expansion (QE) approach BIBREF23, BIBREF24 that leverages word embeddings to find similar words in the latent semantic space  Specifically, we use pre-trained word embeddings based on a large Google News dataset for query expansion  For instance, the top keywords resulting from QE for `politician' are, `deputy',`ministry',`secretary', and `minister'  For each of these keywords, we use the crowd to label a set of tweets and obtain a corresponding expectation 
We observe that our approach consistently outperforms QE by an average of $4 62\%$ and $52 58\%$ AUC on CyberAttack and PoliticianDeath, respectively  The large gap between the performance improvements for the two datasets is mainly due to the fact that microposts that are relevant for PoliticianDeath are semantically more complex than those for CyberAttack, as they encode noun-verb relationship (e g , “the king of     died    ”) rather than a simple verb (e g , “    hacked ”) for the CyberAttack microposts  QE only finds synonyms of existing keywords related to either `politician' or `death', however cannot find a meaningful keyword that fully characterizes the death of a politician  For instance, QE finds the keywords `kill' and `murder', which are semantically close to `death' but are not specifically relevant to the death of a politician  Unlike QE, our approach identifies keywords that go beyond mere synonyms and that are more directly related to the end task, i e , discriminating event-related microposts from non related ones  Examples are `demise' and `condolence'  As a remark, we note that in Figure FIGREF31(b), the increase in QE performance on PoliticianDeath is due to the keywords `deputy' and `minister', which happen to be highly indicative of the death of a politician in our dataset; these keywords are also identified by our approach 
Experiments and Results ::: Cost-Effectiveness Results (Q3)
To demonstrate the cost-effectiveness of using crowdsourcing for obtaining new keywords and consequently, their expectations, we compare the performance of our approach with an approach using crowdsourcing to only label microposts for model training at the same cost  Specifically, we conducted an additional crowdsourcing experiment where the same cost used for keyword discovery in our approach is used to label additional microposts for model training  These newly labeled microposts are used with the microposts labeled in the micropost classification task of our approach (see Section SECREF2: Micropost Classification) and the expectation of the initial keyword to train the model for comparison  The model trained in this way increases AUC by 0 87% for CyberAttack, and by 1 06% for PoliticianDeath; in comparison, our proposed approach increases AUC by 33 42% for PoliticianDeath and by 15 23% for CyberAttack over the baseline presented by BIBREF1)  These results show that using crowdsourcing for keyword discovery is significantly more cost-effective than simply using crowdsourcing to get additional labels when training the model 
Experiments and Results ::: Expectation Inference Results (Q4)
To investigate the effectiveness of our expectation inference method, we compare it against a majority voting approach, a strong baseline in truth inference BIBREF16  Figure FIGREF36 shows the result of this evaluation  We observe that our approach results in better models for both CyberAttack and PoliticianDeath  Our manual investigation reveals that workers' annotations are of high reliability, which explains the relatively good performance of majority voting  Despite limited margin for improvement, our method of expectation inference improves the performance of majority voting by $0 4\%$ and $1 19\%$ AUC on CyberAttack and PoliticianDeath, respectively 
Related Work
Event Detection  The techniques for event extraction from microblogging platforms can be classified according to their domain specificity and their detection method BIBREF0  Early works mainly focus on open domain event detection BIBREF25, BIBREF26, BIBREF27  Our work falls into the category of domain-specific event detection BIBREF21, which has drawn increasing attention due to its relevance for various applications such as cyber security BIBREF1, BIBREF2 and public health BIBREF4, BIBREF5  In terms of technique, our proposed detection method is related to the recently proposed weakly supervised learning methods BIBREF1, BIBREF17, BIBREF3  This comes in contrast with fully-supervised learning methods, which are often limited by the size of the training data (e g , a few hundred examples) BIBREF28, BIBREF29 
Human-in-the-Loop Approaches  Our work extends weakly supervised learning methods by involving humans in the loop BIBREF13  Existing human-in-the-loop approaches mainly leverage crowds to label individual data instances BIBREF9, BIBREF10 or to debug the training data BIBREF30, BIBREF31 or components BIBREF32, BIBREF33, BIBREF34 of a machine learning system  Unlike these works, we leverage crowd workers to label sampled microposts in order to obtain keyword-specific expectations, which can then be generalized to help classify microposts containing the same keyword, thus amplifying the utility of the crowd  Our work is further connected to the topic of interpretability and transparency of machine learning models BIBREF11, BIBREF35, BIBREF12, for which humans are increasingly involved, for instance for post-hoc evaluations of the model's interpretability  In contrast, our approach directly solicits informative keywords from the crowd for model training, thereby providing human-understandable explanations for the improved model 
Conclusion
In this paper, we presented a new human-AI loop approach for keyword discovery and expectation estimation to better train event detection models  Our approach takes advantage of the disagreement between the crowd and the model to discover informative keywords and leverages the joint power of the crowd and the model in expectation inference  We evaluated our approach on real-world datasets and showed that it significantly outperforms the state of the art and that it is particularly useful for detecting events where relevant microposts are semantically complex, e g , the death of a politician  As future work, we plan to parallelize the crowdsourcing tasks and optimize our pipeline in order to use our event detection approach in real-time 
Acknowledgements
This project has received funding from the Swiss National Science Foundation (grant #407540_167320 Tighten-it-All) and from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement 683253/GraphInt) 
Context 2: Introduction
End-to-end speech-to-text translation (ST) has attracted much attention recently BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 given its simplicity against cascading automatic speech recognition (ASR) and machine translation (MT) systems  The lack of labeled data, however, has become a major blocker for bridging the performance gaps between end-to-end models and cascading systems  Several corpora have been developed in recent years  post2013improved introduced a 38-hour Spanish-English ST corpus by augmenting the transcripts of the Fisher and Callhome corpora with English translations  di-gangi-etal-2019-must created the largest ST corpus to date from TED talks but the language pairs involved are out of English only  beilharz2019librivoxdeen created a 110-hour German-English ST corpus from LibriVox audiobooks  godard-etal-2018-low created a Moboshi-French ST corpus as part of a rare language documentation effort  woldeyohannis provided an Amharic-English ST corpus in the tourism domain  boito2019mass created a multilingual ST corpus involving 8 languages from a multilingual speech corpus based on Bible readings BIBREF7  Previous work either involves language pairs out of English, very specific domains, very low resource languages or a limited set of language pairs  This limits the scope of study, including the latest explorations on end-to-end multilingual ST BIBREF8, BIBREF9  Our work is mostly similar and concurrent to iranzosnchez2019europarlst who created a multilingual ST corpus from the European Parliament proceedings  The corpus we introduce has larger speech durations and more translation tokens  It is diversified with multiple speakers per transcript/translation  Finally, we provide additional out-of-domain test sets 
In this paper, we introduce CoVoST, a multilingual ST corpus based on Common Voice BIBREF10 for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  It includes a total 708 hours of French (Fr), German (De), Dutch (Nl), Russian (Ru), Spanish (Es), Italian (It), Turkish (Tr), Persian (Fa), Swedish (Sv), Mongolian (Mn) and Chinese (Zh) speeches, with French and German ones having the largest durations among existing public corpora  We also collect an additional evaluation corpus from Tatoeba for French, German, Dutch, Russian and Spanish, resulting in a total of 9 3 hours of speech  Both corpora are created at the sentence level and do not require additional alignments or segmentation  Using the official Common Voice train-development-test split, we also provide baseline models, including, to our knowledge, the first end-to-end many-to-one multilingual ST models  CoVoST is released under CC0 license and free to use  The Tatoeba evaluation samples are also available under friendly CC licenses  All the data can be acquired at https://github com/facebookresearch/covost 
Data Collection and Processing ::: Common Voice (CoVo)
Common Voice BIBREF10 is a crowdsourcing speech recognition corpus with an open CC0 license  Contributors record voice clips by reading from a bank of donated sentences  Each voice clip was validated by at least two other users  Most of the sentences are covered by multiple speakers, with potentially different genders, age groups or accents 
Raw CoVo data contains samples that passed validation as well as those that did not  To build CoVoST, we only use the former one and reuse the official train-development-test partition of the validated data  As of January 2020, the latest CoVo 2019-06-12 release includes 29 languages  CoVoST is currently built on that release and covers the following 11 languages: French, German, Dutch, Russian, Spanish, Italian, Turkish, Persian, Swedish, Mongolian and Chinese 
Validated transcripts were sent to professional translators  Note that the translators had access to the transcripts but not the corresponding voice clips since clips would not carry additional information  Since transcripts were duplicated due to multiple speakers, we deduplicated the transcripts before sending them to translators  As a result, different voice clips of the same content (transcript) will have identical translations in CoVoST for train, development and test splits 
In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11  1) For German-English, French-English and Russian-English translations, we computed sentence-level BLEU BIBREF12 with the NLTK BIBREF13 implementation between the human translations and the automatic translations produced by a state-of-the-art system BIBREF14 (the French-English system was a Transformer big BIBREF15 separately trained on WMT14)  We applied this method to these three language pairs only as we are confident about the quality of the corresponding systems  Translations with a score that was too low were manually inspected and sent back to the translators when needed  2) We manually inspected examples where the source transcript was identical to the translation  3) We measured the perplexity of the translations using a language model trained on a large amount of clean monolingual data BIBREF14  We manually inspected examples where the translation had a high perplexity and sent them back to translators accordingly  4) We computed the ratio of English characters in the translations  We manually inspected examples with a low ratio and sent them back to translators accordingly  5) Finally, we used VizSeq BIBREF16 to calculate similarity scores between transcripts and translations based on LASER cross-lingual sentence embeddings BIBREF17  Samples with low scores were manually inspected and sent back for translation when needed 
We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint 
Data Collection and Processing ::: Tatoeba (TT)
Tatoeba (TT) is a community built language learning corpus having sentences aligned across multiple languages with the corresponding speech partially available  Its sentences are on average shorter than those in CoVoST (see also Table TABREF2) given the original purpose of language learning  Sentences in TT are licensed under CC BY 2 0 FR and part of the speeches are available under various CC licenses 
We construct an evaluation set from TT (for French, German, Dutch, Russian and Spanish) as a complement to CoVoST development and test sets  We collect (speech, transcript, English translation) triplets for the 5 languages and do not include those whose speech has a broken URL or is not CC licensed  We further filter these samples by sentence lengths (minimum 4 words including punctuations) to reduce the portion of short sentences  This makes the resulting evaluation set closer to real-world scenarios and more challenging 
We run the same quality checks for TT as for CoVoST but we do not find poor quality translations according to our criteria  Finally, we report the overlap between CoVo transcripts and TT sentences in Table TABREF5  We found a minimal overlap, which makes the TT evaluation set a suitable additional test set when training on CoVoST 
Data Analysis ::: Basic Statistics
Basic statistics for CoVoST and TT are listed in Table TABREF2 including (unique) sentence counts, speech durations, speaker demographics (partially available) as well as vocabulary and token statistics (based on Moses-tokenized sentences by sacreMoses) on both transcripts and translations  We see that CoVoST has over 327 hours of German speeches and over 171 hours of French speeches, which, to our knowledge, corresponds to the largest corpus among existing public ST corpora (the second largest is 110 hours BIBREF18 for German and 38 hours BIBREF19 for French)  Moreover, CoVoST has a total of 18 hours of Dutch speeches, to our knowledge, contributing the first public Dutch ST resource  CoVoST also has around 27-hour Russian speeches, 37-hour Italian speeches and 67-hour Persian speeches, which is 1 8 times, 2 5 times and 13 3 times of the previous largest public one BIBREF7  Most of the sentences (transcripts) in CoVoST are covered by multiple speakers with potentially different accents, resulting in a rich diversity in the speeches
Context 3: Introduction
In this publication, we present Sentence-BERT (SBERT), a modification of the BERT network using siamese and triplet networks that is able to derive semantically meaningful sentence embeddings  This enables BERT to be used for certain new tasks, which up-to-now were not applicable for BERT  These tasks include large-scale semantic similarity comparison, clustering, and information retrieval via semantic search 
BERT set new state-of-the-art performance on various sentence classification and sentence-pair regression tasks  BERT uses a cross-encoder: Two sentences are passed to the transformer network and the target value is predicted  However, this setup is unsuitable for various pair regression tasks due to too many possible combinations  Finding in a collection of $n=10\,000$ sentences the pair with the highest similarity requires with BERT $n\cdot (n-1)/2=49\,995\,000$ inference computations  On a modern V100 GPU, this requires about 65 hours  Similar, finding which of the over 40 million existent questions of Quora is the most similar for a new question could be modeled as a pair-wise comparison with BERT, however, answering a single query would require over 50 hours 
A common method to address clustering and semantic search is to map each sentence to a vector space such that semantically similar sentences are close  Researchers have started to input individual sentences into BERT and to derive fixed-size sentence embeddings  The most commonly used approach is to average the BERT output layer (known as BERT embeddings) or by using the output of the first token (the [CLS] token)  As we will show, this common practice yields rather bad sentence embeddings, often worse than averaging GloVe embeddings BIBREF2 
To alleviate this issue, we developed SBERT  The siamese network architecture enables that fixed-sized vectors for input sentences can be derived  Using a similarity measure like cosine-similarity or Manhatten / Euclidean distance, semantically similar sentences can be found  These similarity measures can be performed extremely efficient on modern hardware, allowing SBERT to be used for semantic similarity search as well as for clustering  The complexity for finding the most similar sentence pair in a collection of 10,000 sentences is reduced from 65 hours with BERT to the computation of 10,000 sentence embeddings (5 seconds with SBERT) and computing cosine-similarity (0 01 seconds)  By using optimized index structures, finding the most similar Quora question can be reduced from 50 hours to a few milliseconds BIBREF3 
We fine-tune SBERT on NLI data, which creates sentence embeddings that significantly outperform other state-of-the-art sentence embedding methods like InferSent BIBREF4 and Universal Sentence Encoder BIBREF5  On seven Semantic Textual Similarity (STS) tasks, SBERT achieves an improvement of 11 7 points compared to InferSent and 5 5 points compared to Universal Sentence Encoder  On SentEval BIBREF6, an evaluation toolkit for sentence embeddings, we achieve an improvement of 2 1 and 2 6 points, respectively 
SBERT can be adapted to a specific task  It sets new state-of-the-art performance on a challenging argument similarity dataset BIBREF7 and on a triplet dataset to distinguish sentences from different sections of a Wikipedia article BIBREF8 
The paper is structured in the following way: Section SECREF3 presents SBERT, section SECREF4 evaluates SBERT on common STS tasks and on the challenging Argument Facet Similarity (AFS) corpus BIBREF7  Section SECREF5 evaluates SBERT on SentEval  In section SECREF6, we perform an ablation study to test some design aspect of SBERT  In section SECREF7, we compare the computational efficiency of SBERT sentence embeddings in contrast to other state-of-the-art sentence embedding methods 
Related Work
We first introduce BERT, then, we discuss state-of-the-art sentence embedding methods 
BERT BIBREF0 is a pre-trained transformer network BIBREF9, which set for various NLP tasks new state-of-the-art results, including question answering, sentence classification, and sentence-pair regression  The input for BERT for sentence-pair regression consists of the two sentences, separated by a special [SEP] token  Multi-head attention over 12 (base-model) or 24 layers (large-model) is applied and the output is passed to a simple regression function to derive the final label  Using this setup, BERT set a new state-of-the-art performance on the Semantic Textual Semilarity (STS) benchmark BIBREF10  RoBERTa BIBREF1 showed, that the performance of BERT can further improved by small adaptations to the pre-training process  We also tested XLNet BIBREF11, but it led in general to worse results than BERT 
A large disadvantage of the BERT network structure is that no independent sentence embeddings are computed, which makes it difficult to derive sentence embeddings from BERT  To bypass this limitations, researchers passed single sentences through BERT and then derive a fixed sized vector by either averaging the outputs (similar to average word embeddings) or by using the output of the special CLS token (for example: bertsentenceembeddings1,bertsentenceembeddings2,bertsentenceembeddings3)  These two options are also provided by the popular bert-as-a-service-repository  Up to our knowledge, there is so far no evaluation if these methods lead to useful sentence embeddings 
Sentence embeddings are a well studied area with dozens of proposed methods  Skip-Thought BIBREF12 trains an encoder-decoder architecture to predict the surrounding sentences  InferSent BIBREF4 uses labeled data of the Stanford Natural Language Inference dataset BIBREF13 and the Multi-Genre NLI dataset BIBREF14 to train a siamese BiLSTM network with max-pooling over the output  Conneau et al  showed, that InferSent consistently outperforms unsupervised methods like SkipThought  Universal Sentence Encoder BIBREF5 trains a transformer network and augments unsupervised learning with training on SNLI  hill-etal-2016-learning showed, that the task on which sentence embeddings are trained significantly impacts their quality  Previous work BIBREF4, BIBREF5 found that the SNLI datasets are suitable for training sentence embeddings  yang-2018-learning presented a method to train on conversations from Reddit using siamese DAN and siamese transformer networks, which yielded good results on the STS benchmark dataset 
polyencoders addresses the run-time overhead of the cross-encoder from BERT and present a method (poly-encoders) to compute a score between $m$ context vectors and pre-computed candidate embeddings using attention  This idea works for finding the highest scoring sentence in a larger collection  However, poly-encoders have the drawback that the score function is not symmetric and the computational overhead is too large for use-cases like clustering, which would require $O(n^2)$ score computations 
Previous neural sentence embedding methods started the training from a random initialization  In this publication, we use the pre-trained BERT and RoBERTa network and only fine-tune it to yield useful sentence embeddings  This reduces significantly the needed training time: SBERT can be tuned in less than 20 minutes, while yielding better results than comparable sentence embedding methods 
Model
SBERT adds a pooling operation to the output of BERT / RoBERTa to derive a fixed sized sentence embedding  We experiment with three pooling strategies: Using the output of the CLS-token, computing the mean of all output vectors (MEAN-strategy), and computing a max-over-time of the output vectors (MAX-strategy)  The default configuration is MEAN 
In order to fine-tune BERT / RoBERTa, we create siamese and triplet networks BIBREF15 to update the weights such that the produced sentence embeddings are semantically meaningful and can be compared with cosine-similarity 
The network structure depends on the available training data  We experiment with the following structures and objective functions 
Classification Objective Function  We concatenate the sentence embeddings $u$ and $v$ with the element-wise difference $|u-v|$ and multiply it with the trainable weight $W_t \in \mathbb {R}^{3n \times k}$:
where $n$ is the dimension of the sentence embeddings and $k$ the number of labels  We optimize cross-entropy loss  This structure is depicted in Figure FIGREF4 
Regression Objective Function  The cosine-similarity between the two sentence embeddings $u$ and $v$ is computed (Figure FIGREF5)  We use mean-squared-error loss as the objective function 
Triplet Objective Function  Given an anchor sentence $a$, a positive sentence $p$, and a negative sentence $n$, triplet loss tunes the network such that the distance between $a$ and $p$ is smaller than the distance between $a$ and $n$  Mathematically, we minimize the following loss function:
with $s_x$ the sentence embedding for $a$/$n$/$p$, $||\cdot ||$ a distance metric and margin $\epsilon $  Margin $\epsilon $ ensures that $s_p$ is at least $\epsilon $ closer to $s_a$ than $s_n$  As metric we use Euclidean distance and we set $\epsilon =1$ in our experiments 
Model ::: Training Details
We train SBERT on the combination of the SNLI BIBREF13 and the Multi-Genre NLI BIBREF14 dataset  The SNLI is a collection of 570,000 sentence pairs annotated with the labels contradiction, eintailment, and neutral  MultiNLI contains 430,000 sentence pairs and covers a range of genres of spoken and written text  We fine-tune SBERT with a 3-way softmax-classifier objective function for one epoch  We used a batch-size of 16, Adam optimizer with learning rate $2\mathrm {e}{-5}$, and a linear learning rate warm-up over 10% of the training data

ground_truth: probabilistic model

Logistic Regression, Multilayer Perceptron

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How big is Augmented LibriSpeech dataset?
Context 2:  For example, there are over 1,000 speakers and over 10 accents in the French and German development / test sets  This enables good coverage of speech variations in both model training and evaluation 
Data Analysis ::: Speaker Diversity
As we can see from Table TABREF2, CoVoST is diversified with a rich set of speakers and accents  We further inspect the speaker demographics in terms of sample distributions with respect to speaker counts, accent counts and age groups, which is shown in Figure FIGREF6, FIGREF7 and FIGREF8  We observe that for 8 of the 11 languages, at least 60% of the sentences (transcripts) are covered by multiple speakers  Over 80% of the French sentences have at least 3 speakers  And for German sentences, even over 90% of them have at least 5 speakers  Similarly, we see that a large portion of sentences are spoken in multiple accents for French, German, Dutch and Spanish  Speakers of each language also spread widely across different age groups (below 20, 20s, 30s, 40s, 50s, 60s and 70s) 
Baseline Results
We provide baselines using the official train-development-test split on the following tasks: automatic speech recognition (ASR), machine translation (MT) and speech translation (ST) 
Baseline Results ::: Experimental Settings ::: Data Preprocessing
We convert raw MP3 audio files from CoVo and TT into mono-channel waveforms, and downsample them to 16,000 Hz  For transcripts and translations, we normalize the punctuation, we tokenize the text with sacreMoses and lowercase it  For transcripts, we further remove all punctuation markers except for apostrophes  We use character vocabularies on all the tasks, with 100% coverage of all the characters  Preliminary experimentation showed that character vocabularies provided more stable training than BPE  For MT, the vocabulary is created jointly on both transcripts and translations  We extract 80-channel log-mel filterbank features, computed with a 25ms window size and 10ms window shift using torchaudio  The features are normalized to 0 mean and 1 0 standard deviation  We remove samples having more than 3,000 frames or more than 256 characters for GPU memory efficiency (less than 25 samples are removed for all languages) 
Baseline Results ::: Experimental Settings ::: Model Training
Our ASR and ST models follow the architecture in berard2018end, but have 3 decoder layers like that in pino2019harnessing  For MT, we use a Transformer base architecture BIBREF15, but with 3 encoder layers, 3 decoder layers and 0 3 dropout  We use a batch size of 10,000 frames for ASR and ST, and a batch size of 4,000 tokens for MT  We train all models using Fairseq BIBREF20 for up to 200,000 updates  We use SpecAugment BIBREF21 for ASR and ST to alleviate overfitting 
Baseline Results ::: Experimental Settings ::: Inference and Evaluation
We use a beam size of 5 for all models  We use the best checkpoint by validation loss for MT, and average the last 5 checkpoints for ASR and ST  For MT and ST, we report case-insensitive tokenized BLEU BIBREF22 using sacreBLEU BIBREF23  For ASR, we report word error rate (WER) and character error rate (CER) using VizSeq 
Baseline Results ::: Automatic Speech Recognition (ASR)
For simplicity, we use the same model architecture for ASR and ST, although we do not leverage ASR models to pretrain ST model encoders later  Table TABREF18 shows the word error rate (WER) and character error rate (CER) for ASR models  We see that French and German perform the best given they are the two highest resource languages in CoVoST  The other languages are relatively low resource (especially Turkish and Swedish) and the ASR models are having difficulties to learn from this data 
Baseline Results ::: Machine Translation (MT)
MT models take transcripts (without punctuation) as inputs and outputs translations (with punctuation)  For simplicity, we do not change the text preprocessing methods for MT to correct this mismatch  Moreover, this mismatch also exists in cascading ST systems, where MT model inputs are the outputs of an ASR model  Table TABREF20 shows the BLEU scores of MT models  We notice that the results are consistent with what we see from ASR models  For example thanks to abundant training data, French has a decent BLEU score of 29 8/25 4  German doesn't perform well, because of less richness of content (transcripts)  The other languages are low resource in CoVoST and it is difficult to train decent models without additional data or pre-training techniques 
Baseline Results ::: Speech Translation (ST)
CoVoST is a many-to-one multilingual ST corpus  While end-to-end one-to-many and many-to-many multilingual ST models have been explored very recently BIBREF8, BIBREF9, many-to-one multilingual models, to our knowledge, have not  We hence use CoVoST to examine this setting  Table TABREF22 and TABREF23 show the BLEU scores for both bilingual and multilingual end-to-end ST models trained on CoVoST  We observe that combining speeches from multiple languages is consistently bringing gains to low-resource languages (all besides French and German)  This includes combinations of distant languages, such as Ru+Fr, Tr+Fr and Zh+Fr  Moreover, some combinations do bring gains to high-resource language (French) as well: Es+Fr, Tr+Fr and Mn+Fr  We simply provide the most basic many-to-one multilingual baselines here, and leave the full exploration of the best configurations to future work  Finally, we note that for some language pairs, absolute BLEU numbers are relatively low as we restrict model training to the supervised data  We encourage the community to improve upon those baselines, for example by leveraging semi-supervised training 
Baseline Results ::: Multi-Speaker Evaluation
In CoVoST, large portion of transcripts are covered by multiple speakers with different genders, accents and age groups  Besides the standard corpus-level BLEU scores, we also want to evaluate model output variance on the same content (transcript) but different speakers  We hence propose to group samples (and their sentence BLEU scores) by transcript, and then calculate average per-group mean and average coefficient of variation defined as follows:
and
where $G$ is the set of sentence BLEU scores grouped by transcript and $G^{\prime } = \lbrace g | g\in G, |g|>1, \textrm {Mean}(g) > 0 \rbrace $ 
$\textrm {BLEU}_{MS}$ provides a normalized quality score as oppose to corpus-level BLEU or unnormalized average of sentence BLEU  And $\textrm {CoefVar}_{MS}$ is a standardized measure of model stability against different speakers (the lower the better)  Table TABREF24 shows the $\textrm {BLEU}_{MS}$ and $\textrm {CoefVar}_{MS}$ of our ST models on CoVoST test set  We see that German and Persian have the worst $\textrm {CoefVar}_{MS}$ (least stable) given their rich speaker diversity in the test set and relatively small train set (see also Figure FIGREF6 and Table TABREF2)  Dutch also has poor $\textrm {CoefVar}_{MS}$ because of the lack of training data  Multilingual models are consistantly more stable on low-resource languages  Ru+Fr, Tr+Fr, Fa+Fr and Zh+Fr even have better $\textrm {CoefVar}_{MS}$ than all individual languages 
Conclusion
We introduce a multilingual speech-to-text translation corpus, CoVoST, for 11 languages into English, diversified with over 11,000 speakers and over 60 accents  We also provide baseline results, including, to our knowledge, the first end-to-end many-to-one multilingual model for spoken language translation  CoVoST is free to use with a CC0 license, and the additional Tatoeba evaluation samples are also CC-licensed 
Context 3:  Our default pooling strategy is MEAN 
Evaluation - Semantic Textual Similarity
We evaluate the performance of SBERT for common Semantic Textual Similarity (STS) tasks  State-of-the-art methods often learn a (complex) regression function that maps sentence embeddings to a similarity score  However, these regression functions work pair-wise and due to the combinatorial explosion those are often not scalable if the collection of sentences reaches a certain size  Instead, we always use cosine-similarity to compare the similarity between two sentence embeddings  We ran our experiments also with negative Manhatten and negative Euclidean distances as similarity measures, but the results for all approaches remained roughly the same 
Evaluation - Semantic Textual Similarity ::: Unsupervised STS
We evaluate the performance of SBERT for STS without using any STS specific training data  We use the STS tasks 2012 - 2016 BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF20, the STS benchmark BIBREF10, and the SICK-Relatedness dataset BIBREF21  These datasets provide labels between 0 and 5 on the semantic relatedness of sentence pairs  We showed in BIBREF22 that Pearson correlation is badly suited for STS  Instead, we compute the Spearman's rank correlation between the cosine-similarity of the sentence embeddings and the gold labels  The setup for the other sentence embedding methods is equivalent, the similarity is computed by cosine-similarity  The results are depicted in Table TABREF6 
The results shows that directly using the output of BERT leads to rather poor performances  Averaging the BERT embeddings achieves an average correlation of only 54 81, and using the CLS-token output only achieves an average correlation of 29 19  Both are worse than computing average GloVe embeddings 
Using the described siamese network structure and fine-tuning mechanism substantially improves the correlation, outperforming both InferSent and Universal Sentence Encoder substantially  The only dataset where SBERT performs worse than Universal Sentence Encoder is SICK-R  Universal Sentence Encoder was trained on various datasets, including news, question-answer pages and discussion forums, which appears to be more suitable to the data of SICK-R  In contrast, SBERT was pre-trained only on Wikipedia (via BERT) and on NLI data 
While RoBERTa was able to improve the performance for several supervised tasks, we only observe minor difference between SBERT and SRoBERTa for generating sentence embeddings 
Evaluation - Semantic Textual Similarity ::: Supervised STS
The STS benchmark (STSb) BIBREF10 provides is a popular dataset to evaluate supervised STS systems  The data includes 8,628 sentence pairs from the three categories captions, news, and forums  It is divided into train (5,749), dev (1,500) and test (1,379)  BERT set a new state-of-the-art performance on this dataset by passing both sentences to the network and using a simple regression method for the output 
We use the training set to fine-tune SBERT using the regression objective function  At prediction time, we compute the cosine-similarity between the sentence embeddings  All systems are trained with 10 random seeds to counter variances BIBREF23 
The results are depicted in Table TABREF10  We experimented with two setups: Only training on STSb, and first training on NLI, then training on STSb  We observe that the later strategy leads to a slight improvement of 1-2 points  This two-step approach had an especially large impact for the BERT cross-encoder, which improved the performance by 3-4 points  We do not observe a significant difference between BERT and RoBERTa 
Evaluation - Semantic Textual Similarity ::: Argument Facet Similarity
We evaluate SBERT on the Argument Facet Similarity (AFS) corpus by MisraEW16  The AFS corpus annotated 6,000 sentential argument pairs from social media dialogs on three controversial topics: gun control, gay marriage, and death penalty  The data was annotated on a scale from 0 (“different topic") to 5 (“completely equivalent")  The similarity notion in the AFS corpus is fairly different to the similarity notion in the STS datasets from SemEval  STS data is usually descriptive, while AFS data are argumentative excerpts from dialogs  To be considered similar, arguments must not only make similar claims, but also provide a similar reasoning  Further, the lexical gap between the sentences in AFS is much larger  Hence, simple unsupervised methods as well as state-of-the-art STS systems perform badly on this dataset BIBREF24 
We evaluate SBERT on this dataset in two scenarios: 1) As proposed by Misra et al , we evaluate SBERT using 10-fold cross-validation  A draw-back of this evaluation setup is that it is not clear how well approaches generalize to different topics  Hence, 2) we evaluate SBERT in a cross-topic setup  Two topics serve for training and the approach is evaluated on the left-out topic  We repeat this for all three topics and average the results 
SBERT is fine-tuned using the Regression Objective Function  The similarity score is computed using cosine-similarity based on the sentence embeddings  We also provide the Pearson correlation $r$ to make the results comparable to Misra et al  However, we showed BIBREF22 that Pearson correlation has some serious drawbacks and should be avoided for comparing STS systems  The results are depicted in Table TABREF12 
Unsupervised methods like tf-idf, average GloVe embeddings or InferSent perform rather badly on this dataset with low scores  Training SBERT in the 10-fold cross-validation setup gives a performance that is nearly on-par with BERT 
However, in the cross-topic evaluation, we observe a performance drop of SBERT by about 7 points Spearman correlation  To be considered similar, arguments should address the same claims and provide the same reasoning  BERT is able to use attention to compare directly both sentences (e g  word-by-word comparison), while SBERT must map individual sentences from an unseen topic to a vector space such that arguments with similar claims and reasons are close  This is a much more challenging task, which appears to require more than just two topics for training to work on-par with BERT 
Evaluation - Semantic Textual Similarity ::: Wikipedia Sections Distinction
ein-dor-etal-2018-learning use Wikipedia to create a thematically fine-grained train, dev and test set for sentence embeddings methods  Wikipedia articles are separated into distinct sections focusing on certain aspects  Dor et al  assume that sentences in the same section are thematically closer than sentences in different sections  They use this to create a large dataset of weakly labeled sentence triplets: The anchor and the positive example come from the same section, while the negative example comes from a different section of the same article  For example, from the Alice Arnold article: Anchor: Arnold joined the BBC Radio Drama Company in 1988 , positive: Arnold gained media attention in May 2012 , negative: Balding and Arnold are keen amateur golfers 
We use the dataset from Dor et al  We use the Triplet Objective, train SBERT for one epoch on the about 1 8 Million training triplets and evaluate it on the 222,957 test triplets  Test triplets are from a distinct set of Wikipedia articles  As evaluation metric, we use accuracy: Is the positive example closer to the anchor than the negative example?
Results are presented in Table TABREF14  Dor et al  fine-tuned a BiLSTM architecture with triplet loss to derive sentence embeddings for this dataset  As the table shows, SBERT clearly outperforms the BiLSTM approach by Dor et al 
Evaluation - SentEval
SentEval BIBREF6 is a popular toolkit to evaluate the quality of sentence embeddings  Sentence embeddings are used as features for a logistic regression classifier  The logistic regression classifier is trained on various tasks in a 10-fold cross-validation setup and the prediction accuracy is computed for the test-fold 
The purpose of SBERT sentence embeddings are not to be used for transfer learning for other tasks

ground_truth: Unanswerable

Unanswerable

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What other sentence embeddings methods are evaluated?
Context 3:  Here, we think fine-tuning BERT as described by devlin2018bert for new tasks is the more suitable method, as it updates all layers of the BERT network  However, SentEval can still give an impression on the quality of our sentence embeddings for various tasks 
We compare the SBERT sentence embeddings to other sentence embeddings methods on the following seven SentEval transfer tasks:
MR: Sentiment prediction for movie reviews snippets on a five start scale BIBREF25 
CR: Sentiment prediction of customer product reviews BIBREF26 
SUBJ: Subjectivity prediction of sentences from movie reviews and plot summaries BIBREF27 
MPQA: Phrase level opinion polarity classification from newswire BIBREF28 
SST: Stanford Sentiment Treebank with binary labels BIBREF29 
TREC: Fine grained question-type classification from TREC BIBREF30 
MRPC: Microsoft Research Paraphrase Corpus from parallel news sources BIBREF31 
The results can be found in Table TABREF15  SBERT is able to achieve the best performance in 5 out of 7 tasks  The average performance increases by about 2 percentage points compared to InferSent as well as the Universal Sentence Encoder  Even though transfer learning is not the purpose of SBERT, it outperforms other state-of-the-art sentence embeddings methods on this task 
It appears that the sentence embeddings from SBERT capture well sentiment information: We observe large improvements for all sentiment tasks (MR, CR, and SST) from SentEval in comparison to InferSent and Universal Sentence Encoder 
The only dataset where SBERT is significantly worse than Universal Sentence Encoder is the TREC dataset  Universal Sentence Encoder was pre-trained on question-answering data, which appears to be beneficial for the question-type classification task of the TREC dataset 
Average BERT embeddings or using the CLS-token output from a BERT network achieved bad results for various STS tasks (Table TABREF6), worse than average GloVe embeddings  However, for SentEval, average BERT embeddings and the BERT CLS-token output achieves decent results (Table TABREF15), outperforming average GloVe embeddings  The reason for this are the different setups  For the STS tasks, we used cosine-similarity to estimate the similarities between sentence embeddings  Cosine-similarity treats all dimensions equally  In contrast, SentEval fits a logistic regression classifier to the sentence embeddings  This allows that certain dimensions can have higher or lower impact on the classification result 
We conclude that average BERT embeddings / CLS-token output from BERT return sentence embeddings that are infeasible to be used with cosine-similarity or with Manhatten / Euclidean distance  For transfer learning, they yield slightly worse results than InferSent or Universal Sentence Encoder  However, using the described fine-tuning setup with a siamese network structure on NLI datasets yields sentence embeddings that achieve a new state-of-the-art for the SentEval toolkit 
Ablation Study
We have demonstrated strong empirical results for the quality of SBERT sentence embeddings  In this section, we perform an ablation study of different aspects of SBERT in order to get a better understanding of their relative importance 
We evaluated different pooling strategies (MEAN, MAX, and CLS)  For the classification objective function, we evaluate different concatenation methods  For each possible configuration, we train SBERT with 10 different random seeds and average the performances 
The objective function (classification vs  regression) depends on the annotated dataset  For the classification objective function, we train SBERT-base on the SNLI and the Multi-NLI dataset  For the regression objective function, we train on the training set of the STS benchmark dataset  Performances are measured on the development split of the STS benchmark dataset  Results are shown in Table TABREF23 
When trained with the classification objective function on NLI data, the pooling strategy has a rather minor impact  The impact of the concatenation mode is much larger  InferSent BIBREF4 and Universal Sentence Encoder BIBREF5 both use $(u, v, |u-v|, u*v)$ as input for a softmax classifier  However, in our architecture, adding the element-wise $u*v$ decreased the performance 
The most important component is the element-wise difference $|u-v|$  Note, that the concatenation mode is only relevant for training the softmax classifier  At inference, when predicting similarities for the STS benchmark dataset, only the sentence embeddings $u$ and $v$ are used in combination with cosine-similarity  The element-wise difference measures the distance between the dimensions of the two sentence embeddings, ensuring that similar pairs are closer and dissimilar pairs are further apart 
When trained with the regression objective function, we observe that the pooling strategy has a large impact  There, the MAX strategy perform significantly worse than MEAN or CLS-token strategy  This is in contrast to BIBREF4, who found it beneficial for the BiLSTM-layer of InferSent to use MAX instead of MEAN pooling 
Computational Efficiency
Sentence embeddings need potentially be computed for Millions of sentences, hence, a high computation speed is desired  In this section, we compare SBERT to average GloVe embeddings, InferSent BIBREF4, and Universal Sentence Encoder BIBREF5 
For our comparison we use the sentences from the STS benchmark BIBREF10  We compute average GloVe embeddings using a simple for-loop with python dictionary lookups and NumPy  InferSent is based on PyTorch  For Universal Sentence Encoder, we use the TensorFlow Hub version, which is based on TensorFlow  SBERT is based on PyTorch  For improved computation of sentence embeddings, we implemented a smart batching strategy: Sentences with similar lengths are grouped together and are only padded to the longest element in a mini-batch  This drastically reduces computational overhead from padding tokens 
Performances were measured on a server with Intel i7-5820K CPU @ 3 30GHz, Nvidia Tesla V100 GPU, CUDA 9 2 and cuDNN  The results are depicted in Table TABREF26 
On CPU, InferSent is about 65% faster than SBERT  This is due to the much simpler network architecture  InferSent uses a single BiLSTM layer, while BERT uses 12 stacked transformer layers  However, an advantage of transformer networks is the computational efficiency on GPUs  There, SBERT with smart batching is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder  Smart batching achieves a speed-up of 89% on CPU and 48% on GPU  Average GloVe embeddings is obviously by a large margin the fastest method to compute sentence embeddings 
Conclusion
We showed that BERT out-of-the-box maps sentences to a vector space that is rather unsuitable to be used with common similarity measures like cosine-similarity  The performance for seven STS tasks was below the performance of average GloVe embeddings 
To overcome this shortcoming, we presented Sentence-BERT (SBERT)  SBERT fine-tunes BERT in a siamese / triplet network architecture  We evaluated the quality on various common benchmarks, where it could achieve a significant improvement over state-of-the-art sentence embeddings methods  Replacing BERT with RoBERTa did not yield a significant improvement in our experiments 
SBERT is computationally efficient  On a GPU, it is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder  SBERT can be used for tasks which are computationally not feasible to be modeled with BERT  For example, clustering of 10,000 sentences with hierarchical clustering requires with BERT about 65 hours, as around 50 Million sentence combinations must be computed  With SBERT, we were able to reduce the effort to about 5 seconds 
Acknowledgments
This work has been supported by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1)  It has been co-funded by the German Federal Ministry of Education and Research (BMBF) under the promotional references 03VP02540 (ArgumenText) 

ground_truth: GloVe, BERT, Universal Sentence Encoder, TF-IDF, InferSent

Avg. GloVe embeddings, Avg. fast-text embeddings, Avg. BERT embeddings, BERT CLS-vector, InferSent - GloVe and Universal Sentence Encoder.