prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What other sentence embeddings methods are evaluated?
Context 1: Introduction
In this publication, we present Sentence-BERT (SBERT), a modification of the BERT network using siamese and triplet networks that is able to derive semantically meaningful sentence embeddings  This enables BERT to be used for certain new tasks, which up-to-now were not applicable for BERT  These tasks include large-scale semantic similarity comparison, clustering, and information retrieval via semantic search 
BERT set new state-of-the-art performance on various sentence classification and sentence-pair regression tasks  BERT uses a cross-encoder: Two sentences are passed to the transformer network and the target value is predicted  However, this setup is unsuitable for various pair regression tasks due to too many possible combinations  Finding in a collection of $n=10\,000$ sentences the pair with the highest similarity requires with BERT $n\cdot (n-1)/2=49\,995\,000$ inference computations  On a modern V100 GPU, this requires about 65 hours  Similar, finding which of the over 40 million existent questions of Quora is the most similar for a new question could be modeled as a pair-wise comparison with BERT, however, answering a single query would require over 50 hours 
A common method to address clustering and semantic search is to map each sentence to a vector space such that semantically similar sentences are close  Researchers have started to input individual sentences into BERT and to derive fixed-size sentence embeddings  The most commonly used approach is to average the BERT output layer (known as BERT embeddings) or by using the output of the first token (the [CLS] token)  As we will show, this common practice yields rather bad sentence embeddings, often worse than averaging GloVe embeddings BIBREF2 
To alleviate this issue, we developed SBERT  The siamese network architecture enables that fixed-sized vectors for input sentences can be derived  Using a similarity measure like cosine-similarity or Manhatten / Euclidean distance, semantically similar sentences can be found  These similarity measures can be performed extremely efficient on modern hardware, allowing SBERT to be used for semantic similarity search as well as for clustering  The complexity for finding the most similar sentence pair in a collection of 10,000 sentences is reduced from 65 hours with BERT to the computation of 10,000 sentence embeddings (5 seconds with SBERT) and computing cosine-similarity (0 01 seconds)  By using optimized index structures, finding the most similar Quora question can be reduced from 50 hours to a few milliseconds BIBREF3 
We fine-tune SBERT on NLI data, which creates sentence embeddings that significantly outperform other state-of-the-art sentence embedding methods like InferSent BIBREF4 and Universal Sentence Encoder BIBREF5  On seven Semantic Textual Similarity (STS) tasks, SBERT achieves an improvement of 11 7 points compared to InferSent and 5 5 points compared to Universal Sentence Encoder  On SentEval BIBREF6, an evaluation toolkit for sentence embeddings, we achieve an improvement of 2 1 and 2 6 points, respectively 
SBERT can be adapted to a specific task  It sets new state-of-the-art performance on a challenging argument similarity dataset BIBREF7 and on a triplet dataset to distinguish sentences from different sections of a Wikipedia article BIBREF8 
The paper is structured in the following way: Section SECREF3 presents SBERT, section SECREF4 evaluates SBERT on common STS tasks and on the challenging Argument Facet Similarity (AFS) corpus BIBREF7  Section SECREF5 evaluates SBERT on SentEval  In section SECREF6, we perform an ablation study to test some design aspect of SBERT  In section SECREF7, we compare the computational efficiency of SBERT sentence embeddings in contrast to other state-of-the-art sentence embedding methods 
Related Work
We first introduce BERT, then, we discuss state-of-the-art sentence embedding methods 
BERT BIBREF0 is a pre-trained transformer network BIBREF9, which set for various NLP tasks new state-of-the-art results, including question answering, sentence classification, and sentence-pair regression  The input for BERT for sentence-pair regression consists of the two sentences, separated by a special [SEP] token  Multi-head attention over 12 (base-model) or 24 layers (large-model) is applied and the output is passed to a simple regression function to derive the final label  Using this setup, BERT set a new state-of-the-art performance on the Semantic Textual Semilarity (STS) benchmark BIBREF10  RoBERTa BIBREF1 showed, that the performance of BERT can further improved by small adaptations to the pre-training process  We also tested XLNet BIBREF11, but it led in general to worse results than BERT 
A large disadvantage of the BERT network structure is that no independent sentence embeddings are computed, which makes it difficult to derive sentence embeddings from BERT  To bypass this limitations, researchers passed single sentences through BERT and then derive a fixed sized vector by either averaging the outputs (similar to average word embeddings) or by using the output of the special CLS token (for example: bertsentenceembeddings1,bertsentenceembeddings2,bertsentenceembeddings3)  These two options are also provided by the popular bert-as-a-service-repository  Up to our knowledge, there is so far no evaluation if these methods lead to useful sentence embeddings 
Sentence embeddings are a well studied area with dozens of proposed methods  Skip-Thought BIBREF12 trains an encoder-decoder architecture to predict the surrounding sentences  InferSent BIBREF4 uses labeled data of the Stanford Natural Language Inference dataset BIBREF13 and the Multi-Genre NLI dataset BIBREF14 to train a siamese BiLSTM network with max-pooling over the output  Conneau et al  showed, that InferSent consistently outperforms unsupervised methods like SkipThought  Universal Sentence Encoder BIBREF5 trains a transformer network and augments unsupervised learning with training on SNLI  hill-etal-2016-learning showed, that the task on which sentence embeddings are trained significantly impacts their quality  Previous work BIBREF4, BIBREF5 found that the SNLI datasets are suitable for training sentence embeddings  yang-2018-learning presented a method to train on conversations from Reddit using siamese DAN and siamese transformer networks, which yielded good results on the STS benchmark dataset 
polyencoders addresses the run-time overhead of the cross-encoder from BERT and present a method (poly-encoders) to compute a score between $m$ context vectors and pre-computed candidate embeddings using attention  This idea works for finding the highest scoring sentence in a larger collection  However, poly-encoders have the drawback that the score function is not symmetric and the computational overhead is too large for use-cases like clustering, which would require $O(n^2)$ score computations 
Previous neural sentence embedding methods started the training from a random initialization  In this publication, we use the pre-trained BERT and RoBERTa network and only fine-tune it to yield useful sentence embeddings  This reduces significantly the needed training time: SBERT can be tuned in less than 20 minutes, while yielding better results than comparable sentence embedding methods 
Model
SBERT adds a pooling operation to the output of BERT / RoBERTa to derive a fixed sized sentence embedding  We experiment with three pooling strategies: Using the output of the CLS-token, computing the mean of all output vectors (MEAN-strategy), and computing a max-over-time of the output vectors (MAX-strategy)  The default configuration is MEAN 
In order to fine-tune BERT / RoBERTa, we create siamese and triplet networks BIBREF15 to update the weights such that the produced sentence embeddings are semantically meaningful and can be compared with cosine-similarity 
The network structure depends on the available training data  We experiment with the following structures and objective functions 
Classification Objective Function  We concatenate the sentence embeddings $u$ and $v$ with the element-wise difference $|u-v|$ and multiply it with the trainable weight $W_t \in \mathbb {R}^{3n \times k}$:
where $n$ is the dimension of the sentence embeddings and $k$ the number of labels  We optimize cross-entropy loss  This structure is depicted in Figure FIGREF4 
Regression Objective Function  The cosine-similarity between the two sentence embeddings $u$ and $v$ is computed (Figure FIGREF5)  We use mean-squared-error loss as the objective function 
Triplet Objective Function  Given an anchor sentence $a$, a positive sentence $p$, and a negative sentence $n$, triplet loss tunes the network such that the distance between $a$ and $p$ is smaller than the distance between $a$ and $n$  Mathematically, we minimize the following loss function:
with $s_x$ the sentence embedding for $a$/$n$/$p$, $||\cdot ||$ a distance metric and margin $\epsilon $  Margin $\epsilon $ ensures that $s_p$ is at least $\epsilon $ closer to $s_a$ than $s_n$  As metric we use Euclidean distance and we set $\epsilon =1$ in our experiments 
Model ::: Training Details
We train SBERT on the combination of the SNLI BIBREF13 and the Multi-Genre NLI BIBREF14 dataset  The SNLI is a collection of 570,000 sentence pairs annotated with the labels contradiction, eintailment, and neutral  MultiNLI contains 430,000 sentence pairs and covers a range of genres of spoken and written text  We fine-tune SBERT with a 3-way softmax-classifier objective function for one epoch  We used a batch-size of 16, Adam optimizer with learning rate $2\mathrm {e}{-5}$, and a linear learning rate warm-up over 10% of the training data  Our default pooling strategy is MEAN 
Evaluation - Semantic Textual Similarity
We evaluate the performance of SBERT for common Semantic Textual Similarity (STS) tasks  State-of-the-art methods often learn a (complex) regression function that maps sentence embeddings to a similarity score  However, these regression functions work pair-wise and due to the combinatorial explosion those are often not scalable if the collection of sentences reaches a certain size  Instead, we always use cosine-similarity to compare the similarity between two sentence embeddings  We ran our experiments also with negative Manhatten and negative Euclidean distances as similarity measures, but the results for all approaches remained roughly the same 
Evaluation - Semantic Textual Similarity ::: Unsupervised STS
We evaluate the performance of SBERT for STS without using any STS specific training data  We use the STS tasks 2012 - 2016 BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF20, the STS benchmark BIBREF10, and the SICK-Relatedness dataset BIBREF21  These datasets provide labels between 0 and 5 on the semantic relatedness of sentence pairs  We showed in BIBREF22 that Pearson correlation is badly suited for STS  Instead, we compute the Spearman's rank correlation between the cosine-similarity of the sentence embeddings and the gold labels  The setup for the other sentence embedding methods is equivalent, the similarity is computed by cosine-similarity  The results are depicted in Table TABREF6 
The results shows that directly using the output of BERT leads to rather poor performances  Averaging the BERT embeddings achieves an average correlation of only 54 81, and using the CLS-token output only achieves an average correlation of 29 19  Both are worse than computing average GloVe embeddings 
Using the described siamese network structure and fine-tuning mechanism substantially improves the correlation, outperforming both InferSent and Universal Sentence Encoder substantially  The only dataset where SBERT performs worse than Universal Sentence Encoder is SICK-R  Universal Sentence Encoder was trained on various datasets, including news, question-answer pages and discussion forums, which appears to be more suitable to the data of SICK-R  In contrast, SBERT was pre-trained only on Wikipedia (via BERT) and on NLI data 
While RoBERTa was able to improve the performance for several supervised tasks, we only observe minor difference between SBERT and SRoBERTa for generating sentence embeddings 
Evaluation - Semantic Textual Similarity ::: Supervised STS
The STS benchmark (STSb) BIBREF10 provides is a popular dataset to evaluate supervised STS systems  The data includes 8,628 sentence pairs from the three categories captions, news, and forums  It is divided into train (5,749), dev (1,500) and test (1,379)  BERT set a new state-of-the-art performance on this dataset by passing both sentences to the network and using a simple regression method for the output 
We use the training set to fine-tune SBERT using the regression objective function  At prediction time, we compute the cosine-similarity between the sentence embeddings  All systems are trained with 10 random seeds to counter variances BIBREF23 
The results are depicted in Table TABREF10  We experimented with two setups: Only training on STSb, and first training on NLI, then training on STSb  We observe that the later strategy leads to a slight improvement of 1-2 points  This two-step approach had an especially large impact for the BERT cross-encoder, which improved the performance by 3-4 points  We do not observe a significant difference between BERT and RoBERTa 
Evaluation - Semantic Textual Similarity ::: Argument Facet Similarity
We evaluate SBERT on the Argument Facet Similarity (AFS) corpus by MisraEW16  The AFS corpus annotated 6,000 sentential argument pairs from social media dialogs on three controversial topics: gun control, gay marriage, and death penalty  The data was annotated on a scale from 0 (“different topic") to 5 (“completely equivalent")  The similarity notion in the AFS corpus is fairly different to the similarity notion in the STS datasets from SemEval  STS data is usually descriptive, while AFS data are argumentative excerpts from dialogs  To be considered similar, arguments must not only make similar claims, but also provide a similar reasoning  Further, the lexical gap between the sentences in AFS is much larger  Hence, simple unsupervised methods as well as state-of-the-art STS systems perform badly on this dataset BIBREF24 
We evaluate SBERT on this dataset in two scenarios: 1) As proposed by Misra et al , we evaluate SBERT using 10-fold cross-validation  A draw-back of this evaluation setup is that it is not clear how well approaches generalize to different topics  Hence, 2) we evaluate SBERT in a cross-topic setup  Two topics serve for training and the approach is evaluated on the left-out topic  We repeat this for all three topics and average the results 
SBERT is fine-tuned using the Regression Objective Function  The similarity score is computed using cosine-similarity based on the sentence embeddings  We also provide the Pearson correlation $r$ to make the results comparable to Misra et al  However, we showed BIBREF22 that Pearson correlation has some serious drawbacks and should be avoided for comparing STS systems  The results are depicted in Table TABREF12 
Unsupervised methods like tf-idf, average GloVe embeddings or InferSent perform rather badly on this dataset with low scores  Training SBERT in the 10-fold cross-validation setup gives a performance that is nearly on-par with BERT 
However, in the cross-topic evaluation, we observe a performance drop of SBERT by about 7 points Spearman correlation  To be considered similar, arguments should address the same claims and provide the same reasoning  BERT is able to use attention to compare directly both sentences (e g  word-by-word comparison), while SBERT must map individual sentences from an unseen topic to a vector space such that arguments with similar claims and reasons are close  This is a much more challenging task, which appears to require more than just two topics for training to work on-par with BERT 
Evaluation - Semantic Textual Similarity ::: Wikipedia Sections Distinction
ein-dor-etal-2018-learning use Wikipedia to create a thematically fine-grained train, dev and test set for sentence embeddings methods  Wikipedia articles are separated into distinct sections focusing on certain aspects  Dor et al  assume that sentences in the same section are thematically closer than sentences in different sections  They use this to create a large dataset of weakly labeled sentence triplets: The anchor and the positive example come from the same section, while the negative example comes from a different section of the same article  For example, from the Alice Arnold article: Anchor: Arnold joined the BBC Radio Drama Company in 1988 , positive: Arnold gained media attention in May 2012 , negative: Balding and Arnold are keen amateur golfers 
We use the dataset from Dor et al  We use the Triplet Objective, train SBERT for one epoch on the about 1 8 Million training triplets and evaluate it on the 222,957 test triplets  Test triplets are from a distinct set of Wikipedia articles  As evaluation metric, we use accuracy: Is the positive example closer to the anchor than the negative example?
Results are presented in Table TABREF14  Dor et al  fine-tuned a BiLSTM architecture with triplet loss to derive sentence embeddings for this dataset  As the table shows, SBERT clearly outperforms the BiLSTM approach by Dor et al 
Evaluation - SentEval
SentEval BIBREF6 is a popular toolkit to evaluate the quality of sentence embeddings  Sentence embeddings are used as features for a logistic regression classifier  The logistic regression classifier is trained on various tasks in a 10-fold cross-validation setup and the prediction accuracy is computed for the test-fold 
The purpose of SBERT sentence embeddings are not to be used for transfer learning for other tasks  Here, we think fine-tuning BERT as described by devlin2018bert for new tasks is the more suitable method, as it updates all layers of the BERT network  However, SentEval can still give an impression on the quality of our sentence embeddings for various tasks 
We compare the SBERT sentence embeddings to other sentence embeddings methods on the following seven SentEval transfer tasks:
MR: Sentiment prediction for movie reviews snippets on a five start scale BIBREF25 
CR: Sentiment prediction of customer product reviews BIBREF26 
SUBJ: Subjectivity prediction of sentences from movie reviews and plot summaries BIBREF27 
MPQA: Phrase level opinion polarity classification from newswire BIBREF28 
SST: Stanford Sentiment Treebank with binary labels BIBREF29 
TREC: Fine grained question-type classification from TREC BIBREF30 
MRPC: Microsoft Research Paraphrase Corpus from parallel news sources BIBREF31 
The results can be found in Table TABREF15  SBERT is able to achieve the best performance in 5 out of 7 tasks  The average performance increases by about 2 percentage points compared to InferSent as well as the Universal Sentence Encoder  Even though transfer learning is not the purpose of SBERT, it outperforms other state-of-the-art sentence embeddings methods on this task 
It appears that the sentence embeddings from SBERT capture well sentiment information: We observe large improvements for all sentiment tasks (MR, CR, and SST) from SentEval in comparison to InferSent and Universal Sentence Encoder 
The only dataset where SBERT is significantly worse than Universal Sentence Encoder is the TREC dataset  Universal Sentence Encoder was pre-trained on question-answering data, which appears to be beneficial for the question-type classification task of the TREC dataset 
Average BERT embeddings or using the CLS-token output from a BERT network achieved bad results for various STS tasks (Table TABREF6), worse than average GloVe embeddings  However, for SentEval, average BERT embeddings and the BERT CLS-token output achieves decent results (Table TABREF15), outperforming average GloVe embeddings  The reason for this are the different setups  For the STS tasks, we used cosine-similarity to estimate the similarities between sentence embeddings  Cosine-similarity treats all dimensions equally  In contrast, SentEval fits a logistic regression classifier to the sentence embeddings  This allows that certain dimensions can have higher or lower impact on the classification result 
We conclude that average BERT embeddings / CLS-token output from BERT return sentence embeddings that are infeasible to be used with cosine-similarity or with Manhatten / Euclidean distance  For transfer learning, they yield slightly worse results than InferSent or Universal Sentence Encoder  However, using the described fine-tuning setup with a siamese network structure on NLI datasets yields sentence embeddings that achieve a new state-of-the-art for the SentEval toolkit 
Ablation Study
We have demonstrated strong empirical results for the quality of SBERT sentence embeddings  In this section, we perform an ablation study of different aspects of SBERT in order to get a better understanding of their relative importance 
We evaluated different pooling strategies (MEAN, MAX, and CLS)  For the classification objective function, we evaluate different concatenation methods  For each possible configuration, we train SBERT with 10 different random seeds and average the performances 
The objective function (classification vs  regression) depends on the annotated dataset  For the classification objective function, we train SBERT-base on the SNLI and the Multi-NLI dataset  For the regression objective function, we train on the training set of the STS benchmark dataset  Performances are measured on the development split of the STS benchmark dataset  Results are shown in Table TABREF23 
When trained with the classification objective function on NLI data, the pooling strategy has a rather minor impact  The impact of the concatenation mode is much larger  InferSent BIBREF4 and Universal Sentence Encoder BIBREF5 both use $(u, v, |u-v|, u*v)$ as input for a softmax classifier  However, in our architecture, adding the element-wise $u*v$ decreased the performance 
The most important component is the element-wise difference $|u-v|$  Note, that the concatenation mode is only relevant for training the softmax classifier  At inference, when predicting similarities for the STS benchmark dataset, only the sentence embeddings $u$ and $v$ are used in combination with cosine-similarity  The element-wise difference measures the distance between the dimensions of the two sentence embeddings, ensuring that similar pairs are closer and dissimilar pairs are further apart 
When trained with the regression objective function, we observe that the pooling strategy has a large impact  There, the MAX strategy perform significantly worse than MEAN or CLS-token strategy  This is in contrast to BIBREF4, who found it beneficial for the BiLSTM-layer of InferSent to use MAX instead of MEAN pooling 
Computational Efficiency
Sentence embeddings need potentially be computed for Millions of sentences, hence, a high computation speed is desired  In this section, we compare SBERT to average GloVe embeddings, InferSent BIBREF4, and Universal Sentence Encoder BIBREF5 
For our comparison we use the sentences from the STS benchmark BIBREF10  We compute average GloVe embeddings using a simple for-loop with python dictionary lookups and NumPy  InferSent is based on PyTorch  For Universal Sentence Encoder, we use the TensorFlow Hub version, which is based on TensorFlow  SBERT is based on PyTorch  For improved computation of sentence embeddings, we implemented a smart batching strategy: Sentences with similar lengths are grouped together and are only padded to the longest element in a mini-batch  This drastically reduces computational overhead from padding tokens 
Performances were measured on a server with Intel i7-5820K CPU @ 3 30GHz, Nvidia Tesla V100 GPU, CUDA 9 2 and cuDNN  The results are depicted in Table TABREF26 
On CPU, InferSent is about 65% faster than SBERT  This is due to the much simpler network architecture  InferSent uses a single BiLSTM layer, while BERT uses 12 stacked transformer layers  However, an advantage of transformer networks is the computational efficiency on GPUs  There, SBERT with smart batching is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder  Smart batching achieves a speed-up of 89% on CPU and 48% on GPU  Average GloVe embeddings is obviously by a large margin the fastest method to compute sentence embeddings 
Conclusion
We showed that BERT out-of-the-box maps sentences to a vector space that is rather unsuitable to be used with common similarity measures like cosine-similarity  The performance for seven STS tasks was below the performance of average GloVe embeddings 
To overcome this shortcoming, we presented Sentence-BERT (SBERT)  SBERT fine-tunes BERT in a siamese / triplet network architecture  We evaluated the quality on various common benchmarks, where it could achieve a significant improvement over state-of-the-art sentence embeddings methods  Replacing BERT with RoBERTa did not yield a significant improvement in our experiments 
SBERT is computationally efficient  On a GPU, it is about 9% faster than InferSent and about 55% faster than Universal Sentence Encoder  SBERT can be used for tasks which are computationally not feasible to be modeled with BERT  For example, clustering of 10,000 sentences with hierarchical clustering requires with BERT about 65 hours, as around 50 Million sentence combinations must be computed  With SBERT, we were able to reduce the effort to about 5 seconds 
Acknowledgments
This work has been supported by the German Research Foundation through the German-Israeli Project Cooperation (DIP, grant DA 1600/1-1 and grant GU 798/17-1)  It has been co-funded by the German Federal Ministry of Education and Research (BMBF) under the promotional references 03VP02540 (ArgumenText) 
Context 2: Introduction
Language model pretraining has advanced the state of the art in many NLP tasks ranging from sentiment analysis, to question answering, natural language inference, named entity recognition, and textual similarity  State-of-the-art pretrained models include ELMo BIBREF1, GPT BIBREF2, and more recently Bidirectional Encoder Representations from Transformers (Bert; BIBREF0)  Bert combines both word and sentence representations in a single very large Transformer BIBREF3; it is pretrained on vast amounts of text, with an unsupervised objective of masked language modeling and next-sentence prediction and can be fine-tuned with various task-specific objectives 
In most cases, pretrained language models have been employed as encoders for sentence- and paragraph-level natural language understanding problems BIBREF0 involving various classification tasks (e g , predicting whether any two sentences are in an entailment relationship; or determining the completion of a sentence among four alternative sentences)  In this paper, we examine the influence of language model pretraining on text summarization  Different from previous tasks, summarization requires wide-coverage natural language understanding going beyond the meaning of individual words and sentences  The aim is to condense a document into a shorter version while preserving most of its meaning  Furthermore, under abstractive modeling formulations, the task requires language generation capabilities in order to create summaries containing novel words and phrases not featured in the source text, while extractive summarization is often defined as a binary classification task with labels indicating whether a text span (typically a sentence) should be included in the summary 
We explore the potential of Bert for text summarization under a general framework encompassing both extractive and abstractive modeling paradigms  We propose a novel document-level encoder based on Bert which is able to encode a document and obtain representations for its sentences  Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers to capture document-level features for extracting sentences  Our abstractive model adopts an encoder-decoder architecture, combining the same pretrained Bert encoder with a randomly-initialized Transformer decoder BIBREF3  We design a new training schedule which separates the optimizers of the encoder and the decoder in order to accommodate the fact that the former is pretrained while the latter must be trained from scratch  Finally, motivated by previous work showing that the combination of extractive and abstractive objectives can help generate better summaries BIBREF4, we present a two-stage approach where the encoder is fine-tuned twice, first with an extractive objective and subsequently on the abstractive summarization task 
We evaluate the proposed approach on three single-document news summarization datasets representative of different writing conventions (e g , important information is concentrated at the beginning of the document or distributed more evenly throughout) and summary styles (e g , verbose vs  more telegraphic; extractive vs  abstractive)  Across datasets, we experimentally show that the proposed models achieve state-of-the-art results under both extractive and abstractive settings  Our contributions in this work are three-fold: a) we highlight the importance of document encoding for the summarization task; a variety of recently proposed techniques aim to enhance summarization performance via copying mechanisms BIBREF5, BIBREF6, BIBREF7, reinforcement learning BIBREF8, BIBREF9, BIBREF10, and multiple communicating encoders BIBREF11  We achieve better results with a minimum-requirement model without using any of these mechanisms; b) we showcase ways to effectively employ pretrained language models in summarization under both extractive and abstractive settings; we would expect any improvements in model pretraining to translate in better summarization in the future; and c) the proposed models can be used as a stepping stone to further improve summarization performance as well as baselines against which new proposals are tested 
Background ::: Pretrained Language Models
Pretrained language models BIBREF1, BIBREF2, BIBREF0, BIBREF12, BIBREF13 have recently emerged as a key technology for achieving impressive gains in a wide variety of natural language tasks  These models extend the idea of word embeddings by learning contextual representations from large-scale corpora using a language modeling objective  Bidirectional Encoder Representations from Transformers (Bert; BIBREF0) is a new language representation model which is trained with a masked language modeling and a “next sentence prediction” task on a corpus of 3,300M words 
The general architecture of Bert is shown in the left part of Figure FIGREF2  Input text is first preprocessed by inserting two special tokens  [cls] is appended to the beginning of the text; the output representation of this token is used to aggregate information from the whole sequence (e g , for classification tasks)  And token [sep] is inserted after each sentence as an indicator of sentence boundaries  The modified text is then represented as a sequence of tokens $X=[w_1,w_2,\cdots ,w_n]$  Each token $w_i$ is assigned three kinds of embeddings: token embeddings indicate the meaning of each token, segmentation embeddings are used to discriminate between two sentences (e g , during a sentence-pair classification task) and position embeddings indicate the position of each token within the text sequence  These three embeddings are summed to a single input vector $x_i$ and fed to a bidirectional Transformer with multiple layers:
where $h^0=x$ are the input vectors; $\mathrm {LN}$ is the layer normalization operation BIBREF14; $\mathrm {MHAtt}$ is the multi-head attention operation BIBREF3; superscript $l$ indicates the depth of the stacked layer  On the top layer, Bert will generate an output vector $t_i$ for each token with rich contextual information 
Pretrained language models are usually used to enhance performance in language understanding tasks  Very recently, there have been attempts to apply pretrained models to various generation problems BIBREF15, BIBREF16  When fine-tuning for a specific task, unlike ELMo whose parameters are usually fixed, parameters in Bert are jointly fine-tuned with additional task-specific parameters 
Background ::: Extractive Summarization
Extractive summarization systems create a summary by identifying (and subsequently concatenating) the most important sentences in a document  Neural models consider extractive summarization as a sentence classification problem: a neural encoder creates sentence representations and a classifier predicts which sentences should be selected as summaries  SummaRuNNer BIBREF7 is one of the earliest neural approaches adopting an encoder based on Recurrent Neural Networks  Refresh BIBREF8 is a reinforcement learning-based system trained by globally optimizing the ROUGE metric  More recent work achieves higher performance with more sophisticated model structures  Latent BIBREF17 frames extractive summarization as a latent variable inference problem; instead of maximizing the likelihood of “gold” standard labels, their latent model directly maximizes the likelihood of human summaries given selected sentences  Sumo BIBREF18 capitalizes on the notion of structured attention to induce a multi-root dependency tree representation of the document while predicting the output summary  NeuSum BIBREF19 scores and selects sentences jointly and represents the state of the art in extractive summarization 
Background ::: Abstractive Summarization
Neural approaches to abstractive summarization conceptualize the task as a sequence-to-sequence problem, where an encoder maps a sequence of tokens in the source document $\mathbf {x} = [x_1,    , x_n]$ to a sequence of continuous representations $\mathbf {z} = [z_1,    , z_n]$, and a decoder then generates the target summary $\mathbf {y} = [y_1,    , y_m]$ token-by-token, in an auto-regressive manner, hence modeling the conditional probability: $p(y_1,    , y_m|x_1,    , x_n)$ 
BIBREF20 and BIBREF21 were among the first to apply the neural encoder-decoder architecture to text summarization  BIBREF6 enhance this model with a pointer-generator network (PTgen) which allows it to copy words from the source text, and a coverage mechanism (Cov) which keeps track of words that have been summarized  BIBREF11 propose an abstractive system where multiple agents (encoders) represent the document together with a hierarchical attention mechanism (over the agents) for decoding  Their Deep Communicating Agents (DCA) model is trained end-to-end with reinforcement learning  BIBREF9 also present a deep reinforced model (DRM) for abstractive summarization which handles the coverage problem with an intra-attention mechanism where the decoder attends over previously generated words  BIBREF4 follow a bottom-up approach (BottomUp); a content selector first determines which phrases in the source document should be part of the summary, and a copy mechanism is applied only to preselected phrases during decoding  BIBREF22 propose an abstractive model which is particularly suited to extreme summarization (i e , single sentence summaries), based on convolutional neural networks and additionally conditioned on topic distributions (TConvS2S) 
Fine-tuning Bert for Summarization ::: Summarization Encoder
Although Bert has been used to fine-tune various NLP tasks, its application to summarization is not as straightforward  Since Bert is trained as a masked-language model, the output vectors are grounded to tokens instead of sentences, while in extractive summarization, most models manipulate sentence-level representations  Although segmentation embeddings represent different sentences in Bert, they only apply to sentence-pair inputs, while in summarization we must encode and manipulate multi-sentential inputs  Figure FIGREF2 illustrates our proposed Bert architecture for Summarization (which we call BertSum) 
In order to represent individual sentences, we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it  We also use interval segment embeddings to distinguish multiple sentences within a document  For $sent_i$ we assign segment embedding $E_A$ or $E_B$ depending on whether $i$ is odd or even  For example, for document $[sent_1, sent_2, sent_3, sent_4, sent_5]$, we would assign embeddings $[E_A, E_B, E_A,E_B, E_A]$  This way, document representations are learned hierarchically where lower Transformer layers represent adjacent sentences, while higher layers, in combination with self-attention, represent multi-sentence discourse 
Position embeddings in the original Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings that are initialized randomly and fine-tuned with other parameters in the encoder 
Fine-tuning Bert for Summarization ::: Extractive Summarization
Let $d$ denote a document containing sentences $[sent_1, sent_2, \cdots , sent_m]$, where $sent_i$ is the $i$-th sentence in the document  Extractive summarization can be defined as the task of assigning a label $y_i \in \lbrace 0, 1\rbrace $ to each $sent_i$, indicating whether the sentence should be included in the summary  It is assumed that summary sentences represent the most important content of the document 
With BertSum, vector $t_i$ which is the vector of the $i$-th [cls] symbol from the top layer can be used as the representation for $sent_i$  Several inter-sentence Transformer layers are then stacked on top of Bert outputs, to capture document-level features for extracting summaries:
where $h^0=\mathrm {PosEmb}(T)$; $T$ denotes the sentence vectors output by BertSum, and function $\mathrm {PosEmb}$ adds sinusoid positional embeddings BIBREF3 to $T$, indicating the position of each sentence 
The final output layer is a sigmoid classifier:
where $h^L_i$ is the vector for $sent_i$ from the top layer (the $L$-th layer ) of the Transformer  In experiments, we implemented Transformers with $L=1, 2, 3$ and found that a Transformer with $L=2$ performed best  We name this model BertSumExt 
The loss of the model is the binary classification entropy of prediction $\hat{y}_i$ against gold label $y_i$  Inter-sentence Transformer layers are jointly fine-tuned with BertSum  We use the Adam optimizer with $\beta _1=0 9$, and $\beta _2=0 999$)  Our learning rate schedule follows BIBREF3 with warming-up ($ \operatorname{\operatorname{warmup}}=10,000$):
Fine-tuning Bert for Summarization ::: Abstractive Summarization
We use a standard encoder-decoder framework for abstractive summarization BIBREF6  The encoder is the pretrained BertSum and the decoder is a 6-layered Transformer initialized randomly  It is conceivable that there is a mismatch between the encoder and the decoder, since the former is pretrained while the latter must be trained from scratch  This can make fine-tuning unstable; for example, the encoder might overfit the data while the decoder underfits, or vice versa  To circumvent this, we design a new fine-tuning schedule which separates the optimizers of the encoder and the decoder 
We use two Adam optimizers with $\beta _1=0 9$ and $\beta _2=0 999$ for the encoder and the decoder, respectively, each with different warmup-steps and learning rates:
where $\tilde{lr}_{\mathcal {E}}=2e^{-3}$, and $\operatorname{\operatorname{warmup}}_{\mathcal {E}}=20,000$ for the encoder and $\tilde{lr}_{\mathcal {D}}=0 1$, and $\operatorname{\operatorname{warmup}}_{\mathcal {D}}=10,000$ for the decoder  This is based on the assumption that the pretrained encoder should be fine-tuned with a smaller learning rate and smoother decay (so that the encoder can be trained with more accurate gradients when the decoder is becoming stable) 
In addition, we propose a two-stage fine-tuning approach, where we first fine-tune the encoder on the extractive summarization task (Section SECREF8) and then fine-tune it on the abstractive summarization task (Section SECREF13)  Previous work BIBREF4, BIBREF23 suggests that using extractive objectives can boost the performance of abstractive summarization  Also notice that this two-stage approach is conceptually very simple, the model can take advantage of information shared between these two tasks, without fundamentally changing its architecture  We name the default abstractive model BertSumAbs and the two-stage fine-tuned model BertSumExtAbs 
Experimental Setup
In this section, we describe the summarization datasets used in our experiments and discuss various implementation details 
Experimental Setup ::: Summarization Datasets
We evaluated our model on three benchmark datasets, namely the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), and XSum BIBREF22  These datasets represent different summary styles ranging from highlights to very brief one sentence summaries  The summaries also vary with respect to the type of rewriting operations they exemplify (e g , some showcase more cut and paste operations while others are genuinely abstractive)  Table TABREF12 presents statistics on these datasets (test set); example (gold-standard) summaries are provided in the supplementary material
Context 3: Introduction
Natural Language Processing (NLP) has increasingly attracted the attention of the financial community  This trend can be explained by at least three major factors  The first factor refers to the business perspective  It is the economics of gaining competitive advantage using alternative sources of data and going beyond historical stock prices, thus, trading by analyzing market news automatically  The second factor is the major advancements in the technologies to collect, store, and query massive amounts of user-generated data almost in real-time  The third factor refers to the progress made by the NLP community in understanding unstructured text  Over the last decades the number of studies using NLP for financial forecasting has experienced exponential growth  According to BIBREF0 , until 2008, less than five research articles were published per year mentioning both “stock market” and “text mining” or “sentiment analysis” keywords  In 2012, this number increased to slightly more than ten articles per year  The last numbers available for 2016 indicates this has increased to sixty articles per year 
The ability to mechanically harvest the sentiment from texts using NLP has shed light on conflicting theories of financial economics  Historically, there has been two differing views on whether disagreement among market participants induces more trades  The “non-trade theorem” BIBREF1 states that assuming all market participants have common knowledge about a market event, the level of disagreement among the participants does not increase the number of trades but only leads to a revision of the market quotes  In contrast, the theoretically framework proposed in BIBREF2 advocates that disagreement among market participants increases trading volume  Using textual data from Yahoo and RagingBull com message boards to measure the dispersion of opinions (positive or negative) among traders, it was shown in BIBREF3 that disagreement among users' messages helps to predict subsequent trading volume and volatility  Similar relation between disagreement and increased trading volume was found in BIBREF4 using Twitter posts  Additionally, textual analysis is adding to the theories of medium-term/long-term momentum/reversal in stock markets BIBREF5   The unified Hong and Stein model BIBREF6 on stock's momentum/reversal proposes that investors underreact to news, causing slow price drifts, and overreact to price shocks not accompanied by news, hence inducing reversals  This theoretical predicated behaviour between price and news was systematically estimated and supported in BIBREF7 , BIBREF8 using financial media headlines and in BIBREF9 using the Consumer Confidence Index® published by The Conference Board BIBREF10   Similarly, BIBREF11 uses the Harvard IV-4 sentiment lexicon to count the occurrence of words with positive and negative connotation of the Wall Street Journal showing that negative sentiment is a good predictor of price returns and trading volumes 
Accurate models for forecasting both price returns and volatility are equally important in the financial domain  Volatility measures how wildly the asset is expected to oscillate in a given time period and is related to the second moment of the price return distribution  In general terms, forecasting price returns is relevant to take speculative positions  The volatility, on the other hand, measures the risk of these positions  On a daily basis, financial institutions need to assess the short-term risk of their portfolios  Measuring the risk is essential in many aspects  It is imperative for regulatory capital disclosures required by banking supervision bodies  Moreover, it is useful to dynamically adjust position sizing accordingly to market conditions, thus, maintaining the risk within reasonable levels 
Although, it is crucial to predict the short-term volatility from the financial markets application perspective, much of the current NLP research on volatility forecasting focus on the volatility prediction for very long-term horizons (see BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 )  Predominately, these works are built on extensions of the bag-of-words representation that has the main drawback of not capturing word order  Financial forecasting, however, requires the ability to capture semantics that is dependent upon word order  For example, the headline “Qualcomm sues Apple for contract breach” and “Apple sues Qualcomm for contract breach” trigger different responses for each stock and for the market aggregated index, however, they share the same bag-of-words representation  Additionally, these works use features from a pretrained sentiment analyis model to train the financial forecasting model  A key limitation of this process is that it requires a labelled sentiment dataset  Additionally, the error propagation is not end-to-end  In this work, we fill in the gaps of volatility prediction research in the following manner:
Related work
Previous work in BIBREF12 incorporates sections of the “Form 10-K” to predict the volatility twelve months after the report is released  They train a Support Vector Regression model on top of sparse representation (bag-of-words) with standard term weighting (e g  Term-Frequency)  This work was extended in BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 by employing the Loughran-McDonald Sentiment Word Lists BIBREF20 , which contain three lists where words are grouped by their sentiments (positive, negative and neutral)  In all these works, the textual representation is engineered using the following steps: 1) For each sentiment group, the list is expanded by retrieving 20 most similar words for each word using Word2Vec word embeddings BIBREF21   2) Finally, each 10-K document is represented using the expanded lists of words  The weight of each word in this sparse representation is defined using Information Retrieval (IR) methods such as term-frequency (tf) and term-frequency with inverted document frequency (tfidf)  Particularly, BIBREF16 shows that results can be improved using enhanced IR methods and projecting each sparse feature into a dense space using Principal Component Analysis (PCA) 
The works described above ( BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 ) target long-horizon volatility predictions (one year or quarterly BIBREF16 )  In particular, BIBREF16 and BIBREF15 uses market data (price) features along with the textual representation of the 10-K reports  These existing works that employ multi-modal learning BIBREF22 are based on a late fusion approach  For example, stacking ensembles to take into account the price and text predictions BIBREF16   In contrast, our end-to-end trained model can learn the joint distribution of both price and text 
Predicting the price direction rather than the volatility was the focus in BIBREF23   They extracted sentiment words from Twitter posts to build a time series of collective Profile of Mood States (POMS)  Their results show that collective mood accurately predicts the direction of Down Jones stock index (86 7% accuracy)  In BIBREF24 handcrafted text representations including term count, noun-phrase tags and extracted named entities are employed for predicting stock market direction using Support Vector Machine (SVM)  An extension of Latent Dirichlet Allocation (LDA) is proposed in BIBREF25 to learn a joint latent space of topics and sentiments 
Our deep learning models bear a close resemblance to works focused on directional price forecasting BIBREF26 , BIBREF27   In BIBREF26 , headline news are processed using Stanford OpenIE to generate triples that are fed into a Neural Tensor Network to create the final headline representation  In BIBREF27 , a character-level embedding is pre-trained in an unsupervised manner  The character embedding is used as input to a sequence model to learn the headline representation  Particularly, both works average all headline representations in a given day, rather than attempting to weight the most relevant ones  In this work, we propose a neural attention mechanism to capture the News Relevance and provide experimental evidence that it is a key component of the end-to-end learning process  Our attention extends the previous deep learning methods from BIBREF26 , BIBREF27  
Despite the fact that end-to-end deep learning models have attained state-of-the-art performance, the large number of parameters make them prone to overfitting  Additionally, end-to-end models are trained from scratch requiring large datasets and computational resources  Transfer learning (TL) alleviates this problem by adapting representations learnt from a different and potentially weakly related source domain to the new target domain  For example, in computer vision tasks the convolutional features learnt from ImageNet BIBREF28 dataset (source domain) have been successfully transferred to multiple domain target tasks with much smaller datasets such as object classification and scene recognition BIBREF29   In this work, we consider TL in our experiments for two main reasons  First, it address the question whether our proposed dataset is suitable for end-to-end training since the performance of the transferred representations can be compared with end-to-end learning  Second, it is still to be investigated which dataset transfers better to the forecasting problem  Recently, the NLP community has focused on universal representations of sentences BIBREF17 , BIBREF19 , which are dense representations that carry the meaning of a full sentence  BIBREF17 found that transferring the sentence representation trained on the Stanford Natural Language Inference (SNLI) BIBREF30 dataset achieves state-of-the-art sentence representations to multiple NLP tasks (e g  sentiment analysis, question-type and opinion polarity)  Following BIBREF17 , in this work, we investigate the suitability of SNLI and Reuters RCV1 BIBREF31 datasets to transfer learning to the volatility forecasting task  To the best of our knowledge, the hierarchical attention mechanism at headline level, proposed in our work, has not being applied to volatility prediction so far; neither has been investigated the ability to transfer sentence encoders from source datasets to the target forecasting problem (Transfer Learning) 
Our dataset
Our corpus covers a broad range of news including news around earnings dates and complements the 10-K reports content  As an illustration, the headlines “Walmart warns that strong U S  dollar will cost $15B in sales” and “Procter & Gamble Co raises FY organic sales growth forecast after sales beat” describe the company financial conditions and performance from the management point of view – these are also typical content present in Section 7 of the 10-K reports 
In this section, we describe the steps involved in compiling our dataset of financial news at stock level, which comprises a broad range of business sectors 
Sectors and stocks
The first step in compiling our corpus was to choose the constituents stocks  Our goal was to consider stocks in a broad range of sectors, aiming a diversified financial domain corpus  We found that Exchange Traded Funds (ETF) provide a mechanical way to aggregate the most relevant stocks in a given industry/sector  An ETF is a fund that owns assets, e g  stock shares or currencies, but, unlike mutual funds are traded in stock exchanges  These ETFs are extremely liquid and track different investment themes  We decided to use SPDR Setcor Funds constituents stocks in our work since the company is the largest provider of sector funds in the United States  We included in our analysis the top 5 (five) sector ETFs by financial trading volume (as in Jan/2018)  Among the most traded sectors we also filtered out the sectors that were similar to each other  For example, the Consumer Staples and Consumer Discretionary sectors are both part of the parent Consumer category  For each of the top 5 sectors we selected the top 10 holdings, which are deemed the most relevant stocks  tbl:stockuniverse, details our dataset sectors and its respective stocks 
Stock specific data
We assume that an individual stock news as the one that explicitly mention the stock name or any of its surface forms in the headline  As an illustration, in order to collect all news for the stock code PG, Procter & Gamble company name, we search all the headlines with any of these words: Procter&Gamble OR Procter and Gamble OR P&G  In this example, the first word is just the company name and the remaining words are the company surface forms 
We automatically derived the surface forms for each stock by starting with a seed of surface forms extracted from the DBpedia Knowledge Base (KB)  We then applied the following procedure:
Relate each company name with the KB entity unique identifier 
Retrieve all values of the wikiPageRedirects property  The property holds the names of different pages that points to the same entity/company name  This step sets the initial seed of surface forms 
Manually, filter out some noisy property values  For instance, from the Procter & Glamble entity page we were able to automatically extract dbr:Procter_and_gamble and dbr:P_&_G, but had to manually exclude the noisy associations dbr:Female_pads and dbr:California_Natural 
The result of the steps above is a dictionary of surface forms $wd_{sc}$  
Stock headlines
Our corpus is built at stock code level by collecting headlines from the Reuters Archive  This archive groups the headlines by date, starting from 1 January 2007  Each headline is a html link (<a href> tag) to the full body of the news, where the anchor text is the headline content followed by the release time  For example, the page dated 16 Dec 2016 has the headline “Procter & Gamble appoints Nelson Peltz to board 5:26PM UTC” 
For each of the 50 stocks (5 sectors times 10 stocks per sector) selected using the criteria described in sub:corpussecstock, we retrieved all the headlines from the Reuters Archive raging from 01/01/2007 to 30/12/2017  This process takes the following steps:
For a given stock code ( $sc$ ) retrieve all surface forms $wd_{sc}$  
For each day, store only the headlines content matching any word in $wd_{sc}$   For each stored headline we also store the time and timezone 
Convert the news date and time to Eastern Daylight Time (EDT) 
Categorize the news release time  We consider the following category set: {before market, during market , after market, holidays, weekends}  during market contains news between 9:30AM and 4:00PM  before market before 9:30AM and after market after 4:00PM 
The time categories prevents any misalignment between text and stock price data  Moreover, it prevents data leakage and, consequently, unrealistic predictive model performance  In general, news released after 4:00PM EDT can drastically change market expectations and the returns calculated using close to close prices as in the GARCH(1,1) model (see eq:closingreturn)  Following BIBREF3 , to deal with news misalignment, news issued after 4:00PM (after market) are grouped with the pre-market (before market) on the following trading day 
tbl:stocktimecat shows the distribution of news per sector for each time category  We can see a high concentration of news released before the market opens (55% on average)  In contrast, using a corpus compiled from message boards, a large occurrence of news during market hours was found BIBREF3   This behaviour indicating day traders' activity  Our corpus comprise financial news agency headlines, a content more focused on corporate events (e g  lawsuits, merges & acquisitions, research & development) and on economic news (see tbl:stockheadlinesexmaples for a sample of our dataset)  These headlines are mostly factual  On the other hand, user-generated content such as Twitter and message boards (as in BIBREF3 , BIBREF4 ) tends to be more subjective 
U S  macroeconomic indicators such as Retail Sales, Jobless Claims and GDP are mostly released around 8:30AM (one hour before the market opens)  These numbers are key drivers of market activity and, as such, have a high media coverage  Specific sections of these economic reports impact several stocks and sectors  Another factor that contribute to the high activity of news outside regular trading hours are company earnings reports  These are rarely released during trading hours  Finally, before the market opens news agencies provide a summary of the international markets developments, e g  the key facts during the Asian and Australian trading hours  All these factors contribute to the high concentration of pre-market news 
Background
We start this section by reviewing the GARCH(1,1) model, which is a strong benchmark used to evaluate our neural model  We then review the source datasets proposed in the literature that were trained independently and transfered to our volatility prediction model  Finally, we review the general architectures of sequence modelling and attention mechanisms 
GARCH model
Financial institutions use the concept of “Value at risk” to measure the expected volatility of their portfolios  The widespread econometric model for volatility forecasting is the Generalized Autoregressive Conditional Heteroskedasticity (GARCH) BIBREF32 , BIBREF33   Previous research shows that the GARCH(1,1) model is hard to beat  For example, BIBREF34 compared GARCH(1,1) with 330 different econometric volatility models showing that they are not significantly better than GARCH(1,1)  Let $p_t$ be the price of an stock at the end of a trading period with closing returns $r_t$ given by
$$r_t = \frac{p_t}{p_{t-1}} - 1 $$   (Eq  29)
The GARCH process explicitly models the time-varying volatility of asset returns  In the GARCH(1,1) specification the returns series $r_t$ follow the process:
$$r_t &= \mu + \epsilon _t  \\ \epsilon _t &= \sigma _t z_t  \\ \sigma ^2_t &= a_0 + a_1 \epsilon _{t-1}^2 + b_1 \sigma _{t-1}^2$$   (Eq  30)
where $\mu $ is a constant (return drift) and $z_t$ is a sequence of i i d  random variables with mean zero and unit variance  It is worth noting that although the conditional mean return described in eq:garchcondmean has a constant value, the conditional volatility $\sigma _t$ is time-dependent and modeled by eq:att 
The one-step ahead expected volatility forecast can be computed directly from eq:garchcondvariance and is given by
$$E_T[\sigma _{T+1}^2] = a_0 + a_1 E_T[\epsilon ^2] + b_1 E_T[\sigma _{T}^2] $$   (Eq  32)
In general, the $t^{\prime }$ -steps ahead expected volatility $E_T[\sigma _{T+t^{\prime }}^2]$ can be easily expressed in terms of the previous step expected volatility  It is easy to prove by induction that the forecast for any horizon can be represented in terms of the one-step ahead forecast and is given by
$$E_T[\sigma _{T+t^{\prime }}^2] - \sigma _u^2 = (a_1 + b_1)^{(t^{\prime } -1)} \left(E_T[\sigma _{T+1}^2] - \sigma _u^2\right)$$   (Eq  33)
where $\sigma _u$ is the unconditional volatility:
$$\sigma _u = \sqrt{a_0 / (1 - a_1 - b_1)} $$   (Eq  34)
From the equation above we can see that for long horizons, i e  $t^\prime \rightarrow \infty $ , the volatility forecast in eq:forecastrecursive converges to the unconditional volatility in eq:unvar 
All the works reviewed in sec:introduction ( BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 ) consider GARCH(1,1) benchmark  However, given the long horizon of their predictions (e g  quarterly or annual), the models are evaluated using the unconditional volatility $\sigma _u$ in eq:unvar  In this work, we focus on the short-term volatility prediction and use the GARCH(1,1) one-day ahead conditional volatility prediction in eq:forecastoneperiod to evaluate our models 
Let $\sigma _{t+1}$ denote the ex-post “true” daily volatility at a given time $t$   The performance on a set with $N$ daily samples can be evaluated using the standard Mean Squared Error ( $MSE$ ) and Mean Absolute Error ( $MAE$ )
$$MSE &= \frac{1}{N} \sum _{t=1}^{N} \left( E_t[\sigma _{t+1}] - \sigma _{t+1}\right)^2  \\ MAE &= \frac{1}{N} \sum _{t=1}^{N}\left|E_t[\sigma _{t+1}] - \sigma _{t+1} \right|$$   (Eq  36)
Additionally, following BIBREF35 , the models are also evaluated using the coefficient of determination $R^2$ of the regression
$$\sigma _{t+1} = a + b E_t[\sigma _{t+1}] + e_t$$   (Eq  37)
where
$$R^2 = 1 - \frac{\sum _{t=1}^{N}e^{2}_{t}}{\sum _{t=1}^{N}\left(E_t[\sigma _{t+1}] - \frac{1}{N} \sum _{t=1}^{N}E_t[\sigma _{t+1}]\right)^{2}}$$   (Eq  38)
One of the challenges in evaluating GARCH models is the fact that the ex-post volatility $\sigma _{t+1}$ is not directly observed  Apparently, the squared daily returns $r_{t+1}^{2}$ in eq:closingreturn could stand as a good proxy for the ex-post volatility  However, the squared returns yield very noisy measurements  This is a direct consequence of the term $z^t$ that connects the squared return to the latent volatility factor in eq:garchwhitenoise  The use of intraday prices to estimate the ex-post daily volayility was first proposed in BIBREF35   They argue that volatility estimators using intraday prices is the proper way to evaluate the GARCH(1,1) model, as opposed to squared daily returns  For example, considering the Deutsche Mark the GARCH(1,1) model $R^2$ improves from $0 047$ (squared returns) to $0 33$ (intraday returns) BIBREF35  
It is clear from the previous section that any volatility model evaluation using the noisy squared returns as the ex-post volatility proxy will lead to very poor performance  Therefore, high-frequency intraday data is fundamental to short-term volatility performance evaluation  However, intraday data is difficult to acquire and costly  Fortunately, there are statistically efficient daily volatility estimators that only depend on the open, high, low and close prices  These price “ranges” are widely available  In this section, we discuss these estimators 
Let $O_t$ , $H_t$ , $L_t$ , $C_t$ be the open, high, low and close prices of an asset in a given day $t$   Assuming that the daily price follows a geometric Brownian motion with zero drift and constant daily volatility $\sigma $ , Parkinson (1980) derived the first daily volatility estimator
$$\widehat{\sigma _{PK,t}^2} = \frac{\ln \left(\frac{H_t}{L_t}\right)^2}{4\ln (2)} $$   (Eq  41)
which represents the daily volatility in terms of its price range  Hence, it contains information about the price path  Given this property, it is expected that $\sigma _{PK}$ is less noisy than the volatility calculated using squared returns  The Parkinson's volatility estimator was extended by Garman-Klass (1980) which incorporates additional information about the opening ( $O_t$ ) and closing ( $C_t$ ) prices and is defined as
$$\widehat{\sigma _{GK,t}^{2}} = \frac{1}{2} \ln \left(\frac{H_t}{L_t}\right)^2 - (2\ln (2) - 1) \ln \left(\frac{C_t}{O_t}\right)^2 $$   (Eq  42)
The relative noisy of different estimators $\hat{\sigma }$ can be measured in terms of its relative efficiency to the daily volatility $\sigma $ and is defined as
$$e\left(\widehat{\sigma ^{2}}, \sigma ^2\right) \equiv \frac{Var[\sigma ^2]}{Var[\widehat{\sigma ^{2}}]}$$   (Eq  43)
where $Var[\cdot ]$ is the variance operator  It follows directly from eq:garchwhitenoise that the squared return has efficiency 1 and therefore, very noisy  BIBREF36 reports Parkinson ( $\widehat{\sigma _{PK,t}^2}$ ) volatility estimator has 4 9 relative efficiency and Garman-Klass ( $\widehat{\sigma _{GK,t}^2}$ ) 7 4  Additionally, all the described estimators are unbiased 
Many alternative estimators to daily volatility have been proposed in the literature  However, experiments in BIBREF36 rate the Garman-Klass volatility estimator as the best volatility estimator based only on open, high, low and close prices  In this work, we train our models to predict the state-of-the-art Garman-Klass estimator  Moreover, we evaluate our models and GARCH(1,1) using the metrics described in sub:evalution, but with the appropriate volatility proxies, i e  Parkinson and Garman-Klass estimators 
Transfer Learning from other source domains
Vector representations of words, also known as Word embeddings BIBREF21 , BIBREF37 , that represent a word as a dense vector has become the standard building blocks of almost all NLP tasks  These embeddings are trained on large unlabeled corpus and are able to capture context and similarity among words 
Some attempts have been made to learn vector representations of a full sentence, rather than only a single word, using unsupervised approaches similar in nature to word embeddings  Recently, BIBREF17 showed state-of-the-art performance when a sentence encoder is trained end-to-end on a supervised source task and transferred to other target tasks  Inspired by this work, we investigate the performance of sentence encoders trained on the Text categorization and Natural Language Inference (NLI) tasks and use these encoders in our main short-term volatility prediction task 
A generic sentence encoder $S_e$ receives the sentence words as input and returns a vector representing the sentence  This can be expressed as a mapping
$$S_e \colon \mathbb {R}^{T^{S} \times d_w} \rightarrow \mathbb {R}^{d_S}$$   (Eq  45)
from a variable size sequence of words to a sentence vector $S$ of fixed-size $d_S$ , where $T^{S}$ is the sentence number of words and $d_w$ is the pre-trained word embedding dimension 
In the following sections, we describe the datasets and architectures to train the sentence encoders of the auxiliary transfer learning tasks 
The Reuters Corpus Volume I (RCV1) is corpus containing 806,791 news articles in the English language collected from 20/08/1996 to 19/08/1997 BIBREF31   The topic of each news was human-annotated using a hierarchical structure  At the top of the hierarchy, lies the coarse-grained categories: CCAT (Corporate), ECAT (Economics), GCAT (Government), and MCAT (Markets)  A news article can be assigned to more than one category meaning that the text categorization task is mutilabel  Each news is stored in a separate XML file  lst:rcv1xmlexample shows the typical structure of an article 
<?xml version="1 0" encoding="iso-8859-1" ?>
<newsitem itemid="6159" id="root" date="1996-08-21" xml:lang="en">
<headline>Colombia raises internal coffee price </headline>
<dateline>BOGOTA 1996-08-21</dateline>
<copyright>(c) Reuters Limited 1996</copyright>
<metadata>
<codes class="bip:topics:1 0">
<code code="C13">
<editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/>
</code>
<code code="C31">
<editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/>
</code>
<code code="CCAT">
<editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/>
</code>
<code code="M14">
<editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/>
</code>
<code code="M141">
<editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/>
</code>
<code code="MCAT">
<editdetail attribution="Reuters BIP Coding Group" action="confirmed" date="1996-08-21"/>
</code>
</codes>
</metadata>
</newsitem>
The RCV1 dataset is not released with a standard train, validation, test split  In this work, we separated 15% of samples as a test set for evaluation purposes  The remaining samples were further split leaving 70% and 15% for training and validation, respectively 
Regarding the categories distribution, we found that, from the original 126 categories, 23 categories were never assigned to any news; therefore, were disregarded  From the 103 classes left we found a high imbalance among the labels with a large number of underrepresented categories having less than 12 samples  The very low number of samples for these minority classes brings a great challenge to discriminate the very fine-grained categories  Aiming to alleviate this problem, we grouped into a same class all categories below the second hierarchical level  For example, given the root node CCAT (Corporate) we grouped C151 (ACCOUNTS/EARNINGS), C1511 (ANNUAL RESULTS) and C152 (COMMENT/FORECASTS) into the direct child node C15 (PERFORMANCE)  Using this procedure the original 103 categories where reduced to 55  One of the benefits of this procedure was that the less represented classes end up having around thousand samples compared with only 12 samples in the original dataset 
fig:rcv1arch, shows the architecture for the end-to-end text categorization task  On the bottom of the architecture $S_e$ receives word embeddings and outputs a sentence vector $S$   The $S$ vector pass through a fully connected (FC) layer with sigmoid activation function that outputs a vector $\hat{y} \in \mathbb {R}^{55}$ with each element $\hat{y}_j \in [0,1]$  
The architecture described above is trained under the assumption that each category is independent but not mutually exclusive since a sample can have more than one category assigned (multilabel classification)  The loss per sample is the average log loss across all labels:
$$\mathcal {L}(\hat{y}, y) = - \sum _{i=1}^{55}\left( y_i \log (\hat{y}_i) + (1-y_{i}) \log (1-\hat{y}_{i}) \right)$$   (Eq  48)
where the index $i$ runs over the elements of the predicted and true vectors 
Given the high categories imbalance, during the training we monitor the $F_1$ metric of the validation set and choose the model with the highest value 
Stanford Natural Language Inference (SNLI) dataset BIBREF30 consist of 570,000 pairs of sentences  Each pair has a premise and a hypothesis, manually labeled with one of the three labels: entailment, contradiction, or neutral  The SNLI has many desired properties  The labels are equally balanced, as opposed to the RCV1 dataset  Additionally, language inference is a complex task that requires a deeper understanding of the sentence meaning making this dataset suitable for learning supervised sentence encoders that generalize well to other tasks BIBREF17   tbl:snliexmaples, shows examples of SNLI dataset sentence pairs and its respective labels 
In order to learn sentence encoders that can be transfered to other tasks unambiguously, we consider a neural network architecture for the sentence encoder with shared parameters between the premise and hypothesis pairs as in BIBREF17  
fig:snliarch, describes the neural network architecture  After each premise and hypothesis is encoded into $S_p$ and $S_h$ , respectively, we have a fusion layer  This layer has no trainable weights and just concatenate each sentence embedding  Following BIBREF17 , we add two more matching methods: the absolute difference $\vert S_p - S_h \vert $ and the element-wise $S_p \odot S_h$   Finally, in order to learn the pair representation, $S_ph$ is feed into and FC layer with rectified linear unit (ReLU) activation function, which is expressed as $f(x) = \log (1 + e^x)$   The last softmax layer outputs the probability of each class 
Finally, the NLI classifier weights are optimized in order to minimize the categorical log loss per sample
$$\mathcal {L}(\hat{y}, y) = - \sum _{j=1}^{3}y_i \log (\hat{y}_i)$$   (Eq  52)
During the training, we monitor the validation set accuracy and choose the model with the highest metric value 
Sequence Models
We start this section by reviewing the Recurrent Neural Network (RNN) architecture and its application to encode a sequence of words 
RNN's are capable of handling variable-length sequences, this being a direct consequence of its recurrent cell, which shares the same parameters across all sequence elements  In this work, we adopt the Long Short-Term Memory (LSTM) cell BIBREF38 with forget gates $f_t$ BIBREF39   The LSTM cell is endowed with a memory state that can learn representations that depend on the order of the words in a sentence  This makes LSTM more fit to find relations that could not be captured using standard bag-of-words representations 
Let $x_1, x_2, \cdots , x_T$ be a series of observations of length $T$ , where $x_t \in \mathbb {R}^{d_w}$   In general terms, the LSTM cell receives a previous hidden state $h_{t-1}$ that is combined with the current observation $x_t$ and a memory state $C_t$ to output a new hidden state $h_t$   This internal memory state $C_{t}$ is updated depending on its previous state and three modulating gates: input, forget, and output  Formally, for each step $t$ the updating process goes as follows (see fig:lstmcell for a high level schematic view): First, we calculate the input $i_t$ , forget $T$0 , and output $T$1 gates:
$$i_t &= \sigma _s\left(W_i x_t + U_i h_{t-1} + b_i\right) \\ f_t &= \sigma _s\left(W_f x_t + U_f h_{t-1} + b_f\right) \\ o_t &= \sigma _s\left(W_o x_t + U_o h_{t-1} + b_o\right)$$   (Eq  54)
where $\sigma _s$ is the sigmoid activation

ground_truth: GloVe, BERT, Universal Sentence Encoder, TF-IDF, InferSent

Avg. GloVe embeddings, Avg. fast-text embeddings, Avg. BERT embeddings, BERT CLS-vector, InferSent - GloVe and Universal Sentence Encoder.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the datasets used for evaluation?
Context 2: 
Experimental Setup ::: Summarization Datasets ::: CNN/DailyMail
contains news articles and associated highlights, i e , a few bullet points giving a brief overview of the article  We used the standard splits of BIBREF24 for training, validation, and testing (90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents)  We did not anonymize entities  We first split sentences with the Stanford CoreNLP toolkit BIBREF26 and pre-processed the dataset following BIBREF6  Input documents were truncated to 512 tokens 
Experimental Setup ::: Summarization Datasets ::: NYT
contains 110,540 articles with abstractive summaries  Following BIBREF27, we split these into 100,834/9,706 training/test examples, based on the date of publication (the test set contains all articles published from January 1, 2007 onward)  We used 4,000 examples from the training as validation set  We also followed their filtering procedure, documents with summaries less than 50 words were removed from the dataset  The filtered test set (NYT50) includes 3,452 examples  Sentences were split with the Stanford CoreNLP toolkit BIBREF26 and pre-processed following BIBREF27  Input documents were truncated to 800 tokens 
Experimental Setup ::: Summarization Datasets ::: XSum
contains 226,711 news articles accompanied with a one-sentence summary, answering the question “What is this article about?”  We used the splits of BIBREF22 for training, validation, and testing (204,045/11,332/11,334) and followed the pre-processing introduced in their work  Input documents were truncated to 512 tokens 
Aside from various statistics on the three datasets, Table TABREF12 also reports the proportion of novel bi-grams in gold summaries as a measure of their abstractiveness  We would expect models with extractive biases to perform better on datasets with (mostly) extractive summaries, and abstractive models to perform more rewrite operations on datasets with abstractive summaries  CNN/DailyMail and NYT are somewhat abstractive, while XSum is highly abstractive 
Experimental Setup ::: Implementation Details
For both extractive and abstractive settings, we used PyTorch, OpenNMT BIBREF28 and the `bert-base-uncased' version of Bert to implement BertSum  Both source and target texts were tokenized with Bert's subwords tokenizer 
Experimental Setup ::: Implementation Details ::: Extractive Summarization
All extractive models were trained for 50,000 steps on 3 GPUs (GTX 1080 Ti) with gradient accumulation every two steps  Model checkpoints were saved and evaluated on the validation set every 1,000 steps  We selected the top-3 checkpoints based on the evaluation loss on the validation set, and report the averaged results on the test set  We used a greedy algorithm similar to BIBREF7 to obtain an oracle summary for each document to train extractive models  The algorithm generates an oracle consisting of multiple sentences which maximize the ROUGE-2 score against the gold summary 
When predicting summaries for a new document, we first use the model to obtain the score for each sentence  We then rank these sentences by their scores from highest to lowest, and select the top-3 sentences as the summary 
During sentence selection we use Trigram Blocking to reduce redundancy BIBREF9  Given summary $S$ and candidate sentence $c$, we skip $c$ if there exists a trigram overlapping between $c$ and $S$  The intuition is similar to Maximal Marginal Relevance (MMR; BIBREF29); we wish to minimize the similarity between the sentence being considered and sentences which have been already selected as part of the summary 
Experimental Setup ::: Implementation Details ::: Abstractive Summarization
In all abstractive models, we applied dropout (with probability $0 1$) before all linear layers; label smoothing BIBREF30 with smoothing factor $0 1$ was also used  Our Transformer decoder has 768 hidden units and the hidden size for all feed-forward layers is 2,048  All models were trained for 200,000 steps on 4 GPUs (GTX 1080 Ti) with gradient accumulation every five steps  Model checkpoints were saved and evaluated on the validation set every 2,500 steps  We selected the top-3 checkpoints based on their evaluation loss on the validation set, and report the averaged results on the test set 
During decoding we used beam search (size 5), and tuned the $\alpha $ for the length penalty BIBREF31 between $0 6$ and 1 on the validation set; we decode until an end-of-sequence token is emitted and repeated trigrams are blocked BIBREF9  It is worth noting that our decoder applies neither a copy nor a coverage mechanism BIBREF6, despite their popularity in abstractive summarization  This is mainly because we focus on building a minimum-requirements model and these mechanisms may introduce additional hyper-parameters to tune  Thanks to the subwords tokenizer, we also rarely observe issues with out-of-vocabulary words in the output; moreover, trigram-blocking produces diverse summaries managing to reduce repetitions 
Results ::: Automatic Evaluation
We evaluated summarization quality automatically using ROUGE BIBREF32  We report unigram and bigram overlap (ROUGE-1 and ROUGE-2) as a means of assessing informativeness and the longest common subsequence (ROUGE-L) as a means of assessing fluency  Table TABREF23 summarizes our results on the CNN/DailyMail dataset  The first block in the table includes the results of an extractive Oracle system as an upper bound  We also present the Lead-3 baseline (which simply selects the first three sentences in a document)  The second block in the table includes various extractive models trained on the CNN/DailyMail dataset (see Section SECREF5 for an overview)  For comparison to our own model, we also implemented a non-pretrained Transformer baseline (TransformerExt) which uses the same architecture as BertSumExt, but with fewer parameters  It is randomly initialized and only trained on the summarization task  TransformerExt has 6 layers, the hidden size is 512, and the feed-forward filter size is 2,048  The model was trained with same settings as in BIBREF3  The third block in Table TABREF23 highlights the performance of several abstractive models on the CNN/DailyMail dataset (see Section SECREF6 for an overview)  We also include an abstractive Transformer baseline (TransformerAbs) which has the same decoder as our abstractive BertSum models; the encoder is a 6-layer Transformer with 768 hidden size and 2,048 feed-forward filter size  The fourth block reports results with fine-tuned Bert models: BertSumExt and its two variants (one without interval embeddings, and one with the large version of Bert), BertSumAbs, and BertSumExtAbs  Bert-based models outperform the Lead-3 baseline which is not a strawman; on the CNN/DailyMail corpus it is indeed superior to several extractive BIBREF7, BIBREF8, BIBREF19 and abstractive models BIBREF6  Bert models collectively outperform all previously proposed extractive and abstractive systems, only falling behind the Oracle upper bound  Among Bert variants, BertSumExt performs best which is not entirely surprising; CNN/DailyMail summaries are somewhat extractive and even abstractive models are prone to copying sentences from the source document when trained on this dataset BIBREF6  Perhaps unsurprisingly we observe that larger versions of Bert lead to performance improvements and that interval embeddings bring only slight gains  Table TABREF24 presents results on the NYT dataset  Following the evaluation protocol in BIBREF27, we use limited-length ROUGE Recall, where predicted summaries are truncated to the length of the gold summaries  Again, we report the performance of the Oracle upper bound and Lead-3 baseline  The second block in the table contains previously proposed extractive models as well as our own Transformer baseline  Compress BIBREF27 is an ILP-based model which combines compression and anaphoricity constraints  The third block includes abstractive models from the literature, and our Transformer baseline  Bert-based models are shown in the fourth block  Again, we observe that they outperform previously proposed approaches  On this dataset, abstractive Bert models generally perform better compared to BertSumExt, almost approaching Oracle performance 
Table TABREF26 summarizes our results on the XSum dataset  Recall that summaries in this dataset are highly abstractive (see Table TABREF12) consisting of a single sentence conveying the gist of the document  Extractive models here perform poorly as corroborated by the low performance of the Lead baseline (which simply selects the leading sentence from the document), and the Oracle (which selects a single-best sentence in each document) in Table TABREF26  As a result, we do not report results for extractive models on this dataset  The second block in Table TABREF26 presents the results of various abstractive models taken from BIBREF22 and also includes our own abstractive Transformer baseline  In the third block we show the results of our Bert summarizers which again are superior to all previously reported models (by a wide margin) 
Results ::: Model Analysis ::: Learning Rates
Recall that our abstractive model uses separate optimizers for the encoder and decoder  In Table TABREF27 we examine whether the combination of different learning rates ($\tilde{lr}_{\mathcal {E}}$ and $\tilde{lr}_{\mathcal {D}}$) is indeed beneficial  Specifically, we report model perplexity on the CNN/DailyMail validation set for varying encoder/decoder learning rates  We can see that the model performs best with $\tilde{lr}_{\mathcal {E}}=2e-3$ and $\tilde{lr}_{\mathcal {D}}=0 1$ 
Results ::: Model Analysis ::: Position of Extracted Sentences
In addition to the evaluation based on ROUGE, we also analyzed in more detail the summaries produced by our model  For the extractive setting, we looked at the position (in the source document) of the sentences which were selected to appear in the summary  Figure FIGREF31 shows the proportion of selected summary sentences which appear in the source document at positions 1, 2, and so on  The analysis was conducted on the CNN/DailyMail dataset for Oracle summaries, and those produced by BertSumExt and the TransformerExt  We can see that Oracle summary sentences are fairly smoothly distributed across documents, while summaries created by TransformerExt mostly concentrate on the first document sentences  BertSumExt outputs are more similar to Oracle summaries, indicating that with the pretrained encoder, the model relies less on shallow position features, and learns deeper document representations 
Results ::: Model Analysis ::: Novel N-grams
We also analyzed the output of abstractive systems by calculating the proportion of novel n-grams that appear in the summaries but not in the source texts  The results are shown in Figure FIGREF33  In the CNN/DailyMail dataset, the proportion of novel n-grams in automatically generated summaries is much lower compared to reference summaries, but in XSum, this gap is much smaller  We also observe that on CNN/DailyMail, BertExtAbs produces less novel n-ngrams than BertAbs, which is not surprising  BertExtAbs is more biased towards selecting sentences from the source document since it is initially trained as an extractive model  The supplementary material includes examples of system output and additional ablation studies 
Results ::: Human Evaluation
In addition to automatic evaluation, we also evaluated system output by eliciting human judgments  We report experiments following a question-answering (QA) paradigm BIBREF33, BIBREF8 which quantifies the degree to which summarization models retain key information from the document  Under this paradigm, a set of questions is created based on the gold summary under the assumption that it highlights the most important document content  Participants are then asked to answer these questions by reading system summaries alone without access to the article  The more questions a system can answer, the better it is at summarizing the document as a whole  Moreover, we also assessed the overall quality of the summaries produced by abstractive systems which due to their ability to rewrite content may produce disfluent or ungrammatical output  Specifically, we followed the Best-Worst Scaling BIBREF34 method where participants were presented with the output of two systems (and the original document) and asked to decide which one was better according to the criteria of Informativeness, Fluency, and Succinctness 
Both types of evaluation were conducted on the Amazon Mechanical Turk platform  For the CNN/DailyMail and NYT datasets we used the same documents (20 in total) and questions from previous work BIBREF8, BIBREF18  For XSum, we randomly selected 20 documents (and their questions) from the release of BIBREF22  We elicited 3 responses per HIT  With regard to QA evaluation, we adopted the scoring mechanism from BIBREF33; correct answers were marked with a score of one, partially correct answers with 0 5, and zero otherwise  For quality-based evaluation, the rating of each system was computed as the percentage of times it was chosen as better minus the times it was selected as worse  Ratings thus range from -1 (worst) to 1 (best) 
Results for extractive and abstractive systems are shown in Tables TABREF37 and TABREF38, respectively  We compared the best performing BertSum model in each setting (extractive or abstractive) against various state-of-the-art systems (whose output is publicly available), the Lead baseline, and the Gold standard as an upper bound  As shown in both tables participants overwhelmingly prefer the output of our model against comparison systems across datasets and evaluation paradigms  All differences between BertSum and comparison models are statistically significant ($p<0 05$), with the exception of TConvS2S (see Table TABREF38; XSum) in the QA evaluation setting 
Conclusions
In this paper, we showcased how pretrained Bert can be usefully applied in text summarization  We introduced a novel document-level encoder and proposed a general framework for both abstractive and extractive summarization  Experimental results across three datasets show that our model achieves state-of-the-art results across the board under automatic and human-based evaluation protocols  Although we mainly focused on document encoding for summarization, in the future, we would like to take advantage the capabilities of Bert for language generation 
Acknowledgments
This research is supported by a Google PhD Fellowship to the first author  We gratefully acknowledge the support of the European Research Council (Lapata, award number 681760, “Translating Multiple Modalities into Text”)  We would also like to thank Shashi Narayan for providing us with the XSum dataset 
Context 3:  Second, a candidate memory state $\widetilde{C}_t$ is generated:
$$\widetilde{C}_t = \tanh \left(W_c x_t + U_c h_{t-1} + b_c\right)$$   (Eq  55)
Now we are in a position to set the final memory state $C_t$   Its value is modulated based on the input and forget gates of eq:inputforgetgates and is given by:
$$C_t = i_t \odot \widetilde{C}_t + f_t \odot C_{t-1}$$   (Eq  56)
Finally, based on the memory state and output gate of eq:inputforgetgates, we have the output hidden state
$$h_t = o_t \odot \tanh \left(C_t\right)$$   (Eq  57)
Regarding the trainable weights, let $n$ be the LSTM cell number of units  It follows that $W$ 's and $U$ 's matrices of the affine transformations have ${n \times d_w}$ and ${n \times n}$ dimensions, respectively  Its bias terms $b$ 's are vectors of size $n$   Consequently, the total number of parameters is $4 (n d_w + n^2 + n)$ and does not depend on the sequence number of time steps $T$  
We see that the LSTM networks are able to capture temporal dependencies in sequences of arbitrary length  One straightforward application is to model the Sentence encoder discussed in sec:transferlearning, which outputs a sentence vector representation using its words as input 
Given a sequence of words $\left\lbrace w_t\right\rbrace _{t=1}^{T}$ we aim to learn the words hidden state $\left\lbrace h_t\right\rbrace _{t=1}^{T}$ in a way that each word captures the influence of its past and future words  The Bidirectional LSTM (BiLSTM) proposed in BIBREF40 is an LSTM that “reads” a sentence, or any sequence in general, from the beginning to the end (forward) and the other way around (backward)  The new state $h_t$ is the concatenation
$$h_t = [\overrightarrow{h_t}, \overleftarrow{h_t}]$$   (Eq  59)
where
$$\overrightarrow{h_t} &= \text{LSTM}\left(w_1, \cdots , w_T\right) \\ \overleftarrow{h_t} &= \text{LSTM}\left(w_T, \cdots , w_1\right) \\$$   (Eq  60)
Because sentences have different lengths, we need to convert the $T$ concatenated hidden states of the BiLSTM into a fixed-length sentence representation  One straightforward operation is to apply any form of pooling  Attention mechanism is an alternative approach where the sentence is represented as an weighted average of hidden states where the weights are learnt end-to-end 
In the next sections we describe the sentence encoders using pooling and attention layers 
The max-pooling layer aims to extract the most salient word features all over the sentence  Formally, it outputs a sentence vector representation $S_{MP} \in \mathbb {R}^{2n}$ such that
$$S_{MP} = \max _{t=1}^{T} h_t$$   (Eq  62)
where $h_t$ is defined in eq:htconcat and the $\max $ operator is applied over the time steps dimension  fig:bilstmmaxpool illustrates the BiLSTM max-pooling (MP) sentence encoder 
The efficacy of the max-pooling layer was assessed in many NLP studies  BIBREF41 employed a max-pooling layer on top of word representations and argues that it performs better than mean pooling  Experimental results in BIBREF17 show that among three types of pooling (max, mean and last) the max-pooling provides the most universal sentence representations in terms of transferring performance to other tasks  Grounded on these studies, in this work, we choose the BiLSTM max-pooling as our pooling layer of choice 
Attention mechanisms were introduced in the deep learning literature to overcome some simplifications imposed by pooling operators  When we humans read a sentence, we are able to spot its most relevant parts in a given context and disregard information that is redundant or misleading  The attention model aims to mimic this behaviour 
Attention layers were proposed for different NLP tasks  For example, NLI, with cross-attention between premise and hypothesis, Question & Answering and Machine Translation (MT)  Specifically in the Machine Translation task, each word in the target sentence learns to attend the relevant words of the source sentence in order to generate the sentence translation 
A sentence encoder with attention (or self-attentive) BIBREF42 , BIBREF43 , BIBREF44 assigns different weights to the own words of the sentence; therefore, converting the hidden states into a single sentence vector representation 
Considering the word hidden vectors set $\lbrace h_1, \cdots , h_T\rbrace $ where $h_t \in \mathbb {R}^n$ , the attention mechanism is defined by the equations:
$$\tilde{h}_t &= \sigma \left(W h_t + b \right) \\ \alpha _{t} &= \frac{\exp ({v^{\intercal } \cdot \tilde{h}_t} )}{\sum _{t} \exp ({v \cdot \tilde{h}_t})} \\ S_{A_w} &= \sum _{t} \alpha _{t} h_t$$   (Eq  66)
where $W \in \mathbb {R}^{d_a \times n}$ , $b \in \mathbb {R}^{d_a \times 1}$ , and $v \in \mathbb {R}^{d_a \times 1}$ are trainable parameters 
We can see that the sentence representation $S_{A_w}$ is a weighted average of the hidden states  fig:bilstminneratt provides a schematic view of the BiLSTM attention, where we can account the attention described in eq:att as a two layer model with a dense layer ( $d_a$ units) followed by another dense that predicts $\alpha _t$ (single unit) 
Methodology
In this section, we first introduce our problem in a deep multimodal learning framework  We then present our neural architecture, which is able to address the problems of news relevance and novelty  Finally, we review the methods applied to learn commonalities between stocks (global features) 
Problem statement
Our problem is to predict the daily stock volatility  As discussed in subsub:rangevolestimators, the Gaman-Klass estimator $\widehat{\sigma _{GK,t}}$ in eq:volgk is a very efficient short-term volatility proxy, thus, it is adopted as our target variable 
Our goal is to learn a mapping between the next day volatility $\sigma _{t+1}$ and historical multimodal data available up to day $t$   To this aim, we use a sliding window approach with window size $T$   That is, for each stock $sc$ a sample on day $t$ is expressed as a sequence of historical prices $P^{sc}_t$ and corpus headlines $N^{sc}_t$   The price sequence is a vector of Daily Prices (DP) and expressed as
$$P^{sc}_t = \left[DP^{sc}_{t-T}, DP^{sc}_{t-T+1}, \cdots , DP^{sc}_t \right]$$   (Eq  69)
where $DP^{sc}_{t^{\prime }}$ is a vector of price features  In order to avoid task-specific feature engineering, the daily price features are expressed as the simple returns:
$$DP^{sc}_t = \left[ \frac{O^{sc}_{t}}{C^{sc}_{t-1}} - 1, \frac{H^{sc}_{t}}{C^{sc}_{t-1}} - 1, \frac{L^{sc}_{t}}{C^{sc}_{t-1}} - 1, \frac{C^{sc}_{t}}{C^{sc}_{t-1}} - 1 \right]$$   (Eq  70)
The sequence of historical corpus headlines $N^{sc}_t$ is expressed as
$$N^{sc}_t = \left[n^{sc}_{t-T}, n^{sc}_{t-T+1}, \cdots , n^{sc}_{t} \right]$$   (Eq  71)
where $n^{sc}_{t^{\prime }}$ is a set containing all headlines that influence the market on a given day $t^{\prime }$  
Aiming to align prices and news modes, we consider the explicit alignment method discussed in subsec:stockheadlines  That is, $n^{sc}_{t^{\prime }}$ contains all stock headlines before the market opens ( $\texttt {before market}_{t}$ ), during the trading hours
( $\texttt {during market}_{t}$ ), and previous day after-markets
( $\texttt {after market}_{t-1}$ ) 
As a text preprocessing step, we tokenize the headlines and convert each word to an integer that refers to its respective pre-trained word embedding  This process is described as follows: First, for all stocks of our corpus we tokenize each headline and extract the corpus vocabulary set $V$   We then build the embedding matrix $E_w \in \mathbb {R}^{\vert V \vert \times d_w}$ , where each row is a word embedding vector $d_w$ dimensions  Words that do not have a corresponding embedding, i e  out of vocabulary words, are skipped 
Finally, the input sample of the text mode is a tensor of integers with $T \times l_n \times l_s$ dimensions, where $l_n$ is the maximum number of news occurring in a given day and $l_s$ is the maximum length of a corpus sentence  Regarding the price mode, we have a $T \times 4$ tensor of floating numbers 
Global features and stock embedding
Given the price and news histories for each stock $sc$ we could directly learn one model per stock  However, this approach suffers from two main drawbacks  First, the market activity of one specific stock is expected to impact other stocks, which is a widely accepted pattern named “spillover effect”  Second, since our price data is sampled on a daily basis, we would train the stock model relying on a small number of samples  One possible solution to model the commonality among stocks would be feature enrichment  For example, when modeling a given stock $X$ we would enrich its news and price features by concatenating features from stock $Y$ and $Z$   Although the feature enrichment is able to model the effect of other stocks, it still would consider only one sample per day 
In this work, we propose a method that learns an global model 
The global model is implemented using the following methods:
Multi-Stock batch samples: Since our models are trained using Stochastic Gradient Descent, we propose at each mini-batch iteration to sample from a batch set containing any stock of our stocks universe  As a consequence, the mapping between volatility and multimodal data is now able to learn common explanatory factors among stocks  Moreover, adopting this approach increases the total number of training samples, which is now the sum of the number of samples per stock 
Stock Embedding: Utilizing the Multi-Stock batch samples above, we tackle the problem of modeling commonality among stocks  However, it is reasonable to assume that stocks have part of its dynamic driven by idiosyncratic factors  Nevertheless, we could aggregate stocks per sector or rely on any measure of similarity among stocks  In order to incorporate information specific to each stock, we propose to equip our model with a “stock embedding” mode that is learnt jointly with price and news modes  That is to say, we leave the task of distinguishing the specific dynamic of each stock to be learnt by the neural network  Specifically, this stock embedding is modeled using a discrete encoding as input, i e  $\mathcal {I}^{sc}_t$ is a vector with size equal to the number of stocks of the stocks universe and has element 1 for the i-th coordinate and 0 elsewhere, thus, indicating the stock of each sample 
Formally, we can express the one model per stock approach as the mapping
$$\begin{split} \sigma ^{sc}_{t+1} = f^{sc} ( DN^{sc}_{t-T}, DN^{sc}_{t-T+1}, \cdots , DN^{sc}_t ; \\ DP^{sc}_{t-T}, DP^{sc}_{t-T+1}, \cdots , DP^{sc}_t ) \end{split}$$   (Eq  75)
where $DN^{sc}_{t^{\prime }}$ is a fixed-vector representing all news released on a given day for the stock $sc$ and $DP^{sc}_{t^{\prime }}$ is defined in eq:pricemodevec 
The global model attempts to learn a single mapping $f$ that at each mini-batch iteration randomly aggregates samples across all the universe of stocks, rather than one mapping $f^{sc}$ per stock  The global model is expressed as
$$\begin{split} \sigma ^{sc}_{t+1} = f ( DN^{sc}_{t-T}, DN^{sc}_{t-T+1}, \cdots , DN^{sc}_t ; \\ DP^{sc}_{t-T}, DP^{sc}_{t-T+1}, \cdots , DP^{sc}_t ; \\ \mathcal {I}^{sc}_t) \end{split}$$   (Eq  77)
In the next section, we describe our hierarchical neural model and how the news, price and stock embedding are fused into a joint representation 
Our multimodal hierarchical network
In broad terms, our hierarchical neural architecture is described as follows  First, each headline released on a given day $t$ is encoded into a fixed-size vector $S_t$ using a sentence encoder  We then apply our daily New Relevance Attention (NRA) mechanism that attends each news based on its content and converts a variable size of news released on a given day into a single vector denoted by Daily News ( $DN$ )  We note that this representation take account of the overall effect of all news released on a given day  This process is illustrated in fig:DNencoder  We now are in a position to consider the temporal effect of the past $T$ days of market news and price features  fig:nntimeseriesarch illustrates the neural network architecture from the temporal sequence to the final volatility prediction  For each stock code $sc$ the temporal encoding for news is denoted by Market News $MN^{sc}_t$ and for the price by Market Price $MP^{sc}_t$ and are a function of the past $T$ Daily News representations ${\lbrace DN^{sc}_{t-T}, \cdots , DN^{sc}_t \rbrace }$ (Text mode) and Daily Prices features $S_t$0 (Price mode), where each Daily Price $S_t$1 feature is given by eq:pricemodevec and the $S_t$2 representation is calculated using Daily New Relevance Attention  After the temporal effects of $S_t$3 past days of market activity were already encoded into the Market News $S_t$4 and Market Price $S_t$5 , we concatenate feature-wise $S_t$6 , $S_t$7 and the Stock embedding $S_t$8   The stock embedding $S_t$9 represents the stock code of the sample on a given day $t$   Finally, we have a Fully Connected (FC) layer that learns the Joint Representation of all modes  This fixed-sized joint representation is fed into a FC layer with linear activation that predicts the next day volatility $\hat{\sigma }_{t+1}$  
Below, we detail, for each mode separately, the layers of our hierarchical model 
– Text mode
Word Embedding Retrieval
Standard embedding layer with no trainable parameters  It receives a vector of word indices as input and returns a matrix of word embeddings 
News Encoder
This layer encodes all news on a given day and outputs a set news embeddings $\lbrace S^{1}_t, \cdots , S^{l_n}_t \rbrace $   Each encoded sentence has dimension $d_S$ , which is a hyperparameter of our model  This layer constitutes a key component of our neural architectures and, as such, we evaluate our models considering sentence encoders trained end-to-end, using the BiLSTM attention (subsec:bilstminneratt) and BiLSTM max-pooling (subsec:bilstmmaxpool) architectures, and also transferred from the RCV1 and SNLI as fixed features 
Daily news relevance attention
Our proposed news relevance attention mechanism for all news released on a given day  The attention mechanism is introduced to tackle information overload  It was designed to “filter out” redundant or misleading news and focus on the relevant ones based solely on the news content  Formally, the layer outputs a Daily News (DN) embedding $DN^{sc}_t = \sum _{i=1}^{l_n} \beta _i S^{sc^{i}}_t$ , which is a linear combination of all encoded news on a given day $t$   This news-level attention uses the same equations as in eq:att, but with trainable weights $\lbrace W_{R}, b_{R}, v_{R}\rbrace $ , i e  the weights are segregated from the sentence encoder  fig:DNencoder, illustrates our relevance attention  Note that this layer was deliberately developed to be invariant to headlines permutation, as is the case with the linear combination formula above  The reason is that our price data is sampled daily and, as a consequence, we are not able to discriminate the market reaction for each intraday news 
News Temporal Context
Sequence layer with daily news embeddings $DN^{sc}_t$ as time steps  This layer aims to learn the temporal context of news, i e  the relationship between the news at day $t$ and the $T$ past days  It receives as input a chronologically ordered sequence of $T$ past Daily News embeddings ${\lbrace DN^{sc}_{t-T}, \cdots , DN^{sc}_t \rbrace }$ and outputs the news mode encoding Market News $MN^{sc}_t \in d_{MN}$   The sequence with $T$ time steps is encoded using a BiLSTM attention  The layer was designed to capture the temporal order that news are released and the current news novelty  i e  news that were repeated in the past can be “forgotten” based on the modulating gates of the LSTM network 
– Price mode
Price Encoder
Sequence layer analogous to News Temporal Context, but for the price mode  The input is the ordered sequence Daily Prices ${\lbrace DP^{sc}_{t-T}, \cdots , DP^{sc}_t \rbrace }$ of size $T$ , where each element the price feature defined in eq:pricemodevec  Particularly, the architecture consists of two stacked LSTM's  The first one outputs for each price feature time step a hidden vector that takes the temporal context into account  Then these hidden vectors are again passed to a second independent LSTM  The layer outputs the price mode encoding Market Price $MP^{sc}_t \in d_{MP}$   This encoding is the last hidden vector of the second LSTM Market 
– Stock embedding
Stock Encoder
Stock dense representation  The layer receives the discrete encoding $\mathcal {I}^{sc}_t$ indicating the sample stock code pass through a FC layer and outputs a stock embedding $E_{sc}$  
– Joint Representation
Merging
Feature-wise News, Price, and Stock modes concatenation  No trainable parameters 
Joint Representation Encoder
FC layer of size $d_{JR}$  
Multimodal learning with missing modes
During the training we feed into our neural model the price, news, and stock indicator data  The price and stock indicator modes data occur in all days  However, at the individual stock level we can have days that the company is not covered by the media  This feature imposes challenges to our multimodal training since neural networks are not able to handle missing modes without special intervention  A straightforward solution would be to consider only days with news released, disregarding the remaining samples  However, this approach has two main drawbacks  First, the “missing news” do not happen at random, or are attributed to measurement failure as is, for example, the case of multimodal tasks using mechanical sensors data  Conversely, as highlighted in BIBREF7 , BIBREF8 the same price behaviour results in distinct market reactions when accompanied or not by news  In other words, specifically to financial forecasting problems the absence or existence of news are highly informative 
Some methods were proposed in the multimodal literature to effectively treat informative missing modes or “informative missingness”, which is a characteristic refereed in the literature as learning with missing modalities BIBREF22   In this work, we directly model the news missingness as a feature of our text model temporal sequence by using the method initially proposed in BIBREF45 , BIBREF46 for clinical data with missing measurements and applied in the context of financial forecasting in BIBREF47   Specifically, we implement the Zeros & Imputation (ZI) method BIBREF46 in order to jointly learn the price mode and news relationship across all days of market activity 
The ZI implementation is described as follows: Before the daily news sequence is processed by the text temporal layer (described in itm:newstclayer) we input a 0 vector for all time steps with missing news and leave the news encoding unchanged otherwise  This step is called zero imputation  In addition, we concatenate feature-wise an indicator vector with value 1 for all vectors with zero imputation and 0 for the days with news 
As described in BIBREF47 , the ZI method endow a temporal sequence model with the ability to learn different representations depending on the news history and its relative time position  Moreover, it allows our model to predict the volatility for all days of our time series and, at the same time, take into account the current and past news informative missingness  Furthermore, the learnt positional news encoding works differently than a typical “masking”, where days without news are not passed through the LSTM cell  Masking the time steps would be losing information about the presence or absence of news concomitant with prices 
Experimental results and discussions
We aim to evaluate our hierarchical neural model in the light of three main aspects  First, we asses the importance of the different sentence encoders to our end-to-end models and how it compares to transferring the sentence encoder from our two auxiliary TL tasks  Second, we ablate our proposed news relevance attention (NRA) component to evaluate its importance  Finally, we consider a model that takes into consideration only the price mode (unimodal), i e  ignoring any architecture related to the text mode 
Before we define the baselines to asses the three aspects described above, we review in the next section the scores of the trained TL tasks 
Auxiliary transfer learning tasks
This section reports the performance of the auxiliary TL tasks considered in this work  Our ultimate goal is to indicate that our scores are in line with previous works All the architectures presented in sec:transferlearning are trained for a maximum of 50 epochs using mini-batch SGD with Adam optimizer BIBREF48   Moreover, at the end of each epoch, we evaluate the validation scores, which are accuracy (Stanfor SNLI dataset) and F1 (RCV1 dataset), and save the weights with the best values  Aiming to seeped up training, we implement early stopping with patience set to 8 epochs  That is, if the validation scores do not improve for more than 10 epochs we halt the training  Finally, we use Glove pre-trained word embeddings BIBREF37 as fixed features 
tbl:tlevaluation compares our test scores with state-of-the-art (SOTA) results reported in previous works  We can see that our scores for the SNLI task are very close to state-of-the-art 
Regarding the RCV1 dataset, our results consider only the headline content for training, while the refereed works consider both the news headline and message body  The reason for training using only the headlines is that both tasks are learnt with the sole purpose of transferring the sentence encoders to our main volatility prediction task, whose textual input is restricted to headlines 
Training setup
During the training of our hierarchical neural model described in sub:HAN we took special care to guard against overfitting  To this aim, we completely separate 2016 and 2017 as the test set and report our results on this “unseen” set  The remaining data is further split into training (2007 to 2013) and validation (2014 to 2015)  The model convergence during training is monitored in the validation set  We monitor the validation score of our model at the end of each epoch and store the network weights if the validation scores improves between two consecutive epochs  Additionally, we use mini-batch SGD with Adam optimizer and early stopping with patience set to eight epochs  The hyperparameter tunning is performed using grid search 
All training is performed using the proposed global model approach described in sub:globalmodel, which learns a model that takes into account the features of all the 40 stocks of our corpus  Using this approach our training set has a total of 97,903 samples  Moreover, during the SGD mini-batch sampling the past $T$ days of price and news history tensors and each stock sample stock indicator are randomly selected from the set of all 40 stocks 
Stocks universe result
In order to evaluate the contributions of each component of our neural model described in sub:HAN and the effect of using textual data to predict the volatility, we report our results using the following baselines:
- News (unimodal price only): This baseline completely ablates (i e  removes) any architecture related to the news mode, considering only the price encoding and the stock embedding components  Using this ablation we aim to evaluate the influence of news to the volatility prediction problem 
+ News (End-to-end Sentence Encoders) - NRA: This baseline ablates our proposed new relevance attention (NRA) component, and instead, makes use of the same Daily Averaging method in BIBREF26 , BIBREF27 , where all fixed-sized headline representations on a given day are averaged without taking into account the relevance of each news  We evaluate this baseline for both BiLSTM attention (Att) and BiLSTM max-pooling (MP) sentence encoders  Here, our goal is to asses the true contribution of our NRA component in the case SOTA sentence encoders are taken into account 
+ News (End-to-End W-L Att Sentence Encoder) + NRA: The Word-Level Attention (W-L Att) sentence encoder implements an attention mechanism directly on top of word embeddings, and, as such, does not consider the order of words in a sentence  This baseline complements the previous one, i e  it evaluates the influence of the sentence encoder when our full specification is considered 
+ News (TL Sentence Encoders) + NRA: Makes use of sentence encoders of our two auxiliary TL tasks as fixed features  This baseline aims to address the following questions, namely: What dataset and models are more suitable to transfer to our specific volatility forecasting problem; How End-to-End models, which are trained on top of word embeddings, perform compared to sentence encoders transferred from other tasks 
tbl:comparativeallsectors summarizes the test scores for the ablations discussed above  Our best model is the + News (BiLSTM Att) + NRA, which is trained end-to-end and uses our full architecture  The second best model, i e  + News (BiLSTM MP) + NRA, ranks slightly lower and only differs form the best model in terms of the sentence encoder  The former sentence encoder uses an attention layer (subsec:bilstminneratt) and the the last a max-pooling layer (subsec:bilstmmaxpool), where both layers are placed on top of the LSTM hidden states of each word 
Importantly, our experiments show that using news and price (multimodal) to predict the volatility improves the scores by 11% (MSE) and 9% (MAE) when compared with the – News (price only unimodal) model that considers only price features as explanatory variables 
When comparing the performance of End-to-End models and the TL auxiliary tasks the following can be observed: The end-to-end models trained with the two SOTA sentence encoders perform better than transferring sentence encoder from both auxiliary tasks  However, our experiments show that the same does not hold for models trained end-to-end relying on the simpler WL-Att sentence encoder, which ignores the order of words in a sentence  In other words, considering the appropriate TL task, it is preferable to transfer a SOTA sentence encoder trained on a larger dataset than learning a less robust sentence encoder in an end-to-end fashion  Moreover, initially, we thought that being the RCV1 a financial domain corpus it would demonstrate a superior performance when compared to the SNLI dataset  Still, the SNLI transfers better than RCV1  We hypothesize that the text categorization task (RCV1 dataset) is not able to capture complex sentence structures at the same level required to perform natural language inference  Particularly to the volatility forecasting problem, our TL results corroborates the same findings in BIBREF17 , where it was shown that SNLI dataset attains the best sentence encoding for a broad range of pure NLP tasks, including, among other, text categorization and sentiment analysis 
Significantly, experimental results in tbl:comparativeallsectors clearly demonstrate that our proposed news relevance attention (NRA) outperforms the News Averaging method proposed in previous studies BIBREF26 , BIBREF27   Even when evaluating our NRA component in conjunction with the more elementary W-L Att sentence encoder it surpass the results of sophisticated sentence encoder using a News Averaging approach  In other words, our results strongly points to the advantage of discriminating noisy from impacting news and the effectiveness of learning to attend the most relevant news 
Having analyzed our best model, we now turn to its comparative performance with respect to the widely regarded GARCH(1,1) model described in sec:GARCH 
We asses our model performance relative to GARCH(1,1) using standard loss metrics (MSE and MAE) and the regression-based accuracy specified in eq:regressionloss and measured in terms of the coefficient of determination $R^2$   In addition, we evaluate our model across two different volatility proxies: Garman-Klass ( $\widehat{\sigma _{GK}}$ ) (eq:volgk) and Parkinson ( $\widehat{\sigma _{PK}}$ ) (eq:volpk)  We note that, as reviewed in sub:evalution, these two volatility proxies are statically efficient and proper estimators of the next day volatility 
tbl:garchallsectors reports the comparative performance among our best Price + News model (+ News BiLSTM (MP) + NRA), our Price only (unimodal) model and GARCH(1,1)  The results clearly demonstrate the superiority of our model, being more accurate than GRACH for both volatility proxies  We note that evaluating the GARCH(1,1) model relying on standard MSE and MAE error metrics should be taken with a grain of salt  BIBREF35 provides the background theory and arguments supporting $R^2$ as the metric of choice to evaluate the predictive power of a volatility model  In any case, the outperformance or our model with respect to GARCH(1,1) permeates all three metrics, name $R^2$ , $MSE$ and $MAE$  
Sector-level results
Company sectors are expected to have different risk levels, in the sense that each sector is driven by different types of news and economic cycles  Moreover, by performing a sector-level analysis we were initially interested in understanding if the outperformance of our model with respect to GARCH(1,1) was the result of a learning bias to a given sector or if, as turned out to be the case, the superior performance of our model spreads across a diversified portfolio of sectors 
In order to evaluate the performance per sector, we first separate the constituents stocks for each sector in tbl:stockuniverse  Then, we calculate the same metrics discussed in the previous section for each sector individually 
tbl:garcheachsector reports our experimental results segregated by sector  We observe that the GRACH model accuracy, measured using the $R^2$ score, has a high degree of variability among sectors  For example, the accuracy ranges from 0 15 to 0 44 for the HealthCare and Energy sector, respectively  This high degree of variability is in agreement with previous results reported in BIBREF16 , but in the context of long-term (quarterly) volatility predictions  Although the GARCH(1,1) accuracy is sector-dependent, without any exception, our model using price and news as input clearly outperforms GRACH sector-wise  This fact allow us to draw the following conclusions:
Our model outperformance is persistent across sectors, i e  the characteristics of the results reported in tbl:garchallsectors permeates all sectors, rather than being composed of a mix of outperforming and underperforming sector contributions  This fact provides a strong evidence that our model is more accurate than GARCH(1,1) 
The proposed Global model approach discussed in sub:globalmodel is able to generalize well, i e  the patterns learnt are not biased to a given sector or stock 
One of the limitations of our work is to rely on proxies for the volatility estimation  Although these proxies are handy if only open, high, low and close daily price data is available, having high frequency price data we could estimate the daily volatility using the sum of squared intraday returns to measure the true daily latent volatility  For example, in evaluating the performance for the one-day-ahead GARCH(1,1) Yen/Dollar exchange rate BIBREF35 reports $R^2$ values of 0 237 and 0 392 using hourly and five minutes sampled intraday returns, respectively  However, we believe that utilizing intraday data would further improve our model performance 
Since our experimental results demonstrate the key aspect of the news relevance attention to model architecture we observe that intraday data would arguably ameliorate the learning process  Having intraday data would allow us to pair each individual news release with the instantaneous market price reaction  Using daily data we are losing part of this information by only measuring the aggregate effect of all news to the one-day-ahead prediction 
Conclusion
We study the joint effect of stock news and prices on the daily volatility forecasting problem  To the best of our knowledge, this work is one of the first studies aiming to predict short-term (daily) rather than long-term (quarterly or yearly) volatility taking news and price as explanatory variables and using a comprehensive dataset of news headlines at the individual stock level  Our hierarchical end-to-end model benefits from state-of-the-art approaches to encode text information and to deal with two main challenges in correlating news with market reaction: news relevance and novelty  That is, to address the problem of how to attend the most important news based purely on its content (news relevance attention) and to take into account the temporal information of past news (temporal context)  Additionally, we propose a multi-stock mini-batch + stock embedding method suitable to model commonality among stocks 
The experimental results show that our multimodal approach outperforms the GARCH(1,1) volatility model, which is the most prevalent econometric model for daily volatility predictions  The outperformance being sector-wise and demonstrates the effectiveness of combining price and news for short-term volatility forecasting  The fact that we outperform GARCH(1,1) for all analyzed sectors confirms the robustness of our proposed architecture and evidences that our global model approach generalizes well 
We ablated (i e  removed) different components of our neural architecture to assess its most relevant parts  To this aim, we replaced our proposed news relevance attention layer, which aims to attend the most important news on a given day, with a simpler architecture proposed in the literature, which averages the daily news  We found that our attention layer improves the results  Additionally, we ablated all the architecture related to the news mode and found that news enhances the forecasting accuracy 
Finally, we evaluated different sentence encoders, including those transfered from other NLP tasks, and concluded that they achieve better performance as compared to a plain Word-level attention sentence encoder trained end-to-end  However, they do not beat state-of-the-art sentence encoders trained end-to-end 
In order to contribute to the literature of Universal Sentence Encoders, we evaluated the performance of transferring sentence encoders from two different tasks to the volatility prediction problem  We showed that models trained on the Natural Language Inference (NLI) task are more suitable to forecasting problems than a financial domain dataset (Reuters RCV1)  By analyzing different architectures, we showed that a BiLSTM with max-pooling for the SNLI dataset provides the best sentence encoder 
In the future, we plan to make use of intraday prices to better assess the predictive power of our proposed models  Additionally, we would further extend our analysis to other stock market sectors 

ground_truth: CNN/DailyMail news highlights, New York Times Annotated Corpus, XSum

the CNN/DailyMail news highlights dataset BIBREF24, the New York Times Annotated Corpus (NYT; BIBREF25), XSum BIBREF22

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which stock market sector achieved the best performance?
Context 4:  This way, we obtain a synthetic parallel simplified-ordinary sentences  Both the synthetic sentences and the available parallel data are used as training data for the original NMT system 
Evaluation
We evaluate the performance of text simplification using neural machine translation on available parallel sentences and additional simplified sentences 
Dataset  We use two simplification datasets (WikiSmall and WikiLarge)  WikiSmall consists of ordinary and simplified sentences from the ordinary and simple English Wikipedias, which has been used as benchmark for evaluating text simplification BIBREF17 , BIBREF18 , BIBREF8   The training set has 89,042 sentence pairs, and the test set has 100 pairs  WikiLarge is also from Wikipedia corpus whose training set contains 296,402 sentence pairs BIBREF19 , BIBREF20   WikiLarge includes 8 (reference) simplifications for 2,359 sentences split into 2,000 for development and 359 for testing 
Metrics  Three metrics in text simplification are chosen in this paper  BLEU BIBREF5 is one traditional machine translation metric to assess the degree to which translated simplifications differed from reference simplifications  FKGL measures the readability of the output BIBREF23   A small FKGL represents simpler output  SARI is a recent text-simplification metric by comparing the output against the source and reference simplifications BIBREF20  
We evaluate the output of all systems using human evaluation  The metric is denoted as Simplicity BIBREF8   The three non-native fluent English speakers are shown reference sentences and output sentences  They are asked whether the output sentence is much simpler (+2), somewhat simpler (+1), equally (0), somewhat more difficult (-1), and much more difficult (-2) than the reference sentence 
Methods  We use OpenNMT BIBREF24 as the implementation of the NMT system for all experiments BIBREF5   We generally follow the default settings and training procedure described by Klein et al (2017)  We replace out-of-vocabulary words with a special UNK symbol  At prediction time, we replace UNK words with the highest probability score from the attention layer  OpenNMT system used on parallel data is the baseline system  To obtain a synthetic parallel training set, we back-translate a random sample of 100K sentences from the collected simplified corpora  OpenNMT used on parallel data and synthetic data is our model  The benchmarks are run on a Intel(R) Core(TM) i7-5930K CPU@3 50GHz, 32GB Mem, trained on 1 GPU GeForce GTX 1080 (Pascal) with CUDA v  8 0 
We choose three statistical text simplification systems  PBMT-R is a phrase-based method with a reranking post-processing step BIBREF18   Hybrid performs sentence splitting and deletion operations based on discourse representation structures, and then simplifies sentences with PBMT-R BIBREF25   SBMT-SARI BIBREF19 is syntax-based translation model using PPDB paraphrase database BIBREF26 and modifies tuning function (using SARI)  We choose two neural text simplification systems  NMT is a basic attention-based encoder-decoder model which uses OpenNMT framework to train with two LSTM layers, hidden states of size 500 and 500 hidden units, SGD optimizer, and a dropout rate of 0 3 BIBREF8   Dress is an encoder-decoder model coupled with a deep reinforcement learning framework, and the parameters are chosen according to the original paper BIBREF20   For the experiments with synthetic parallel data, we back-translate a random sample of 60 000 sentences from the collected simplified sentences into ordinary sentences  Our model is trained on synthetic data and the available parallel data, denoted as NMT+synthetic 
Results  Table 1 shows the results of all models on WikiLarge dataset  We can see that our method (NMT+synthetic) can obtain higher BLEU, lower FKGL and high SARI compared with other models, except Dress on FKGL and SBMT-SARI on SARI  It verified that including synthetic data during training is very effective, and yields an improvement over our baseline NMF by 2 11 BLEU, 1 7 FKGL and 1 07 SARI  We also substantially outperform Dress, who previously reported SOTA result  The results of our human evaluation using Simplicity are also presented in Table 1  NMT on synthetic data is significantly better than PBMT-R, Dress, and SBMT-SARI on Simplicity  It indicates that our method with simplified data is effective at creating simpler output 
Results on WikiSmall dataset are shown in Table 2  We see substantial improvements (6 37 BLEU) than NMT from adding simplified training data with synthetic ordinary sentences  Compared with statistical machine translation models (PBMT-R, Hybrid, SBMT-SARI), our method (NMT+synthetic) still have better results, but slightly worse FKGL and SARI  Similar to the results in WikiLarge, the results of our human evaluation using Simplicity outperforms the other models  In conclusion, Our method produces better results comparing with the baselines, which demonstrates the effectiveness of adding simplified training data 
Conclusion
In this paper, we propose one simple method to use simplified corpora during training of NMT systems, with no changes to the network architecture  In the experiments on two datasets, we achieve substantial gains in all tasks, and new SOTA results, via back-translation of simplified sentences into the ordinary sentences, and treating this synthetic data as additional training data  Because we do not change the neural network architecture to integrate simplified corpora, our method can be easily applied to other Neural Text Simplification (NTS) systems  We expect that the effectiveness of our method not only varies with the quality of the NTS system used for back-translation, but also depends on the amount of available parallel and simplified corpora  In the paper, we have only utilized data from Wikipedia for simplified sentences  In the future, many other text sources are available and the impact of not only size, but also of domain should be investigated 

ground_truth: Energy with accuracy of 0.538

Energy