HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

Abstract: Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods. Training the hierarchical encoder with these \emph{inaccurate} labels is challenging. Inspired by the recent work on pre-training transformer sentence encoders \cite{devlin:2018:arxiv}, we propose {\sc Hibert} (as shorthand for {\bf HI}erachical {\bf B}idirectional {\bf E}ncoder {\bf R}epresentations from {\bf T}ransformers) for document encoding and a method to pre-train it using unlabeled data. We apply the pre-trained {\sc Hibert} to our summarization model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times dataset. We also achieve the state-of-the-art performance on these two datasets.

Automatic document summarization is the task of rewriting a document into its shorter form while still retaining its important content. Over the years, many paradigms for document summarization have been explored (see Nenkova:McKeown:2011 for an overview). The most popular two among them are extractive approaches and abstractive approaches. As the name implies, extractive approaches generate summaries by extracting parts of the original document (usually sentences), while abstractive methods may generate new words or phrases which are not in the original document.
Extractive summarization is usually modeled as a sentence ranking problem with length constraints (e.g., max number of words or sentences). Top ranked sentences (under constraints) are selected as summaries. Early attempts mostly leverage manually engineered features BIBREF1 . Based on these sparse features, sentence are selected using a classifier or a regression model. Later, the feature engineering part in this paradigm is replaced with neural networks. cheng:2016:acl propose a hierarchical long short-term memory network (LSTM; BIBREF2 ) to encode a document and then use another LSTM to predict binary labels for each sentence in the document. This architecture is widely adopted recently BIBREF3 , BIBREF4 , BIBREF5 . Our model also employs a hierarchical document encoder, but we adopt a hierarchical transformer BIBREF6 rather a hierarchical LSTM. Because recent studies BIBREF6 , BIBREF0 show the transformer model performs better than LSTM in many tasks.
Abstractive models do not attract much attention until recently. They are mostly based on sequence to sequence (seq2seq) models BIBREF7 , where a document is viewed a sequence and its summary is viewed as another sequence. Although seq2seq based summarizers can be equipped with copy mechanism BIBREF8 , BIBREF9 , coverage model BIBREF9 and reinforcement learning BIBREF10 , there is still no guarantee that the generated summaries are grammatical and convey the same meaning as the original document does. It seems that extractive models are more reliable than their abstractive counterparts.
However, extractive models require sentence level labels, which are usually not included in most summarization datasets (most datasets only contain document-summary pairs). Sentence labels are usually obtained by rule-based methods (e.g., maximizing the ROUGE score between a set of sentences and reference summaries) and may not be accurate. Extractive models proposed recently BIBREF11 , BIBREF3 employ hierarchical document encoders and even have neural decoders, which are complex. Training such complex neural models with inaccurate binary labels is challenging. We observed in our initial experiments on one of our dataset that our extractive model (see Section "Extractive Summarization" for details) overfits to the training set quickly after the second epoch, which indicates the training set may not be fully utilized. Inspired by the recent pre-training work in natural language processing BIBREF12 , BIBREF13 , BIBREF0 , our solution to this problem is to first pre-train the “complex”' part (i.e., the hierarchical encoder) of the extractive model on unlabeled data and then we learn to classify sentences with our model initialized from the pre-trained encoder. In this paper, we propose Hibert, which stands for HIerachical Bidirectional Encoder Representations from Transformers. We design an unsupervised method to pre-train Hibert for document modeling. We apply the pre-trained Hibert to the task of document summarization and achieve state-of-the-art performance on both the CNN/Dailymail and New York Times dataset.
In this section, we introduce work on extractive summarization, abstractive summarization and pre-trained natural language processing models. For a more comprehensive review of summarization, we refer the interested readers to Nenkova:McKeown:2011 and Mani:01.
In this section, we present our model Hibert. We first introduce how documents are represented in Hibert. We then describe our method to pre-train Hibert and finally move on to the application of Hibert to summarization.
Let $\mathcal {D} = (S_1, S_2, \dots , S_{| \mathcal {D} |})$ denote a document, where $S_i = (w_1^i, w_2^i, \dots , w_{|S_i|}^i)$ is a sentence in $\mathcal {D}$ and $w_j^i$ a word in $S_i$ . Note that following common practice in natural language processing literatures, $w_{|S_i|}^i$ is an artificial EOS (End Of Sentence) token. To obtain the representation of $\mathcal {D}$ , we use two encoders: a sentence encoder to transform each sentence in $\mathcal {D}$ to a vector and a document encoder to learn sentence representations given their surrounding sentences as context. Both the sentence encoder and document encoder are based on the Transformer encoder described in vaswani:2017:nips. As shown in Figure 1 , they are nested in a hierarchical fashion. A transformer encoder usually has multiple layers and each layer is composed of a multi-head self attentive sub-layer followed by a feed-forward sub-layer with residual connections BIBREF30 and layer normalizations BIBREF31 . For more details of the Transformer encoder, we refer the interested readers to vaswani:2017:nips. To learn the representation of $S_i$ , $S_i= (w_1^i, w_2^i, \dots , w_{|S_i|}^i)$ is first mapped into continuous space 
$$\begin{split}
\mathbf {E}_i = (\mathbf {e}_1^i, \mathbf {e}_2^i, \dots , \mathbf {e}_{|S_i|}^i) \\
\quad \quad \text{where} \quad \mathbf {e}_j^i = e(w_j^i) + \mathbf {p}_j
\end{split}$$   (Eq. 6) 
 where $e(w_j^i)$ and $\mathbf {p}_j$ are the word and positional embeddings of $w_j^i$ , respectively. The word embedding matrix is randomly initialized and we adopt the sine-cosine positional embedding BIBREF6 . Then the sentence encoder (a Transformer) transforms $\mathbf {E}_i$ into a list of hidden representations $(\mathbf {h}_1^i, \mathbf {h}_2^i, \dots , \mathbf {h}_{|S_i|}^i)$ . We take the last hidden representation $\mathbf {h}_{|S_i|}^i$ (i.e., the representation at the EOS token) as the representation of sentence $S_i$ . Similar to the representation of each word in $S_i$ , we also take the sentence position into account. The final representation of $S_i$ is 
$$\hat{\mathbf {h}}_i = \mathbf {h}_{|S_i|}^i + \mathbf {p}_i$$   (Eq. 8) 
Note that words and sentences share the same positional embedding matrix.
In analogy to the sentence encoder, as shown in Figure 1 , the document encoder is yet another Transformer but applies on the sentence level. After running the Transformer on a sequence of sentence representations $( \hat{\mathbf {h}}_1, \hat{\mathbf {h}}_2, \dots , \hat{\mathbf {h}}_{|\mathcal {D}|} )$ , we obtain the context sensitive sentence representations $( \mathbf {d}_1, \mathbf {d}_2, \dots , \mathbf {d}_{|\mathcal {D}|} )$ . Now we have finished the encoding of a document with a hierarchical bidirectional transformer encoder Hibert. Note that in previous work, document representation are also learned with hierarchical models, but each hierarchy is a Recurrent Neural Network BIBREF3 , BIBREF21 or Convolutional Neural Network BIBREF11 . We choose the Transformer because it outperforms CNN and RNN in machine translation BIBREF6 , semantic role labeling BIBREF32 and other NLP tasks BIBREF0 . In the next section we will introduce how we train Hibert with an unsupervised training objective.
Most recent encoding neural models used in NLP (e.g., RNNs, CNNs or Transformers) can be pre-trained by predicting a word in a sentence (or a text span) using other words within the same sentence (or span). For example, ELMo BIBREF12 and OpenAI-GPT BIBREF13 predict a word using all words on its left (or right); while word2vec BIBREF33 predicts one word with its surrounding words in a fixed window and BERT BIBREF0 predicts (masked) missing words in a sentence given all the other words.
All the models above learn the representation of a sentence, where its basic units are words. Hibert aims to learn the representation of a document, where its basic units are sentences. Therefore, a natural way of pre-training a document level model (e.g., Hibert) is to predict a sentence (or sentences) instead of a word (or words). We could predict a sentence in a document with all the sentences on its left (or right) as in a (document level) language model. However, in summarization, context on both directions are available. We therefore opt to predict a sentence using all sentences on both its left and right.
Specifically, suppose $\mathcal {D} = (S_1, S_2, \dots , S_{| \mathcal {D} |})$ is a document, where $S_i = (w_1^i, w_2^i, \dots , w_{|S_i|}^i)$ is a sentence in it. We randomly select 15% of the sentences in $\mathcal {D}$ and mask them. Then, we predict these masked sentences. The prediction task here is similar with the Cloze task BIBREF34 , BIBREF0 , but the missing part is a sentence. However, during test time the input document is not masked, to make our model can adapt to documents without masks, we do not always mask the selected sentences. Once a sentence is selected (as one of the 15% selected masked sentences), we transform it with one of three methods below. We will use an example to demonstrate the transformation. For instance, we have the following document and the second sentence is selected:
William Shakespeare is a poet . He died in 1616 . He is regarded as the greatest writer .
In 80% of the cases, we mask the selected sentence (i.e., we replace each word in the sentence with a mask token [MASK]). The document above becomes William Shakespeare is a poet . [MASK] [MASK] [MASK] [MASK] [MASK] He is regarded as the greatest writer . (where “He died in 1616 . ” is masked).
In 10% of the cases, we keep the selected sentence as it is. This strategy is to simulate the input document during test time (with no masked sentences).
In the rest 10% cases, we replace the selected sentence with a random sentence. In this case, the document after transformation is William Shakespeare is a poet . Birds can fly . He is regarded as the greatest writer . The second sentence is replaced with “Birds can fly .” This strategy intends to add some noise during training and make the model more robust.
After the application of the above procedures to a document $\mathcal {D} = (S_1, S_2, \dots , S_{| \mathcal {D} |})$ , we obtain the masked document $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$ . Let $\mathcal {K} $ denote the set of indicies of selected sentences in $\mathcal {D}$ . Now we are ready to predict the masked sentences $\mathcal {M} = \lbrace S_k | k \in \mathcal {K} \rbrace $ using $\widetilde{ \mathcal {D} }$ . We first apply the hierarchical encoder Hibert in Section "Conclusions" to $\widetilde{ \mathcal {D} }$ and obtain its context sensitive sentence representations $( \tilde{ \mathbf {d}_1 }, \tilde{ \mathbf {d}_2 }, \dots , \tilde{ \mathbf {d}_{| \mathcal {D} |} } )$ . We will demonstrate how we predict the masked sentence $S_k = (w_0^k, w_1^k, w_2^k, \dots , w_{|S_k|}^k)$ one word per step ( $w_0^k$ is an artificially added BOS token). At the $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$0 th step, we predict $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$1 given $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$2 and $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$3 . $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$4 already encodes the information of $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$5 with a focus around its $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$6 th sentence $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$7 . As shown in Figure 1 , we employ a Transformer decoder BIBREF6 to predict $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$8 with $\widetilde{ \mathcal {D} }= (\tilde{S_1}, \tilde{S_2}, \dots , \tilde{S_{| \mathcal {D} |}})$9 as its additional input. The transformer decoder we used here is slightly different from the original one. The original decoder employs two multi-head attention layers to include both the context in encoder and decoder, while we only need one to learn the decoder context, since the context in encoder is a vector (i.e., $\mathcal {K} $0 ). Specifically, after applying the word and positional embeddings to ( $\mathcal {K} $1 ), we obtain $\mathcal {K} $2 (also see Equation 6 ). Then we apply multi-head attention sub-layer to $\mathcal {K} $3 : 
$$\begin{split}
\tilde{\mathbf {h}_{j-1}} &= \text{MultiHead}(\mathbf {q}_{j-1}, \mathbf {K}_{j-1}, \mathbf {V}_{j-1}) \\
\mathbf {q}_{j-1} &= \mathbf {W}^Q \: \tilde{\mathbf {e}_{j-1}^k} \\
\mathbf {K}_{j-1} &= \mathbf {W}^K \: \widetilde{ \mathbf {E} }^k_{1:j-1} \\
\mathbf {K}_{j-1} &= \mathbf {W}^V \: \widetilde{ \mathbf {E} }^k_{1:j-1}
\end{split}$$   (Eq. 13) 
 where $\mathbf {q}_{j-1}$ , $\mathbf {K}_{j-1}$ , $\mathbf {V}_{j-1}$ are the input query, key and value matrices of the multi-head attention function BIBREF6 $\text{MultiHead}(\cdot , \cdot , \cdot )$ , respectively. $\mathbf {W}^Q \in \mathbb {R}^{d \times d}$ , $\mathbf {W}^K \in \mathbb {R}^{d \times d}$ and $\mathbf {W}^V \in \mathbb {R}^{d \times d}$ are weight matrices.
Then we include the information of $\widetilde{ \mathcal {D} }$ by addition: 
$$\tilde{\mathbf {x}_{j-1}} = \tilde{\mathbf {h}_{j-1}} + \tilde{ \mathbf {d}_k }$$   (Eq. 14) 
We also follow a feedforward sub-layer (one hidden layer with ReLU BIBREF35 activation function) after $\tilde{\mathbf {x}_{j-1}}$ as in vaswani:2017:nips: 
$$\tilde{\mathbf {g}_{j-1}} = \mathbf {W}^{ff}_2 \max (0, \mathbf {W}^{ff}_1 \tilde{\mathbf {x}_{j-1}} + \mathbf {b}_1) + \mathbf {b}_2$$   (Eq. 15) 
Note that the transformer decoder can have multiple layers by applying Equation ( 13 ) to ( 15 ) multiple times and we only show the computation of one layer for simplicity.
The probability of $w_j^k$ given $w_0^k,\dots ,w_{j-1}^k$ and $\widetilde{ \mathcal {D} }$ is: 
$$p( w_j^k | w_{0:j-1}^k, \widetilde{ \mathcal {D} } ) = \text{softmax}( \mathbf {W}^O \: \tilde{\mathbf {g}_{j-1}} )$$   (Eq. 16) 
Finally the probability of all masked sentences $ \mathcal {M} $ given $\widetilde{ \mathcal {D} }$ is 
$$p(\mathcal {M} | \widetilde{ \mathcal {D} }) = \prod _{k \in \mathcal {K}} \prod _{j=1}^{|S_k|} p(w_j^k | w_{0:j-1}^k, \widetilde{ \mathcal {D} })$$   (Eq. 17) 
The model above can be trained by minimizing the negative log-likelihood of all masked sentences given their paired documents. We can in theory have unlimited amount of training data for Hibert, since they can be generated automatically from (unlabeled) documents. Therefore, we can first train Hibert on large amount of data and then apply it to downstream tasks. In the next section, we will introduce its application to document summarization.
Extractive summarization selects the most important sentences in a document as its summary. In this section, summarization is modeled as a sequence labeling problem. Specifically, a document is viewed as a sequence of sentences and a summarization model is expected to assign a True or False label for each sentence, where True means this sentence should be included in the summary. In the following, we will introduce the details of our summarization model based Hibert.
Let $\mathcal {D} = (S_1, S_2, \dots , S_{| \mathcal {D} |})$ denote a document and $Y = (y_1, y_2, \dots , y_{| \mathcal {D} |})$ its sentence labels (methods for obtaining these labels are in Section "Datasets" ). As shown in Figure 2 , we first apply the hierarchical bidirectional transformer encoder Hibert to $\mathcal {D}$ and yields the context dependent representations for all sentences $( \mathbf {d}_1, \mathbf {d}_2, \dots , \mathbf {d}_{|\mathcal {D}|} )$ . The probability of the label of $S_i$ can be estimated using an additional linear projection and a softmax: 
$$p( y_i | \mathcal {D} ) = \text{softmax}(\mathbf {W}^S \: \mathbf {d}_i)$$   (Eq. 20) 
where $\mathbf {W}^S \in \mathbb {R}^{2 \times d}$ . The summarization model can be trained by minimizing the negative log-likelihood of all sentence labels given their paired documents.
In this section we assess the performance of our model on the document summarization task. We first introduce the dataset we used for pre-training and the summarization task and give implementation details of our model. We also compare our model against multiple previous models.
We conducted our summarization experiments on the non-anonymous version CNN/Dailymail (CNNDM) dataset BIBREF36 , BIBREF9 , and the New York Times dataset BIBREF37 , BIBREF38 . For the CNNDM dataset, we preprocessed the dataset using the scripts from the authors of see:2017:acl. The resulting dataset contains 287,226 documents with summaries for training, 13,368 for validation and 11,490 for test. Following BIBREF38 , BIBREF37 , we created the NYT50 dataset by removing the documents whose summaries are shorter than 50 words from New York Times dataset. We used the same training/validation/test splits as in xu:2019:arxiv, which contain 137,778 documents for training, 17,222 for validation and 17,223 for test. To create sentence level labels for extractive summarization, we used a strategy similar to nallapati:2017:aaai. We label the subset of sentences in a document that maximizes Rouge BIBREF39 (against the human summary) as True and all other sentences as False.
To unsupervisedly pre-train our document model Hibert (see Section "Pre-training" for details), we created the GIGA-CM dataset (totally 6,626,842 documents and 2,854 million words), which includes 6,339,616 documents sampled from the English Gigaword dataset and the training split of the CNNDM dataset. We used the validation set of CNNDM as the validation set of GIGA-CM as well. As in see:2017:acl, documents and summaries in CNNDM, NYT50 and GIGA-CM are all segmented and tokenized using Stanford CoreNLP toolkit BIBREF40 . To reduce the vocabulary size, we applied byte pair encoding (BPE; BIBREF41 ) to all of our datasets. To limit the memory consumption during training, we limit the length of each sentence to be 50 words (51th word and onwards are removed) and split documents with more than 30 sentences into smaller documents with each containing at most 30 sentences.
Our model is trained in three stages, which includes two pre-training stages and one finetuning stage. The first stage is the open-domain pre-training and in this stage we train Hibert with the pre-training objective (Section "Pre-training" ) on GIGA-CM dataset. In the second stage, we perform the in-domain pre-training on the CNNDM (or NYT50) dataset still with the same pre-training objective. In the final stage, we finetune Hibert in the summarization model (Section "Extractive Summarization" ) to predict extractive sentence labels on CNNDM (or NYT50).
The sizes of the sentence and document level Transformers as well as the Transformer decoder in Hibert are the same. Let $L$ denote the number of layers in Transformer, $H$ the hidden size and $A$ the number of attention heads. As in BIBREF6 , BIBREF0 , the hidden size of the feedforward sublayer is $4H$ . We mainly trained two model sizes: $\text{\sc Hibert}_S$ ( $L=6$ , $H=512$ and $A=8$ ) and $\text{\sc Hibert}_M$ ( $L=6$ , $H$0 and $H$1 ). We trained both $H$2 and $H$3 on a single machine with 8 Nvidia Tesla V100 GPUs with a batch size of 256 documents. We optimized our models using Adam with learning rate of 1e-4, $H$4 , $H$5 , L2 norm of 0.01, learning rate warmup 10,000 steps and learning rate decay afterwards using the strategies in vaswani:2017:nips. The dropout rate in all layers are 0.1. In pre-training stages, we trained our models until validation perplexities do not decrease significantly (around 45 epochs on GIGA-CM dataset and 100 to 200 epochs on CNNDM and NYT50). Training $H$6 for one epoch on GIGA-CM dataset takes approximately 20 hours.
Our models during fine-tuning stage can be trained on a single GPU. The hyper-parameters are almost identical to these in the pre-training stages except that the learning rate is 5e-5, the batch size is 32, the warmup steps are 4,000 and we train our models for 5 epochs. During inference, we rank sentences using $p( y_i | \mathcal {D} ) $ (Equation ( 20 )) and choose the top $K$ sentences as summary, where $K$ is tuned on the validation set.
We evaluated the quality of summaries from different systems automatically using ROUGE BIBREF39 . We reported the full length F1 based ROUGE-1, ROUGE-2 and ROUGE-L on the CNNDM and NYT50 datasets. We compute ROUGE scores using the ROUGE-1.5.5.pl script.
Additionally, we also evaluated the generated summaries by eliciting human judgments. Following BIBREF11 , BIBREF4 , we randomly sampled 20 documents from the CNNDM test set. Participants were presented with a document and a list of summaries produced by different systems. We asked subjects to rank these summaries (ties allowed) by taking informativeness (is the summary capture the important information from the document?) and fluency (is the summary grammatical?) into account. Each document is annotated by three different subjects.
Our main results on the CNNDM dataset are shown in Table 1 , with abstractive models in the top block and extractive models in the bottom block. Pointer+Coverage BIBREF9 , Abstract-ML+RL BIBREF10 and DCA BIBREF42 are all sequence to sequence learning based models with copy and coverage modeling, reinforcement learning and deep communicating agents extensions. SentRewrite BIBREF26 and InconsisLoss BIBREF25 all try to decompose the word by word summary generation into sentence selection from document and “sentence” level summarization (or compression). Bottom-Up BIBREF27 generates summaries by combines a word prediction model with the decoder attention model. The extractive models are usually based on hierarchical encoders (SummaRuNNer; BIBREF3 and NeuSum; BIBREF11 ). They have been extended with reinforcement learning (Refresh; BIBREF4 and BanditSum; BIBREF20 ), Maximal Marginal Relevance (NeuSum-MMR; BIBREF21 ), latent variable modeling (LatentSum; BIBREF5 ) and syntactic compression (JECS; BIBREF38 ). Lead3 is a baseline which simply selects the first three sentences. Our model $\text{\sc Hibert}_S$ (in-domain), which only use one pre-training stage on the in-domain CNNDM training set, outperforms all of them and differences between them are all significant with a 0.95 confidence interval (estimated with the ROUGE script). Note that pre-training $\text{\sc Hibert}_S$ (in-domain) is very fast and it only takes around 30 minutes for one epoch on the CNNDM training set. Our models with two pre-training stages ( $\text{\sc Hibert}_S$ ) or larger size ( $\text{\sc Hibert}_M$ ) perform even better and $\text{\sc Hibert}_M$ outperforms BERT by 0.5 ROUGE. We also implemented two baselines. One is the hierarchical transformer summarization model (HeriTransfomer; described in "Extractive Summarization" ) without pre-training. Note the setting for HeriTransfomer is ( $L=4$ , $H=300$ and $A=4$ ) . We can see that the pre-training (details in Section "Pre-training" ) leads to a +1.25 ROUGE improvement. Another baseline is based on a pre-trained BERT BIBREF0 and finetuned on the CNNDM dataset. We used the $\text{BERT}_{\text{base}}$ model because our 16G RAM V100 GPU cannot fit $\text{BERT}_{\text{large}}$ for the summarization task even with batch size of 1. The positional embedding of BERT supports input length up to 512 words, we therefore split documents with more than 10 sentences into multiple blocks (each block with 10 sentences). We feed each block (the BOS and EOS tokens of each sentence are replaced with [CLS] and [SEP] tokens) into BERT and use the representation at [CLS] token to classify each sentence. Our model $\text{\sc Hibert}_S$1 outperforms BERT by 0.4 to 0.5 ROUGE despite with only half the number of model parameters ( $\text{\sc Hibert}_S$2 54.6M v.s. BERT 110M). Results on the NYT50 dataset show the similar trends (see Table 2 ). EXTRACTION is a extractive model based hierarchical LSTM and we use the numbers reported by xu:2019:arxiv. The improvement of $\text{\sc Hibert}_S$3 over the baseline without pre-training (HeriTransformer) becomes 2.0 ROUGE. $\text{\sc Hibert}_S$4 (in-domain), $\text{\sc Hibert}_S$5 (in-domain), $\text{\sc Hibert}_S$6 and $\text{\sc Hibert}_S$7 all outperform BERT significantly according to the ROUGE script.
We also conducted human experiment with 20 randomly sampled documents from the CNNDM test set. We compared our model $\text{\sc Hibert}_M$ against Lead3, DCA, Latent, BERT and the human reference (Human). We asked the subjects to rank the outputs of these systems from best to worst. As shown in Table 4 , the output of $\text{\sc Hibert}_M$ is selected as the best in 30% of cases and we obtained lower mean rank than all systems except for Human. We also converted the rank numbers into ratings (rank $i$ to $7-i$ ) and applied student $t$ -test on the ratings. $\text{\sc Hibert}_M$ is significantly different from all systems in comparison ( $p < 0.05$ ), which indicates our model still lags behind Human, but is better than all other systems.
As mentioned earlier, our pre-training includes two stages. The first stage is the open-domain pre-training stage on the GIGA-CM dataset and the following stage is the in-domain pre-training on the CNNDM (or NYT50) dataset. As shown in Table 3 , we pretrained $\text{\sc Hibert}_S$ using only open-domain stage (Open-Domain), only in-domain stage (In-Domain) or both stages (Open+In-Domain) and applied it to the CNNDM summarization task. Results on the validation set of CNNDM indicate the two-stage pre-training process is necessary.
The core part of a neural extractive summarization model is the hierarchical document encoder. We proposed a method to pre-train document level hierarchical bidirectional transformer encoders on unlabeled data. When we only pre-train hierarchical transformers on the training sets of summarization datasets with our proposed objective, application of the pre-trained hierarchical transformers to extractive summarization models already leads to wide improvement of summarization performance. Adding the large open-domain dataset to pre-training leads to even better performance. In the future, we plan to apply models to other tasks that also require hierarchical document encodings (e.g., document question answering). We are also interested in improving the architectures of hierarchical document encoders and designing other objectives to train hierarchical transformers.

Figure and table captions:
Figure 1: The architecture of HIBERT during training. senti is a sentence in the document above, which has four sentences in total. sent3 is masked during encoding and the decoder predicts the original sent3.
Figure 2: The architecture of our extractive summarization model. The sentence and document level transformers can be pretrained.
Table 1: Results of various models on the CNNDM test set using full-length F1 ROUGE-1 (R-1), ROUGE-2 (R2), and ROUGE-L (R-L).
Table 4: Human evaluation: proportions of rankings and mean ranks (MeanR; lower is better) of various models.
Table 2: Results of various models on the NYT50 test set using full-length F1 ROUGE. HIBERTS (indomain) and HIBERTM (in-domain) only uses one pretraining stage on the NYT50 training set.
Table 3: Results of summarization model (HIBERTS setting) with different pre-training strategies on the CNNDM validation set using full-length F1 ROUGE.

Questions about this paper:
Is the baseline a non-heirarchical model like BERT?

Answers:
There were hierarchical and non-hierarchical baselines; BERT was one of those baselines