Year: 2019

Abstract:
Pretrained language models attract lots of attentions, and they take advantage of the two-stages training process: pretraining on huge corpus and finetuning on specific tasks. Thereinto, BERT (Devlin et al., 2019) is a Transformer (Vaswani et al., 2017) based model and has been the state-of-the-art for many kinds of Nature Language Processing (NLP) tasks. However, BERT cannot take text longer than the maximum length as input since the maximum length is predefined during pretraining. When we apply BERT to long text tasks, e.g., document-level text summarization: 1) Truncating inputs by the maximum sequence length will decrease performance, since the model cannot capture long dependency and global information ranging the whole document. 2) Extending the maximum length requires re-pretraining which will cost a mass of time and computing resources. Whats even worse is that the computational complexity will increase quadratically with the length, which will result in an unacceptable training time. To resolve these problems, we propose to apply Transformer to only model local dependency and recurrently capture long dependency by inserting multi-channel LSTM into each layer of BERT. The proposed model is named as BERT-AL (BERT for Arbitrarily Long Document Understanding) and it can accept arbitrarily long input without re-pretraining from scratch. We demonstrate BERT-ALs effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset. Furthermore, our method can be adapted to other Transformer based models, e.g., XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), for various NLP tasks with long text.

heading: 1 INTRODUCTION
text: In recent years, neural networks are proposed to solve various NLP tasks. Especially, pretrained language models (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019) attract lots of attentions, which take advantage of the two-stages training process: pretraining on unlabeled corpus and then finetuning on specific tasks. The most famous model is BERT. BERT and its varieties are the state-of-the-art for many kinds of NLP tasks. The power of BERT does not only come from its architecture of networks, but also because it can be pretrained on a mass of text as a masked language model.
BERT can be used to solve almost all NLP tasks, and especially it can perform best on datasets with short text, e.g., GLUE (Wang et al., 2018) and Squad (Rajpurkar et al., 2016). However, there are still many document-level tasks, e.g., document-level text summarization (Hermann et al., 2015), long-document machine reading comprehension (Hewlett et al., 2016) and long text classification (Zhang et al., 2015). BERT cannot be finetuned for such tasks with long text directly or perform good on these tasks, since it is limited by the fixed-length position embedding which was determined during pretraining. We employ document-level text summarization as an example, which usually has longer text than the maximum sequence length of BERT.
Intuitively, there are two alternative solutions: 1) Truncating inputs by the maximum sequence length to fit the BERTs constraint. 2) Increasing the length of position embedding and re-pretraining the BERT from scratch. The first method will decrease performance, obviously, since some useful information placing behind the maximum sequence length is discarded by truncating. E.g., for text summarization, if the key point sentence locates at the end of the document, it never be recalled
even though the model is powerful. For the second method, re-pretraining the BERT from scratch will cost a mass of computing time and resources. Whats even worse is that the computational complexity will increase quadratically with the length, which will result in an unacceptable training time. For example, the XLNet-Large (Yang et al., 2019) costs 2.5 days on 512 TPU v3 chips and the RoBERTa (Liu et al., 2019) costs 1 day on 1024 V100 GPUs.
To resolve these problems, we propose BERT-AL (BERT for Arbitrarily Long Document Understanding) model that extracts local features by applying parallel multi-layer Transformers into chunked input and employs multi-channel LSTMs to capture global information crossing Transformers. This fusion breaks the limitation of BERT by the ability of capture unlimited sequential information from LSTM, and makes it be able to process arbitrarily long text. On the other hand, it also skillfully avoids gradient vanishing and exploding problem (Li et al., 2018) of LSTM since only few steps are required by multi-channel LSTM. Therefore, BERT-AL can solve the problems of original BERT: 1) For document-level tasks, BERT-AL can directly take all text as the input without truncating. 2) When the input length is longer then maximum sequence length of BERT, BERT-AL still can load the pretrained BERT model, which avoids pretraining much longer BERT model from scratch.
We demonstrate BERT-ALs effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset (Hermann et al., 2015) with various maximum sequence lengths of pretrained BERT. The results prove that BERT-AL can consistently outperform BERTSUM (Liu, 2019) which is the BERT-based state-of-the-art, when finetuning from the pretrained BERT model with the same maximum sequence length. Additionally, BERT-AL is a general NLP model which has no specific setting for text summarization, so it can be easily adapted to other tasks with long text, e.g., document-level machine reading comprehension and long text classification. Furthermore, the method, applying multi-channel LSTM on hidden states from transformers, also can be used in other Transformer based pretrained models, e.g., XLNet and RoBERTa.
In summary, contributions of this paper are shown as follow.
1) We propose a new architecture that combines Transformer and LSTM, which resolve the problem Transformer cannot be used in very long text, and skillfully avoid LSTMs drop backs.
2) We propose multi-channel LSTM only applied on the corresponding position across different Transformers, which can take fully advantage of LSTM without hurting Transformers representation too much.
3) We conduct experiments to prove BERT-AL can outperform other models with the BERT pretrained under the same maximum sequence length, and can perform very close to BERTSUM with at least twice the maximum length than ours.

heading: 2 BACKGROUND

heading: 2.1 BERT
text: According to the original implementation described in Vaswani et al. (2017), BERT is a multi-layer bidirectional Transformer encoder, and Multi-Head Self-Attention is the key structure of Transformer encoder.
For the input H ∈ RL×D of each Transformer layer, H will be mapped to three different spaces, named as Q, K and V , respectively. Self-attention computes the dot products between Q and K to roughly get the weight matrix, and then multiply V to get the hidden representation. Multi-head mechanism promotes the power of Transformer because it allows the model to jointly attend to information from different representation subspaces at different positions.
BERT employs two tasks for pretraining: Masked Language Model and Next Sentence Prediction.
Masked Language Model: BERT proposes a bidirected language model, which replaces 15% words in text by [MASK] label and then predicts which words they are. To mitigate the mismatch between pretraining and finetuning, i.e., the [MASK] token does not appear during finetuning, they do not always replace masked words with the actual [MASK] token. Then, BERT will predict the original token with cross entropy loss.
Next Sentence Prediction: BERT is also pretrained by the next sentence prediction task. Specifically, when choosing the sentences A and B for each pretraining example, 50% of the time B is the actual next sentence that follows A (labeled as IsNext), and 50% of the time it is a random sentence from the corpus (labeled as NotNext). Then, BERT will predict whether the sentence B is the actual next sentence that follows A or not with cross entropy loss.
For most of all NLP tasks, BERT concatenates different parts of input into a sequence beginning with [CLS] token and inserts [SEP ] token between two different parts. Before going through Transformer layers, BERT merges three different embeddings into one, i.e., word embedding, position embedding and segment embedding. Thereinto, position embedding is the information about the relative or absolute position of the tokens in the sequence. Since the model contains no recurrence and no convolution, position embedding is added to make use of the order of the sequence. There are two implements of position embedding, i.e., learned position embedding and sine and cosine functions of different frequencies, but both of them cannot expand to longer without a performance regression (Wang et al., 2019a).

heading: 2.2 BERTSUM
text: BERTSUM is an extension of BERT on extractive text summarization task and it truncates only the first 512 tokens as input. To select sentences, BERTSUM adds [CLS] to the head of each sentence and [SEP ] to the tail of each sentence indicating the end of that sentence. The following summarization layer will score each [CLS] which presents the importance of that sentence. Finally, sentences with top 3 highest scores compose the summary. There are three summarization layers proposed in Liu (2019):
1) Simple Classifier: only adds a linear layer on each [CLS] and uses a sigmoid function to get the predicted score:
Ŷi = σ(WoTi + bo) (1)
where Ti is the logit of ith sentence, σ is the Sigmoid function.
2) Inter-Sentence Transformer: applies more Transformer layers into sentence-level representations as follows.
h̃l = LN(hl−1 +MHAtt(hl−1))
hl = LN(h̃l + FFN(h̃l))
Ŷi = σ(Wohi + bo)
(2)
where h0 = PosEmb(T ) and T are the sentence vectors output by BERT, PosEmb is the function of adding position embeddings to T ; LN is the layer normalization operation; FFN is a feedforward network; MHAtt is the multi-head attention operation; the superscript l indicates the depth of the stacked layer. The final output layer is still a sigmoid classifier and Ti is the logit of ith sentence.
3) Recurrent Neural Network: applies an LSTM layer over the BERT outputs to learn summarization-specific features. At time step i, the input to the LSTM layer is the BERT output Ti, and the output is calculated as:(
Fi Ii Oi Gi
) = LNh(Whhi−1 + LNx(WxTi))
Ci = σ(Fi ⊙ Ci−1) + σ(Ii) ⊙ tanh(Gi−1)
hi = σ(Ot) ⊙ tanh(LNc(Ct))
Ŷi = σ(Wohi + bo)
(3)
where Fi, Ii, Oi are forget gates, input gates, output gates; Gi is the hidden vector and Ci is the memory vector; hi is the output vector; LNh, LNx, LNc are there difference layer normalization operations; The final output layer is also a sigmoid classifier and Ti is the logit of ith sentence.
However, we argue that the way of truncation will result in the loss of information in the latter part of the document. If the key sentences locate at the end of the document, they will never be recalled
even though the model is powerful. In contrast, if model can take all the tokens of the document as input, it will get these key sentences and produce better result. Therefore, we propose BERT-AL to solve this problem. For simplicity, we only employ the Simple Classifier as the summarization layer in the following sections.

heading: 3 BERT-AL
text: In this section, we will introduce the detail about BERT-AL, and as illustrated in Figure 1, BERTAL has mainly two key components different with BERTSUM: one is the multi-channel LSTM and another is the positional encoding.

heading: 3.1 MULTI-CHANNEL LSTM
text: We first give the definition of our scenario: assume that the length of the document is ldoc, and we have only a pretrained BERT model with the maximum sequence length lBERT . We first split the document into nsegment segments and each segment has the length lBERT (the length of nsegmentth segment is shorter than lBERT ). We set lBERT−AL = nsegment ∗ lBERT as the maximum sequence length of BERT-AL, and then we have lBERT−AL ≥ ldoc, which means BERT-AL can take arbitrarily long document as input.
As we know, LSTM has the capability of capturing information across arbitrarily long steps but is weak in capturing long-term dependencies. Therefore, within a segment, we use Transformer to capture long-term dependencies and extract the feature via self-attention among local positions, and then fully represent the segment at each layer. To this end, we propose a multi-channel LSTM to chain the representation of different segments. For the above definition, ldoc can be arbitrarily long and so is nsegment. Thus, we apply LSTM on nsegment dimension, and the steps of LSTM should also be nsegment.
As Figure 1 shows, LSTM is following the segment-wise Transformers at each layer and takes the hidden states from Transformers as its input. To fit the next layer, we set LSTM’s hidden size also as dmodel, i.e., 768 for BERT-Base and 1024 for BERT-Large. Thus, the computing in each layer is shown as follows.
Hi−1 = Concat(Hi−1,1, Hi−1,2, ...,Hi−1,nsegment)
HTransi,j = Transformer(Hi−1,j ∈ R[lBERT ,dmodel])
Hi = LSTM(H Trans i ∈ R[lBERT ,nsegment,dmodel])
(4)
where Hi is the hidden states from ith layer, Hi,j is jth segment in Hi, and HTransi,j is the hidden states from jth Transformer. All segment-wise Transformers at the same layer share parameters.
BERT-AL employs a single layer of unidirectional LSTM as the implement of multi-channel LSTM, and a channel is corresponding to a relative position within a segment. Per-gate layer normalization is applied in each LSTM cell (Ba et al., 2016). We do not split the segments according to natural sentences, so [CLS] may be at the beginning, in the middle or at the end of a block. Similarly, the output at [CLS] positions also do not come from the same LSTM channel. The reason we apply multi-channel LSTM on nsegment rather than a 2D LSTM on the whole sequence is taking advantage of LSTM’s expandability on variable-length segments, but meanwhile, reducing the interference to self-attention among positions within a segment.

heading: 3.2 EMBEDDING
text: BERT-AL take the same input format as BERTSUM does, i.e., adding [CLS] to the head of each sentence and [SEP ] to the tail of each sentence. Since BERT-AL has multiple segment-wise Transformers at the first layer, its input embedding is also changed. More details are shown as follows.
Before feeding embedding to Transformers, the sequence is divided into segments whose lengths are the same with the pretrained BERT model’s maximum sequence length. Token embedding and segment embedding are also same as the original BERT. For position embedding, we also reserve the learned position embedding used in the original BERT. However, if we do not re-pretrain the BERT, the dimension of position embedding matrix in the pretrained BERT model is [lBERT , dmodel], while the length of task’s input is lBERT−AL. To resolve this problem, we copy the original position embedding matrix nsegment times and concatenate them. Therefore, we can get a position embedding matrix with dimension [lBERT−AL, dmodel]. Through this way, each position in the input will get a positional encoding. Therefore, the model can work correctly. In the Transformer layer, each segment does not interact with each other. Therefore, although the position embeddings are same in different segments, it does not affect the model to extract feature in its own segment.

heading: 3.3 OTHER DETAILS
text: It is obvious that each sentence’s length is different, but segments have the same length. Therefore, the start of each segment may not be [CLS] and the tail may not be [SEP ], either. In Figure 1, the Ti is the representation of i sentence through N Transformers and LSTMs. After summarization layer, Yi represents the score of that sentence, and then we choose sentences with top 3 highest scores compose the summary. Finally, Table 1 shows difference of BERT-AL and BERTSUM.

heading: 4 EXPERIMENT
text: In this section, we demonstrate BERT-ALs effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset. We compare BERTSUM with our models on various different settings, since BERTSUM is the state-of-the-art on CNN/Daily Mail dataset.

heading: 4.1 CNN/DAILY MAIL DATASET
text: The CNN/Daily Mail dataset contains news articles and associated highlights, i.e., a few bullet points giving a brief overview of the article. We used the standard splits as training, validation and testing sets as BERTSUM did. The statistics of datasets is shown in Tabel 2.
We also perform the same preprocessing for data as BERTSUM did, including keeping entities, splitting sentences by CoreNLP and following methods in See et al. (2017).
We employ ROUGE-1, 2, L (Lin, 2004) to evaluate the performance for different methods and focus on the F1 score, as follows.
RROUGE−N =
∑ S∈{ReferenceSummaries} ∑ gramn∈S Countmatch(gramn)∑
S∈{ReferenceSummaries} ∑ gramn∈S Count(gramn)
PROUGE−N =
∑ S∈{ReferenceSummaries} ∑ gramn∈S Countmatch(gramn)∑
S∈{CandidateSummaries} ∑ gramn∈S Count(gramn)
FROUGE−N = (1 + β2)RROUGE−NPROUGE−N RROUGE−N + β2PROUGE−N
(5)
where n stands for the length of the n-gram, gramn, and Countmatch(gramn) is the maximum number of n-grams co-occurring in a candidate summary and a set of reference summaries.
Rlcs = LCS(X,Y )
m
Plcs = LCS(X,Y )
n
Flcs = (1 + β2)RlcsPlcs Rlcs + β2Plcs
(6)
where X is a reference summary sentence and Y is a candidate summary sentence, m is the length of X and n is the length of Y , LCS(X,Y ) is the length of a longest common subsequence of X and Y .
To adapt dataset to suit extractive summarization task, we also use a greedy algorithm which is same as BERTSUM to generate an oracle summary for each document. The algorithm selects a set of sentences as the oracle set, which can maximize the ROUGE scores.

heading: 4.2 EXPERIMENT SETTING
text: Assume that we have only a pretrained BERT model and the length of its position embedding is lBERT . Our task is to produce summaries on documents with ldoc tokens. The BERT-AL can take lBERT−AL length tokens as input (lBERT−AL = nsegment ∗ lBERT and lBERT−AL >= ldoc). We design four groups of experiments and set lBERT = 8, 16, 128, 256 for them. More details about settings are shown in Table 3. Under each group setting, we compare three different methods, including Baseline-1, Baseline-2 and BERT-AL. More details about methods are shown as follows.
1) Baseline-1 applies BERTSUM to the dataset with length = lBERT , and then truncates text longer than lBERT .
2) Baseline-2 concatenates the first lBERT position embedding of the original BERT with a randomly initialized embedding with the length (nsegment − 1) ∗ lBERT , and then employs the concatenated embedding as BERTSUM’s position embedding. Finally, it applies BERTSUM to long documents with length = lBERT−AL.
3) BERT-AL copies the position embedding matrix nsegment times and concatenates them along the dimension of length, and then employs the embedding as the position embedding of segment-wise Transformers at corresponding layers. Finally, it applies BERT-AL to long documents with length = lBERT−AL
The Baseline-1 aims to compare BERTSUM and BERT-AL while the Baseline-2 aims to reveal the effectiveness of multi-channel LSTM. Furthermore, Baseline-2 can be used to indicate whether the promotion of BERT-AL is due to the longer input or the multi-channel LSTM.
For fairly comparison, we set all hyperparameters equal to BERTSUM reported in Liu (2019). Specifically, we set the number of layers as 12, the hidden size as 768, the number of self-attention heads as 12 and the feed-forward size as 3072. We use Adam with β1 = 0.9, β2 = 0.999 and = 10−9. We also use a linear learning rate decreasing scheduler with warming-up on first 10,000 steps. All models are trained for 50,000 steps with gradient accumulation per two steps. We also select the top-3 checkpoints based on their evaluation losses on the validations set and report the averaged results on the test set.

heading: 4.3 RESULTS AND ANALYSIS
text: The results are showed in Table 4, including BERTSUM1. We can obtain the following observations:
1) For all of the four groups, BERT-AL outperforms baselines, consistently. It proves that BERT-AL is effective on long document summarization task, which comes from merging Transformer’s local feature extraction ability and LSTM’s global time capturing ability.
2) Comparing Baseline-1 across four groups, performance increases with lBERT is longer. It implies longer input contains more useful information and truncating input leads to a performance drop.
3) Comparing Baseline-1 with Baseline-2, Baseline-2 performs better than Baseline-1 under nearly all settings. It implies randomly initialized position embedding also can help capture longer information even only with finetuning. However, this improvement will decrease with the maximum sequence length increases, and Baseline-2 has a worse perform than Baseline-1 when lBERT = 256.
4) Comparing BERT-AL and Baseline-1 in Group-4, BERT-AL can further promote the ROUGE score and break up the bottleneck of Baseline-2 when the lBERT is large. It implies repeated position embedding with multi-channel LSTM is more effective than random initialized one.
1The result comes from our reproduction
5) Comparing Baseline-2 with BERT-AL in all four groups, the promotion of ROUGE score is more significant when the number of segments (i.e., LSTM’s time-steps) is larger. It implies that the LSTM’s capability of timing capture is not be fully utilized when the LSTM has less time-steps.

heading: 4.4 DISCUSSION
text: BERT-AL is designed as the model with a shorter pretrained BERT, and it still can achieve a comparable performance to BERTSUM. In Group 4 experiments, we can find BERT-AL (256)s performance has been very closed to BERTSUM (512). Simultaneously, BERT-AL has a much faster training and inference speed, since Transformers runtime is proportional to l2 and multi-channel LSTM is high parallelly applied on nsegment steps. Therefore, BERT-AL can be a good alternative under the following situations: 1) For a NLP task, the input text is too long to feed into a BERT model due to GPU memory or other limitations. 2) The time of pretraining a longer model from scratch is unacceptable under a limited computing resource. 3) There is restraint for inference speed, and then we also can split the text into small segments and feed them into BERT-AL.

heading: 5 RELATED WORK
text: There are several works related to modeling recurrence for self-attention network (SAN) in Transformer. Dehghani et al. (2018) recurrently refines the representations of each layer to improve SAN encoder. Shen et al. (2018) introduces a directional self-attention network (DiSAN), which only allows each token to attend previous (or following) tokens. Both Hao et al. (2019) and Chen et al. (2018) propose to combine SAN encoder with an additional RNN encoder. The former enhances the Transformer with recurrence information, while the latter augments RNN-based models with SAN encoder. Wang et al. (2019b) adds a local RNN layer in front of SAN in each Transformer layer, which aims to capture both local structures and global long-term dependencies in sequences. Wang et al. (2019a) adds LSTM after all the Transformer by the guide of Coordinate Architecture Search. However, these above works all target to solve the limitation of positional encoding, while our model aims to apply pretrained BERT to longer text and need not to pretrain from scratch, which reduces the cost of time and computing resources. Transformer-XL (Dai et al., 2019) contains segment-level recurrence with state reuse and relative positional encoding for language model beyond a fixed-length context, which is similar with our model and also can processes longer text. However, Transformer-XL aims at language model and need to pretrain the model from scratch.
For processing long input, Bengio et al. (2013) proposes ”conditional computation” which is to only compute a subset of a networks units for a given input by gating different parts of the network. Ling & Rush (2017) proposes a coarse-to-fine attention model that uses hard attention to find the text chunks of importance and then only attend to words in that chunk. Liu et al. (2018) first coarsely selects a subset of the input, then trains an model while conditioning on this subset. Cohan et al. (2018) proposes a hierarchical encoder which represents each section using word-level RNN and then represents all sections using section-level RNN. These works is all similar with our method, which is splitting the long input into several subsets to process them respectively and then merging them. However, they are all need re-pretrain the model from scratch. Furthermore, we use the most powerful model–BERT.

heading: 6 CONCLUSION
text: BERT has been the state-of-the-art for all kinds of NLP tasks. However, BERT cannot be applied to long text tasks, e.g., document-level text summarization, because it cannot take text longer than the maximum length as input. However, the maximum length is predefined during pretraining, and expanding maximum length need re-pretrain which will cost lots of computing resource. We propose a novel model named BERT-AL which combines the advantages of Transformer and LSTM. The BERT-AL can take arbitrarily long text as its input and need not re-pretrain from scratch. We demonstrate BERT-ALs effectiveness on the text summarization task by conducting experiments on the CNN/Daily Mail dataset, and the experimental results prove that BERT-AL is effective on NLP tasks with very long text as input. Furthermore, our model can be easily adapted to various tasks and pretrained models.

Review summaries:
The paper proposes a methodology to overcome the problem of processing long sequences with a pre-trained Transformer model, which suffers from high computational costs due to the complexity being quadratic in the length of the sequence. The authors also point out that BERT needs to be retrained from scratch if sequences longer than the specified maximum length (512) are to be processed. Their method (BERT-AL) chunks the input text into segments of maximum length. Each segment is propagated through several layers of a pre-trained Transformer layer followed by a multi-channel LSTM. Herein, the positional embeddings in each segment are the same. The model is applied to extractive summarization, and directly compared to the BERTSUM model, which can only process documents up to length 512. The comparison is made for 4 application scenarios, each corresponding to an artificial maximum length after which the BERTSUM model truncates the input: after 8, 16, 128, and 256 tokens. The experimental results suggest that BERT-AL outperforms BERTSUM in all 4 scenarios: Substantially for max length 8 and 16, and marginally for 128 and 256. BERTSUM without any truncation still performs best, however.

This paper proposed another variant of BERT, called BERT-AL, which can deal with arbitrarily long inputs. The authors constructed the proposed method by combining the segment-wise BERT with the multi-channel LSTM. The authors validated the proposed method on the text summarization task and achieved higher performance than existing works. Their proposed method can be combined with other transformer-based approaches, such as XLNet or RoBERTa.