Title: BERT-AL: BERT FOR ARBITRARILY LONG DOCU-MENT UNDERSTANDING

Abstract: Pretrained language models attract lots of attentions, and they take advantage of the two-stages training process: pretraining on huge corpus and finetuning on specific tasks. Thereinto, BERT (Devlin et al., 2019) is a Transformer (Vaswani et al., 2017) based model and has been the state-of-the-art for many kinds of Nature Language Processing (NLP) tasks. However, BERT cannot take text longer than the maximum length as input since the maximum length is predefined during pretraining. When we apply BERT to long text tasks, e.g., document-level text summarization: 1) Truncating inputs by the maximum sequence length will decrease performance, since the model cannot capture long dependency and global information ranging the whole document. 2) Extending the maximum length requires re-pretraining which will cost a mass of time and computing resources. Whats even worse is that the computational complexity will increase quadratically with the length, which will result in an unacceptable training time. To resolve these problems, we propose to apply Transformer to only model local dependency and recurrently capture long dependency by inserting multi-channel LSTM into each layer of BERT. The proposed model is named as BERT-AL (BERT for Arbitrarily Long Document Understanding) and it can accept arbitrarily long input without re-pretraining from scratch. We demonstrate BERT-ALs effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset. Furthermore, our method can be adapted to other Transformer based models, e.g., XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), for various NLP tasks with long text.

1. INTRODUCTION:

In recent years, neural networks are proposed to solve various NLP tasks. Especially, pretrained language models (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019; Yang et al., 2019) attract lots of attentions, which take advantage of the two-stages training process: pretraining on unlabeled corpus and then finetuning on specific tasks. The most famous model is BERT. BERT and its varieties are the state-of-the-art for many kinds of NLP tasks. The power of BERT does not only come from its architecture of networks, but also because it can be pretrained on a mass of text as a masked language model. BERT can be used to solve almost all NLP tasks, and especially it can perform best on datasets with short text, e.g., GLUE (Wang et al., 2018) and Squad (Rajpurkar et al., 2016) . However, there are still many document-level tasks, e.g., document-level text summarization (Hermann et al., 2015) , long-document machine reading comprehension (Hewlett et al., 2016) and long text classification (Zhang et al., 2015) . BERT cannot be finetuned for such tasks with long text directly or perform good on these tasks, since it is limited by the fixed-length position embedding which was determined during pretraining. We employ document-level text summarization as an example, which usually has longer text than the maximum sequence length of BERT.

Intuitively, there are two alternative solutions: 1) Truncating inputs by the maximum sequence length to fit the BERTs constraint. 2) Increasing the length of position embedding and re-pretraining the BERT from scratch. The first method will decrease performance, obviously, since some useful information placing behind the maximum sequence length is discarded by truncating. E.g., for text summarization, if the key point sentence locates at the end of the document, it never be recalled even though the model is powerful. For the second method, re-pretraining the BERT from scratch will cost a mass of computing time and resources. Whats even worse is that the computational complexity will increase quadratically with the length, which will result in an unacceptable training time. For example, the XLNet-Large (Yang et al., 2019) costs 2.5 days on 512 TPU v3 chips and the RoBERTa (Liu et al., 2019 ) costs 1 day on 1024 V100 GPUs.

To resolve these problems, we propose BERT-AL (BERT for Arbitrarily Long Document Understanding) model that extracts local features by applying parallel multi-layer Transformers into chunked input and employs multi-channel LSTMs to capture global information crossing Transformers. This fusion breaks the limitation of BERT by the ability of capture unlimited sequential information from LSTM, and makes it be able to process arbitrarily long text. On the other hand, it also skillfully avoids gradient vanishing and exploding problem (Li et al., 2018) of LSTM since only few steps are required by multi-channel LSTM. Therefore, BERT-AL can solve the problems of original BERT: 1) For document-level tasks, BERT-AL can directly take all text as the input without truncating. 2) When the input length is longer then maximum sequence length of BERT, BERT-AL still can load the pretrained BERT model, which avoids pretraining much longer BERT model from scratch.

We demonstrate BERT-ALs effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset (Hermann et al., 2015) with various maximum sequence lengths of pretrained BERT. The results prove that BERT-AL can consistently outperform BERTSUM (Liu, 2019) which is the BERT-based state-of-the-art, when finetuning from the pretrained BERT model with the same maximum sequence length. Additionally, BERT-AL is a general NLP model which has no specific setting for text summarization, so it can be easily adapted to other tasks with long text, e.g., document-level machine reading comprehension and long text classification. Furthermore, the method, applying multi-channel LSTM on hidden states from transformers, also can be used in other Transformer based pretrained models, e.g., XLNet and RoBERTa.

In summary, contributions of this paper are shown as follow.

1) We propose a new architecture that combines Transformer and LSTM, which resolve the problem Transformer cannot be used in very long text, and skillfully avoid LSTMs drop backs.

2) We propose multi-channel LSTM only applied on the corresponding position across different Transformers, which can take fully advantage of LSTM without hurting Transformers representation too much.

3) We conduct experiments to prove BERT-AL can outperform other models with the BERT pretrained under the same maximum sequence length, and can perform very close to BERTSUM with at least twice the maximum length than ours.

2. BACKGROUND:



2.1. BERT:

According to the original implementation described in Vaswani et al. (2017) , BERT is a multi-layer bidirectional Transformer encoder, and Multi-Head Self-Attention is the key structure of Transformer encoder.

For the input H ∈ R L×D of each Transformer layer, H will be mapped to three different spaces, named as Q, K and V , respectively. Self-attention computes the dot products between Q and K to roughly get the weight matrix, and then multiply V to get the hidden representation. Multi-head mechanism promotes the power of Transformer because it allows the model to jointly attend to information from different representation subspaces at different positions.

BERT employs two tasks for pretraining: Masked Language Model and Next Sentence Prediction.

Masked Language Model: BERT proposes a bidirected language model, which replaces 15% words in text by [M ASK] label and then predicts which words they are. To mitigate the mismatch between pretraining and finetuning, i.e., the [M ASK] token does not appear during finetuning, they do not always replace masked words with the actual [M ASK] token. Then, BERT will predict the original token with cross entropy loss.

Next Sentence Prediction: BERT is also pretrained by the next sentence prediction task. Specifically, when choosing the sentences A and B for each pretraining example, 50% of the time B is the actual next sentence that follows A (labeled as IsN ext), and 50% of the time it is a random sentence from the corpus (labeled as N otN ext). Then, BERT will predict whether the sentence B is the actual next sentence that follows A or not with cross entropy loss.

For most of all NLP tasks, BERT concatenates different parts of input into a sequence beginning with [CLS] token and inserts [SEP ] token between two different parts. Before going through Transformer layers, BERT merges three different embeddings into one, i.e., word embedding, position embedding and segment embedding. Thereinto, position embedding is the information about the relative or absolute position of the tokens in the sequence. Since the model contains no recurrence and no convolution, position embedding is added to make use of the order of the sequence. There are two implements of position embedding, i.e., learned position embedding and sine and cosine functions of different frequencies, but both of them cannot expand to longer without a performance regression (Wang et al., 2019a) .

2.2. BERTSUM:

BERTSUM is an extension of BERT on extractive text summarization task and it truncates only the first 512 tokens as input. To select sentences, BERTSUM adds [CLS] to the head of each sentence and [SEP ] to the tail of each sentence indicating the end of that sentence. The following summarization layer will score each [CLS] which presents the importance of that sentence. Finally, sentences with top 3 highest scores compose the summary. There are three summarization layers proposed in Liu (2019):

1) Simple Classifier: only adds a linear layer on each [CLS] and uses a sigmoid function to get the predicted score:

EQUATION

where T i is the logit of ith sentence, σ is the Sigmoid function.

2) Inter-Sentence Transformer: applies more Transformer layers into sentence-level representations as follows. hl = LN (h l-1 + M HAtt(h l-1 ))

EQUATION

where h 0 = P osEmb(T ) and T are the sentence vectors output by BERT, P osEmb is the function of adding position embeddings to T ; LN is the layer normalization operation; F F N is a feedforward network; M HAtt is the multi-head attention operation; the superscript l indicates the depth of the stacked layer. The final output layer is still a sigmoid classifier and T i is the logit of ith sentence.

3) Recurrent Neural Network: applies an LSTM layer over the BERT outputs to learn summarization-specific features. At time step i, the input to the LSTM layer is the BERT output T i , and the output is calculated as:

Fi Ii Oi Gi = LN h (W h h i-1 + LN x (W x T i )) C i = σ(F i C i-1 ) + σ(I i ) tanh(G i-1

)

EQUATION

where F i , I i , O i are forget gates, input gates, output gates; G i is the hidden vector and C i is the memory vector; h i is the output vector; LN h , LN x , LN c are there difference layer normalization operations; The final output layer is also a sigmoid classifier and T i is the logit of ith sentence.

However, we argue that the way of truncation will result in the loss of information in the latter part of the document. If the key sentences locate at the end of the document, they will never be recalled even though the model is powerful. In contrast, if model can take all the tokens of the document as input, it will get these key sentences and produce better result. Therefore, we propose BERT-AL to solve this problem. For simplicity, we only employ the Simple Classifier as the summarization layer in the following sections.

3. BERT-AL:

In this section, we will introduce the detail about BERT-AL, and as illustrated in Figure 1 , BERT-AL has mainly two key components different with BERTSUM: one is the multi-channel LSTM and another is the positional encoding. We first give the definition of our scenario: assume that the length of the document is l doc , and we have only a pretrained BERT model with the maximum sequence length l BERT . We first split the document into n segment segments and each segment has the length l BERT (the length of n segment th segment is shorter than l BERT ). We set l BERT-AL = n segment * l BERT as the maximum sequence length of BERT-AL, and then we have l BERT-AL ≥ l doc , which means BERT-AL can take arbitrarily long document as input.

As we know, LSTM has the capability of capturing information across arbitrarily long steps but is weak in capturing long-term dependencies. Therefore, within a segment, we use Transformer to capture long-term dependencies and extract the feature via self-attention among local positions, and then fully represent the segment at each layer. To this end, we propose a multi-channel LSTM to chain the representation of different segments. For the above definition, l doc can be arbitrarily long and so is n segment . Thus, we apply LSTM on n segment dimension, and the steps of LSTM should also be n segment .

As Figure 1 shows, LSTM is following the segment-wise Transformers at each layer and takes the hidden states from Transformers as its input. To fit the next layer, we set LSTM's hidden size also as d model , i.e., 768 for BERT-Base and 1024 for BERT-Large. Thus, the computing in each layer is shown as follows.

H i-1 = Concat(H i-1,1 , H i-1,2 , ..., H i-1,nsegment ) H T rans i,j = T ransf ormer(H i-1,j ∈ R [l BERT ,d model ]

)

EQUATION

where H i is the hidden states from ith layer, H i,j is jth segment in H i , and H T rans i,j

is the hidden states from jth Transformer. All segment-wise Transformers at the same layer share parameters.

BERT-AL employs a single layer of unidirectional LSTM as the implement of multi-channel LSTM, and a channel is corresponding to a relative position within a segment. Per-gate layer normalization is applied in each LSTM cell (Ba et al., 2016) . We do not split the segments according to natural sentences, so [CLS] may be at the beginning, in the middle or at the end of a block. Similarly, the output at [CLS] positions also do not come from the same LSTM channel. The reason we apply multi-channel LSTM on n segment rather than a 2D LSTM on the whole sequence is taking advantage of LSTM's expandability on variable-length segments, but meanwhile, reducing the interference to self-attention among positions within a segment.

3.2. EMBEDDING:

BERT-AL take the same input format as BERTSUM does, i.e., adding [CLS] to the head of each sentence and [SEP ] to the tail of each sentence. Since BERT-AL has multiple segment-wise Transformers at the first layer, its input embedding is also changed. More details are shown as follows.

Before feeding embedding to Transformers, the sequence is divided into segments whose lengths are the same with the pretrained BERT model's maximum sequence length. Token embedding and segment embedding are also same as the original BERT. For position embedding, we also reserve the learned position embedding used in the original BERT. However, if we do not re-pretrain the BERT, the dimension of position embedding matrix in the pretrained BERT model is [l BERT , d model ], while the length of task's input is l BERT-AL . To resolve this problem, we copy the original position embedding matrix n segment times and concatenate them. Therefore, we can get a position embedding matrix with dimension [l BERT-AL , d model ]. Through this way, each position in the input will get a positional encoding. Therefore, the model can work correctly. In the Transformer layer, each segment does not interact with each other. Therefore, although the position embeddings are same in different segments, it does not affect the model to extract feature in its own segment.

3.3. OTHER DETAILS:

It is obvious that each sentence's length is different, but segments have the same length. Therefore, the start of each segment may not be [CLS] and the tail may not be [SEP ], either. In Figure 1 , the T i is the representation of i sentence through N Transformers and LSTMs. After summarization layer, Y i represents the score of that sentence, and then we choose sentences with top 3 highest scores compose the summary. Finally, Table 1 shows difference of BERT-AL and BERTSUM. 

4. EXPERIMENT:

In this section, we demonstrate BERT-ALs effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset. We compare BERTSUM with our models on various different settings, since BERTSUM is the state-of-the-art on CNN/Daily Mail dataset.

4.1. CNN/DAILY MAIL DATASET:

The CNN/Daily Mail dataset contains news articles and associated highlights, i.e., a few bullet points giving a brief overview of the article. We used the standard splits as training, validation and testing sets as BERTSUM did. The statistics of datasets is shown in Tabel 2. We also perform the same preprocessing for data as BERTSUM did, including keeping entities, splitting sentences by CoreNLP and following methods in See et al. (2017) .

We employ ROUGE-1, 2, L (Lin, 2004) to evaluate the performance for different methods and focus on the F 1 score, as follows.

EQUATION

where n stands for the length of the n-gram, gram n , and Count match (gram n ) is the maximum number of n-grams co-occurring in a candidate summary and a set of reference summaries.

EQUATION

where X is a reference summary sentence and Y is a candidate summary sentence, m is the length of X and n is the length of Y , LCS(X, Y ) is the length of a longest common subsequence of X and Y .

To adapt dataset to suit extractive summarization task, we also use a greedy algorithm which is same as BERTSUM to generate an oracle summary for each document. The algorithm selects a set of sentences as the oracle set, which can maximize the ROUGE scores. Assume that we have only a pretrained BERT model and the length of its position embedding is l BERT . Our task is to produce summaries on documents with l doc tokens. The BERT-AL can take l BERT-AL length tokens as input (l BERT-AL = n segment * l BERT and l BERT-AL >= l doc ). We design four groups of experiments and set l BERT = 8, 16, 128, 256 for them. More details about settings are shown in Table 3 . Under each group setting, we compare three different methods, including Baseline-1, Baseline-2 and BERT-AL. More details about methods are shown as follows.

4.2. EXPERIMENT SETTING:

1) Baseline-1 applies BERTSUM to the dataset with length = l BERT , and then truncates text longer than l BERT .

2) Baseline-2 concatenates the first l BERT position embedding of the original BERT with a randomly initialized embedding with the length (n segment -1) * l BERT , and then employs the concatenated embedding as BERTSUM's position embedding. Finally, it applies BERTSUM to long documents with length = l BERT-AL .

3) BERT-AL copies the position embedding matrix n segment times and concatenates them along the dimension of length, and then employs the embedding as the position embedding of segment-wise Transformers at corresponding layers. Finally, it applies BERT-AL to long documents with length = l BERT-AL

The Baseline-1 aims to compare BERTSUM and BERT-AL while the Baseline-2 aims to reveal the effectiveness of multi-channel LSTM. Furthermore, Baseline-2 can be used to indicate whether the promotion of BERT-AL is due to the longer input or the multi-channel LSTM.

For fairly comparison, we set all hyperparameters equal to BERTSUM reported in Liu (2019) . Specifically, we set the number of layers as 12, the hidden size as 768, the number of self-attention heads as 12 and the feed-forward size as 3072. We use Adam with β 1 = 0.9, β 2 = 0.999 and = 10 -9 . We also use a linear learning rate decreasing scheduler with warming-up on first 10,000 steps. All models are trained for 50,000 steps with gradient accumulation per two steps. We also select the top-3 checkpoints based on their evaluation losses on the validations set and report the averaged results on the test set.

4.3. RESULTS AND ANALYSIS:

The results are showed in Table 4 , including BERTSUM1 . We can obtain the following observations: Search. However, these above works all target to solve the limitation of positional encoding, while our model aims to apply pretrained BERT to longer text and need not to pretrain from scratch, which reduces the cost of time and computing resources. Transformer-XL (Dai et al., 2019) contains segment-level recurrence with state reuse and relative positional encoding for language model beyond a fixed-length context, which is similar with our model and also can processes longer text. However, Transformer-XL aims at language model and need to pretrain the model from scratch.

For processing long input, Bengio et al. (2013) proposes "conditional computation" which is to only compute a subset of a networks units for a given input by gating different parts of the network. Ling & Rush (2017) proposes a coarse-to-fine attention model that uses hard attention to find the text chunks of importance and then only attend to words in that chunk. Liu et al. (2018) first coarsely selects a subset of the input, then trains an model while conditioning on this subset. Cohan et al. (2018) proposes a hierarchical encoder which represents each section using word-level RNN and then represents all sections using section-level RNN. These works is all similar with our method, which is splitting the long input into several subsets to process them respectively and then merging them. However, they are all need re-pretrain the model from scratch. Furthermore, we use the most powerful model-BERT.

6. CONCLUSION:

BERT has been the state-of-the-art for all kinds of NLP tasks. However, BERT cannot be applied to long text tasks, e.g., document-level text summarization, because it cannot take text longer than the maximum length as input. However, the maximum length is predefined during pretraining, and expanding maximum length need re-pretrain which will cost lots of computing resource. We propose a novel model named BERT-AL which combines the advantages of Transformer and LSTM. The BERT-AL can take arbitrarily long text as its input and need not re-pretrain from scratch. We demonstrate BERT-ALs effectiveness on the text summarization task by conducting experiments on the CNN/Daily Mail dataset, and the experimental results prove that BERT-AL is effective on NLP tasks with very long text as input. Furthermore, our model can be easily adapted to various tasks and pretrained models.

:

The result comes from our reproduction

Figures and Tables:

Figure 1: The architecture of BERT-AL 3.1 MULTI-CHANNEL LSTM

The difference of BERT-AL and BERT-SUM

Table content: <table><tr><td/><td>BERTSUM</td><td>BERT-AL</td></tr><tr><td>Architeture</td><td>BERT + summarization layer</td><td>segment-wise BERT + multi-channel LSTM + summarization layer</td></tr><tr><td>Position embedding</td><td>same as BERTs position embedding</td><td>copy and concatenate BERTs position embedding</td></tr><tr><td>Max input length</td><td colspan="2">BERTs input length arbitrarily long</td></tr></table>

Statistics of CNN/Daily Mail dataset

Table content: <table><tr><td/><td>CNN</td><td colspan="2">Daily Mail Total</td></tr><tr><td>Train</td><td colspan="2">90266 196961</td><td>287227</td></tr><tr><td colspan="2">Validate 1220</td><td>12148</td><td>13368</td></tr><tr><td>Test</td><td>1093</td><td>10397</td><td>11490</td></tr></table>

Setting of experiment groupsl BERT l BERT-AL n segment

Table content: <table><tr><td>Group-1 8</td><td>512</td><td>64</td></tr><tr><td>Group-2 16</td><td>512</td><td>32</td></tr><tr><td>Group-3 128</td><td>512</td><td>4</td></tr><tr><td>Group-4 256</td><td>1024</td><td>4</td></tr></table>

Experiment results Comparing Baseline-2 with BERT-AL in all four groups, the promotion of ROUGE score is more significant when the number of segments (i.e., LSTM's time-steps) is larger. It implies that the LSTM's capability of timing capture is not be fully utilized when the LSTM has less time-steps. AL is designed as the model with a shorter pretrained BERT, and it still can achieve a comparable performance to BERTSUM. In Group 4 experiments, we can find BERT-AL (256)s performance has been very closed to BERTSUM (512). Simultaneously, BERT-AL has a much faster training and inference speed, since Transformers runtime is proportional to l 2 and multi-channel LSTM is high parallelly applied on n segment steps. Therefore, BERT-AL can be a good alternative under the following situations: 1) For a NLP task, the input text is too long to feed into a BERT model due to GPU memory or other limitations. 2) The time of pretraining a longer model from scratch is unacceptable under a limited computing resource. 3) There is restraint for inference speed, and then we also can split the text into small segments and feed them into BERT-AL.5 RELATED WORKThere are several works related to modeling recurrence for self-attention network (SAN) in Transformer.Dehghani et al. (2018) recurrently refines the representations of each layer to improve SAN encoder.Shen et al. (2018) introduces a directional self-attention network (DiSAN), which only allows each token to attend previous (or following) tokens. BothHao et al. (2019) andChen et al. (2018) propose to combine SAN encoder with an additional RNN encoder. The former enhances the Transformer with recurrence information, while the latter augments RNN-based models with SAN encoder.Wang et al. (2019b)  adds a local RNN layer in front of SAN in each Transformer layer, which aims to capture both local structures and global long-term dependencies in sequences.Wang et al. (2019a)  adds LSTM after all the Transformer by the guide of Coordinate Architecture

Table content: <table><tr><td>Model</td><td colspan="3">ROUGE-1 ROUGE-2 ROUGE-L</td></tr><tr><td colspan="2">BERTSUM 42.94</td><td>20.14</td><td>39.38</td></tr></table>

Peer review (iclr2020):
This paper proposed another variant of BERT, called BERT-AL, which can deal with arbitrarily long inputs. The authors constructed the proposed method by combining the segment-wise BERT with the multi-channel LSTM. The authors validated the proposed method on the text summarization task and achieved higher performance than existing works. Their proposed method can be combined with other transformer-based approaches, such as XLNet or RoBERTa. The paper tackles a relevant topic that is interesting to the attendees of ICLR, which is, “How do we get the overall information of a document, which has a number of sentences?”. However, the paper has several issues that I will try to cover in the following. (1) (No sufficient results which support authors’ claim) First: In the introduction, authors claim that the model for NLP tasks whose inputs are usually long texts should be able to take arbitrarily long inputs so that the model can extract important information from the entire long documents. However, there is no concrete experimental example, where the existing methods (standard BERT or BERTSUM) fail to extract information. Second: In the discussion of the experiment section, the authors claim that the proposed method BERT-AL can be parallelized and runs faster than existing methods. However, there is no comparison with existing methods about the running time. (2) (Clarity about the proposed method) The presentation of the proposed method is slightly confusing. There is no concrete explanation about the learning procedure. Are any pre-training and fine-tuning procedures the same with those of the standard BERT? The main part of the proposed method is to combine the segment-wise BERT with the multi-channel LSTM. How is the pre-training procedure conducted with parallelized computation? A detailed explanation should be shown. (3) (Clarity about the experimental results) In table 4, the experimental results are shown. There are results of the baseline 1 & 2 (existing methods), the proposed method, and the existing method (BERTSUM). There is no discussion about the comparison of the proposed method with BERTSUM. It seems that BERTSUM’s result is superior to the proposed method’s result. Please explain more details of those results. The paper includes an explanation about the dataset and the task but does not include the experimental environment (used machines and the time to conduct the experiment). For the fair evaluation of results, we need the information on the overall experimental environment. (4) (Presentation) The presentation around mathematical formulas is a bit confusing. Please fix the following. - Subscripts of the formula should be ‘mathrm’ - ‘<=‘ should be ‘le’ - The ’n’ of the right-hand side of equation (5) is ’N’? - The multiplication ‘*’ should be ‘	imes’