BERT-AL: BERT for Arbitrarily Long Document Understanding
Authors: Ruixuan Zhang
Zhuoyu Wei
Yu Shi
Yining Chen
Venue:  (2020)

Abstract: Pretrained language models attract lots of attentions, and they take advantage of the two-stages training process: pretraining on huge corpus and finetuning on specific tasks. Thereinto, BERT (Devlin et al., 2019) is a Transformer (Vaswani et al., 2017) based model and has been the state-of-the-art for many kinds of Nature Language Processing (NLP) tasks. However, BERT cannot take text longer than the maximum length as input since the maximum length is predefined during pretraining. When we apply BERT to long text tasks, e.g., document-level text summarization: 1) Truncating inputs by the maximum sequence length will decrease performance, since the model cannot capture long dependency and global information ranging the whole document. 2) Extending the maximum length requires re-pretraining which will cost a mass of time and computing resources. What's even worse is that the computational complexity will increase quadratically with the length, which will result in an unacceptable training time. To resolve these problems, we propose to apply Transformer to only model local dependency and recurrently capture long dependency by inserting multi-channel LSTM into each layer of BERT. The proposed model is named as BERT-AL (BERT for Arbitrarily Long Document Understanding) and it can accept arbitrarily long input without re-pretraining from scratch. We demonstrate BERT-AL's effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset. Furthermore, our method can be adapted to other Transformer based models, e.g., XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019), for various NLP tasks with long text.

Under review as a conference paper at ICLR 2020
BERT-AL: BERT FOR ARBITRARILY LONG DOCU- MENT UNDERSTANDING
Anonymous authors Paper under double-blind review
ABSTRACT
Pretrained language models attract lots of attentions, and they take advantage of the two-stages training process: pretraining on huge corpus and finetuning on specific tasks. Thereinto, BERT (Devlin et al., 2019) is a Transformer (Vaswani et al. 2017) based model and has been the ofate-of-the-ard for many kinds of Na- ture Language Processing (NLP) tasks. However, BERT cannot take text longer than the maximum length as input since the maximum length is predefined during pretraining. When apply BERT to long text tasks, e.g., document-level text summarization: 1) Truncating inputs by the maximum sequence length will de- crease performance, since the model cannot capture long dependency and global information ranging the whole document. 2) Extending the maximum length re- quires re- re-prerraining which will cost mass of time and computing resources. Whats even worse is that the computational complexity will increase quadratically with the length, which will result in an unacceptable training time. To resolve these problems, we propose to apply Transformer to only model local dependency and recurrently capture long dependency by inserting multi-channel LSTTM into each layer of BERT. The proposed model is named as BERT-AL (BERT for Ar- bitrarily Long Document Understanding) and it can accept arbitrarily long input without re- re-pretraining from scratch. We demonstrate BERT-ALs effectiveness on text summarization by conducting experiments on the CNN/Daily Mail dataset. Furthermore, our method can be adapted to other Transformer based models, e.g.. XLNet (Yang et al. 2019) and RoBERTa (Liu et al. 2019). for various NL. tasks with long text.
INTRODUCTION
In recent years, neural networks are proposed to solve various NLP tasks, Especially, pretrained language models (Peters et al., 2018 Radford et al. 2018 Devlin et al. 2019 Yang et al. 2019) attract lots of attentions, which take advantage of the two- stages training process: pretraining
unlabeled corpus and then finetuning on specific tasks. The most famous model is BERT. BERT and its varieties are the state-of-the-art for many kinds of NLP tasks. The power of BERT does not only come from its architecture of networks, but also because it can be pretrained on mass of text as masked language model.
BERT can be used to solve almost all NLP tasks, and especially it can perform best on datasets with short text, e.g., GLUE (Wang et al. 2018) and Squad (Rajpurkar et al. 2016). However, there are still many document-level tasks, e.g., document-level text summarization (Hermann et al. 2013), long-document machine reading comprehension (Hewlett al.) 2016) and long text classifictiin (Zhang et al. 2015). BERT cannot be finetuned for such tasks with long text directly or perform good on these tasks, since it limited by the fixed-length position embedding which was determined during pretraining. We employ document-level text summarization an example, which usually has longer text than the maximum sequence length of BERT.
Intuitively, there are two alternative solutions: 1) Truncating inputs by the maximum sequence length to fit the BERT's constraint. 2) Increasing the length of position embedding and re- re-pretraining the BERT from scratch. The first method will decrease performance, obviously, since some useful information placing behind the maximum sequence length is discarded by truncating. E.g., for text summarization, if the key point sentence locates the end of the document, it never be recalled

---

Under review as a conference paper at ICLR 2020
though the model is powerful. For the second method, re- re-pretraining the BERT from scratch will cost a mass of computing time and resources. Whats even worse is that the computational complexity will increase quadratically with the length, which will result in unacceptable training time. For example, the Xl.Net-Large (Yang et al., 2019) costs 2.5 days on 512 TPU v3 chips and the RoBERTa (Liu al. 2019) costs day 1024 VT00 GPUs.
To resolve these problems, we propose BERT-AL (BERT for Arbitrarily Long Document Under- standing) model that extracts local features by applying parallel multi-layer Transformers into chun- ked input and employs multi-channel LSTMs to capture global information crossing Transformers. This fusion breaks the limitation of BERT by the ability of capture unlimited sequential information from LSTM, and makes it be able to process arbitrarily long text. On the other hand, it also skillfully avoids gradient vanishing and exploding problem (Li et al. 2018) of LSTT since only few steps are required by multi-channel LSTM. Therefore, BERT-AL can solve the problems of original BERT: 1) For document-level tasks, BERT-AL can directly take all text as the input without truncating. 2) When the input length is longer then maximum sequence length of BERT, BERT-AL still can load the pretrained BERT model, which avoids pretraining much longer BERT model from scratch.
We demonstrate BERT-ALS effectiveness text summarization by conducting experiments on the CNN/Daily Mail dataset (Hermann et al., 2015) with various maximum sequence lengths of pre- trained BERT. The results prove that BERT-AL can consistently outperform BERTSUM (Liu 2019) which the BERT-based sate-offthheaart when finetuning from the pretrained BERT model with the same maximum sequence length. Additionally, BERT-AL is general NLP model which has no specific setting for text summarization, so it can be easily adapted to other tasks with long text, e.g., document-level machine reading comprehension and long text classification. Furthermore, the method, applying multi-channel LSTM on hidden states from transformers, also can be used in other Transformer based pretrained models, e.g., XLNet and RoBERTa.
In summary, contributions of this paper are shown as follow.
1) We propose new architecture that combines Transformer and LSTM, which resolve the problem Transformer cannot be used in very long text, and skillfully avoid LSTMs drop backs.
2) We propose multi-channel LSTM only applied on the corresponding position across different Transformers, which can take fully advantage of LSTM without hurting Transformers representation too much.
3) We conduct experiments to prove BERT-AL can outperform other models with the BERT pre- trained under the same maximum sequence length, and can perform very close BERTSUM with at least twice the maximum length than ours.
2 BACKGROUND
2.1 BERT
According to the original implementation described in Vaswani et al. (2017) BERT is multi-layer bidirectional Transformer encoder, and Multi-Head Self-Attention is the key structure of Trans- former encoder.
For the input H E LR of each Transformer layer, H will be mapped to three different spaces, named as Q, K and V, respectively. Self-attention computes the dot products between Q and K to roughly get the weight matrix, and then multiply V to get the hidden representation. Multi-head mechanism promotes the power of Transformer because it allows the model to jointly attend to information from different representation subspaces at different positions.
BERT employs two tasks for pretraining: Masked Language Model and Next Sentence Prediction.
Masked Language Model: BERT proposes a bidirected language model, which replaces 15% words in text by [MASK] label and then predicts which words they are. To mitigate the mismatch between pretraining and finetuning, ie.. the [MASK token does not appear during finetuning, they do not always replace masked words with the actual [MASK] token. Then, BERT will predict the original token with cross entropy loss.
2

---

Under review as a conference paper at ICLR 2020
Next Sentence Prediction: BERT is also pretrained by the next sentence prediction task. Specif- ically, when choosing the sentences A and В for each pretraining example, 50% of the time B is the actual next sentence that follows A (labeled sentence from the corpus (labeled as Not.Nett) Then, BERT will predict whether the sentence B is the actual next sentence that follows A not with cross entropy loss.
IsNext) and 50% of the time it is random
For most of all NLP tasks, BERT concatenates different parts of input into a sequence beginning with [CLS] token and inserts [SEP] token between two different parts. Before going through Transformer layers, BERT merges three different embeddings into one, i.e., word embedding, posi- tion embedding and segment embedding. Thereinto, position embedding the information about the relative or absolute position of the tokens in the sequence. Since the model contains no recurrence and no convolution, position embedding is added to make use of the order of the sequence. There are two implements of position embedding, i.e., learned position embedding and sine and cosine functions of different frequencies, but both of them cannot expand to longer without performance regression (Wang et al. 2019a).
2.2 BERTSUM
BERTSUM is an extension of BERT on extractive text summarization task and it truncates only the first 512 tokens input. To select sentences, BERTSUM adds [CLS] the head of each sentence and [SEP] to the tail of each sentence indicating the end of that sentence. The following summarization layer will score each [CLS] which presents the importance of that sentence. Finally, sentences with top highest scores compose the summary. There are three summarization layers proposed in Liu (209))
1) Simple Classifier: only adds a linear layer on each [CLS] and uses sigmoid function to get the
predicted score:
Y (WoT, b.) where T, is the logit of ith sentence, a is the Sigmoid function.
2) Inter-Sentence Transformer: applies more Transformer layers into sentence-level representa- tions as follows.
ht LN + MH
h' LN + FFN(())
Y, = oWWhh bb)
where nt (ob(T and 7 are the sentence vectors output by BERT, Pos Emb is the function of adding position embeddings to T: LN the layer normalization operation; FFN is feedfor- ward network; M H Atl is the multi-head attention operation; the superscript indicates the depth of the stacked layer. The final output layer is still a sigmoid classifier and T, the logit of ith sentence.
3) Recurrent Neural Network: applies an LSTM layer over the BERT outputs to learn summarization-spceee features. At time step the input the LSTM layer is the BERT output T; and the output is calculated as:
LNN (W,TT.
6.
C. o(F. O C+-1) (II,) tanh(G
(3)
h4 (O.) tanh((NN.C..))
Y, oWWn, + b,)
where F,, I,, O, forget gates, input gates, output gates; G; is the hidden vector and C2 is the memory vector; h; is the output vector; LN, LN,, LN, are there difference layer normalization operations; The final output layer is also sigmoid classifier and T; is the logit of ith sentence.
However, we argue that the way of truncation will result in the loss information in the latter part of the document. If the key sentences locate the end of the document, they will never be recalled
3

---

Under review as a conference paper at ICLR 2020
though the model is powerful. In contrast, if model take all the tokens of the document as input, will get these key sentences and produce better result. Therefore, we propose BERT-AL to solve this problem. For simplicity, we only employ the Simple Classifier the summarization layer in the following sections.
3 BERT-AL
In this section, will introduce the detail about BERT-AL, and illustrated in Figure 1 BERT- AL has mainly two key components different with BERTSUM: one is the multi-channel LSTM and another is the positional encoding.
summarization layer
LSTM
X N
LSTM
Transformer
Transformer
Transformer
Embeddings
Sugmart En
Irput Doaument
Figure 1: The architecture of BERT-AL
3.1 MULTI-CHANNEL LSTM
We first give the definition of our scenario: assume that the length of the document is Ioees and we have only a pretrained BERT model with the maximum sequence length IBERT- We first split the document into Naegment segments and each segment has the length / BERT (the length of neegmene segment is shorter than (ERT)). We set (ERTTAA. = nesemmeen BRER as the maximum sequence length of BERT-AL, and then we have (BERT-AL Idocs which means BERT-AL can take arbitrar- ily long document as input.
As we know, LSTM has the capability of capturing information across arbitrarily long steps but is weak capturing long-term dependencies. Therefore, within a segment, we use Transformer to capture long-tern term dependencies and extract the feature via self-attention among local positions, and then fully represent the segment at each layer. To this end, we propose a multi-channel LSTM to chain the representation of different segments. For the above definition, Laoee can be arbitrarily long and so is Raggment- Thus, we apply LSTM on Rsegment dimension, and the steps of LSTM should also be Reegment.
As Figure D shows, LSTM is following the segment-wise Transformers at each layer and takes the hidden states from Transformers as its input. To fit the next layer, we LSTM's hidden size also as dmodel. i.e., 768 for BERT-Bas Base and 1024 for BERT-Large. Thus, the computing in each layer is shown as follows.
H Concal ( H;,1, H 1, Iegmmeet
HTrans Trans (aansforde former (oammerrH,,p (H,1,5 1 Rllnarr dmadal))
H. LH. Trans E AllaERt. dmmntel)
4

---

Under review as a conference paper at ICLR 2020
where H, is the hidden states from ith layer, Hy, is jth segment in H,, and H is the hidden states from jth Transformer. All segment-wise Transformers at the same layer share parameters.
BERT-AL employs single layer of unidirectional LSTM as the implement of multi-channel LSTM, and a channel is corresponding to a relative position within a segment. Per-gate layer normalization is applied in each LSTM cell (Ba al. 2016). We do not split the segments according natural sentences, so [CLSS] may be at the beginning, in the middle or at the end of a block. Similarly, the output at [CLLS] positions also do not come from the same LSTM channel. The reason we apply multi-channel LSTM on Nlsegment rather than 2D LSTM on the whole sequence is taking advantage of LSTM's 's expandability on variable-length segments, but meanwhile, reducing the interference to self-attention among positions within a segment.
3.2 EMBEDDING
BERT-AL take the same input format as BERTSUM does, i.e., adding [CLS] to the head of each sentence and [SEP] to the tail of each sentence. Since BERT-AL has multiple segment-wise Trans- formers at the first layer, its input embedding is also changed. More details are shown as follows.
Before feeding embedding to Transformers, the sequence is divided into segments whose lengths are the same with the pretrained BERT model's maximum sequence length. Token embedding and segment embedding learned position embedding used in the original BERT. However, if we do not re-pretrain the BERT, the dimension of position embedding matrix in the pretrained BERT model is [EETT, dmodet] while the length of task's s input is |RERT-AL- To resolve this problem, we copy the original position embedding matrix Peeegment times and concatenate them. Therefore, we can get a position embed- ding matrix with dimension [BBERTALL dmodel). Through this way, each position in the input will get a positional encoding. Therefore, the model can work correctly. In the Transformer layer, each segment does not interact with each other. Therefore, although the position embeddings are same in different segments, it does not affect the model to extract feature its own segment.
also same the original BERT. For position embedding, we also reserve the
3.3 OTHER DETAILS
It is obvious that each sentence's 's length is different, but segments have the same length. Therefore, the start of each segment may not be [CLS] and the tail may not be [SFPP] either. In Figure [ the T; is the representation of sentence through N Transformers and LSTMs. After summarization layer, Y; represents the score of that sentence, and then we choose sentences with top 3 highest scores compose the summary. Finally, Table shows difference of BERT-AL and BERTSUM.
Table 1: The difference of BERT-AL and BERT-SUM
BERTSUM
BERT-AL
BERT +
segment-wist BERT multi-channel LSTM summarization layer
Architeture
summarization layer same as BERTs
copy and concatenate BERTs
Position embedding
position embedding
position embedding
Max input length
BERTs input length
arbitrarily long
EXPERIMENT
In this section, we demonstrate BERT-ALs effectiveness on text summarization by conducting ex- periments on the CNN/Daily Mail dataset. We compare BERTSUM with our models on various different settings, since BERTSUM is the state-of-the-art on CNN/Daily Mail dataset.
4.1 CNN/DAILY MAIL DATASET
The CNN/Daily Mail dataset contains news articles and associated highlights, i.e.. few bullet points giving a brief overview of the article. We used the standard splits as training, validation and testing sets BERTSUM did. The statistics of datasets is shown in Tabel 2
5

---

Under review as a conference paper at ICLR 2020
Table 2: Statistics of CNN/Daily Mail dataset
CNN
Daily Mail
Total
Train
90266
196961
287227
Validate
1220
12148
13368
Test
1093
10397
11490
We also perform the same preprocessing for data as BERTSUM did, including keeping entities, splitting sentences CoreNLP and following methods in See al. (2017).
We employ ROUGE-I, L. Lin 2004) to evaluate the performance for different methods and focus on the Fi score, as follows.
Σ Se (RoffrrenceSvmmariic) E gram,,es ES Countmateh (gramp)
BROUGE-N =
E Se(Ree (RefereecceSummariic gram, Counl(
Ese RRe fieferrnceSummrrces Σ gram,es Es Coustlaateh (gramp)
PROUGE-N
(5)
Ese See(aandidattsummn
Egram,,s Counl(gramm
FROUGE-N= = (1+
BROUGE-N 32 BRRRGGG-NN
where stands for the length of the n-gram, gram,, and Count (gram,)) is the maximum number n-grams co-occurring a candidate summary and a set of reference summaries.
Rios LCS(X,Y)
m Pies
LCS(X,Y)
(6)
n1 Rics Pies
(1 (1+)))Riiss
Fics
Rice + Pres
where is reference summary sentence and is candidate summary sentence, m the length of X and is the length of Y, LCS(X,Y) is the length of longest common subsequence of X and
To adapt dataset to suit extractive summarization task, we also use greedy algorithm which is same as BERTSUM to generate an oracle summary for each document. The algorithm selects a set of sentences as the oracle set, which can maximize the ROUGE scores.
4.2 EXPERIMENT SETTING
Table 3: Setting of experiment groups
(BERT
[BERT
Tseemment
Group-1 I
512
64
Group-2
16
512
32
Group-3
128
512
4
Group-4
256
1024
4
Assume that we have only a pretrained BERT model and the length of its position embedding is IBERT. Our task is to produce summaries documents with Lave tokens. The BERT-AL can take (BERT-AL length tokens as input (LBERT-AL = n negmeen BEER and LBERT-AL >= (doe). We design four groups of experiments and set |BER 8, 16, 128, 256 for them. More details about settings are shown in T Under each group setting, we compare three different methods, including Baseline-l, Baseline-2 and BERT-AL. More details about methods are shown follows.
1) Baseline-1 applies BERTSUM to the dataset with length LEERT. and then truncates text longer than BERT-

---

Under review as a conference paper at ICLR 2020
2) Baseline-2 concatenates the first (EER position embedding of the original BERT with domly initialized embedding with the length (Nzegment - 1] LBERT, and then employs the con- catenated embedding BERTSUM's position embedding. Finally, it applies BERTSUM to long documents with length IBERT-AL-
3) BERT-AL copies the position embedding matrix neegment times and concatenates them along the dimension of length, and then employs the embedding as the position embedding of segment-wise Transformers corresponding layers. Finally, applies BERT-AL to long documents with length IBERT-AL
The Baseline-1 aims to compare BERTSUM and BERT-AL while the Baseline-2 aims reveal the effectiveness of multi-channel LSTM. Furthermore, Baseline-2 can be used to indicate whether the promotion of BERT-AL is due the longer input or the multi-channel LSTM.
For fairly comparison, we set all hyperparameters equal to BERTSUM reported in Liu 2019) Specifically, we the number of layers as 12, the hidden size as 768, the number of self-attention heads as 12 and the feed-forward size as 3072. We use Adam with $1 0.9, B2 0.999 and
e 10-9 We also use a linear learning rate decreasing scheduler with warming-up on first 10,000 steps. All models are trained for 50,000 steps with gradient accumulation per two steps. We also select the top-3 checkpoints based on their evaluation losses on the validations set and report the averaged results the test set.
4.3 RESULTS AND ANALYSIS
The results are showed in Table 4 including BERTSUM
We can obtain the following observations:
Table Experiment results
Model
ROUGE-
ROUGE-2
ROUGE-L
BERTSUM
42.94
20.14
39.38
Baseline-1
6.47
1.24
5.68
Group-1
Baseline-2
40.38
17.86
36.77
BERT-AL
41.26
18.63
37.69
Baseline-1
11.81
3.69
10.47
Group-2
Baseline-2
40.44
17.93
36.83
BERT-AL
41.73
19.00
38.18
Baseline-1
41.19
18.57
37.50
Group-3
Baseline-2
41.44
18.77
37,83
BERT-AL
42.14
19.38
38.60
Baseline-1 I
42.30
19.58
38.73
Group-4
Baseline-2
42.27
19.57
38.72
BERT-AL
42.61
19.79
39.07
1) For all of the four groups, BERT-AL outperforms baselines, consistently. It proves that BERT-AL effective on long document summarization task, which comes from merging Transformer's local feature extraction ability and LSTM's global time capturing ability.
2) Comparing Baseline-1 across four groups, performance increases with L BET is longer. It implies longer input contains more useful information and truncating input leads to performance drop.
3) Comparing Baseline-1 with Baseline-2, Baseline-2 performs better than Baseline- under nearly all settings. It implies randomly initialized position embedding also help capture longer in- formation even only with finetuning. However, this improvement will decrease with the maximum sequence length increases, and Baseline-2 has worse perform than Bascline-1 when / BE 256.
4) Comparing BERT-AL and Baseline-1 in Group-4, BERT-AL can further promote the ROUGE score and break up the bottleneck of Baseline-2 when the / BE is large. It implies repeated position embedding with multi- multi-channel LSTM is more effective than random initialized one.
'The result from reproduction

---

Under review as a conference paper at ICLR 2020
5) Comparing 2 with BERT-AL in all four groups, the promotion of ROUGE score is more significant when the number of segments (i.e., LSTM's time-steps) is larger. implies that the LSTM's capability of timing capture is not be fully utilized when the LSTM has less time-steps.
4.4 DISCUSSION
BERT-AL is designed as the model with shorter pretrained BERT, and it still can achieve com- parable performance to BERTSUM. formance has been very closed to BERTSUM (512). Simultaneously, BERT-AL has much faster training and inference speed, since Transformers runtime is proportional to 12 and multi-channel LSTM is high parallelly applied Nsegmentt steps. Therefore, BERT-AL can be good alterna- tive under the following situations: 1) For a NLP task, the input text is too long to feed into a BERT model due to GPU memory other limitations. 2) The time of pretraining longer model from scratch unacceptable under a limited computing resource. 3) There is restraint for inference speed, and then we also can split the text into small segments and feed them into BERT-AL.
Group 4 experiments, we can find BERT-AL (256)s per-
5 RELATED WORK
There are several works related modeling recurrence for self-attention network (SAN) in Trans- former. Debghani et al. (2018) recurrently refines the representations of each layer to improve SAN encoder. [Shen et al. 2018) introduces directional self-attention network (DiSAN), which only allows each token to attend previous (or following) tokens. Both Hao et al. (2019) and Chen et al.) (0188) propose to combine SAN encoder with an additional RNN encoder. The former enhances the Transformer with recurrence information, while the latter augments RNN-based models with SAN encoder. WWang et al. (2019b) adds a local RNN layer in front of SAN in each Transformer layer. which aims to capture both local structures and global long-teng term-termme dependencies in sequences. Wang et al. (2019)) adds LSTM after all the Transformer by the guide of Coordinate Architecture Search. However, these above works all target solve the limitation of positional encoding, while our model aims to apply pretrained BERT to longer text and need not to pretrain from scratch, which reduces the cost of time and computing resources. Transformer-XL. (Dai et al., 2019) con- tains segment-level recurrence with state reuse and relative positional encoding for language model beyond lixed-length context, which similar with our model and also can processes longer text. However, Transformer-X XL aims at language model and need to pretrain the model from scratch.
For processing long input, Bengio et al. (2013) proposes "conditional computation" which to only compute a subset of networks units for given input by gating different parts of the network. Ling Rush|ee (2017) proposes a coarse-to- fine attention model that uses hard attention to find the text chunks of importance and then only attend to words in that chunk. [iu et al. (2018) first coarsely selects a subset of the input, then trains an model while conditioning on this subset. Cohan et al. (2018) proposes hierarchical encoder which represents each section using word-level RNN and then represents all sections using section-level RNN. These works all similar with our method, which is splitting the long input into several subsets to process them respectively and then merging them. However, they are all need re-pretrain the model from scratch. Furthermore, we use the most powerful model-BERT
6 CONCLUSION
BERT has been the state-of-the-art for all kinds of NLP tasks. However, BERT cannot be applied to long text tasks, e.g., document-level text summarization, because it cannot take text longer than the maximum length as input. However, the maximum length predefined during pretraining, and expanding maximum length need re-pretrain which will cost lots of computing resource. We propose a novel model named BERT-AL which combines the advantages of Transformer and LSTM. The BERT-AL can take arbitrarily long text as its input and need not re-pretrain from scratch. We demonstrate BERT-ALS effectiveness on the text summarization task by conducting experiments on the CNN/Daily Mail dataset, and the experimental results prove that BERT-AL is effective on NLP tasks with very long text as input. Furthermore, our model can be easily adapted to various tasks and pretrained models.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv: (ee::06070655 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1rXXi::308.3432. 2013.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, et al. The best of both worlds: Combin- ing recent advances in neural machine translation. arXiv preprint arXiv: /804. 09849, 2018.
Arman Cohan, Franck Demonccurt, Doo Soon Kim, Trung Bui, Seokhwan Kim. Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of NAACL-HLT, pp. 615-621, 2018
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-ke xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv:1901.0288 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint urXiv:/807.0333 0.3899...99,.. 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171-4186 2019.
Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. Modeling recurrence for transformer. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1198-1207, 2019.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693-1701, 2015.
Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin. Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. Wikireading: A novel large-scale language understanding task over wikipedia. In Proceedings of the 54th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 1535-1545, 2016.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmn): Building longer and deeper rnn. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp- 5457-5466, 2018
Chin-Yer Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 81, 2004.
Jeffrey Ling and Alexander Rush. Coarse-to-fine attention models for document summarization. In Proceedings of the Workshop on New Frontiers Summarization, pp. 33 42, 2017.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations, 2018.
Yang Liu. Fine-tune bert for extractive summarization. arXiv preprint 1rXiv:1903.1033: 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HLT, pp. 2227-2237, 2018.

---

Under review as a conference paper at ICLR 2020
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language derstanding by generative pre-training. URL https://s3-us- usee-s--2- amazonaws. com/openai- aassttreenehooev/uugqgquuugreeer
understanding paper. pdf. 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Abigail See, Peter J Liu, and Christopher Manning. Get the point: Summarization with pointer- generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume Long Pupers), pp. 1073-1083, 2017.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. Disan: Di- rectional self-attention network for mn/cnn-free language understanding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, pp. 5998-9990000 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, 2018.
Chenguang Wang, Mu Li, and Alexander J. Smola. Language models with transformers. CoRR, abs/1904.09408, 2019a.
Zhiwei Wang, Yao Ma, Zitao Liu, and Jiliang Tang. R-transformer: Recurrent neural network enhanced transformer. arXiv preprint arXiv: 1907.0557, 2019b.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint /9t:::/906.0823333 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
10


Reviews:
[{"review_id": "SkeVjGKGqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a BERT based document summary model that has the capability of modeling arbitrarily long documents. Experiments on CNN/Daily dataset show the improvement of the proposed model compared with its baselines.\nThe advantage of this paper is that the time-consuming training process of BERT can be avoided.\n\nI think this paper is not good enough to be accepted. The model does not have the ability of understanding “Arbitrarily” long documents, it just aggregates multiple (n_segment) BERT segments and extends the BERT capability from l_bert to n_segment*l_bert. It is not as flexible as the LSTM which can capture the real arbitrary long document. The model looks bloated and the performance is not persuasive. The authors reproduced the baseline BERTSUM but the reproduced performance is significantly lower than performance in the original BERTSUM paper. (without explanation) The proposed model can’t beat the performance in the original BERTSUM paper. Even to their reproduced BERTSUM results, the proposed model gives very closed performances.\n\nBesides, this paper doesn’t compare their model with other SOTA models and all the experiments are conducted in one dataset. The result analysis is not enough. As the main contribution in this paper is to understand the longer documents, the performance on long documents should be evaluated separated.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJelPR1a9B", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a methodology to overcome the problem of processing long sequences with a pre-trained Transformer model, which suffers from high computational costs due to the complexity being quadratic in the length of the sequence. The authors also point out that BERT needs to be retrained from scratch if sequences longer than the specified maximum length (512) are to be processed. Their method (BERT-AL) chunks the input text into segments of maximum length. Each segment is propagated through several layers of a pre-trained Transformer layer followed by a multi-channel LSTM. Herein, the positional embeddings in each segment are the same. The model is applied to extractive summarization, and directly compared to the BERTSUM model, which can only process documents up to length 512. The comparison is made for 4 application scenarios, each corresponding to an artificial maximum length after which the BERTSUM model truncates the input: after 8, 16, 128, and 256 tokens. The experimental results suggest that BERT-AL outperforms BERTSUM in all 4 scenarios: Substantially for max length 8 and 16, and marginally for 128 and 256. BERTSUM without any truncation still performs best, however.\n\nI think the paper should be rejected for three main reasons: (1) The idea of using RNNs to overcome the long sequence problem in Transformers is already well studied. The specific proposed architecture may be new, but the design choices are not well justified. (2) The experimental evaluation is not convincing: The application scenarios are unrealistic, the dataset is not well-chosen, and the results are not impressive. (3) The structure and language of the paper needs to be polished.\n\nRegarding (1): The problem of dealing with long sequences in Transformer models has been known for a long time, and there already exist several proposed solutions based on an RNN component, which are acknowledged in this paper. The authors propose another model for a similar purpose. While the model is interesting, the individual design decisions are not well-justified, e.g., why the LSTM is applied at each layer and why it needs to be a multi-channel LSTM. Neither a comparison with models from previous works nor an ablation study is provided, so that there is also no empirical justification for the model design. \nThe authors argue that the novelty in their model comes from its applicability to pre-trained models, e.g., BERT and XLNET. While the motivation for BERT is reasonable, it is questionable whether this will still be a relevant for future research, as XLNET already mitigates the problem through the reliance on Transformer-XL, which is conceptually very similar to BERT-AL. If future pretrained models account for long sequences already during pre-training, the motivation for this work is rather low.\n\nRegarding (2): In order to show the superiority of BERT-AL over BERTSUM, it would've been natural to employ it to datasets whose data are beyond what standard BERT can handle, i.e., longer than 512 tokens. Instead, the authors chose to evaluate on 4 rather unrealistic application scenarios derived from the CNN/DailyMail dataset, which artificially limit the range of the pre-trained BERT model to much less than what it can actually handle. While I understand the idea behind this setup, the insight that BERT performs poorly if the text is truncated to 8 or 16 tokens is not helpful. Since the improvement of BERT-AL over the baselines becomes marginal for longer sequences, and the performance of BERTSUM, without truncating the text, is not reached, I am not convinced of the model's value. An evaluation on more suitable datasets could help here.\n\nRegarding (3): The paper is difficult to read for two reasons. First, the paper is not structured in a way that facilitates the understanding of the proposed method. For example, it is not clear why the pre-training tasks of BERT (masked language modeling and next sentence prediction) are relevant enough to be so prominently described. If the method is applicable to many pre-trained Transformers (as was claimed), then there is no need to go into this level of detail. On the same note, there is no need to explain BERTSUM's Inter-Sentence Transformer and Recurrent Neural Network variants for half a page if they are dismissed in the next paragraph. Second, the paper contains a lot of typos and grammatical mistakes. This is not a deal-breaker by itself, but it makes it obvious that the paper needs substantial polishing before it should be published. \n\nQuestions and suggestions to the authors:\n\n1) What is the advantage of choosing a multi-channel LSTM, where each segment is a channel? Wouldn't a single channel suffice that processes aggregated information from the segment (as the Transformer computes)? I think your paper would benefit from exploring different options like that experimentally to better justify your model decisions.\n\n2) It is not clear to me how the information exchange between layers happens. If each layer consists of a single layer from a pre-trained Transformer followed by an LSTM, the input to the following layer of the pre-trained Transformer is the output of the LSTM, which is not part of the pre-trained model. That is, the following Transformer layer receives an input that is very different from what it has seen during pre-training, which we can not expect to work. If there is a misunderstanding on my side, could you please clarify that, or otherwise make it clearer in the paper?\n\n3) I think a good baseline would be to do segment-wise encoding via the pre-trained Transformer, and then feed all sentence representations (i.e., the respective CLS tokens) from all segments into the Transformer-based summarization layer from BERTSUM. This would be a compromise between your model (doing segment-wise encoding but not using an LSTM) and BERTSUM, and thus could potentially highlight the importance of the LSTM.\n\n4) I don't think showing the number of examples in your datasets as in Table 2 contributes much to the paper. Instead, I suggest to show statistics on the length of the documents / number of sentences etc., because these are directly relevant to your study.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1gQytXw5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The author proposes an extended version of the BERT architecture, BERTAL for text summarization. BERTAL aims to overcome the limit of maximal allowable text length in BERT, and the experiments show some consistent performance enhancement to baseline architecture using BERT and approaching performance to state-of-the-art architecture using BERT (BERTSUM), where BERTSUM has the maximal length limit .\n           The experimental procedures as well as the choice of architectural design are well explained and designed in a logical way. Substantial comparison experiments also pinpoint the performance.\n            However, it doesn’t compare the result of BERTAL to other text summarizers with arbitrary-length text, which is not based on BERT.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hke48Wc_KS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed another variant of BERT, called BERT-AL, which can deal with arbitrarily long inputs. The authors constructed the proposed method by combining the segment-wise BERT with the multi-channel LSTM. The authors validated the proposed method on the text summarization task and achieved higher performance than existing works. Their proposed method can be combined with other transformer-based approaches, such as XLNet or RoBERTa.\n\nThe paper tackles a relevant topic that is interesting to the attendees of ICLR, which is, “How do we get the overall information of a document, which has a number of sentences?”.  However, the paper has several issues that I will try to cover in the following.\n\n(1) (No sufficient results which support authors’ claim) \nFirst: In the introduction, authors claim that the model for NLP tasks whose inputs are usually long texts should be able to take arbitrarily long inputs so that the model can extract important information from the entire long documents. However, there is no concrete experimental example, where the existing methods (standard BERT or BERTSUM) fail to extract information.\nSecond: In the discussion of the experiment section, the authors claim that the proposed method BERT-AL can be parallelized and runs faster than existing methods. However, there is no comparison with existing methods about the running time. \n\n(2) (Clarity about the proposed method)\nThe presentation of the proposed method is slightly confusing. There is no concrete explanation about the learning procedure. Are any pre-training and fine-tuning procedures the same with those of the standard BERT?\nThe main part of the proposed method is to combine the segment-wise BERT with the multi-channel LSTM. How is the pre-training procedure conducted with parallelized computation? A detailed explanation should be shown.\n\n(3) (Clarity about the experimental results)\nIn table 4, the experimental results are shown. There are results of the baseline 1 & 2 (existing methods), the proposed method, and the existing method (BERTSUM). There is no discussion about the comparison of the proposed method with BERTSUM. It seems that BERTSUM’s result is superior to the proposed method’s result. Please explain more details of those results.\nThe paper includes an explanation about the dataset and the task but does not include the experimental environment (used machines and the time to conduct the experiment). For the fair evaluation of results, we need the information on the overall experimental environment.\n\n(4) (Presentation)\nThe presentation around mathematical formulas is a bit confusing. Please fix the following.\n- Subscripts of the formula should be ‘\\mathrm’\n- ‘<=‘ should be ‘\\le’\n- The ’n’ of the right-hand side of equation (5) is ’N’?\n- The multiplication ‘*’ should be ‘\\times’", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a hybrid LSTM-Transformer method to use pretrained Transformers like BERT that have a fixed maximum sequence lengths on texts longer than that limit.

The consensus of the reviewers is that the results aren't sufficient to justify the primary claims of the paper, and that—in addition—the missing details and ablations cast doubt on the reliability of those results. This is an interesting research direction, but substantial further experimental work would be needed to turn this into something that's ready for publication at a top venue.
Decision: Reject