ADAPTING PRETRAINED LANGUAGE MODELS FOR LONG DOCUMENT CLASSIFICATION
Authors: Matthew Lyle Olson
Lisa Zhang
Chun-Nam Yu
Venue:  (2020)
Keywords: NLP
Deep Learning
Language Models
Long Document

Abstract: Pretrained language models (LMs) have shown excellent results in achieving human like performance on many language tasks. However, the most powerful LMs have one significant drawback: a fixed-sized input. With this constraint, these LMs are unable to utilize the full input of long documents. In this paper, we introduce a new framework to handle documents of arbitrary lengths. We investigate the addition of a recurrent mechanism to extend the input size and utilizing attention to identify the most discriminating segment of the input. We perform extensive validating experiments on patent and Arxiv datasets, both of which have long text. We demonstrate our method significantly outperforms state-of-the-art results reported in recent literature.

Under review as a conference paper at ICLR 2020
ADAPTING PRETRAINED LANGUAGE MODELS FOR LONG DOCUMENT CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Pretrained Language Models (LMs) have shown excellent results in achieving human-like performance on many language tasks. However, the most powerful LMs have one significant drawback: a fixed-sized input. Due to this constraint, these LMs are unable utilize the full input of long documents. In this paper, we introduce a new framework to handle documents of arbitrary lengths. We investi- gate the addition of different mechanisms to handle an extended input size, and we demonstrate how attention can be used for parameter updates based on the most discriminating segments of input. We perform extensive validating experiments on patent and Arxiv datasets, both of which have long text. We show our method consistently outperforms state-of-the-art results reported in recent literature.
INTRODUCTION
Neural Network based Language Models (LMs) have flurry of work, where new design and implementation improvements have advanced state-of-the-art performance in a variety of natural language tasks over the past few years (Devlin et al. 2018 Dai et al.] 2019 Radford et al.) 2019 Yang et al. 2019 Liu et al. 2019). LMs are powerful tools because they process a collection of unlabeled text and learn rich embedding of natural language without supervision. This represen- tation can be re- purposed on subsequent tasks such as classification and sentiment analysis (Korde & Mahender 2012). This technique is essential for reaching state-of-the-arr performance, as LM- based systems are able to achieve much better results than techniques that only use a small, labeled dataset. Modern LMs achieve thier success by utilizing powerful mechanism called "The Trans- former" (Vaswani et al. 2017). The transformer learns strong dependencies between its inputs and has the ability to be stacked as many times as hardware can handle. This mechanism allows LMs to take in relatively large, yet still fixed, sized input.
For the largest LMs, the input size can reach up to four thousand tokens; however, this is still a limitation as they cannot process arbitrarily long documents. On many natural language tasks, this fixed input size is sufficient. For example, reading comprehension tasks are often used to analyze the quality of LMs, contain relatively few words average, and do not have sufficiently long de- pendencies (Wang et al.] 2019). Furthermore, there are multiple tasks where the input data is too long and must be truncated before being processed (Lee & Hsiang 2019). Truncation is unsuitable because long complex text often contains inter-referential pieces of information. For instance, read- ing the final chapter of book after all the previous ones takes on a different meaning compared to reading the same text by itself.
Solving the problem of arbitrarily long input requires more than cursory glance. first intuition may be to take a pretrained LM, separate the text into segments, place Recursive Neural Network (RNN) after the embedding, and simply pass in the segments sequentially. This seems reasonable as RNNs have been used in the past for sequential text-based tasks, e.g. sentiment analysis (Socher ctt al. 201T). Unfortunately, using RNNs in this way causes two problems. First, RNNs are typ- ically trained via backpropagation through time, making them prone to the problem of vanishing or exploding gradients (Pascanu et al. 2012). While many techniques exist to deal with this issue Williams & Peng 1990, Mujika al. 2018), they do not solve the secondary problem of signifi- cant memory requiements for LM parameter updates. Each forward pass through the LM produces set of parameter gradients. With multiple forward passes, the number of stored gradients quickly

---

Under review as a conference paper at ICLR 2020
grows segmented document increases in length. To further complicate the issue, the number of parameters in transformer- based networks is quadratic in relation to the fixed input size.
In order to solve these problems, we look the brain's ability to guide attention and provide be- havioral updates as a biological inspiration. The brain processes an extraordinary amount of data, yet from moment to moment, much of that information is filtered out. What becomes filtered is not arbitrary, but is directly influenced by one's 's objective. This filtering can be observed when an individual is tasked with counting the number of basketball passes in video, yet fails register large gorilla that appears in center frame (Chabris & Simons, 201T). Even outside the moment to moment, changes in an individual's behavior is guided by the structure of their values, self-selected or otherwise. An individual's 's value structure imposes framework for determining significance of events (Peterson 1999). An event that may have been insignificant in the past can go on to take new meaning once new value has been gained or once an old one has changed (Laudet et al.) 2006). This increase in valence may cause a behavioral change, reorientation of goals, or a shift in the interpretation of experiences.
Unfortunately, when it comes to modeling this selective learning Artificial Intelligence systems, common gradient-based methods fall short. A typical neural network model will update every pa- rameter based on all of the inputs to minimize an objective function. While this is desired for many applications, full input based learning can cause issues for others (Pascanu et al., 2012). [e et al. (2088) attempt to solve this issue via selective attentive backtracking, though they focus on remem- bering long-term dependencies rather than utilizing the full context of the input. In this work, introduce a method that performs objective-based filtering during learning, but still utilizes the entire input.
We focus on two domains, specifically for classification: patents and scientific papers. Patent reading a typical activity for lawyers trying to find relevant documents. Since 2003, the number of patents filed has increased nearly every year (WIPO) 2018); and using an automated system to perform classification is a continuously growing area of interest (Trappey et al. 2006). Scientific papers are another significant area of investigation. Many papers are uploaded to the internet every day, and their automatic categorization is becoming necessity. A statistics paper, for example, may not be categorized machine learning by its authors, but could be of interest to the machine learning community. Additionally, [shitoyan et al. (2019) show material science concepts can be learned from scientific papers by language model. They demonstrated materials for functional applications could be recommended several years before their initial discovery.
In this work, we use attention mechanism to discover the significant portions of text to be used in updating a pretrained LM. We find attention-based updates to be essential because the most sig- nificant portion of an input sequence may occur anywhere throughout the document. While the datasets we study often start with highly discriminative features (titles and abstracts), we conduct experiments to show our attention mechanism can find the important parts of text even when it does not occur in the first segment. In either case, updating a LM's parameters from only the first input segment performs well- often better than just using baseline of the original LM with the input truncated to fit the max size. However, we find using an attention mechanism consistently improves performance and achieves the best results in our experimented language models. Therefore, our contributions are as follows:
1. We introduce a new framework for performing inference over arbitrary length documents.
2. We perform extensive validating experiments of our methods, showing how our attention- based framework consistently outperforms alternative methods.
3. We demonstrate the attention mechanism can be utilized to perform selective language model parameter updates.
RELATED WORKS
Language model pretraining is popular method for tackling many natural language understanding tasks. Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al. 2018) and Generative Pre- Training (GPT) (Radford et al. 20188) are two well-known language model pretrain- ing methods that utilize in this work. BERT is trained by selectively conditioning on most of
2

---

Under review as a conference paper at ICLR 2020
X1
X1, X2..... Xm
LM
LM
Enhancement
C
p
Figure 1: Left: The base language model (Base BERT/GPT) for classification. Right: The frame- work of language model enhancements for classification.
the input sequence, masking the rest, and attempting to predict the masked tokens. BERT is concur- rently trained with a next sentence prediction task. The token used for the next sentence prediction task reused for classification after BERT has been trained. GPT trains via next word prediction by conditioning on sequence of text and trying predict the next token. By training Wikipedia and books, GPT is able to generate novel sequences of text. GPT able to perform classification by appending special classifying token the end of an imput, encoding the token into latent representation, and using the representation as input to a new linear classifier set specifically for the task. These language model pretraining methods became state-of-art natural language process- ing benchmarks such as GLUE (Wang et al. 2019) and SQUAD (Rajpurkar et al. 2016), achieving close to human-level performance.
As both BERT and GPT are based on transformers, their computational and storage costs scale quadratically with the input sequence length. This limits their application mostly relatively short pieces of texts. To the best of our knowledge, we are not aware of any works applying these pretrained language models for long document classification. Next, we discuss a few other deep learning-basee text classification approaches, with special attention to the classification of scientific papers and patents.
Dai & Le 2015) consider pretraining recurrent neural networks with large corpus of texts, and show improved performance on several text classification tasks. Kim (2014) introduced convolutional neural networks (CNN) for text/sentence classification. Yang et al. (2016) introduced hierarchical attention mechanism for document classification that attends to interesting sentences and words in document. The length of documents considered in latter two works are relatively short, with the corpus consisting of mostly individual sentences or online reviews.
For patent classification, Li Lt, al. (2018) present deep learning algorithm called Deep Patent, which based on convolutional neural networks and continuous skip-gram embedding. They were the first to apply deep learning large scale real-world patent classification. Lee & Hsiang (2019) used the pretrained BERT model to classify patents at the section and subclass level, only taking the title and abstract, the first claim, as input.
For scientific paper classification, He et al. (2019) introduce a relatively large scientific paper dataset and perform classification through multi-network approach. They introduce RNN Attention- based reinforcement learning scheme. They select short text sequences to be parsed by a CNN. The representation learned by the CNN is sequentially fed back into the RNN for subsequent text selections and eventual classification.
Very recently, Cohan al. 2019) construct a joint sentence representation that allows BERT trans- former layers to directly utilize contextual information from all words in every sentence. However, their task is classification at the sentence level for single text input.
METHODS
Our approach for classifying long documents is to divide the long document into a sequence of segments, each of which is short enough be processed by pretrained language model. The in-
3

---

Under review as a conference paper at ICLR 2020
21, 21.21......
Z1, 21.2
LSTM
Zi. 41.42...2
LSTM
Attention
argmax
21
Z1
Zi b 2o ..... 7m
hm
hg hm
a
a
Z
21 t
11
h2
MMiii
Figure 2:
Three enhancements for base LM. Colors used represent the progressive enhance- ment of the combination strategies. Left: The concatenation model (Cat-BERT/GPT), Center: The RNN based model (RNN-BERT/GPT) Right: The full attention model (ATT-BERT/GPT),
formation from the language model's representation of each segment utilized in different manners to produce a classifier (see Figure rright)). We develop multiple strategies to combine these rep- resentations, progressively enhancing the combinations: starting from simple concatenation to an attention-based strategy. In this work, one of our primary contributions is to investigate the effec- tiveness of these different enhancements.
Let (x1, .,, be document, where is fixed-length sequence of tokens (segment), and m the number of segments in the document. Let E be the respective labels k-class classification problem. We z; LM(x)) E R to denote the d-dimensional latent representa- tion, the segment 2;, for classification (e.g. the representation of the "CLS" token in BERT) from selected language model LM. Let Cw (v) = o (W b) be linear classifier followed by the softmax function, and let the vector of probabilities æ being assigned to each class.
Base Language Model (Base LM) the usual application of deep language models such as BERT and GPT for text classification, the imput text truncated fixed length (256, 512, etc) due to limits in the size of the model. This corresponds to our basic model:
= Cw.& (LM(x1)),
(1)
where W Rkxd b TR and assume the segment length of in equals the input size limit of the language model. This model is depicted in Figure [lfft).
Next describe the three progressive enhancements the base LM, shown in Figure
Concatenated Language Model (Cat-LM) The first enhancement is natural extension im- prove the basic model by including information from more segments T2...., Ty- A very simple way to do this is concatenate the representation ž1, ..,..., .m before the classification layer. This leads to the model:
p = Cw Cw.6(LM(z)) + LM(x)) ... LM(xm)).
(2)
where W kx b R This model difficult perform backpropagation directly because we cannot hold m copies of the LM parameters in memory at the same time. We solve this problem by stopping the backpropgation paths of some of the segments, namely discusses this approximation.
Zm Section 3.1
RNN-augmented Language Model (RNN-LM) The third model we want to consider is one that , 2m using bidirectional LSTM (Hochreiter & Schmid- .ee....,h,,, hn)) biLSTM(ZI.... 7m) be the dimensional hidden state representations from bidirectional LSTM, where h, E R². The biLSTM--aad model can be written as:
summarizes the information from 21.....
huber 1997). Let (h₃, hz....,
Cw.n(LM(rz) + h2 e hm),
(3)
where W E (d) b E R For this model we also stop the gradient computation at 22, ... Sm, and do not backpropagate beyond the LSTM parameters.
We do not include 22...., Zm as input to the classifier as the size of W and b scale directly with m, and with large the number of parameters would grow counter-productee pondute-roduutivvely However, still
4

---

Under review as a conference paper at ICLR 2020
include 21 the text from that segment typically contains the most discriminative features. By in- cluding 21, there is direct path for gradient updates to the language model. Additionally, we found that including h1, instead of h2, in the classifier consistently produced worse results. We believe this is from the forward direction of h not providing any contextual information. Comparatively. the backwards direction of h2 provides enough context that, when concatenated with 21, achieves the best performance.
Attention-baser Language Model (ATT-LM) For our attention- preentio-bbased model, we utilize the structure of RNN-LM and add the attention mechanism as described in Yang et al. (2016). We define our attention variables as follows:
u, tanh( + ba), .
0; = ouu us)
(4)
Σ ochi
where W, E R2xx4 E R4, u, IR9 are learned attention parameters for attention over seg- ments. Let M(a) = arg max ag be the index the segment that gives the highest attention weight. These definitions give us our equation for the attention-base model by concatenating set of relevant features:
P Cw,CCw/.LLM. (x1 @ h2 @ hm @ a ① a + LM(ZM(a))),
(5)
where W (2)+ b E RRtt are the parameters of the linear classifier. For this model, we stop the gradient computations paths for all z, 's apart from 2 and the selected Z)ía)-
For all four methods, we let n be the number of training documents and use the negative log likeli- hood as the loss function:
L
log Pgg.
(6)
3.1 PARAMETER UPDATES
We run mini-batch stochastic gradient descent for parameter updates. We treat the parameters of the language models and the other parameters (weight matrix W for classification, LSTM param- eters, etc) differently. We perform full gradient computation on non-LM parameters, and only ap- proximate gradient computation for the LM's parameters by stopping backpropagation on selected segments (see the model description above and the Appendix Section
4 EXPERIMENTS
Implementation Details. We use PyTorch (Paszke et al. 2017) to conduct all our experiments. For our pretrained language models, we use the HuggingFace (2019) implementation of the Base- BERT model (1 (1M parameters) and use the GPT model of similar size (117M parameters). Due to computational constraints, we do not use BERT-Large or larger GPT models. We use BERT's lower-case tokenizer and GPT's tokenizer with an added classification token. Both models take fixed input size of 256 tokens, contain 12 transformer blocks, and have a hidden size of 768 neurons. We apply dropout (Srivastava et al. 2014) (p .1) before the final linear layer.
Unless otherwise noted we use learning rate of -5 use a scheduled ADAM optimizer (Kingma & Ba 2015), train for 3 epochs over each training dataset, use training mini-batch size of 32 documents, and set all other hyper parameters to their default values. While the number segments does not need to be fixed, for of processing have a relatively constrained length and allows for minimal padding to be used. We also use document typically contains 6k words, many of
set m 8 for the patent datasets as patents
16 segments for the Arxiv dataset which are removed as non-meaningful or are intentionally truncated as part of the bibliography.
Next, describe the different datasets in our experiments.
5

---

Under review as a conference paper at ICLR 2020
Dataset
inverted wireless
Arxiv-4
Arxiv-11
section
subclass
wireless
Method
Li et al.
2018)
<43
Deep Patent
Lee & Hsiang 2019)
80.98
66.80
PatentBert
He et al. (019))
94.18
80.47
Local Word Glimpses
BPse-GPT
96.59
84.62
83.32
67.29
89.82
87.69
Base-BERT
97.06
87.42
83.85
68.31
90.21
87.72
Cat-GPT
96.82
80.03
83.43
66.17
89.34
88.80
Cat-BERT
97.06
87.34
83,99
68.34
90.64
89.39
RNN-GPT
96.98
85.31
83.52
67.72
90.16
89.19
RNN-BERT
97.62
87.72
83.99
68.72
90.51
89.41
ATT-GPT
97.62
85.94
83.66
68.13
90.31
90.08
ATT-BERT
97.70
87.96
84.13
69.01
90.69
90.25
Table 1: Micro Fl results on our datasets.
Patents. Patents can be broken down into multiple levels of resolution according to the Interna- tional Patent Classification System (IPC): Section, Class, Subclass, ... elc. The most broad category, Section, has eight labels (A-H). For instance, Section A is concerned with Human Necessities, while Section H is concerned with Electricity. We also perform classification experiments more de- tailed level of categorization: Subclass, which contains 638 labels. Patents were gathered from the Google Patents Public Dataset via SQL queries.
We gathered all documents from the United States Patent Office (USPTO) from 2006-200 2014-004-200 as our training and all patents from 2015 our set. We have 1,917,334 training and 296,724 testing documents, where 15, 172 and 1,835 documents were respectively skipped for missing ab- stracts. The text of patent composed of different parts: title, abstract, and list of claims, For our purposes, we consider one patent to be first the title, then the abstract, followed by each claim in order- claim 1, claim 2, ... until the last claim.
(Inverted) Wireless Patents. We selected subset of patent data perform additional experi- ments. We chose the wireless (H04) Class due to its large number of training and test examples (the second most of all the Class data). The Class with the most examples (computing) was not chosen due to its large imbalance in Subclasses (dominated by one label with over 75% examples). We use the wireless Class construct an inverted patent dataset, where a single patent starts with its last claim, up to the first claim in the reverse order, then the abstract, and lastly the title. It is commonly believed that the abstract and the first claim is the most useful in classifying a patent. We create this dataset to present information in reverse order of relevance to test models that bias towards the beginning documents (e.g., models that truncate beyond a fixed number of tokens). After processing, the wireless Class contains 250,982 training and 42,892 testing documents, where 15,172 and 97 documents were skipped, respectively, for missing abstracts.
Arxiv papers. We study the long document dataset provided by He et al. (011)). consists of 33,388 papers, from 11 different categories, downloaded from the scientific article hosting website Arxiv. The least occurring category is "math.AC" with 2885 documents, and the most occurring "cs.DS" with 4136 documents. We call this dataset Arxiv-11. They also provide subset of the data using four categories and 12,195 documents, which we refer to as Arxiv-4. All downloaded pdf documents converted txt files, with document less than 1,000 words. We randomly sample 90% for training and use the remaining 10% for test.
5 RESULTS
Patents and Arxiv Datasets. We report our main results in Table [ where the numbers micro- Fl scores. We make three observations. First, we compare our methods to the previous work using DeepPatent (Li al. 2018), PatentBert (Lee & Hsiang 2019) and Local Word Glimpses (He

---

Under review as a conference paper at ICLR 2020
Param
3 epochs
3 epochs
3 epochs 4e-5 Ir
3 epochs 5e-5 Ir
8 epochs 2e-5 Ir
8 epochs 5e-5
Method
2e-5
3e-5 Ir
Base GPT
96.19
96.59
96.43
96.19
96.27
96.35
Base BERT
96.90
97.06
96.43
96.27
96.58
97.06
Cat-Grt GPT
93.88
94.28
95.63
95.31
96.74
96.82
Cat-BERT
96.82
96.98
96.27
96.66
96.90
97.06
RNN-GPT
96.03
96.82
96.59
96.98
96.90
96.74
RNN-BERT 97.62
96.74
97.06
96.74
96.51
96.90
ATT-GPT
95.79
97.38
97.62
97.62
96.43
97.14
ATT-BERT 97.06
97.70
96.27
95.87
96.98
97.14
Table The effect of changing the learning rate, and epoch, hyper- syper--arameters for all models on the Arxiv-4 dataset.
et al. 2019). Table T shows the Base LMs perform well. We note Base BERT implementation outperforms PatentBert as they do not combine the title, abstract, and claims- but only the title and abstract or just the first claim. Second, we compare Base LMs against the three enhancements across variants of the patent dataset and variants of the Arxiv dataset. The RNN-based and attention-baal based LMs show consistent improvement over base LMs, while the simple concatenation- based LM is not consistent. Furthermore, the usage of attention is superior in all cases (highlighted the last line). Lastly, is also interesting to note that BERT-based models consistently outperform GPT-based models. This is likely due to the use of bidirectional contexts in BERT.
Inverted Patents. As shown in recent work (Lee & Hsiang 2019 Li et al. 2018). only the abstract or first claim patent is needed for good classification performance. To analyze the effect of at- tention discovering the location discriminative content, we invert the structure of the patents in our wireless dataset and train new models. Comparing the results from the second last column (wireless) and last column (inverted wireless) in Table [ we see that Base LMs, which only take into account of first 256 tokens, suffer from drop of F1 scores of more than 2.0. Cat-LMs and RNN-LMs reduce the gap in F1 scores to about 1.0, while the attention based models perform the best, with the gap between wireless and inverted wireless reducing smaller than 0.5.
Exploration of Hyper-parameters. Next, we investigate the effects of different hyper-parameters on the various models using small set of training data. For this analysis, While the attention-based models seem to do well, demonsttesssss the unpredictable nature of using different learning rates along with different training epochs. This behavior aligns with the claims of Devlin et al. (2018), who also found fine-tuning on small datasets sometimes lead to unstable results.
the Arxiv-4 dataset.
Training and Evaluation Time. Lastly, we compare the training time and evaluation speed of our models on the Arxiv-11 dataset. As shown Table all of the enhanced variants of the LMs require nearly 3-4x training time and over 2x to evaluate. Considering there are substantially more operations required to process the full input text via multiple forward passes, this slow down is better than expected. The difference in training time between the attention-basee models and non- inteentionbbsed is surprising, given the fact that the LM's parameters must be updated with more than one set of gradients. This points towards further gradient computations being feasible for architectures and hardware that can handle the additional required memory.
Base LM
Cat-LM
RNN-LM
ATT-LM
BERT Training time
1.000
2.711
3.079
3.610
BERT Evaluation time
1.000
2.712
2.940
3.147
GPT Training time
1.000
3.088
3,551
3.558
GPT Evaluation time
1.000
2.520
2.689
2.839
Table 3: An analysis of the different model run times factor of the baseline method.

---

Under review as a conference paper at ICLR 2020
Ablated
Base BERT
Cat-BERT
RNN-BERT
ATT-BERT
ATT-BERT
21 gradients
11.92
12.96
11.92
81.05
81.11
No 21
gradients 11.92
46.72
70.59
81.08
81.11
Table 4: BERT model results when shuffling every in the Arxiv-1 dataset.
5.1 ABLATION EXPERIMENT
In order to understand the effects of the attention, we introduce an ablated model that allows to investigate how attention influences and guides backpropagation. The model we use is similar to ATT-BERT, but we remove (M())) h2, and hm from the classifier. Therefore it can be written
= Cw.o(a @ a + LM(IM(()))
with W E (Rxx(++++d)) b R& being the parameters of the linear classifier. This model performs well, but not better than ATT-BERT. The full results of this model can seen in table S in the appendix. But here, use this model to carry out two additional experiments.
Shuffling Experiment. First, we perform a shuffled input experiment to examine the effect of using attention guide language model parameter updates. We all the same setup the exper- iments for the Arxiv-11 column in Table T except for each training iteration, we randomly permute each segment 21 between all mini-batch examples. This means methods that use xı will longer be able to rely upon those gradients to give an informative update to the model's parameters. Table shows how the aon-attention-basee methods guess the maximum occurring class when gradients are updated using Z1. The table also shows how these methods perform poorly without gradients to update the language model parameters. However, the crention-based methods are able to perform well despite the loss of information from shuffling, and ATT-BERT is relatively unaffected by the loss of gradients from 21.
ATT-BLRT on wireless 1-BLH wireless inverted
ATT-BERT
07
94
BUT-BURT on inverted wrrleess
Ablozed ATT-SENT on inverted wirelets
Alpha Value
Alpha Value
2:5
0.5
011
Segment Index
Segment Index
(a)
(b)
Figure 3: (a) An analysis of the effect attention values between the original wireless dataset, and the inverted wireless dataset. (b) A comparison of a values with the ablated model.
Attention a Comparisons. Second, we measure and compare the a values on the wireless and inverted wireless patent dataset. We average the a value for each of the eight input segments over the entire test set. Figure 3 (a) comparatively shows the a values our attention-based model on the wireless and inverted wireless dataset; this clearly demonstrates how the attention mechanism is able to accurately pick up on the important sections text. Figure b)) shows a comparison between ATT-BERT and the ablated model on both the wireless and inverted wireless datasets. The values for both models on the inverted dataset seem relatively similar, with the ablated model placing heavier emphasis the first few segments. But for the wireless dataset, the effect of including the
8

---

Under review as a conference paper at ICLR 2020
gradients of 2 can been seen on, where the dj value is over 0.7 on the ablated model. This means enabling backpropagation to occur for input #1 has a positive effect for ATT-BERT and, moreover, that the title and abstract in patent are of high discriminative importance.
CONCLUSIONS
In this work, achieved state-of-the-an results multiple long document classification tasks by utilizing pretrained language models with attention- baated enhancements. With language modeling continuing
improvements every month, showed how different models be utilized with our method. We performed numerous experiments to clearly demonstrate the value added by using attention to learn from important segments. We showed that the additional gradient computation as result of attention is marginal when compared to the consistent improvement of results. We ana- lyzed the effects of the attention mechanism through the loss of input information via both shuffling experiment and carefully constructed dataset augmentation.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Christopher F Chabris and Daniel Simons. The invisible gorilla: And other ways our intuitions deceive us. Harmony, 2011.
Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, and Daniel S. Weld. Pretrained language
models for sequential sentence classification. CoRR, abs/1909.04054 2019. URL https:// 04054. pdf
axxiv.org/paf/220
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in Neural Infor- mation Processing Systems, pp. 3079-3087, 2015.
Zihang Dai, Zhilin Yang, Yiming Yang. William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-XL. Attentive language models beyond fixed-length context. arXiv preprint arXiv:I901 020860, 2019.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint wrXiv:1810004805 2018.
Jun He, Liqun Wang, Liu Liu, Jíao Feng, and Hao Wu. Long document classification from local word glimpses via recurrent attention learning. IEEE Access, 7:40707-40718, 2019. URL https:
//doi. 10. 1ww.11111199////// ACCESS
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
HuggingFace.
Pytorch-transformers.
Bytorch-transformeer 2019.
Nan Rosemary Ke, Anirudh Goyal GOYAL. Olexa Bilaniuk, Jonathan Binas, Michael C Mozer, Chris Pal, and Yoshua Bengio. Sparse attentive backtracking: Temporal credit assignment through reminding. In Advances in Neural Information Processing Systems, pp. 7640-7651, 2018
Yoon Kim. Convolutional neural networks for sentence classification.
arXiv preprint
arXiv: arXiv:1408.555 188..588, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations, ICLR, 2015.
Vandana Korde and C Namrata Mahender. Text classification and elassifiers: survey, International
Journal of Artificial Intelligence & Applicatioos,, 3(2):85, 2012.
Alexandre B Laudet, Keith Morgen, and William L White. The role of social supports, spirituality, religiousness, life meaning and affiliation with 12-step fellowships in quality of life satisfaction among individuals in recovery from alcohol and drug problems. Alcoholism trealment quarterly, 24(1-2):33-73, 2006.
Jieh-Sheng Lee and Jieh Hsiang. PatentBERT: Patent classification with fine-tuning a pre-trained BERT model. CoRR, abs/1906.02124, 2019. URL httt:://axxiv.orr/abbb/// 02124
Shaobo Li, Jie Hu, Yuxin Cui, and Jianjun Hu. DeepPatent: patent classification with convolutional neural networks and word embedding. Scientometrics, 117(2):211-744, 2018. doi: 10.1007/
111192001--205-55
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Asier Mujika, Florian Meier, and Angelika Steger. Approximating real-time recurrent learning with random kronecker factors. In Advances in Neural Information Processing Systems, pp. 6594 6603, 2018.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob- lem. CoRR, abs/1211.500 2, 2012.
10

---

Under review as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPN-WWW W, 2017.
J.B. Peterson. Maps of Meaning: The Architecture of Belief. Routledge, 1999. ISBN
9780415922227.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training- URL https://3-uu-wwstt- amazonaws. com/openai- understanding paper. pdf. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan-
guage models are unsupervised multitask learners. 2019. /BLL:tttss///4mmtpkeeee/ c oege://eeeeeeetteeaaaaaagneneeee
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Richard Socher, Jeffrey Pennington, Eric H Huang. Andrew Y Ng. and Christopher D Manning. Semi-supervissed recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 151-161, Association for Computational Linguistics, 2011.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 152991192911958 2014.
Amy JC Trappey, Fu-Chiang Hsu, Charles V Trappey, and Chia-I Lin. Development of a patent document classification and search platform using back pook-propagation network. Expert Systems with Applications, 31(4):755-765, 2006.
Vahe Tshitoyan, John Dagdelen, Leigh Weston, Alexander Dunn, Ziqin Rong, Olga Kononova, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571(7763)::5,, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR, 2019.
Ronald Williams and Jing Peng. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural computation, 2(4):490-501, 1990.
WIPO. World intellectual property indicators. Geneva: World Intellectual Property Organization, 2018.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08233 0823770660887....... 2019.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. Hierarchical attention networks for document classification. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016.

---

Under review as a conference paper at ICLR 2020
Dataset
inverted wireless
Arxiv-4
Arxiv-11 11
section
subclass
wireless
Method
Ablated
97.38
87.22
84.01
68.87
90.31
89.98
ATT-BERT
ATT-BERT
97.70
87.96
84.13
69.01
90.69
90.25
Table 5: Micro Fl results of our ablated BERT model compared the ATT-BERT.
A GRADIENT COMPUTATION FOR LANGUAGE MODEL PARAMETERS
Let us denote our language model representation be = LM(x;) = fo(r;), where 0 are the language model parameters. Let he be the function we compute top of the LM representations. e.g., the classifier, LSTM, etc. The parameter e can contain the classification weights W and the LSTM weights. The models considered this paper can be written as:
p ha ( ........................ (Fm))
Coupled with the loss function I (log loss) and the target label y. we have
((: y) = 1 (Jo (x1), fo fo(xm)),
Computing the gradient over the LM parameters 0, by chain rule we have
① (p. v) 1(2, v) la-ho(-.)
i Ou h fo( ................................ fo(a ))
θ
00 fo(ar)
00
0z
θ S0 fo( + Du
à u)) fo (xm))
ho (fo(z)), u. ... fo(mm) 00 ru
ho
00
By stopping the gradient computation over 22,...,, Tm. we are dropping the terms related to 8
20 (xx)). > 2, from the above formula. In optimization, we say q a descent direction
(q, 0gl) < 0. The negative gradient -0gl is clearly a descent direction. The gradient above in Equation g is in the form ög! Σx= 9i. where Ji are the gradient term of the ith segment in the equation. By dropping terms in the backpropagation, we are assuming either the contribution from gi (or (M(()) in the attention model of Equation (5) dominates the contributions from other segments, or the gradients 9i from different segments point towards similar directions, so that the truncated gradient still a descent direction.
In some cases, if the storage of partial gradients Du h2 u.... Zm) is feasible, it is possible to compute the full gradient in two passes over the mini- batch. For example, in the basic Cat-BERT model for classification, this partial gradient is W, scaled by the derivative of softmax, where is the index of the segment. We can compute these W, with the derivative of the softmax as scaling factors in a first pass, and accumulate gradients over fo (xx) scaled by these factors in second pass over the mini-batch.
12


Reviews:
[{"review_id": "SkxR9kKJ5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to compare different methods to build BERT/GPT  representations of long documents, to bypass the limitation of the input size of these models. One of the proposed method uses attention mechanism to discover the most significant portion of the text which are use to backpropagate the error on the language model. Three combination methods (concatenation, RNN and attention)  are tested on 2 databases plus one modified version of one of the databases to show the impact of the presentation bias in the texts (most important part are at the beginning). \nResults show that the largest improvement is the base BERT model over the previously proposed model : this aspect should be comment : what is the reason of the improvement ? \nCombination of textual part also yields improvement, but to a smaller extend. Hyper-parameter and Training/Testing time are reported, which is useful from a practical point of view if one should decide to implement the proposed method or not, considering the extra computational load and the relatively small improvement. The Shuffling experiment demonstrate an interesting behaviour of the models, that should be confirmed on a real dataset.\n\n ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkxafCjfqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an attention-based document classifier based on BERT-LSTM structure. Experiments on patent and ArXiv datasets show the structure is slightly better than three baselines. This paper also analyzed the effection of gradient update settings in different models. \n\nThis proposed model is lack of novelty, it applied the attention sum of LSTM output on top of BERT in the document classification task. Although this model concatenated more vectors (original LSTM output, argmax, attention weight) into the final prediction vector, the reason of integration those vectors is not explained and no experiment prove the effectiveness of this setting. The experiments are only evaluated on two less studied datasets which makes the comparison with other works less persuasive, more results on general document classification datasets are needed.\n\nQuestions: \n1. Why the RNN-BERT and the ATT-BERT only include h2~hm but exclude h1? \n2. As the segment is fixed in each model, why not include all the z1~zm but only include z1?\n3. “Recurrent Neural Networks (RNNs) have been used for short text, e.g. sentiment analysis by Socher et al. (2011) “, the cited paper is a recursive neural network. It is not a recurrent neural network.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxSKurAYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a methods to combine multiple pre-trained language models using an attention mechanism in order to take the whole document into account. The effectiveness of the methods is evaluated on two long document classification tasks (Arxiv publication and patent classification) with state-of the art results.\n\nPros:\n\n- the effectiveness of the methods is experimentally demonstrated using two relevant datasets\n- the inverted wireless experiment clearly shows the interest of the attention combination strategy \n\nCons:\n\n- the methods is very simple (combining multiple segment prediction to perform document classifications) making the contribution of this paper quite weak. In the own terms of the authors, the \"main contribution [...] is to investigate the effectiveness of different combination strategies\". I am not sure that it is sufficient for the ICLR standards.\n- the paper is difficult to read and should be proofread to improve readability\n\nMinor issues:\n\n- Socher et al (2011) uses recursive neural networks (ReNN) and not recurrent neural networks (RNN). RNN are ReNN but restrained to a linear chain structure\n- In Introduction: ... in the  domain extremely complex data that is language ... -> I'm not sure the sentence is correct\n- In Introduction: the last sentence should be shorten and rephrased.\n- many typos ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates ways of using pretrained transformer models like BERT for classification tasks on documents that are longer than a standard transformer can feasibly encode. 

This seems like a reasonable research goal, and none of the reviewers raised any concerns that seriously questioned the claims of the paper. However, neither of the more confident reviewers were convinced by the experiments in the paper (even after some private discussion) that the methods presented here represent a useful contribution. 

This is not an area that I (the area chair) know well, but it seems as though there aren't any easy fixes to suggest: Additional discussion of the choice of evaluation data (or new data), further ablations, and general refinement of the writing could help.
Decision: Reject