ADAPTING PRETRAINED LANGUAGE MODELS FOR LONG DOCUMENT CLASSIFICATION

Pretrained Language Models ( LMs ) have shown excellent results in achieving human-like performance on many language tasks . However , the most powerful LMs have one significant drawback : a fixed-sized input . Due to this constraint , these LMs are unable to utilize the full input of long documents . In this paper , we introduce a new framework to handle documents of arbitrary lengths . We investigate the addition of different mechanisms to handle an extended input size , and we demonstrate how attention can be used for parameter updates based on the most discriminating segments of input . We perform extensive validating experiments on patent and Arxiv datasets , both of which have long text . We show our method consistently outperforms state-of-the-art results reported in recent literature . 1 INTRODUCTION . Neural Network based Language Models ( LMs ) have seen a flurry of work , where new design and implementation improvements have advanced state-of-the-art performance in a variety of natural language tasks over the past few years ( Devlin et al. , 2018 ; Dai et al. , 2019 ; Radford et al. , 2019 ; Yang et al. , 2019 ; Liu et al. , 2019 ) . LMs are powerful tools because they process a collection of unlabeled text and learn a rich embedding of natural language without supervision . This representation can be re-purposed on subsequent tasks such as classification and sentiment analysis ( Korde & Mahender , 2012 ) . This technique is essential for reaching state-of-the-art performance , as LMbased systems are able to achieve much better results than techniques that only use a small , labeled dataset . Modern LMs achieve thier success by utilizing a powerful mechanism called “ The Transformer ” ( Vaswani et al. , 2017 ) . The transformer learns strong dependencies between its inputs and has the ability to be stacked as many times as hardware can handle . This mechanism allows LMs to take in relatively large , yet still fixed , sized input . For the largest LMs , the input size can reach up to four thousand tokens ; however , this is still a limitation as they can not process arbitrarily long documents . On many natural language tasks , this fixed input size is sufficient . For example , reading comprehension tasks are often used to analyze the quality of LMs , contain relatively few words on average , and do not have sufficiently long dependencies ( Wang et al. , 2019 ) . Furthermore , there are multiple tasks where the input data is too long and must be truncated before being processed ( Lee & Hsiang , 2019 ) . Truncation is unsuitable because long complex text often contains inter-referential pieces of information . For instance , reading the final chapter of a book after all the previous ones takes on a different meaning compared to reading the same text by itself . Solving the problem of arbitrarily long input requires more than a cursory glance . A first intuition may be to take a pretrained LM , separate the text into segments , place a Recursive Neural Network ( RNN ) after the embedding , and simply pass in the segments sequentially . This seems reasonable as RNNs have been used in the past for sequential text-based tasks , e.g . sentiment analysis ( Socher et al. , 2011 ) . Unfortunately , using RNNs in this way causes two problems . First , RNNs are typically trained via backpropagation through time , making them prone to the problem of vanishing or exploding gradients ( Pascanu et al. , 2012 ) . While many techniques exist to deal with this issue ( Williams & Peng , 1990 ; Mujika et al. , 2018 ) , they do not solve the secondary problem of significant memory requirements for LM parameter updates . Each forward pass through the LM produces a set of parameter gradients . With multiple forward passes , the number of stored gradients quickly grows as a segmented document increases in length . To further complicate the issue , the number of parameters in transformer-based networks is quadratic in relation to the fixed input size . In order to solve these problems , we look at the brain ’ s ability to guide attention and provide behavioral updates as a biological inspiration . The brain processes an extraordinary amount of data , yet from moment to moment , much of that information is filtered out . What becomes filtered is not arbitrary , but is directly influenced by one ’ s objective . This filtering can be observed when an individual is tasked with counting the number of basketball passes in a video , yet fails to register a large gorilla that appears in center frame ( Chabris & Simons , 2011 ) . Even outside the moment to moment , changes in an individual ’ s behavior is guided by the structure of their values , self-selected or otherwise . An individual ’ s value structure imposes a framework for determining significance of events ( Peterson , 1999 ) . An event that may have been insignificant in the past can go on to take a new meaning once a new value has been gained or once an old one has changed ( Laudet et al. , 2006 ) . This increase in valence may cause a behavioral change , a reorientation of goals , or a shift in the interpretation of experiences . Unfortunately , when it comes to modeling this selective learning in Artificial Intelligence systems , common gradient-based methods fall short . A typical neural network model will update every parameter based on all of the inputs to minimize an objective function . While this is desired for many applications , full input based learning can cause issues for others ( Pascanu et al. , 2012 ) . Ke et al . ( 2018 ) attempt to solve this issue via selective attentive backtracking , though they focus on remembering long-term dependencies rather than utilizing the full context of the input . In this work , we introduce a method that performs objective-based filtering during learning , but still utilizes the entire input . We focus on two domains , specifically for classification : patents and scientific papers . Patent reading is a typical activity for lawyers trying to find relevant documents . Since 2003 , the number of patents filed has increased nearly every year ( WIPO , 2018 ) ; and using an automated system to perform classification is a continuously growing area of interest ( Trappey et al. , 2006 ) . Scientific papers are another significant area of investigation . Many papers are uploaded to the internet every day , and their automatic categorization is becoming a necessity . A statistics paper , for example , may not be categorized as machine learning by its authors , but could be of interest to the machine learning community . Additionally , Tshitoyan et al . ( 2019 ) show material science concepts can be learned from scientific papers by a language model . They demonstrated materials for functional applications could be recommended several years before their initial discovery . In this work , we use an attention mechanism to discover the significant portions of text to be used in updating a pretrained LM . We find attention-based updates to be essential because the most significant portion of an input sequence may occur anywhere throughout the document . While the datasets we study often start with highly discriminative features ( titles and abstracts ) , we conduct experiments to show our attention mechanism can find the important parts of text even when it does not occur in the first segment . In either case , updating a LM ’ s parameters from only the first input segment performs well– often better than just using a baseline of the original LM with the input truncated to fit the max size . However , we find using an attention mechanism consistently improves performance and achieves the best results in our experimented language models . Therefore , our contributions are as follows : 1 . We introduce a new framework for performing inference over arbitrary length documents . 2 . We perform extensive validating experiments of our methods , showing how our attentionbased framework consistently outperforms alternative methods . 3 . We demonstrate the attention mechanism can be utilized to perform selective language model parameter updates . 2 RELATED WORKS . Language model pretraining is a popular method for tackling many natural language understanding tasks . Bidirectional Encoder Representations from Transformers ( BERT ) ( Devlin et al. , 2018 ) and Generative Pre-Training ( GPT ) ( Radford et al. , 2018 ) are two well-known language model pretraining methods that we utilize in this work . BERT is trained by selectively conditioning on most of the input sequence , masking the rest , and attempting to predict the masked tokens . BERT is concurrently trained with a next sentence prediction task . The token used for the next sentence prediction task is reused for classification after BERT has been trained . GPT trains via next word prediction by conditioning on a sequence of text and trying to predict the next token . By training on Wikipedia and books , GPT is able to generate novel sequences of text . GPT is able to perform classification by appending a special classifying token at the end of an input , encoding the token into a latent representation , and using the representation as input to a new linear classifier set specifically for the task . These language model pretraining methods became state-of-art on natural language processing benchmarks such as GLUE ( Wang et al. , 2019 ) and SQUAD ( Rajpurkar et al. , 2016 ) , achieving close to human-level performance . As both BERT and GPT are based on transformers , their computational and storage costs scale quadratically with the input sequence length . This limits their application to mostly relatively short pieces of texts . To the best of our knowledge , we are not aware of any works applying these pretrained language models for long document classification . Next , we discuss a few other deep learning-based text classification approaches , with special attention to the classification of scientific papers and patents . Dai & Le ( 2015 ) consider pretraining recurrent neural networks with large corpus of texts , and show improved performance on several text classification tasks . Kim ( 2014 ) introduced convolutional neural networks ( CNN ) for text/sentence classification . Yang et al . ( 2016 ) introduced a hierarchical attention mechanism for document classification that attends to interesting sentences and words in a document . The length of documents considered in latter two works are relatively short , with the corpus consisting of mostly individual sentences or online reviews . For patent classification , Li et al . ( 2018 ) present a deep learning algorithm called Deep Patent , which is based on convolutional neural networks and continuous skip-gram embedding . They were the first to apply deep learning to large scale real-world patent classification . Lee & Hsiang ( 2019 ) used the pretrained BERT model to classify patents at the section and subclass level , only taking the title and abstract , or the first claim , as input . For scientific paper classification , He et al . ( 2019 ) introduce a relatively large scientific paper dataset and perform classification through a multi-network approach . They introduce a RNN Attentionbased reinforcement learning scheme . They select short text sequences to be parsed by a CNN . The representation learned by the CNN is sequentially fed back into the RNN for subsequent text selections and eventual classification . Very recently , Cohan et al . ( 2019 ) construct a joint sentence representation that allows BERT transformer layers to directly utilize contextual information from all words in every sentence . However , their task is classification at the sentence level for a single text input .

Summary: This paper proposes to compare different methods to build BERT/GPT  representations of long documents, to bypass the limitation of the input size of these models. One of the proposed method uses attention mechanism to discover the most significant portion of the text which are use to backpropagate the error on the language model. Three combination methods (concatenation, RNN and attention)  are tested on 2 databases plus one modified version of one of the databases to show the impact of the presentation bias in the texts (most important part are at the beginning).