ADAPTING PRETRAINED LANGUAGE MODELS FOR LONG DOCUMENT CLASSIFICATION

Abstract: Pretrained language models (LMs) have shown excellent results in achieving human like performance on many language tasks. However, the most powerful LMs have one significant drawback: a fixed-sized input. With this constraint, these LMs are unable to utilize the full input of long documents. In this paper, we introduce a new framework to handle documents of arbitrary lengths. We investigate the addition of a recurrent mechanism to extend the input size and utilizing attention to identify the most discriminating segment of the input. We perform extensive validating experiments on patent and Arxiv datasets, both of which have long text. We demonstrate our method significantly outperforms state-of-the-art results reported in recent literature.

Reviews and author responses:

Thank you for taking the time to review our paper. We address the issues below, and have updated the manuscript accordingly.

Issue:  simple methods, weak contributions

While it is the case our primary model (ATT-LM) combines existing techniques, the intention with our method is to have a model that is ready to use, simple, and effective. Long document classification is an area of practical importance, and we hope to encourage further research in this area with our work. 
Our method, though simple, has shown consistent improvement demonstrated by our extensive experiments. Nevertheless, we have updated the language in our manuscript to more accurately reflect our other two non-trivial contributions (in addition to combination strategies): 

1. The analysis of our methods via timing, hyperparameter search, multiple dataset augmentations, and ablation experiments.
2. The utilization of attention to perform selective LM parameter updates.


Issue: readability

We have revised the manuscript to address the issues with readability. We believe the updated manuscript is more comprehensible and has significantly fewer typos.


Thank you for taking the time to review our paper. We clarify some points below, and we have added further discussion to the paper.

Q: Why does the BERT-base model show the most improvement over the previously proposed models?

Our BERT-base improves over the PatentBert implementation because PatentBert does not combine the title, abstract, and claim-- but only the title+abstract or just the first claim. This was done to prevent truncation, but these combinations don't always reach the max input size. Additionally, our BERT-base improves over the Deep Patent methods and Local Word Glimpses as they use word level embeddings instead of language model based embeddings. 


Comment: Shuffling should be confirmed on a real dataset

While we would like to confirm the results of our shuffling experiment on other datasets, we were unable to find any sufficiently long document classification task that would be illuminative. For future work, we would like to find more datasets for this task.


Thank you for taking the time to review our paper. We clarify some points below, and we will add further discussion to the paper.
Issue: This proposed model is lack of novelty

Our intention with our method is to have a ready to use, simple, and effective model. While other methods have been suggested to incorporate LSTMs with Transformer-based LMs, they are primarily focused on the pre-training step and have not been applied to long document classification.  This area of long document classification is a practical problem, yet has received relatively little attention by the research community when compared to other NLP tasks. 

Issue: the reason of integration those vectors is not explained and no experiment prove the effectiveness of this setting

The reason of integration for our ATT-LM method is to iteratively expand upon the other combination strategies. We have updated Figure 2 to reflect this more accurately. We have also run additional experiments with the ablated model to provide justification for the vector integration of ATT-LM. The ablated model performs relatively well, but not better than ATT-BERT.


Issue: evaluated on two less studied datasets 

Other datasets were purposefully ignored for our setting as they do not illuminate the issues present with Transformer-based LMs, as the input typically does not need to be truncated when used with larger LMs [1].  The idea of co-opting traditional datasets for long document classification has been suggested, E.G. separating the examples in a traditional dataset into many shorter segments much smaller than the maximum input size [2]. But it is not clear how this can be done without hampering the power of a Transformer-based LM. We hope that in future work our chosen datasets will be studied more actively, and that new long document datasets will be created.


Q: Why do our models only include h2~hm but exclude h1?

In our initial development, we found that including h1, instead of h2, produced worse results. We believe this is from the forward direction of h1 not providing any contextual information. Comparatively, the backwards direction of h2 provides enough context that when concatenated with z1 achieve the best performance.

Q: "As the segment is fixed in each model, why not include all the z1~zm but only include z1?"

Even though our specific implementation uses a fixed segment size, we designed our model to be more general during training and inference. Despite being able to include z1~zm for the CAT-LM models on our datasets, the size of W (the linear classifier's parameters) scales directly with m. If there were a sufficiently long document, the number of parameters could be unreasonably large. More importantly, by directly using the representation for each z, the linear classifier overfits as demonstrated by our results. Lastly, we include z1 as that text sequence contains the most discriminative features and by including it there is a direct path for gradient updates to the LM.

[1] Radford, Alec, et al. "Language models are unsupervised multitask learners." OpenAI Blog 1.8 (2019).
[2] https://openreview.net/forum?id=SklnVAEFDB

This paper proposed an attention-based document classifier based on BERT-LSTM structure. Experiments on patent and ArXiv datasets show the structure is slightly better than three baselines. This paper also analyzed the effection of gradient update settings in different models. 

This proposed model is lack of novelty, it applied the attention sum of LSTM output on top of BERT in the document classification task. Although this model concatenated more vectors (original LSTM output, argmax, attention weight) into the final prediction vector, the reason of integration those vectors is not explained and no experiment prove the effectiveness of this setting. The experiments are only evaluated on two less studied datasets which makes the comparison with other works less persuasive, more results on general document classification datasets are needed.

Questions: 
1. Why the RNN-BERT and the ATT-BERT only include h2~hm but exclude h1? 
2. As the segment is fixed in each model, why not include all the z1~zm but only include z1?
3. “Recurrent Neural Networks (RNNs) have been used for short text, e.g. sentiment analysis by Socher et al. (2011) “, the cited paper is a recursive neural network. It is not a recurrent neural network.

The paper presents a methods to combine multiple pre-trained language models using an attention mechanism in order to take the whole document into account. The effectiveness of the methods is evaluated on two long document classification tasks (Arxiv publication and patent classification) with state-of the art results.

Pros:

- the effectiveness of the methods is experimentally demonstrated using two relevant datasets
- the inverted wireless experiment clearly shows the interest of the attention combination strategy 

Cons:

- the methods is very simple (combining multiple segment prediction to perform document classifications) making the contribution of this paper quite weak. In the own terms of the authors, the "main contribution [...] is to investigate the effectiveness of different combination strategies". I am not sure that it is sufficient for the ICLR standards.
- the paper is difficult to read and should be proofread to improve readability

Minor issues:

- Socher et al (2011) uses recursive neural networks (ReNN) and not recurrent neural networks (RNN). RNN are ReNN but restrained to a linear chain structure
- In Introduction: ... in the  domain extremely complex data that is language ... -> I'm not sure the sentence is correct
- In Introduction: the last sentence should be shorten and rephrased.
- many typos 

This paper proposes to compare different methods to build BERT/GPT  representations of long documents, to bypass the limitation of the input size of these models. One of the proposed method uses attention mechanism to discover the most significant portion of the text which are use to backpropagate the error on the language model. Three combination methods (concatenation, RNN and attention)  are tested on 2 databases plus one modified version of one of the databases to show the impact of the presentation bias in the texts (most important part are at the beginning). 
Results show that the largest improvement is the base BERT model over the previously proposed model : this aspect should be comment : what is the reason of the improvement ? 
Combination of textual part also yields improvement, but to a smaller extend. Hyper-parameter and Training/Testing time are reported, which is useful from a practical point of view if one should decide to implement the proposed method or not, considering the extra computational load and the relatively small improvement. The Shuffling experiment demonstrate an interesting behaviour of the models, that should be confirmed on a real dataset.

 

Meta-review: This paper investigates ways of using pretrained transformer models like BERT for classification tasks on documents that are longer than a standard transformer can feasibly encode. 

This seems like a reasonable research goal, and none of the reviewers raised any concerns that seriously questioned the claims of the paper. However, neither of the more confident reviewers were convinced by the experiments in the paper (even after some private discussion) that the methods presented here represent a useful contribution. 

This is not an area that I (the area chair) know well, but it seems as though there aren't any easy fixes to suggest: Additional discussion of the choice of evaluation data (or new data), further ablations, and general refinement of the writing could help.

Decision: reject