# Mining the Frequent Patterns of Named Entities for Long Document Classification
## 2 Related Work
### Deep Learning-Based Methods  
Deep learning-based methods employ the neural-network structure, which can generate low-dimensional vectors to represent text and capture better semantic relationships between words than the statistical methods [39]. Deep learning-based methods usually use the distributed representations of words, sentences, and documents, such as Word2 Vec [40] and Doc2 Vec [41]; and some studies [42; 43] fed the word representations obtained by these two methods into the classifier to accomplish the text classification task. The early deep learning methods for text classification used convolutional neural networks (CNN). [PERSON] [44] proposed a convolutional neural network model called TextCNN to analyze movie reviews [45], Stanford Sentiment Treebank [46], and customer reviews [47]. TextCNN produces good results in short text classification. Other experiments [48] show that when the text length exceeds 2000, the performance of TextCNN is not significantly improved compared with the traditional methods. TextRNN [49] is a well-known text classification model; however, it is not effective at handling long documents because a long text leads to a vanishing gradient. The author also proposed a recurrent neural network (RNN) for text classification to capture contextual information. Based on long short term memory (LSTM) [50], [PERSON] et al. [51] proposed an end-to-end model to classify the entity relationships in long texts. RNN and LSTM both have no restriction on text length, but they have difficulty in learning long-range correlations of sequences, and their training is time consuming. The current best systems for text classification leverage transformer-based pretrained language models [52], such as BERT [53], as a critical encoding component to achieve state-of-the-art performance. Unfortunately, transformer-based models are incapable of processing long documents due to their quadratically increasing memory and time consumption. Consequently, most transformer-based models have a length limit of 512, and recent works [54; 55; 56; 57; 58; 59; 60] have been devoted to alleviating this constraint. They provided several solutions for this problem:  
* The truncation method [54; 56]: These solutions only use some (head or tail) of the tokens from the text instance, which will lose information but reduce the computational cost.
* The hierarchical structure method [57; 58; 59]: A long document will be split into multiple parts to fit the length limit, and the representations of the parts are sent into neural networks to construct final representations. A typical model, the HAN [59] proposed by [PERSON] et al., includes two levels of attention mechanisms: word level and sentence level, which allow the model to pay more or less attention to individual words or sentences in constructing the representation of a document.
* Key sentence selection [60]: These approaches aim to select the core parts of the original text instance in an intelligent way to preserve the most relevant elements.  
However, all those solutions have their own drawbacks: the truncation methods lose key information from the original text; the hierarchical structure methods inevitably fail to capture long-range dependencies of text for representation learning; the key sentence selection models are complex, and their computational cost is substantially higher than that of the truncation method. Furthermore, all of these methods suffer from poor interpretability.