Abstract: One of the principal tasks of machine learning with major applications is text classification. This paper focuses on the legal domain and, in particular, on the classification of lengthy legal documents. The main challenge that this study addresses is the limitation that current models impose on the length of the input text. In addition, the present paper shows that dividing the text into segments and later combining the resulting embeddings with a BiLSTM architecture to form a single document embedding can improve results. These advancements are achieved by utilising a simpler structure, rather than an increasingly complex one, which is often the case in NLP research. The dataset used in this paper is obtained from an online public database containing lengthy legal documents with highly domain-specific vocabulary and thus, the comparison of our results to the ones produced by models implemented on the commonly used datasets would be unjustified. This work provides the foundation for future work in document classification in the legal field.

Introduction: Text classification is a problem in library, information and computer science and one of the most classical and prominent tasks in Natural Language Processing (NLP). In particular, document classification is a procedure of assigning one or more labels to a document from a predetermined set of labels. Automatic document classification tasks can be divided into three categories: supervised, unsupervised and semi-supervised. This study focuses on supervised document classification. Research so far has focused on short text BIBREF0, BIBREF1, BIBREF2, BIBREF3, whereas the main objective of this paper is to address the classification of lengthy legal documents. In fact, pre-existing models could not be applied on our corpus, which consists of excessively lengthy legal documents. In the legal field, companies manage millions of documents per year, depending on the size of the company. Therefore, automatic categorisation of documents into different groups significantly enhances the efficiency of document management and decreases the time spent by legal experts analysing documents. Recently, several quite sophisticated frameworks have been proposed to address the document classification task. However, as proven by BIBREF3 regarding the document classification task, complex neural networks such as Bidirectional Encoder Representations from Transformers (BERT; BIBREF4) can be distilled and yet achieve similar performance scores. In addition, BIBREF5 shows that complex architectures are more sensitive to hyperparameter fluctuations and are susceptible to domains that consist of data with dissimilar characteristics. In this study, rather than employing an overly complex neural architecture, we focus on a relatively simpler neural structure that, in short, creates text embeddings using Doc2Vec BIBREF6 and then passes them through a Bi-directional LSTM (BiLSTM) with attention before making the final prediction. Furthermore, an important contribution of this paper to automatic document classification is the concept of dividing documents into chunks before processing. It is demonstrated that the segmentation of lengthy documents into smaller chunks of text allows the context of each document to be encapsulated in an improved way, leading to enhanced results. The intuition behind this idea was formed by investigating automatic audio segmentation research. Audio segmentation (also known as audio classification) is an essential pre-processing step in audio analysis that separates different types of sound (e.g. speech, music, silence etc.) and splits audio signals into chunks in order to further improve the comprehension of these signals BIBREF7. Analogously, the present paper shows that splitting overly lengthy legal documents into smaller parts before processing them, boosts the final results.

Question: How is the text segmented?
Answer: They simply split document in chunks, get embedding for each chunk and train BiLSTM models with embeddings.