SlowGuess/NAACL_2022_OCR huggingface dataset:
Qiqi Wang $^{1\dagger}$ , Kaiqi Zhao $^{1‡}$ , Robert Amor $^{1‡}$ , Benjamin Liu $^{2‡}$ , Ruofan Wang $^{1\dagger}$
<sup>1</sup>School of Computer Science, Faculty of Science
$^{2}$ Department of Commercial Law, Faculty of Business and Economics
University of Auckland, New Zealand
$\dagger$  {qwan857, rwan551}@aucklanduni.ac.nz,
$\ddagger$  {kaiqi.zhao,trebor,b.liu}@auckland.ac.nz
Legal document classification is an essential task in law intelligence to automate the labor-intensive law case filing process. Unlike traditional document classification problems, legal documents should be classified by reasons and facts instead of topics. We propose a Document-to-Graph Classifier (D2GCLF), which extracts facts as relations between key participants in the law case and represents a legal document with four relation graphs. Each graph is responsible for capturing different relations between the litigation participants. We further develop a graph attention network on top of the four relation graphs to classify the legal documents. Experiments on a real-world legal document dataset show that D2GCLF outperforms the state-of-the-art methods in terms of accuracy.
Legal Artificial Intelligence (LegalAI) (Zhong et al., 2020; Rissland et al., 2003) is a specific subject to apply the artificial intelligence technology into legal tasks including legal judgement prediction (Rosca et al., 2020; Gan et al., 2021), similar case matching (Tran et al., 2019) and law case classification (Noguti et al., 2020; Lin et al., 2012; Li et al., 2019a). In this paper, we focus on law case classification. The traditional case filing process requires experts to categorize the civil complaints manually, which is labor-intensive. For example, a court in a small town has more than 10,000 civil complaint cases per year in China. This calls for accurate machine learning techniques to improve the efficiency of the case filing process.
Recently, deep learning approaches (Wang et al., 2018a; Johnson and Zhang, 2017; Wang, 2018; Shen et al., 2018; Zhang et al., 2018; Lin et al., 2017) have significantly improved the accuracy of text classification on well-known datasets, e.g., Amazon reviews. Existing methods learn a
latent representations for each document by considering the semantics and themes of the documents. In this way, documents are classified into high-level topics such as "Sport" and "Medical". Unlike traditional datasets, legal documents of different types exhibit high semantic similarity. For example, Table 1 shows two semantically similar cases in different Chinese civil categories.
Given the nature of legal classification tasks, there are two limitations of applying existing text classification models. First, document structure is a key to accurate classification, but it is ignored in most existing methods. Legal documents in different categories may only differ in some particular parts. Some previous research proposed to utilize the graph with sentence relations to address the problem (Hu et al., 2019). But not all sentences are relevant to the document class. Second, the facts and reasons in the legal cases are essential for distinguishing different dispute types, but they are not considered in existing methods. Existing word co-occurrence graphs (Zhang and Zhang, 2020; Ragesh et al., 2021; Zhang et al., 2020) are not only hard to represent the key facts but also include words that are irrelevant to the document class.
We argue that the key to accurate legal document classification is to understand the facts, which are expressed as relations between entities. For example, the debtor-creditor relationship denotes the fact that someone borrows other entities' money. Motivated by this, we designed a Document-to-Graph Classifier (D2GCLF) to address the two limitations above. D2GCLF extracts four graphs from each legal document to represent the facts about the key entities, i.e., plaintiffs and defendants. The four graphs are (1) Entity-Matter graph to describe the matters (e.g., money) associated with the entities; (2) Entity-Action graph to describe the actions of entities; (3) Entity-Keyword graph to model the general topic of the facts about
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2208 - 2221
Table 1: Examples of translated legal documents of two different types in Chinese law. See Appendix A for the original Chinese documents. The class "Dispute Over Contract of Sale of Commercial Residential Housing Property" (DOCSCRH) covers extremely similar words as the other class "Dispute Over Contract for Sale and Purchase of Housing Property" (DOCSPHP). The only evidence to distinguish the two classes is the seller's identity – only real estate companies sell commercial residential house. For ethical concerns, we hide the parties' names for both cases.
Class	Documents
DOCS
CRH	Plaintiff: LF
Defendant: GDYZ co.,Ltd
Fact and Reason: On June 3, 2016, the plaintiff and the defendant signed a contract to purchase a house located in D City with a construction area of 19.36 square meters and a purchase amount of 367,724 yuan. The defendant promised to deliver the relevant documents of the commercial housing transfer registration to the plaintiff before May 15, 2018, which was overdue Liquidated damages were paid according to one ten thousandth of the total purchase price per day; after the signing of the contract, the plaintiff paid the defendant all the purchase price, but the defendant has not yet delivered to the plaintiff the relevant documents for the transfer of commercial housing. The defendant’s delay in handling the housing ownership certificate breached the contract, which harmed the interests of the plaintiff. Therefore, a lawsuit was filed in the court.
DOCS
PHP	Plaintiff: ZB
Defendant: YJ
Defendant: WS
Fact and Reason: The two defendants are relatives, and WS was YJ’s mother-in-law. On November 24, 2015, the plaintiff and the defendant reached an agreement to purchase a house. On November 27, 2015, the plaintiff and the defendant signed a house purchase contract. On December 3, 2015, the plaintiff paid the purchase price and has been living until now. At the beginning of 2019, the plaintiff failed to urge the defendant to go through the house transfer procedures, so he sued to the court.
the two entities; and (4) Semantic Role Labeling (SRL) graph to model broader relations including those among third-party persons and facts. We will elaborate on each graph in Section 3.2. To learn the document representation, D2GCLF combines the four graphs and passes them to a graph representation learning module, based on the idea of graph attention networks (Velickovic et al., 2018). Then, D2GCLF uses the document representation for classification. Our main contributions include:
- We propose a new idea of leveraging facts, i.e., the relations between entities, for legal document classification.
- We propose a novel document-to-graph model for legal document classification.
- We conduct extensive experiments on a real-world legal document dataset. The proposed model outperforms the state-of-the-art text classifiers.
2.1 Traditional Classification Methods
Traditional text classifiers are based on machine learning models, including Naïve Bayes, Logistic regression, etc. Naïve Bayes assumes the words in a document are independent given the class and estimates the class label with Maximum a Posteriori (Zhang and Hawkins, 2018; Fang et al., 2020). Logistic regression (Genkin et al., 2005; Ifrim et al., 2008; Pranckevicius and
Marcinkevicius, 2016) and Support Vector Machine (SVM) (Sathe and Aggarwal, 2019; Pranckevicius and Marcinkevicius, 2016) find decision boundaries for the document classes in the feature space. Bagging (Li et al., 2011) and AdaBoost (Bloehdorn and Hotho, 2004) classifiers ensemble multiple classification models. The bagging model chooses the best result of multiple sub-classifiers as the final decision. AdaBoost uses the sub-classifiers to focus on the fallible classification cases when training. In the legal domain, some early studies utilize these models to classify judgments (Lin et al., 2012; Li et al., 2019a).
2.2 Deep Neural Networks
Deep neural network classifiers learn latent semantic representations of the input documents, which are then used for classification. Convolutional Neural Networks (CNNs) perform convolution operations for capturing the local context in the latent representation (Li et al., 2019b; Wang et al., 2018b). Recurrent Neural Networks (RNNs) consider the sequential information of a sentence (Das et al., 2019). Long-Short Term Memory (LSTM), Gated Recurrent Unit (GRU), and other modified RNNs (Liu et al., 2016) address the gradient vanishing problem of RNN with well-designed gates in a recurrent unit. Hybrid models, such as Recurrent Convolutional Neural Networks (RCNN), combine RNN and CNN. RCNN is more robust to noise when encoding the sequential information (Lin et al., 2018).
Bidirectional Encoder Representations from Transformers (BERT) series are widely utilized in
NLP tasks. BERT is proven effective in many LegalAI tasks, such as judgment prediction. Previous studies of BERT in legal domain include the legal BERT(Cui et al., 2021; Chalkidis et al., 2020), RoBERTa(Cui et al., 2020), Chinese legal Longformer (Xiao et al., 2021), etc.
2.3 Graph-based Classifications
All the methods above focus on the semantics of documents. However, the document structure, especially for long documents, is also vital for text classification. Recent studies (Kipf and Welling, 2016; Yao et al., 2019; ?; Liu et al., 2020) propose to represent the documents in a homogeneous or heterogeneous graph. The homogeneous graph consists of all coherent words as nodes (Ragesh et al., 2021; Zhang et al., 2020) - two words are connected if they co-occur. The heterogeneous graph utilizes the entities-sentence co-occurrence (Hu et al., 2019). Given the representative graph, a graph neural network is used to classify the documents (Velicković et al., 2017; Pal et al., 2020; ?).
To the best of our knowledge, all existing methods classify the documents by topics, reflected by either word frequency or word co-occurrence. None of the existing methods leverage the facts described in the documents. Different from existing methods, we propose to explore the facts in the legal documents for accurate classification.
3 Document-to-Graph Classifier
Referring back to the previous example in Table 1, we observe that a civil complaint case often contains four main components: (1) Entity information sections that cover the information of litigation participants; (2) Facts between the entities, which are essential for identifying the type of dispute; (3) Reason why the plaintiff sued; (4) Miscellaneous items that include discussions on the relevant law, procedure, evidence. Miscellaneous items are less relevant to document types, because the same law may be used in different types of disputes. Figure 1 shows the common structures for civil complaints.
The relations among the key participants are often implied in the sentences in the fact and reason sections. These relations vary across different types. Table 2 shows four example sentences relevant to different types of lending disputes. The
Figure 1: The common structure of Chinese Civil Complaints (Left) and a specific type (Right).
Table 2: Example of different lending disputes.
No.	Example sentence	Class
1	A borrowed B 5,000 dollars.	DOCPL
2	A borrowed B 5,000 dollars to buy C's house.	DOCPL
3	A failed in business and bor- rowed money from B.	DOCPL
4	A borrowed B's money, C is the guarantor. Because A did not re- turn the money, B ask C to re- turn.	DOCS
first three examples belong to "Dispute Over Contract for Private Lending" (DOCPL). Example 1 discusses a lending action, while Example 2 and 3 mention the purpose and reason for the lending action. In Example 4 of a "Dispute Over Contract of Suretyship" (DOCS) document, the "guarantor" is pivotal to identifying the type. However, this keyword may be overlooked by a classifier unless it understands the relations among participants from the sentences.
Motivated by the observations above, we propose a Document-to-Graph Classifier (D2GCLF). D2GCLF represents a legal document by relation graphs constructed from the facts. Figure 2 shows the architecture of the D2GCLF. Our method extracts four relation graphs, each graph represents the facts associated with the main participants from different aspects (Section 3.2). Once we obtain the graphs, we learn a latent representation of each document by aggregating the information from all graphs, based on the idea of graph attention networks (Section 3.3). The document representation is then passed to a log-softmax layer to generate the class label.
Figure 2: The architecture of D2GCLF.
Figure 3: An example of Entity-Matter graph extracted from text: "A borrowed B dollars".
Figure 4: An example of Entity-Action graph extracted from text: "A borrowed B dollars".
3.2 Graph Extraction
Entity-Matter Graph The Entity-Matter graph is motivated by the observation that each dispute case should correspond to some matters between the plaintiffs and defendants. For example, the first case in Table 1 discusses the matter of commercial housing transfer and housing ownership. We consider the "matters" as the key evidence for identifying the class of a dispute case. Observing that "matters" are often nouns that appear in the same sentence of the plaintiffs and defendants, we extract the nouns from every sentence that contains both plaintiffs and defendants using a part-of-speech parser<sup>1</sup>. To understand the actions that have been taken on the "matters", we also extract the verb that describes each extracted noun. For example in Table 2, the verb "borrowed" will be extracted together with the noun "dollar". Then, we construct the Entity-Matter graph as follows: (1) create a document node, a plaintiff node (denoted by A) and a defendant node (denoted by B); (2) connect both A and B to the document node; and (3) connect the nouns extracted to both A and B, and to the verbs describing them. Figure 3 shows an example of Entity-Matter graph.
Entity-Action Graph The Entity-Action graph is motivated by the observation that each dispute case should correspond to some actions between the plaintiffs and defendants. For example, the
first case in Table 1 indicates that the plaintiff has "paid" the defendant money to purchase the house. The verb "paid" is an action that has been taken by the plaintiff. Motivated by this, we extract the verb from every sentence that contains both plaintiffs and defendants as actions. To understand the actions, we also extract the object of each action. For example in Table 2, the noun "dollar" will be extracted together with the verb "borrowed". Similar to the construction of Entity-Matter graph, we connect the plaintiff and defendant to a document node. The difference is to connect the plaintiff and defendant with each "action". In addition, we attach the corresponding object to each "action" node. Figure 4 shows an example of Entity-Action graph.
Entity-Keyword Graph The Entity-Keyword graph is designed to capture the topics related to the plaintiffs and defendants. For example, the first case in Table 1 includes keywords such as "house", "transfer" and "buy", which describe the general topics of the dispute. We extract keywords from every sentence that contains all litigation participants using Textrank (Mihalcea and Taurau, 2004). For example in Table 2, "dollar" and "borrowed" are the keywords relevant to the key stakeholders. To construct the Entity-Keyword graph, we create the document node and participant nodes in the same way as the previous two graphs. Then, we connect the keywords extracted to the litigation participants. Figure 5 shows an example of the Entity-Keyword graph.
SRL Graph The above graphs only focus on the noun or verb related to the litigation participants. However, besides the main participants, other peo
1https://pypi.org/project/pkuseg/
Figure 5: An example of Entity-Keyword graph extracted from text: "A borrowed B dollars".
Figure 6: An example of SRL graph extracted from text: "A borrowed B dollars. C provide guarantee for the loan". In the example, "borrowed" is the predicate associated with A and B.
ple and facts may be involved in a law case. For instance, in Example 4 of Table 2, C is a third party individual, who is not the direct participant in the case but acts as a guarantor. Therefore, we construct a Semantic Role Lableing (SRL) graph to extract relations in the form of (subject, predicate, object) from every sentence in the document using the LTP tool<sup>2</sup>. For example in Table 2, (A, borrowed, B) and (C, guarantee, "loan") will be extracted. Then, we construct the SRL graph by: (1) connecting the extracted subjects and objects to the document node; and (2) connecting subjects and objects through the predicates. Figure 6 shows an example of the SRL graph.
Combined graph The four graphs extracted above represent the facts in the documents from different aspects, and thus they have limited presentation power if applied individually. As such, we combine the four graphs by merging the nodes that denote the same concepts from the four graphs into one, e.g., the document nodes, the plaintiff and defendant nodes, and other noun or verb nodes. The edges from the four graphs are preserved in the combined graph. Figure 7 shows an example of the combined graph.
3.3 Graph-based Document Representation Learning
Given the combined graph that represents the facts in a document, we aim to learn a latent representation for the document that encodes information
Figure 7: An example of the combined graph.
of the graph. In this work, we apply a graph attention network (Veličković et al., 2017) to aggregate the information from all nodes to the "document" node for learning document representations. Note that, our method is flexible to apply any existing graph neural networks for document representation learning. We will leave the comparison of different graph neural networks as future work.
The input of the graph attention network is a set of node features:
$$
h = \left\{\vec {h} _ {1}, \vec {h} _ {2}, \dots , \vec {h} _ {N} \right\}, \vec {h} _ {i} \in \mathbb {R} ^ {F}, \tag {1}
$$
where  $N$  is the number of nodes and  $F$  is the number of features for each node. We use pre-trained word embeddings (detailed in Section 4.3) as the input feature  $h$  for all nodes.
The outputs are the  $F^{\prime}$ -dimension latent representations of all nodes:
$$
h ^ {\prime} = \left\{\vec {h} _ {1} ^ {\prime}, \vec {h} _ {2} ^ {\prime}, \dots , \vec {h} _ {N} ^ {\prime} \right\}, \vec {h} _ {i} ^ {\prime} \in \mathbb {R} ^ {F ^ {\prime}}. \tag {2}
$$
For each node, the graph attention network performs self-attention and the cross-correlation attention with neighbor nodes:
$$
a: \mathbb {R} ^ {F ^ {\prime}} \times \mathbb {R} ^ {F ^ {\prime}} \rightarrow \mathbb {R} \tag {3}
$$
$$
e _ {i j} = a (\mathbf {W} \vec {h} _ {i}, \mathbf {W} \vec {h} _ {j}),
$$
where  $\mathbf{W}$  represents the weight,  $a$  is a single-layer feedforward neural network,  $e_{ij}$  means the importance of node  $j$  to node  $i$ , vector  $h$  is the feature vector and subscripts  $i, j$  denote the  $i$ -th node and the  $j$ -th node, respectively. Then, a softmax function is applied to regularize the attention scores  $\alpha_{ij}$  for the adjacent nodes such that they sum to one.
$$
\alpha_ {i j} = \operatorname {s o f t m a x} _ {j} \left(e _ {i j}\right) = \frac {\exp \left(e _ {i f}\right)}{\sum_ {k \in \mathcal {N} _ {i}} \exp \left(e _ {i k}\right)}. \tag {4}
$$
Then, the latent representation of a node is computed as an aggregation of its neighbors:
$$
\vec {h} _ {i} ^ {\prime} = \sigma \left(\sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i j} \mathbf {W} \vec {h} _ {j}\right), \tag {5}
$$
$^{2}$ http://ltp.ai/docs/quickstart.html
where the  $\sigma (\cdot)$  is an activation function and  $\mathbf{W}$  is a learnable weight matrix. In this work, we use the Rectified Linear Unit  $RELU(\cdot)$  as the activation function. In order to be able to focus on the information from the scattered nodes to Document node, we apply multi-head attention, which calculate  $K$  different attention scores  $\alpha_{i,j}^{k}$  and weights  $\mathbf{W}^{k}$ :
$$
\vec {h} _ {i} ^ {\prime} = \sigma \left(\frac {1}{K} \sum_ {k = 1} ^ {K} \sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i, j} ^ {k} \mathbf {W} ^ {k} \vec {h} _ {j}\right). \tag {6}
$$
By the above formulation, the "document" node will aggregate information from its neighbors, which in turn, aggregate information from their neighbors.
Let the latent vector representation of the document be  $\vec{h}_{doc}^{\prime}$ , which is obtained by Eq. 6. We feed the document representation  $\vec{h}_{doc}^{\prime}$  to a log-softmax layer to compute the probability for a document to be a particular type by:
$$
p (\text {c l a s s} | \text {d o c}) = \log - \text {s o f t m a x} \left(\vec {h} _ {\text {d o c}} ^ {\prime}\right). \tag {7}
$$
To learn the parameters of the model, we minimise the Negative Log Likelihood Loss:
$$
\mathcal {L} = - \sum_ {(d, l _ {d}) \in \mathscr {D} _ {\text {t r a i n}}} \log p (l _ {d} | d), \tag {8}
$$
where  $(d,l_d)$  is a pair of document  $d$  and its class label  $l_{d}$  in the training set  $\mathcal{D}_{train}$ .
Chinese civil law contains more than 400 classes. Among those 400 classes, we asked legal experts to pick up 20 classes, which are the most semantically close to each other. Table 5 shows the name and the label of each of the 20 selected classes. We then collected the 4,000 judgments from China Judgments Online<sup>3</sup> by choosing the latest published 200 cases in the searching result for each of the 20 classes. Following the format of the Chinese judgment, we selected the parties' information paragraph and the plaintiff's allegation paragraph from each judgment to create the statement of claim. The resulting dataset contains
4,000 civil cases, 200 cases per class. Ethnic issues are discussed in Section 6.
We randomly split the dataset into  $70\%$  training set and  $30\%$  test set. For the anaphora and co-references in the indictments, we replace them with the actual name of the litigation participants.
We compare the proposed method with several baselines. As described in Section 2, multiple word representation methods can be used, such as TF-IDF and word embeddings. For word embedding, We use the pre-trained word embedding on People's Daily News<sup>4</sup>. Because, the linguistic style of the newspaper is very similar to the Chinese legal document. The comparison of different word representations, initialization, and TF-IDF, for baselines are presented in Appendix C. For neural network-based model, we keep the same parameter as the original papers and we selected the 10 best checkpoints based on performance on the validation set and report averaged results on the test set.
Machine learning methods. We compare our model with Naïve Bayes, SVM, Logistic Regression, Boosting model, Bagging model in legal document classification. The best embedding result for these models is Char-level TF-IDF representation. We utilize the suitable pre-trained Chinese word embeddings<sup>4</sup>.
Deep learning. We compare D2GCLF with deep learning techniques including CNN, RNN, RNN-Bidirectional (BiRNN), BiLSTM, RCNN, HN-ATT (?) using pre-train word embeddings as input.
BERT series. We utilize the pre-trained BERT (Devlin et al., 2018), RoBERTa (Cui et al., 2021), and Chinese Legal Longformer (Xiao et al., 2021) to compare with the proposed model. After encoding the document by the pre-trained model, we add a fully-connected layer to predict the label. Based on the pre-trained language model, BERT, we also select the Task-Scaling mechanisms (TaSc) (Chrysostomou and Aletras,
3All indictments used in this paper were collected on China Judgment Onlinehttps://wenshu.court.gov.cn/. For ethical concern, we only release the Reference Number and url for the cases we used. The index file is available on: https://drive.google.com/file/d/1bZVv0TPSjIRsRj00P67v8Y-K-tb-o7IE/view?usp=sharing.
4https://github.com/Embedding/Chinese-Word-Vectors
5https://storage.googleapis.com/bert_models/