SlowGuess/NAACL_2022_OCR huggingface dataset:
2018_11_03/chinese_L-12_H-768_A-12.zip
$^{6}$ https://huggingface.co/hfl/chinese-roberta-wwm-ext-large
<sup>7</sup>https://github.com/thunlp/LegalPLMs
2021) $^{8}$  as baseline. TaSc allows us to learn non-contextualised information from category text.
Graph-Based series. We also include graph-based models, such as Graph Convolutional Networks (GCN) (Yao et al., 2019), GrapSAGE (?) and Graph Attention Networks (GAT) (Pal et al., 2020), TextING (Zhang et al., 2020). For GCN, GAT, and GrapSAGE following Yao et al. (Yao et al., 2019), we construct a document-word graph from the dataset. Specifically, we create a graph node for each document and each word. Then, a word node connects to a document node if the word is in the document. Word nodes are connected if they are in the same document. Then, we also strictly follow TextING instructions<sup>9</sup>.
4.3 Configuration
We utilize the same pre-trained Chinese word embeddings $^4$  as the baseline methods. For the parameters of D2GCLF, we set the size of node representation  $F' = 25$  in Eq. 2 and the number of attention heads  $K = 8$  in Eq. 6, and utilize the Adam optimizer with the learning rate  $5 * 10^{-5}$ , and weight decay  $5 * 10^{-4}$ . In the experiments, the parameters of all methods are obtained by cross-validation on the training data. To validate the performance of the classifiers, we use AUC as the evaluation metric. Experiments are repeated five times and take the average on a workstation with an Nvidia GeForce RTX 3090 GPU with 24 GB memory.
4.4 Result and Discussion
Overall Comparison: The overall comparison among all methods is shown in Table 3. We observe that D2GCLF outperforms all methods and gains a  $4\%$  AUC improvement over the best baseline method. Among the traditional classification methods, the boosting model achieves the best AUC because it ensembles multiple models, each of which focus on different features. Naïve Bayes performs the worst. This is because it heavily relies on the class distribution in the training data, which could be different in the test set.
For deep neural networks, CNN performs the best while RNN performs the worst (70% of AUC). This is because RNN embeds sentences and
Table 3: Classification performance
Model Name	AUC
Machine Learning	Naive Bayes	81.93%
Logistic Regression	85.81%
SVM	85.15%
Bagging Model	85.35%
Boosting Model	87.17%
Deep Learning	CNN	85.75%
RNN	79.08%
BiRNN	83.07%
BiLSTM	84.67%
RCNN	83.55%
HN-ATT	85.14%
BERT Series	BERT	85.75%
TaSc	85.27%
RoBERTa	81.00%
Legal Longformer	82.25%
Graph based	GCN	79.08%
GAT	83.07%
GraphSAGE	83.55%
TextING	83.71%
Ours	D2GCLF	91.33%
words that may not be relevant to the class. Conversely, CNN aggregates information in a local context, which is less sensitive to noise. Among the deep learning methods, BERT performs the best, because the transformer can capture the impact from key sentences, paragraphs, or sections to some extent. As we have mentioned, the key factors for determining the class of a legal document often lie in particular sentences, paragraphs, or sections. The graph neural networks with a document-word graph perform the worst, because the graph contains irrelevant words. TextING constructs a word graph by connecting co-coherence words to represent structure information for message passing. However, it does not work well for the legal data because the graph does not capture the key facts.
Besides, deep learning methods such as CNN and RNN are less effective in legal document classification than in other domains. This is because legal documents imply complex relations among the participants, while the size of real-world legal document datasets, especially the indictment documents, is relatively small for learning a complex and deep model from plain texts. D2GCLF outperforms existing methods by representing the facts in a document as a graph.
$^{8}$ https://github.com/GChrysostomou/tasc
$^{9}$ https://github.com/CRIPAC-DIG/TextING
Figure 8: Number of wrong classification cases in different types.
Table 4: Ablation study for the four graphs.  $\ominus$  denotes the variant that removes one of the four graphs from D2GCLF.
Model Name	AUC
D2GCLFEntity-Matter	84.83%
D2GCLFEntity-Action	83.42%
D2GCLFSRL	82.25%
D2GCLFEntity-Keyword	82.25%
D2GCLFΘEntity-Matter	87.08%
D2GCLFΘEntity-Action	87.67%
D2GCLFΘSRL	87.75%
D2GCLFΘEntity-Keyword	88.08%
D2GCLF	91.33%
Ablation Study: Next, we test the impact of different relation graphs. We first compare the performance of each graph, denoted by  $D2GCLF_{graph}$ , where graph  $\in$  {Entity-Matter, Entity-Action, SRL, Entity-Keyword}. Then, we compare the full model with its variants by removing each graph, denoted by  $D2GCLF_{\text{graph}}$ .
Table 4 shows the AUC of all variants. The Entity-Matter graph performs the best among individual graphs. Besides, when the Entity-Matter graph is removed from the full model, the performance decreases the most. The result justifies that matters involving both participants are the most important factors for classifications. The performance drop of  $D2GCLF_{\ominus Entity-Action}$  implies the actions of the participants provide additional information for classification.  $D2GLF_{\ominus Entity-Keyword}$  results in the least performance drop. This justifies that the topics represented by keywords may have little impact on the task. The significant difference between the full and ablation models shows that the four graphs complement each other.
Case Study: We report the numbers of wrong classification cases for each model in Figure 8. Due to the space limit, we only report the results of Boosting, CNN, RNN and D2GCLF. Over
all, D2GCLF generates the least wrong classifications in most classes (e.g., DPDL, DOCD and DOCSP). Specifically, the two classes Disputes of employer liability (DEL) and Dispute of voluntary workers injured liability (DVWIL) are the most difficult to identify, because they are highly related to other types. The difference between the two is whether work is paid or volunteered. Besides, the injured liability includes traffic accidents, medical liability, etc., which are similar to other dispute classes. D2GCLF performs the best, even for difficult classes because it considers the facts, e.g. traffic accidents.
This study explores a novel idea of classifying legal documents based on the facts discussed in the documents. We propose a Document-to-Graph Classifier (D2GCLF) to implement our idea by modeling the facts as four relation graphs, and applying graph attention network to learn the document representation for classification. Experiments on a real-world legal document dataset show the effectiveness of the four relation graphs and the superior performance (91.33% AUC) of D2GCLF.
The 4,000 Chinese cases that this research is based upon are drawn from the China Judgements Online (https://wenshu.court.gov.cn/), which is available for everyone to search cases' judgments once logged in. We notice that previous researches (Tsarapatsanis and Aletras, 2021; Leins et al., 2020) are worried about the ethical concerns raised in terms of applying the NLP technique into the legal domain. Some researchers believe that processed datasets cause people to harm the relevant parties as such datasets enable a more straightforward retrieval process than searching
through the original datasets published by government agencies. We would like to point out that the judgments published on China Judgment Online already redacted most of the important personal information. Nevertheless, to mitigate against any remaining concerns, in our dataset, we only publish the case reference number and URL instead of the full text which is available. Besides, we only allow the index dataset to be used in research, not for any other purpose. To download or any other way to use judgments in China Judgment Online, please follow the website's terms and conditions.
Stephan Bloehdorn and Andreas Hotho. 2004. Boosting for text classification with semantic features. In International workshop on knowledge discovery on the web, pages 149-166. Springer.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898-2904.
George Chrysostomou and Nikolaos Aletras. 2021. Improving the faithfulness of attention-based explanations with task-specific information for text classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 477-488.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pretrained models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657-668.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese bert.
Monidipa Das, Mahardhika Pratama, Septiviana Savitri, and Jie Zhang. 2019. MUSE-RNN: A multi-layer self-evolving recurrent neural network for data stream classification. In 2019 IEEE International Conference on Data Mining, pages 110-119.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Shikai Fang, Shandian Zhe, Kuang-chih Lee, Kai Zhang, and Jennifer Neville. 2020. Online bayesian sparse learning with spike and slab priors. In 20th IEEE International Conference on Data Mining, ICDM, pages 142-151.
Leilei Gan, Kun Kuang, Yi Yang, and Fei Wu. 2021. Judgment prediction via injecting legal knowledge into neural networks. In Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), pages 12866-12874.
Alexander Genkin, David D Lewis, and David Madigan. 2005. Sparse logistic regression for text categorization. DIMACS Working Group on Monitoring Message Streams Project Report.
Linmei Hu, Tianchi Yang, Chuan Shi, Houye Ji, and Xiaoli Li. 2019. Heterogeneous graph attention networks for semi-supervised short text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 4820-4829.
Georgiana Ifrim, Gokhan Bakir, and Gerhard Weikum. 2008. Fast logistic regression for text categorization with variable-length n-grams. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 354-362.
Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 562-570.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Kobi Leins, Joy Han Lau, and Timothy Baldwin. 2020. Give me convenience and give her death: Who should decide what uses of NLP are appropriate, and on what basis? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2908-2913.
FengGang Li, JiLi Fan, Li Wang, HuLin Zhang, and Rui Duan. 2011. A method based on manifold learning and bagging for text classification. In 2011 2nd International Conference on Artificial Intelligence, Management Science and Electronic Commerce, pages 2713-2716. IEEE.
Guodong Li, Zhe Wang, and Yinglong Ma. 2019a. Combining domain knowledge extraction with graph long short-term memory for learning classification of chinese legal documents. IEEE Access, 7:139616-139627.
Xiaoyu Li, Buyue Qian, Jishang Wei, An Li, Xuan Liu, and Qinghua Zheng. 2019b. Classify eeg and reveal latent graph structure with spatio-temporal graph convolutional neural network. In 2019 IEEE International Conference on Data Mining (ICDM), pages 389-398. IEEE.
Ronghua Lin, Chengzhou Fu, Chengjie Mao, Jingmin Wei, and Jianguo Li. 2018. Academic news text classification model based on attention mechanism
and rcnn. In CCF Conference on Computer Supported Cooperative Work and Social Computing, pages 507-516. Springer.
Wan-Chen Lin, Tsung-Ting Kuo, Tung-Jia Chang, Chueh-An Yen, Chao-Ju Chen, and Shou-de Lin. 2012. Exploiting machine learning models for Chinese legal documents labeling, case classification, and sentencing prediction. In Proceedings of the 24th Conference on Computational Linguistics and Speech Processing (ROCLING 2012), pages 140-141.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. CoRR, abs/1703.03130.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101.
Xien Liu, Xinxin You, Xiao Zhang, Ji Wu, and Ping Lv. 2020. Tensor graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8409-8416.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing.
Mariana Y. Noguti, Eduardo Vellasques, and Luiz S. Oliveira. 2020. Legal document classification: An application to law area prediction of petitions to public prosecution service. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8.
Ankit Pal, Muru Selvakumar, and Malaikannan Sankarasubbu. 2020. Magnet: Multi-label text classification using attention-based graph neural network. In ICAART (2), pages 494-505.
Tomas Pranckevicius and Virginijus Marcinkevicius. 2016. Application of logistic regression with part-of-the-speech tagging for multi-class text classification. In 2016 IEEE 4th Workshop on Advances in Information, Electronic and Electrical Engineering (AIEEE), pages 1-5. IEEE.
Rahul Ragesh, Sundararajan Sellamanickam, Arun Iyer, Ramakrishna Bairi, and Vijay Lingam. 2021. Heterocn: Heterogeneous graph convolutional networks for text classification. In The Fourteenth ACM International Conference on Web Search and Data Mining, pages 860-868.
Edwina L. Rissland, Kevin D. Ashley, and R.P. Loui. 2003. Ai and law: A fruitful synergy. Artificial Intelligence, 150(1):1-15.
Constanta Rosca, Bogdan Covrig, Catalina Goanta, Gijs van Dijck, and Gerasimos Spanakis. 2020. Return of the AI: an analysis of legal research on artificial intelligence using topic modeling. In Proceedings of the Natural Legal Language Processing Workshop 2020, pages 3-10.
Saket Sathe and Charu C Aggarwal. 2019. Nearest neighbor classifiers versus random forests and support vector machines. In 2019 IEEE International Conference on Data Mining (ICDM), pages 1300-1305. IEEE.
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. 2018. Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 440-450.
Vu D. Tran, Minh Le Nguyen, and Ken Satoh. 2019. Building legal case retrieval systems with lexical matching and summarization using A pre-trained phrase scoring model. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law (ICAIL), pages 275-282.
Dimitrios Tsarapatsanis and Nikolaos Aletras. 2021. On the ethical limits of natural language processing on legal text. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 3590–3599.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In 6th International Conference on Learning Representations (ICLR).
Baoxin Wang. 2018. Disconnected recurrent neural networks for text categorization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2311-2320, Melbourne, Australia. Association for Computational Linguistics.
Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018a. Joint embedding of words and labels for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2321-2331. Association for Computational Linguistics.
Shiyao Wang, Minlie Huang, and Zhidong Deng. 2018b. Densely connected cnn with multi-scale feature attention for text classification. In *IJCAI*, pages 4468-4474.
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021. Lawformer: A pre-trained language model for chinese legal long documents. AI Open, 2:79-84.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7370-7377.
Haopeng Zhang and Jiawei Zhang. 2020. Text graph transformer for document classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8322-8327. Association for Computational Linguistics.
Tianyang Zhang, Minlie Huang, and Li Zhao. 2018. Learning structured representation for text classification via reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 6053-6060.
Yufeng Zhang, Xueli Yu, Zeyu Cui, Shu Wu, Zhongzhen Wen, and Liang Wang. 2020. Every document owns its structure: Inductive text classification via graph neural networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 334-339.
Zheng Zhang and Cole Hawkins. 2018. Variational bayesian inference for robust streaming tensor factorization and completion. In IEEE International Conference on Data Mining, pages 1446-1451.
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218-5230.
A Example of Legal Documents
Table 6 shows two examples of real cases with similar topics but categories in different types.
B Classes in Dataset
There are twenty classes in our experiment, such as Dispute over contract for sale and purchase, etc. Table 5 shows the full name and abbreviation of these classes.
Except for the baselines, which are introduced in Section 4, we also utilize the other word embedding methods to test the baselines. The full result is shown in Table 7.
Table 5: Classes in dataset
No.	Class
1	Dispute over contract for sale and purchase (DOCSP)
2	Dispute over contract of sale of commercial residential housing property (DOC-SCRHP)
3	Dispute over contract for sale and purchase of housing property (DOCSPHP)
4	Disputes over contract for housing rental (DOCHR)
5	Disputes over contract for vehicle rental (DOCVR)
6	Dispute over contract of earnest money (DOCEM)
7	Dispute over contract for private lending (DOCPL)
8	Dispute over contract for housing demolition, relocation and compensation (DOCHDRC)
9	Dispute over contract for decoration (DOCD)
10	Dispute over partnership contract (DOPC)
11	Dispute over compensation for personal injury resulting from traffic accidents (DCPIRTA)
12	Disputes of product liability (DPL)
13	Disputes over liability for personal injury from animals (DLPIA)
14	Dispute of medical liability (DML)
15	Dispute of voluntary workers injured liability (DVWIL)
16	Disputes of educational institutions liability (DEIL)
17	Disputes of property damage liability (DPDL)
18	Dispute of online infringement liability (DOIL)
19	Disputes of labor providers victimization liability (DLPVL)
20	Disputes of employer liability (DEL)
Word representation techniques Most existing methods take word representations as input. As such, we apply different word representation techniques: (1) Word-level TF-IDF that calculates each word's TF-IDF in the document; (2) N-grams TF-IDF that calculates the word frequency on the n-gram basis; (3) Char-level TF-IDF that calculates the frequency of n consecutive characters; (4) Pre-trained word embedding<sup>10</sup> that represents the semantics of each word with a low-dimensional vector.
Machine learning methods We choose the Naive Bayes, SVM, Logistic Regression, Boosting model, Bagging model to classifier the legal documents. Given that these methods will not update the word representation vector, We run these classifiers on Word-level, N-grams and Char-level TF-IDF representations.
Deep learning We compare our method to deep learning techniques including CNN, RNN, RNN-Bidirectional (BiRNN), BiLSTM, RCNN. We compare the classification performance of this category of methods with both random initialized and pre-train word embeddings.
$^{10}$ https://github.com/Embedding/Chinese-Word-Vectors
Table 6: Example legal documents of two different types
Class	Documents
DOCS CRH	原告: LF
被告: GDYZ co.,Ltd
事实及理由: 2016年6月3日原告与被告签订合同,购买位于都匀市,建筑面积19.36平方米,购房金额为367724元,被告承诺于2018年5月15日前将商品房转移登记有关文件交付给原告,逾期按按照总购房款每日万分之一支付违约金;合同签订后,原告向被告交付了全部购房款,但被告至今未向原告交付商品房转移登记有关文件。被告迟延办理房屋产权证书存在违约,损害了原告的利益。故向法院提起诉讼。
(Translated)
Plaintiff: LF
Defendant: GDYZ co.,Ltd
Fact and Reason: On June 3, 2016, the plaintiff and the defendant signed a contract to purchase a house located in Duyun City with a construction area of 19.36 square meters and a purchase amount of 367,724 yuan. The defendant promised to deliver the relevant documents of the commercial housing transfer registration to the plaintiff before May 15, 2018, which was overdue Liquidated damages were paid according to one ten thousandth of the total purchase price per day; after the signing of the contract, the plaintiff paid the defendant all the purchase price, but the defendant has not yet delivered to the plaintiff the relevant documents for the transfer of commercial housing. The defendant's delay in handling the housing ownership certificate breached the contract, which harmed the interests of the plaintiff. Therefore, a lawsuit was filed in the court.
DOCS PHP	原告: ZB
被告: YJ
被告: WS
事实及理由: 二被告是亲戚关系,WS曾是YJ的岳母。2015年11月24日,原、被告达成购房合意。2015年11月27日,原、被告签订房屋买卖合同。2015年12月3日,原告交付完购房款,一直居住至今。2019年初,原告催促被告办理房屋过户手续未果,故诉至法院。
(Translated)
Plaintiff: ZB
Defendant: YJ
Defendant: WS
Fact and Reason: The two defendants are relatives, and WS was YJ's mother-in-law. On November 24, 2015, the plaintiff and the defendant reached an agreement to purchase a house. On November 27, 2015, the plaintiff and the defendant signed a house purchase contract. On December 3, 2015, the plaintiff paid the purchase price and has been living until now. At the beginning of 2019, the plaintiff failed to urge the defendant to go through the house transfer procedures, so he sued to the court.
Table 7: Classification performance (AUC) of all methods
Model
Name	Result
WordLevel TF-IDF	N-Gram Vectors	CharLevel Vectors
Naive Bayes	45.75%	25.38%	81.93%
Logistic Regression	43.76%	26.18%	85.81%
SVM	38.78%	24.63%	85.15%
Bagging Model	39.74%	26.13%	85.35%
Boosting Model	35.41%	13.62%	87.17%
Random Initialization	Word Embedding
CNN	72.72%	85.75%
RNN	39.47%	79.08%
RNN-Bidirectional	54.45%	83.07%
BiLSTM	58.58%	84.67%
RCNN	72.75%	83.55%
HN-ATT	-	85.14%
BERT	85.75%
TaSc	85.27%
RoBERTa	81.00%
Legal Longformer	82.25%
GCN	79.08%
GAT	83.07%
GraphSAGE	83.55%
TextING	83.71%
D2GCLF	91.33%