SlowGuess/ACL_2020_OCR huggingface dataset:
We concatenate the basic representation  $\mathbf{v}_f^{\mathrm{b}}$  and the distinguishable representation  $\mathbf{v}_f^{\mathrm{d}}$  as the final representation of fact description  $f$ , i.e.,  $\tilde{\mathbf{v}}_f = [\mathbf{v}_f^{\mathrm{b}}, \mathbf{v}_f^{\mathrm{d}}]$ . Based on  $\tilde{\mathbf{v}}_f$ , we generate a corresponding feature vector  $\tilde{\mathbf{v}}_f^j$  for each subtask  $t_j$ ,  $j = 1, 2, 3$  mentioned in Sec. 3, i.e.,  $t_1$ : law article prediction;  $t_2$ : charge prediction;  $t_3$ : term of penalty prediction. To obtain the prediction for each subtask, we use a linear classifier:
$$
\hat {y} _ {j} = \mathrm {s o f t m a x} (\mathbf {W} _ {p} ^ {j} \tilde {\mathbf {v}} _ {f} ^ {j} + \mathbf {b} _ {p} ^ {j}),
$$
where  $\mathbf{W}_p^j$  and  $\mathbf{b}_p^j$  are parameters specific to task  $t_j$ . For training, we compute a cross-entropy loss function for each subtask and take the loss sum of all subtasks as the overall prediction loss:
$$
\mathcal {L} _ {p} = - \sum_ {j = 1} ^ {3} \sum_ {k = 1} ^ {| Y _ {j} |} y _ {j, k} \log (\hat {y} _ {j, k}),
$$
where  $|Y_j|$  denotes the number of different classes (or, labels) for task  $t_j$  and  $[y_{j,1}, y_{j,2}, \dots, y_{j,|Y_j|}]$  refers to the ground-truth vector of task  $t_j$ . Besides, we also consider the loss of law article community prediction (i.e., Eq. 1):
$$
\mathcal {L} _ {c} = - \lambda \sum_ {j = 1} ^ {M} X _ {j} \log (\hat {X} _ {j}),
$$
where  $[X_1, X_2, \ldots, X_M]$  is the ground-truth vector of the community including the correct law article applied to the law case. In summary, our final overall loss function is:
$$
\mathcal {L} = \mathcal {L} _ {p} + \mathcal {L} _ {c} \tag {2}
$$
To evaluate the performance of our method, we use the publicly available datasets of the Chinese AI and Law challenge (CAIL2018)<sup>1</sup> (Xiao et al., 2018): CAIL-small (the exercise stage dataset) and CAIL-big (the first stage dataset). The case samples in both datasets contain fact description, applicable law articles, charges, and the terms of penalty. For data processing, we first filter out samples with fewer than 10 meaningful words. To be consistent with state-of-the-art methods, we filter out the case samples with multiple applicable law articles and multiple charges. Meanwhile, referring to (Zhong et al., 2018), we only keep the law articles and charges that apply to not less than 100 corresponding case samples and divide the terms of penalty into non-overlapping intervals. The detailed statistics of the datasets are shown in Table 1.
5.2 Baselines and Settings
Baselines. We compare LADAN with some baselines, including:
<http://cail.cipsc.org.cn/index.html
Dataset	CAIL-small	CAIL-big
#Training Set Cases	101,619	1,587,979
#Test Set Cases	26,749	185,120
#Law Articles	103	118
#Charges	119	130
#Term of Penalty	11	11
- CNN (Kim, 2014): a CNN-based model with multiple filter window widths for text classification.
- HARNN (Yang et al., 2016): an RNN-based neural network with a hierarchical attention mechanism for document classification.
- FLA (Luo et al., 2017): a charge prediction method that uses an attention mechanism to capture the interaction between fact description and applicable laws.
- Few-Shot (Hu et al., 2018): a discriminating confusing charge method, which extracts features about ten predefined attributes from fact descriptions to enforce semantic information.
- TOPJUDGE (Zhong et al., 2018): a topological multi-task learning framework for LJP, which formalizes the explicit dependencies over subtasks in a directed acyclic graph.
- MPBFN-WCA (Yang et al., 2019): a multitask learning framework for LJP with multiperspective forward prediction and backward verification, which is the state-of-the-art method.
Similar to existing works (Luo et al., 2017; Zhong et al., 2018), we train the baselines CNN, HLSTM and FLA using a multi-task framework (recorded as MTL) and select a set of the best experimental parameters according to the range of the parameters given in their original papers. Besides, we use our method LADAN with the same multi-task framework (i.e., Landan+MTL, LADAN+TOPJUDGE, and LADAN+MPBFN) to demonstrate our superiority in feature extraction.
Experimental Settings. We use the THU-LAC (Sun et al., 2016) tool to get the word segmentation because all case samples are in Chinese. Afterward, we use the Skip-Gram model (Mikolov et al., 2013) to pre-train word embeddings on these case documents, where the model's embedding size and frequency threshold are
set to 200 and 25 respectively. Meanwhile, we set the maximum document length as 512 words for CNN-based models in baselines and set the maximum sentence length to 100 words and maximum document length to 15 sentences for LSTM-based models. As for hyperparameters setting, we set the dimension of all latent states (i.e.,  $d_w$ ,  $d_s$ ,  $d_l$  and  $d_f$ ) as 256 and the threshold  $\tau$  as 0.3. In our method LADAN, we use two graph distillation layers, and a Bi-GRU with a randomly initialized attention vector  $u$  is adopted as the basic document encoder. For training, we set the learning rate of Adam optimizer to  $10^{-3}$ , and the batch size to 128. After training every model for 16 epochs, we choose the best model on the validation set for testing.
To compare the performance of the baselines and our methods, we choose four metrics that are widely used for multi-classification tasks, including accuracy (Acc.), macro-precision (MP), macro-recall (MR), and macro-F1 (F1). Since the problem of confusing charges often occurs between a few categories, the main metric is the F1 score. Tables 2 and 3 show the experimental results on datasets CAIL-small and CAIL-big, respectively. Our method LADAN performs the best in terms of all evaluation metrics. Because both CAIL-small and CAIL-big are imbalanced datasets, we focus on comparing the F1-score, which more objectively reflects the effectiveness of our LADAN and other baselines. Compared with the state-of-the-art MPBFN-WCA, LADAN improved the F1-scores of law article prediction, charge prediction, and term of penalty prediction on dataset CAIL-small by  $2.02\%$ ,  $2.42\%$  and  $4.20\%$  respectively, and about  $3.18\%$ ,  $1.44\%$  and  $5.79\%$  on dataset CAIL-big. Meanwhile, the comparison under the same multi-task framework (i.e., MTL, TOPJUDGE, and MPBFN) shows that our LADAN extracted more effective features from fact descriptions than all baselines. Meanwhile, we can observe that the performance of Few-shot on charge prediction is close to LADAN, but its performance on the term of penalty prediction is far from ideal. It is because the ten predefined attributes of Few-Shot are only effective for identifying charges, which also proves the robustness
2Our source codes are available at https://github. com/prometheusXN/LADAN
Tasks Metrics	Law Articles	Charges	Term of Penalty
Acc.	MP	MR	F1	Acc.	MP	MR	F1	Acc.	MP	MR	F1
FLA+MTL	77.74	75.32	74.36	72.93	80.90	79.25	77.61	76.94	36.48	30.94	28.40	28.00
CNN+MTL	78.71	76.02	74.87	73.79	82.41	81.51	79.34	79.61	35.40	33.07	29.26	29.86
HARNN+MTL	79.79	75.26	76.79	74.90	83.80	82.44	82.78	82.12	36.17	34.66	31.26	31.40
Few-Shot+MTL	79.30	77.80	77.59	76.09	83.65	80.84	82.01	81.55	36.52	35.07	26.88	27.14
TOPJUDGE	79.88	79.77	73.67	73.60	82.10	83.60	78.42	79.05	36.29	34.73	32.73	29.43
MPBFN-WCA	79.12	76.30	76.02	74.78	82.14	82.28	80.72	80.72	36.02	31.94	28.60	29.85
LADAN+MTL	81.20	78.24	77.38	76.47	85.07	83.42	82.52	82.74	38.29	36.16	32.49	32.65
LADAN+TOPJUDGE	81.53	78.62	78.29	77.10	85.12	83.64	83.57	83.14	38.34	36.39	32.75	33.53
LADAN+MPBFN	82.34	78.79	77.59	76.80	84.83	83.33	82.80	82.85	39.35	36.94	33.25	34.05
Table 2: Judgment prediction results on CAIL-small.
Tasks Metrics	Law Articles	Charges	Term of Penalty
Acc.	MP	MR	F1	Acc.	MP	MR	F1	Acc.	MP	MR	F1
FLA+MTL	93.23	72.78	64.30	66.56	92.76	76.35	68.48	70.74	57.63	48.93	45.00	46.54
CNN+MTL	95.84	83.20	75.31	77.47	95.74	86.49	79.00	81.37	55.43	45.13	38.85	39.89
HARNN+MTL	95.63	81.48	74.57	77.13	95.58	85.59	79.55	81.88	57.38	43.50	40.79	42.00
Few-Shot+MTL	96.12	85.43	80.07	81.49	96.04	88.30	80.46	83.88	57.84	47.27	42.55	43.44
TOPJUDGE	95.85	84.84	74.53	77.50	95.78	86.46	78.51	81.33	57.34	47.32	42.77	44.05
MPBFN-WCA	96.06	85.25	74.82	78.36	95.98	89.16	79.73	83.20	58.14	45.86	39.07	41.39
LADAN+MTL	96.57	86.22	80.78	82.36	96.45	88.51	83.73	85.35	59.66	51.78	45.34	46.93
LADAN+TOPJUDGE	96.62	86.53	79.08	81.54	96.39	88.49	82.28	84.64	59.70	51.06	45.46	46.96
LADAN+MPBFN	96.60	86.42	80.37	81.98	96.42	88.45	83.08	84.95	59.85	51.75	45.59	47.18
of our LADAN. The highest MP- and MR-scores of LADAN also demonstrates its ability to distinguish confusing law articles. Note that all methods' performance on dataset CAIL-big is better than that on CAIL-small, which is because the training set on CAIL-big is more adequate.
To further illustrate the significance of considering the difference between law articles, we conducted ablation experiments on model  $LADAN + MTL$  with dataset CAIL-small. To prove the effectiveness of our graph construction layer (GCL), we build a LADAN model with the GCL's removing threshold  $\tau = 0$  (i.e., "no GCL" in Table 4), which directly applies the GDO on the fully-connected graph  $G^{*}$  to generate a global distinction vector  $\beta_{g}$  for re-encoding the fact description. To verify the effectiveness of our graph distillation operator (GDO), we build a no-GDO LADAN model (i.e., "no GDO" in Table 4), which directly pools each subgraph  $g_{i}$  to a distinction vector  $\beta_{i}$  without GDOs. To evaluate the importance of considering the difference among law articles, we remove both GCL and GDO from LADAN by setting  $\tau = 1.0$  (i.e., "no both" in Table 4), i.e., each law article independently extracts the attentive feature from fact description. In Table 4, we
Table 3: Judgment prediction results on CAIL-big.
Tasks Metrics	Law	Charge	Penalty
Acc.	F1	Acc.	F1	Acc.	F1
LADAN+MTL	81.20	76.47	85.07	83.14	38.29	32.65
-no GCL	80.46	75.98	84.04	82.33	37.80	31.85
-no GDO	80.82	76.19	84.65	82.50	36.69	31.62
-no both	79.79	74.97	83.72	82.02	34.87	31.34
Table 4: Ablation analysis on CAIL-small.
see that both GCL and GDO effectively improve the performance of LADAN. GCL is more critical than GDO because GDO has a limited performance when the law article communities obtained by GCL are not accurate. When removing both GCL and GDO, the accuracy of LADAN decreases to that of HARNN+MTL, which powerfully demonstrates the effectiveness of our method exploiting differences among similar law articles.
To intuitively verify that LADAN effectively extracts distinguishable features, we visualize the attention of LADAN's encoders. Figure 5 shows two law case examples, each for Article 385 and Article 163, respectively, where the darker the word is, the higher the attention weight it gets in the corresponding encoder, i.e., its information is more important to the encoder. For the basic encoder, we see that the vital information in these two cases is very similar, which both contain the
Basic Encoder:
In the second half of 2017, the defendant Tian, while working in the Cadastral Unit of Luocheng Branch of Luohe City Land and Resources Bureau, used his position responsibility for cadastral registration to provide Zhao with a blank cadastral registration form, forged a false cadastral file, and accepted Zhao a certain amount of cash of 25,000 yuan for personal consumption.
Fact Re-encoder:
Case example of Law Article 185:
Crimeof acceptance of bribes
From Jan. 2006 to Mar.2007, the defendant Gong used the position of signing and clearing the contract for the coal unloading business of the plant's steam coal during his service as a company manager to obtain benefits for a loading and unloading team. And he illegally accepted the benefit fee of 40,000 yuan from the legal person of the loading and unloading team......
From Jan. 2006 to Mar.2007, the defendant Gong used the position of signing and clearing the contract for the coal unloading business of the plant's steam coal during his service as a company manager to obtain benefits for a loading and unloading team. And he illegally accepted the benefit fee of 40,000 yuan from the legal person of the loading and unloading team....
Case example of Law Article 163: Bribery crime of non-state emplote
Figure 5: The attention visualization on case examples for Article 185 and Article 163.
word like "use position" "accept benefit" "accept ... cash", etc. Therefore, when using just the representation of basic encoder to predict acceptable law articles, charges and terms of penalty, these two cases tend to be misjudged. As we mentioned in Sec. 4.3, with the distinction vector, our fact re-encoder focuses on extracting distinguishable features like defendants' identity information (e.g., "company manager" "working in the Cadastral Unit of Luocheng Branch of Luohoe City Land and Resources Bureau" in our examples), which effectively distinguish the applicable law articles and charges of these two cases.
In this paper, we present an end-to-end model, LADAN, to solve the issue of confusing charges in LJP. In LADAN, a novel attention mechanism is proposed to extract the key features for distinguishing confusing law articles attentively. Our attention mechanism not only considers the interaction between fact description and law articles but also the differences among similar law articles, which are effectively extracted by a graph neural network GDL proposed in this paper. The experimental results on real-world datasets show that our LADAN raises the F1-score of state-of-the-art by up to  $5.79\%$ . In the future, we plan to study complicated situations such as a law case with multiple defendants and charges.
The research presented in this paper is supported in part by National Key R&D Program of China (2018YFC0830500), Shenzhen Basic Research Grant (JCYJ20170816100819428), National Nat
ural Science Foundation of China (61922067, U1736205, 61902305), MoE-CMCC "Artificial Intelligence" Project (MCM20190701), National Science Basic Research Plan in Shaanxi Province of China (2019JM-159), National Science Basic Research Plan in Zhejiang Province of China (LGG18F020016).
Stephen Bonner, Ibad Kureshi, John Brennan, Georgios Theodoropoulos, Andrew Stephen McGough, and Boguslaw Obara. 2019. Exploring the semantic content of unsupervised graph embeddings: an empirical study. Data Science and Engineering, 4(3):269-289.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In ICML.
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In NeurIPS.
Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-shot charge prediction with discriminative legal attributes. In COLING.
R Keown. 1980. Mathematical models for legal prediction. Computer/lj, 2:829.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP.
Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICML.
Fred Kort. 1957. Predicting supreme court decisions mathematically: A quantitative analysis of the "right to counsel" cases. American Political Science Review, 51(1):1-12.
Benjamin E. Lauderdale and Tom S Clark. 2012. The supreme court's many median justices. *American Political Science Review*, 106(4):847-866.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI.
Wan-Chen Lin, Tsung-Ting Kuo, Tung-Jia Chang, Chueh-An Yen, Chao-Ju Chen, and Shou-de Lin. 2012. Exploiting machine learning models for chinese legal documents labeling, case classification, and sentencing prediction. *Proceedings of ROCLING*.
Chao-Lin Liu, Cheng-Tsung Chang, and Jim-How Ho. 2004. Case instance generation and refinement for case-based criminal summary judgments in chinese.
Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to predict charges for criminal cases with legal basis. arXiv preprint arXiv:1707.09168.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NeurIPS.
Stuart S Nagel. 1963. Applying correlation analysis to case prediction. *Tex. L. Rev.*, 42:1006.
Octavia-Maria Sulea, Marcos Zampieri, Shervin Malmasi, Mihaela Vela, Liviu P Dinu, and Josef van Genabith. 2017. Exploring the use of text classification in the legal domain. arXiv preprint arXiv:1710.09306.
Maosong Sun, Xinxiong Chen, Kaixu Zhang, Zhipeng Guo, and Zhiyuan Liu. 2016. Thulac: An efficient lexical analyzer for chinese.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv:1807.02478.
Nuo Xu, Pinghui Wang, Long Chen, Jing Tao, and Junzhou Zhao. 2019. Mr-gnn: Multi-resolution and dual graph neural network for predicting structured entity interactions. In IJCAI.
Wenmian Yang, Weijia Jia, XIIaojie Zhou, and Yutao Luo. 2019. Legal judgment prediction via multiperspective bi-feedback network. arXiv preprint arXiv:1905.03969.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In NAACL.
Haoxi Zhong, Guo Zhipeng, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judgment prediction via topological learning. In EMNLP.