Through the MUD: A Multi-Defendant Charge Prediction Benchmark with Linked Crime Elements (2024)

Abstract: The current charge prediction datasets mostly focus on single-defendant criminal cases. However, real-world criminal cases usually involve multiple defendants whose criminal facts are intertwined. In an early attempt to fill this gap, we introduce a new benchmark that encompasses legal cases involving multiple defendants, where each defendant is labeled with a charge and four types of crime elements, i.e., Object Element, Objective Element, Subject Element, and Subjective Element. Based on the dataset, we further develop an interpretable model called EJudge that incorporates crime elements and legal rules to infer charges. We observe that predicting crime charges while providing corresponding rationales benefits the interpretable AI system. Extensive experiments show that EJudge significantly surpasses state-of-the-art methods, which verify the importance of crime elements and legal rules in multi-defendant charge prediction. Source code and dataset available at https://github.com/welchxu/MCP.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2864–2878 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
The charge prediction task aims to automatically recommend charges given a fact description (Luo et al., 2017; Nair and Modani, 2023). It has attracted substantial attention recently, leading researchers to construct high-quality datasets for its advancement, such as CAIL2018 (Xiao et al., 2018) and ECHR (Medvedeva et al., 2018).
Commonly, existing datasets mainly support coarse-grained prediction, recommending charges for each defendant based on the whole criminal facts, without specifying relevant details. For example, Fig. 1 (a) shows a case from CAIL2018 (Xiao et al., 2018) that has only one defendant and is labeled solely with the charge, without any justification or rationale for the conviction.
*corresponding author
While in real-world scenarios a single case may involve multiple defendants, as shown in Fig. 1 (b) with two defendants, namely, Liu and Wang, whose criminal facts intertwine and overlap. Intuitively, addressing intricate cases with multiple defendants necessitates providing clear and compelling explanations for the criminal facts relevant to each defendant, ensuring the precision of charge predictions. Unfortunately, most of the existing datasets lack fine-grained annotations of criminal facts, consequently impairing the performance of current advanced methods.
As illustrated in Fig. 2 (a), several popular methods, e.g., LegalBERT (Chalkidis et al., 2020), LawFormer (Xiao et al., 2021), and RoBERTa (Cui et al., 2021), show inferior performance on multiple-defendant cases (our new benchmark) compared to single-defendant cases (CAIL2018), with drops of 49%, 38%, and 32%, respectively. In this work, we construct a new benchmark with fine-grained annotations for multi-defendant legal cases, named MUD.
2864
It focuses on multi-defendant criminal cases, comprising 2,865 cases and 7,128 defendantcharge pairs, spanning across 22 different charges. Moreover, each defendant in MUD is annotated with four types of crime elements. It is notable that, in the Chinese legal system, crime elements play a critical role in determining whether a particular action constitutes a crime or not (Cohen, 1982). These consist of the Subject Element, Object Element, Objective Element, and Subjective Element. Their precise definitions according to the Criminal Law of the People’s Republic of China are shown in Table 1. By deeply understanding crime elements, legal professionals can more accurately determine criminal facts, ensuring fair trials. Similarly, in legal artificial intelligence (LegalAI) systems, the incorporation of crime elements is expected to enhance charge prediction accuracy, as well as improve model explainability and credibility. In Fig. 2 (b), our probing experiments confirm the efficacy of crime elements in boosting the performance of existing methods on MUD.
It is not trivial to annotate crime elements for each defendant. To ensure the quality, we adopted a three-stage annotation approach and engaged three legal experts. The experts’ review of 500 randomly selected cases shows 99.3% agreement on annotated crime elements, confirming the high quality of our MUD benchmark.
Our new benchmark fills the gap in the absence of annotated crime elements, facilitating the creation of interpretable models. Based on MUD, we propose a new method named EJudge which jointly leverages crime elements and legal rules to infer charges. The extensive experiments show that MUD poses challenges to existing state-of-the-art models and verify the advancements of our method. Our contributions are as follows:
1) We propose a new multi-defendant charge prediction benchmark named MUD, in which four types of crime elements are annotated.

heading: LegalBERT LawFormer RoBERTa LegalBERT LawFormer RoBERTa
text: 2) We design a crime-element-informed model named EJudge, which jointly leverages crime elements and legal rules to predict charges.
3) Extensive experiments verify the effectiveness of the proposed EJudge in leveraging crime elements with +9.4% F1 increase over existing methods for multi-defendant prediction.
2 Related Work
Legal Datasets. Recently, various datasets have been constructed for LegalAI, such as FLA (Luo et al., 2017), RACP (Jiang et al., 2018), Criminal (Hu et al., 2018), ECHR (Aletras et al., 2016), ECHR-Case (Chalkidis et al., 2019), ECHR-Crystal-Ball (Medvedeva et al., 2018), CAIL2018 (Xiao et al., 2018), QAjudge (Zhong et al., 2020) and FEDLEGAL (Zhang et al., 2023). However, these datasets mainly support coarsegrained charge prediction lacking detailed annotations.
To alleviate these problems, RACP (Jiang et al., 2018) and ACI (Paul et al., 2020) are constructed by randomly selecting 1,000 and 120 documents for sentence-level annotation, respectively. MNLM (Ge et al., 2021) provides fine-grained factarticle annotations for 1,189 legal cases, but there are only two charges. Yue et al. (2021b) constructs a dataset for charge prediction and court view generation that contains the crime circumstances.
SCE (An et al., 2022) provides sentence-level crime elements for 685 signal-defendant cases. Some works also delved into the practical scenarios of multi-defendant cases. MSA (Pan et al., 2019) is developed with the multi-scale attention model to predict the charge for each defendant. However, their used dataset only contains 100 legal cases. Later, MultiLJP (Lyu et al., 2023) is constructed for legal prediction containing a large-scale collection of multi-defendant cases. Following this line, we construct a new dataset with criminal elements for multi-defendant cases. In this work, we create a new benchmark consisting of 2,865 legal cases, with an average of 2.5 defendants per case and covering 22 different charges, and each defendant is annotated with crime elements.
Interpretable Methods. LegalAI has attracted attention in both research and practical applications, yielding notable achievements (Xiao et al., 2021; Feng et al., 2022). There has been a growing emphasis on the significance of interpretability in LegalAI, aiming to diminish the opacity of black-box models and improve the transparency of legal predictions (Jiang et al., 2018; Lyu et al., 2022; Zhao et al., 2022a; Li et al., 2022a; Luo et al., 2023; Barale et al., 2023). For example, Luo et al. (2017) show that manually designed ten elements such as Violence and Death are effective in distinguishing confusing charges. Jiang et al. (2018) and Zhong et al. (2020) verify the usefulness of contributory spans. Luo et al. (2023) make legal decisions by providing precedents and Legislations as inputs. Zhao et al. (2022b) design a multi-task learning method CPEE to explore the practical judicial process and analyzes comprehensive legal essential elements to make judgment predictions. NeurJudge (Yue et al., 2021a) separates the fact description into different circumstances and exploits them to make predictions. Recently, several works (Lyu et al., 2022; Zhao et al., 2022a;
Li et al., 2022b; An et al., 2022) reveal the importance of crime elements for interpretable charge prediction. In line with this, we contribute a new benchmark annotated with crime elements and introduce a novel crime-element-informed method.
3 A New Benchmark: MUD
3.1 Data Collection Our benchmark is sourced from China Judgment Online (CJO)1, a Chinese government website that is widely used in LegalAI tasks (Xiao et al., 2018; Yao et al., 2022). We focus on multi-defendant cases. Specifically, we extract the fact description and defendant-charge pairs from the documents following Xiao et al. (2018). We discard fact descriptions with fewer than 50 characters or those involving only a single defendant. Then, charges with a frequency of less than 100 are filtered out. Through this process, we collect 2,856 documents containing 7,128 defendant-charge pairs covering 22 different charges for annotation.
3.2 Crime Elements Annotation The identification of crime elements (as outlined in Table 1) is crucial in determining if a behavior constitutes a crime in the real-world conviction process (Cohen, 1982). This annotation process is conducted by senior Ph.D. students in law, who possess extensive legal knowledge and a comprehensive understanding of the four elements of crime.
Given a fact description and a defendant (Subject Element), annotators are required to label Object Element and select sentences mentioning Objective Element (i.e., Harmful Action, Harmful Result) and Subjective Element. Fig. 3 shows an annotation example. The annotators are required to spend a minimum of 10 minutes on each fact and are compensated at a rate of $20 per hour based on the time required to complete the annotations.
1 https://wenshu.court.gov.cn/
Commonly used Legal Judgment Prediciton dataset CAIL-2018 (Xiao et al., 2018) relies on automatic extraction for annotation inevitably leading to some errors (as shown in Appendix A). In contrast, we design a three-stage annotation process. In the first stage, annotators are required to familiarize themselves with the annotation process by annotating a subset containing 500 cases that are randomly selected from MUD. Moving to the second stage, each case is annotated three times independently. We discard annotation results if the overlap ratio is less than 0.96. In the third stage, legal experts specifically focus on annotating cases discarded in the second stage, engaging in discussions to reach inter-annotator agreement.
3.3 Data Analysis Dataset Statistics. There are 2,856 cases and 7,128 defendant-charge pairs covering 22 distinct charges in MUD. As shown in Fig. 4, we divide MUD into two subsets: the Easy set, where each case involves all defendants accused of the same charge; and the Hard set, where at least two defendants face different charges, posing a greater challenge for charge prediction.
Dataset Quality. To evaluate the dataset quality, we randomly sample 500 cases labeled three times independently from MUD. The legal experts’ review shows 99.3% agreement on annotated crime elements, demonstrating that the MUD is a highquality manually annotated benchmark.
Annotation Scale. The annotation process for datasets in the legal domain is complex and rigorous, and annotators are required to possess a strong legal background. As far as we know, our new benchmark provides the largest fine-grained annotation scale for multi-defendant charge prediction. Some legal datasets also provide fine-
grained annotations beside the charge labels. Fig. 5 shows the annotations scale of MUD and existing fine-grained annotated datasets in the legal domain. SCE (An et al., 2022) is annotated with sentence-level criminal elements for 685 cases, but they only consider the signal-defendant cases from CAIL. RACP (Jiang et al., 2018) and ACI (Paul et al., 2020) randomly select 1,000 and 120 documents for sentence-level annotation, respectively. MNLM (Ge et al., 2021) provides fine-grained factarticle annotations for 1,189 legal cases covering two different charges. Our benchmark MUD provides crime element annotations for 2,856 cases, which is much larger than the existing datasets.
4 Crime-Element-Informed Method
4.1 Task Definition
Given a multiple-defendant case, its fact description is denoted as f and the involved defendants are denoted as D = {d1, d2, ..., dl}, where l is the number of defendants. The task is to predict the charge for each defendant. To enhance interpretability, legal knowledge is incorporated into the prediction process. In this work, we leverage category information for the legal system, denoted as C = {C1, C2, · · · , Cm}, where Ci represents a crime category encompassing ni charges, i.e., Ci = {ci,1, ci,2, · · · , ci,ni}. Additionally, we use the legal rules R = {r1,1, r1,2, .., rm,nm} defined by law, where ri,j is the legal rule of charge ci,j .
4.2 EJudge
Overview. Fig. 6 shows the overall architecture of EJudge. The basic idea is to deduce the charges against each defendant by analyzing the elements of the crime in conjunction with relevant legal rules. The Element Selector extracts crime elements for each defendant from the fact description. Subse-
quently, the Category Selector predicts charge categories, and the Rule Selector improves the differentiation of confusing charges within each category using legal rules. The Verifier integrates predicted charge categories and legal rules to infer charges. We detail the four modules below.
Element Selector. The conviction process is rigorous and requires consideration of the crime elements in the facts. This module aims to extract the sentences mentioning the crime elements for a given defendant. First, for each defendant di, we generate the representation of defendant-aware fact description f by passing them into a pre-trained encoder (e.g., RoBERTa (Cui et al., 2021)):
Hdi = Encoder([CLS] di [SEP] f [SEP]), (1)
where [CLS] and [SEP] are the special tokens, and Hdi is the output embeddings for all input tokens. We use the NLTK tool2 to split the fact description into sentences, i.e., f = {s1, s2, s3, · · · }, and generate the sentence embeddings using an averagepooling layer:
hsi = avg_pool(hwi,1 ,hwi,2 , ...,hwi,j , ...), (2)
where wi,j is the j-th word in sentence si, hwi,j is its word embedding in Hdi , and hsi is the sentence embedding of si. Then, we apply a linear classifier on hsi followed by a softmax function to predict the element probabilities p̂ ∈ RKe for four types of elements, where Ke is the number of element types, i.e., 4. We train the module by the element classification loss, which can be formulated as:
Les = E[− Ke∑
ke=1
p(ke|hsi) log(p̂(ke|hsi))], (3)
where E denotes the average expectation, and p(ke|si) represents the ground-truth probability of
2 https://www.nltk.org/
crime elements based on the sentence si. For the ground-truth crime element class ke, the p(ke|si) equals to 1 otherwise 0.
Category Selector. In the legal domain, charges are divided into different categories depending on the Object Element. Appendix C shows several examples of charge categories. Generally, given the fact, it’s easy to identify the crime categorize, such as distinguishing between Public Social Security and Market Economic Order). However, the difficulty arises when trying to differentiate between confusing charges within the same category, such as Intentional Homicide and Involuntary Manslaughter. Inspired by this observation, we first predict the charge category for each defendant. Specifically, for each defendant di, we obtain the embedding sequence of the fact description as defined in Eq. (1), and employ an average-pooling layer to get the defendant-aware fact description, denoted as hf . Then, we use the same encoder to encode the extracted crime elements for di, and leverage an average pooling layer over the output sequence embeddings to get the context representation of crime elements, denoted as he. We concatenate hf and he and pass them through a linear layer as the category feature µdi . We train the module by the category classification loss as:
Lcs = E[− Kc∑
kc=1
p(kc|µdi) log(p̂(kc|µdi))], (4)
where E denotes the average expectation, and Kc is the number of charge categories. p̂(kc|µdi) ∈ RKc denote the predicted category probabilities, and p(kc|µdi) represent the ground-truth probability of charge categories based on the category feature µdi , which equals to 1 otherwise 0.
Rule Selector. In our method, convictions are based on aligning crime elements with the relevant
legal rules. This module is designed to calculate matching scores between legal rules in the selected categories and extracted crime elements, identifying the most probable legal rules for charging. The legal rule of each charge is clearly defined, and several examples are shown in Appendix D. In this module, we use the pre-trained encoder (e.g., RoBERTa (Cui et al., 2021)) to separately encode the word sequence of the legal rule ri,j , and sentences containing crime elements of defendant di. Then we obtain the hidden vector sequence of the legal rule Hr = {h1,h2, ...,hlr} and crime elements He = {h1,h2, ...,hle}, where lr and le represent the sequence length. We apply an averagepooling layer to Hr and He to get the embedding of the legal rule and crime elements, which are denoted as hri,j , and he, respectively. Finally, we use the cosine similarity function to measure the matching score between them:
λri,j = he · hri,j
|he| · |hri,j | , (5)
where λri,j represents the matching score. We train the Rule Selector by optimizing contrastive loss. Specifically, given the legal rule ri,j of charge ci,j , we sample sentences mentioning crime elements of the charge ci,j as s+. We sample sentences mentioning crime elements of charge ci,t(t ̸= j), which we denote as s−. With above steps , we construct positive pairs (ri,j , s+) and negative pairs (ri,j , s−). The contrastive loss is defined as:
Lrs = −log e sim(hri,j ,hs+ )/τ
∑N j=1(e sim(hri,j ,hs+ )/τ + e sim(hri,j ,hs− )/τ ) ,
(6)
where N , τ , and sim represent the mini-batch size, temperature hyperparameter, and cosine similarity function, respectively.
Verifier. This module aims to aggregate the scores generated by the Category Selector and Rule Selector to make a final decision. Specifically, we select categories with the top-η highest logits generated by the Category Selector, where the selected categories set is denoted as C′η = {C ′1, C ′2, ..., C ′η}. We choose the charge for which the corresponding legal rule has the highest probability pri,j as the final charge prediction ĉi,j :
q(kc|µdi) = softmax(αµdi), (7)
q(kr|λri,j ) = softmax(βλri,j ), (8)
argmax i,j {pri,j |pri,j = q(kc|µdi) ∗ q(kr|λri,j ), kc ∈ C′η}, (9)
where λri,j represents the similarity score which is generated by the Rule Selector (Eq. 5), α and β are the temperature hyperparameters.
5 Experiment
5.1 Experiment Setting
Dataset and Metrics. We conduct experiments for multi-defendant charge prediction on our MUD, which is randomly split into the training set, development set, and test set, following a ratio of 3 : 1 : 1. We also conduct experiments on the commonly used dataset CAIL (Xiao et al., 2018) with single-defendant cases to verify the effectiveness of EJudge. The details of used datasets are shown in Table 2.
Each case in MUD contains multiple defendants. Following Lyu et al. (2022), we adopt Accuracy (Acc), Macro Precision (MaP), Macro Recall (MaR), and Macro F1 (MaF) to evaluate the model’s ability to predict charges for defendants. In addition, we use Accuracy (Acc*) to measure the model’s ability to predict the charge for cases, i.e., whether correctly assign the charge for all defendants in a case.
Baseline Models. To verify the effectiveness of our model Ejudge, we compare EJudge with the following methods which are summarized in the three groups: Single-Defendant Methods including DPAM (Wang et al., 2018), which incorporate law articles to help charge prediction; CECP (Zhao et al., 2022a), DCSCP (Li et al., 2022a) and GEEN (Lyu et al., 2022), which predict charges by extracted crime elements; HMN (Wang et al., 2019), which formulates charge prediction as a hierarchical multi-label classification problem; NeurJudge (Yue et al., 2021a), which splits the facts into several parts to predict charges; CTM (Liu et al., 2022), which takes case triples as input to predict charges. Multi-Defendant Methods including MSA (Pan et al., 2019), which predict charge by using a multi-scale attention model; Pretrained Language Models including RoBERTa,
which is pre-trained language model of Chinese version (Cui et al., 2021); LegalBERT (Zhong et al., 2019) and LawFormer (Xiao et al., 2021), which are pre-trained language models in the legal domain. Moreover, we also explore the performance of large language models (LLMs) on our MUD, including GPT-4.0 (OpenAI, 2023), GPT3.5 (Ouyang et al., 2022), and GLM-130B (Zeng et al., 2023).
Implementation Details. We use the released source codes to implement baseline models (i.e., DPAM, HMN, NeurJudge, CTM, MSA, GEEN, CECP, DCSCP, HRN). For EJudge, we set the dropout rate, learning rate, batch size, warmup steps, and max length of fact as 0.1, 1× 10−5, 12, 800, and 500, respectively. For the Rule Selector, we sample four negative samples and one positive sample for each instance. We search τ in Eq. (6), top-η of Verifier, α in Eq. (7), and β in Eq. (8) with grid searching, where τ ∈ {0.01, 0.05, 0.1}, top-η ∈ {1, 3, 5}, α ∈ {0.1, 0.3, 0.5}, and β ∈ {0.1, 0.3, 0.5}. The implementation is based on Pytorch and trained on a Tesla V100 GPU with AdamW (Loshchilov and Hutter, 2019) optimizer
for 20 epochs. We choose the checkpoints with the best average performance on the development set and report performance on the test set. In terms of crime elements, the experiments are implemented under three settings: (1) Without elements (w/o E): Only the fact description is used for charge prediction. (2) With extracted elements (w/ E, marked with "∗"): The fact description and extracted crime elements are used for charge prediction. (3) With annotated elements (Oracle, marked with "+"): The fact description and annotated crime elements are used for charge prediction.
5.2 Main Results Table 3 shows the overall performance on MUD. It is observed that EJudge outperforms all baselines by a large margin. For example, in the Hard set of MUD, our EJudge outperforms the prior SOTA method without elements (i.e., RoBERTa) by improving MaF by 9.4% and 13.8% using extracted and annotated elements, respectively. Table 4 reports the performance of single-defendant charge prediction on CAIL (Xiao et al., 2018). It is observed that the element-aware methods, i.e., CECP, DCSCP, GEEN, and EJudge, surpass other meth-
ods by 1.7% in terms of average MaR, showing the importance of crime elements. Moreover, EJudge achieves the best performance, indicating the superiority of our method in leveraging crime elements. Furthermore, we explore the performance of LLMs under zero- and few-shot settings on the full test dataset of MUD. The best ACC* achieved by LLMs is 59.73%, worse than all baseline models trained with labeled cases. This indicates LLMs’ limitations in dealing with professional and intricate legal scenarios. Please refer to Appendix E for details about experiment settings and results analysis.
5.3 In-Depth Analysis Ablation Study. We conduct an ablation study to illustrate the effectiveness of each component of EJudge∗. Table 5 shows that removing the Element Selector (-ES), Category Selector (-CS), Rule Selector (-RS), or Verifier (-V) leads to performance drops, indicating each component is useful for multi-defendant charge prediction.
Element Selector. In Table 3, the performance of EJudge+ (which utilizes annotated crime elements) surpasses that of EJudge∗ (which relies on extracted elements), by an average margin of 3.68%, showing that enhancing the quality of extracted crime elements can benefit charge prediction. To investigate the quality of extracted elements, we fine-tune LegalBERT (Chalkidis et al., 2020), LawFormer (Xiao et al., 2021), and RoBERTa (Cui et al., 2021) in the Element Selector module, and report exact match scores at both sentence- and tokenlevel in Table 6. The averaged exact match scores are 74.1% and 75.6% on sentence- and token-level respectively, indicating scope for improvement.
Category Selector and Rule Selector. Considering the charges of Fraud, Contract Fraud, and Extortion, we investigate the model’s ability to dis-
tinguish confusing charges. Fig. 7 shows that when removing the Category Selector (a) from EJudge∗ (w/o CS), it is hard to distinguish confusing charges (Fraud and Contract Fraud) in different categories. Removing the Rule Selector (b) makes it difficult to differentiate between Fraud and Extortion, which are in the same category. These observations verify the effectiveness of the Rule Selector and Category Selector for accurate charge prediction, by leveraging interpretable crime elements and legal rules.
Case Study and Interpretability Analysis. Fig. 8 presents a case with three defendants Zhu, Jiang, and Wang whose criminal facts intertwine and overlap. The existing methods, such as LegalBERT (Chalkidis et al., 2020), and EJudge*-ES can correctly predict the charges relevant to the whole case but fail to accurately assign the charges for each defendant. Our model EJudge* correctly predicts the charge for each defendant. Notably, our EJudge* method provides the extracted crime elements and matched legal rules, enhancing both prediction accuracy and model interpretability.
6 Conclusion
In this study, we introduce a new charge prediction benchmark called MUD that comprises multidefendant legal cases. We annotate the crime elements for each defendant, which benefits interpretable model development. Moreover, we
propose a crime-element-informed model named EJudge, which outperforms existing methods for multi-defendant charge prediction. In the future, we will work on more accurate crime element extraction for interpretable charge prediction.
Limitations
In this work, we aim to promote the development of LegalAI, providing a new benchmark with annotated crime elements to the community. The limitation of this work is that the proposed EJudge represents an initial exploration into incorporating crime elements for charge prediction. In EJudge, we integrate crime elements by directly concatenating implicit representations of extracted elements with fact descriptions. Although this straightforward method shows the advantage of crime elements, the potential to take full advantage of these constitutive elements is still under-explored.
Ethics Statement
Each case included in the MUD benchmark has been obtained from the Chinese government website, with sensitive information appropriately anonymized to protect privacy. During the document selection stage, we filter out any segments that might contain personal information, such as name, gender, age, address, and more. For the annotation task, we initially annotated a subset of cases ourselves, and then we established annota-
tor wages based on local standards to ensure fair compensation. It is important to note that while our work aims to alleviate the workload of legal professionals, our LegalAI model, like any other, may occasionally make mistakes. Therefore, we emphasize that our model should only serve as an additional auxiliary tool in the legal field. The ultimate decision-making should always depend on legal professionals.
Acknowledgments
This work is supported by National Natural Science Foundation of China under Grant No.62302287. The research reported in this article was supported by the Shanghai Committee of Science and Technology, China (Grant No.23ZR1423500). We thank the anonymous reviewers for their insightful comments.
References Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel
Preoţiuc-Pietro, and Vasileios Lampos. 2016. Predicting judicial decisions of the european court of human rights: A natural language processing perspective. PeerJ Computer Science, 2:e93.
Zhenwei An, Quzhe Huang, Cong Jiang, Yansong Feng, and Dongyan Zhao. 2022. Do charge prediction models learn legal theory? In Findings of the Association for Computational Linguistics: EMNLP 2022.
Claire Barale, Michael Rovatsos, and Nehal Bhuta. 2023. Automated refugee case analysis: A NLP pipeline for supporting legal practitioners. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 2992–3005.
Erik Cambria, Qian Liu, Sergio Decherchi, Frank Xing, and Kenneth Kwok. 2022. Senticnet 7: A commonsense-based neurosymbolic AI framework for explainable sentiment analysis. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pages 3829–3839. European Language Resources Association.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in english. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4317–4323. Association for Computational Linguistics.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of
law school. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics.
Jerome Alan Cohen. 1982. The criminal procedure law of the people’s republic of china. The Journal of Criminal Law and Criminology (1973-), 73(1):171– 203.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504–3514.
Yi Feng, Chuanyi Li, and Vincent Ng. 2022. Legal judgment prediction via event extraction with constraints. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 648–664, Dublin, Ireland. Association for Computational Linguistics.
Jidong Ge, Yunyun Huang, Xiaoyu Shen, Chuanyi Li, and Wei Hu. 2021. Learning fine-grained fact-article correspondence in legal cases. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3694–3706.
Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-shot charge prediction with discriminative legal attributes. In Proceedings of the 27th International Conference on Computational Linguistics, pages 487–498, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Xin Jiang, Hai Ye, Zhunchen Luo, WenHan Chao, and Wenjia Ma. 2018. Interpretable rationale augmented charge prediction system. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 146–151, Santa Fe, New Mexico. Association for Computational Linguistics.
Lin Li, Lingyun Zhao, Peiran Nai, and Xiaohui Tao. 2022a. Charge prediction modeling with interpretation enhancement driven by double-layer criminal system. World Wide Web, 25(1):381–400.
Lin Li, Lingyun Zhao, Peiran Nai, and Xiaohui Tao. 2022b. Charge prediction modeling with interpretation enhancement driven by double-layer criminal system. World Wide Web, 25(1):381–400.
Dugang Liu, Weihao Du, Lei Li, Weike Pan, and Zhong Ming. 2022. Augmenting legal judgment prediction with contrastive case relations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2658–2667, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to predict charges for criminal cases with legal basis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2727– 2736, Copenhagen, Denmark. Association for Computational Linguistics.
Chu Fei Luo, Rohan Bhambhoria, Samuel Dahan, and Xiaodan Zhu. 2023. Prototype-based interpretability for legal citation prediction. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 4883–4898.
Yougang Lyu, Jitai Hao, Zihan Wang, Kai Zhao, Shen Gao, Pengjie Ren, Zhumin Chen, Fang Wang, and Zhaochun Ren. 2023. Multi-defendant legal judgment prediction via hierarchical reasoning. pages 2198–2209. Association for Computational Linguistics.
Yougang Lyu, Zihan Wang, Zhaochun Ren, Pengjie Ren, Zhumin Chen, Xiaozhong Liu, Yujun Li, Hongsong Li, and Hongye Song. 2022. Improving legal judgment prediction through reinforced criminal element extraction. Information Processing & Management, 59(1):102780.
Rui Mao, Qian Liu, Kai He, Wei Li, and Erik Cambria. 2023. The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection. IEEE Trans. Affect. Comput.
Masha Medvedeva, Michel Vols, and Martijn Wieling. 2018. Judicial decisions of the european court of human rights: Looking into the crystal ball. In Proceedings of the conference on empirical legal studies, page 24.
Inderjeet Nair and Natwar Modani. 2023. Exploiting language characteristics for legal domain-specific language model pretraining. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2516–2526, Dubrovnik, Croatia. Association for Computational Linguistics.
OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.
Sicheng Pan, Tun Lu, Ning Gu, Huajuan Zhang, and Chunlin Xu. 2019. Charge prediction for multidefendant cases with multi-scale attention. In Computer Supported Cooperative Work and Social Computing: 14th CCF Conference, ChineseCSCW 2019,
Kunming, China, August 16–18, 2019, Revised Selected Papers 14, pages 766–777. Springer.
Shounak Paul, Pawan Goyal, and Saptarshi Ghosh. 2020. Automatic charge identification from facts: A few sentence-level charge annotations is all you need. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1011–1022, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Ruihao Shui, Yixin Cao, Xiang Wang, and Tat-Seng Chua. 2023. A comprehensive evaluation of large language models on legal judgment prediction. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 7337–7348.
Pengfei Wang, Yu Fan, Shuzi Niu, Ze Yang, Yongfeng Zhang, and Jiafeng Guo. 2019. Hierarchical matching network for crime classification. In proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pages 325–334.
Pengfei Wang, Ze Yang, Shuzi Niu, Yongfeng Zhang, Lei Zhang, and ShaoZhang Niu. 2018. Modeling dynamic pairwise attention for crime classification over legal articles. In the 41st international ACM SIGIR conference on research & development in information retrieval, pages 485–494.
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021. Lawformer: A pre-trained language model for chinese legal long documents. arXiv preprint arXiv:2105.03887.
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv:1807.02478.
Feng Yao, Chaojun Xiao, Xiaozhi Wang, Zhiyuan Liu, Lei Hou, Cunchao Tu, Juanzi Li, Yun Liu, Weixing Shen, and Maosong Sun. 2022. LEVEN: A largescale Chinese legal event detection dataset. In Findings of the Association for Computational Linguistics: ACL 2022, pages 183–201, Dublin, Ireland. Association for Computational Linguistics.
Wei Jie Yeo, Ranjan Satapathy, Rick Siow Mong Goh, and Erik Cambria. 2024. How interpretable are reasoning explanations from prompting large language models? CoRR, abs/2402.11863.
Linan Yue, Qi Liu, Binbin Jin, Han Wu, Kai Zhang, Yanqing An, Mingyue Cheng, Biao Yin, and Dayong Wu. 2021a. Neurjudge: A circumstance-aware neural framework for legal judgment prediction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 973–982.
Linan Yue, Qi Liu, Han Wu, Yanqing An, Li Wang, Senchao Yuan, and Dayong Wu. 2021b. Circumstances enhanced criminal court view generation. In The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1855–1859.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. GLM-130B: an open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
Zhuo Zhang, Xiangjing Hu, Jingyuan Zhang, Yating Zhang, Hui Wang, Lizhen Qu, and Zenglin Xu. 2023. FEDLEGAL: The first real-world federated learning benchmark for legal NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3492– 3507.
Jie Zhao, Ziyu Guan, Cai Xu, Wei Zhao, and Enze Chen. 2022a. Charge prediction by constitutive elements matching of crimes. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, pages 4517–4523.
Lili Zhao, Linan Yue, Yanqing An, Yuren Zhang, Jun Yu, Qi Liu, and Enhong Chen. 2022b. CPEE: civil case judgment prediction centering on the trial mode of essential elements. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management(CIKM), pages 2691–2700.
Haoxi Zhong, Yuzhong Wang, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):1250– 1257.
Haoxi Zhong, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2019. Open chinese language pretrained model zoo. Technical report.
A Errors in CAIL-2018
Commonly used dataset cail-2018 (Xiao et al., 2018) in LegalAI task relies on automatic extraction for annotation, which inevitably leads to some errors. As shown in Fig. 9, we list out some error types, i.e. Mislabeling and Non-existent label. Mislabeling refers to labeling the case with the wrong charge. Non-existent label means labeling the case with the charge that undefined in the Criminal Law of China.
B Existing Datasets
In LegalAI field, there are several wildly used datasets. To compare with our MUD, We summarize the existing datasets in Table 7. In the early stage, Most of the works, such as Xiao et al. (2018), and QAjudge (Zhong et al., 2020), focus on singledefendant charge prediction. Recently, Lyu et al. (2023) construct a new legal judgment prediction dataset, where each criminal case contains an average of 3.4 defendants. However, it mainly supports black-box model development. In our study,
we propose a new benchmark with high-quality crime element annotation, which can support interpretable model development.
C Category of Charges
Charges are arranged into different categories according to the Criminal Law of China, as shown in Fig. 12.
D Rule of Charges
The rule of charges expresses the conviction process, and the specific crime elements have corresponding formal terms in the rule. Table 8 shows their definitions according to the Criminal Law of China.
E Charge Prediction via Large Language Models
Large Language Models (LLMs) have shown remarkable performance in many domain-specific tasks, such as the sentiment analysis (Yeo et al., 2024; Mao et al., 2023; Cambria et al., 2022), and law domain (Shui et al., 2023). In this section, we conduct zero and few-shot experiments to evaluate LLMs on our benchmark MUD. We hope that these results can supplement previous research on the multi-defendant charge prediction capability of LLMs and serve as baselines for future studies.
GPT-4.0. A state-of-the-art commercial model from OpenAI (OpenAI, 2023). We choose the versions of GPT-4-0314.
GPT-3.5. To ensure reproducibility, we choose the GPT-3.5-turbo-0301 a Snapshot of GPT-3.5turbo (Ouyang et al., 2022) from March 1st, 2023.
GLM-130B. GLM-130B (Zeng et al., 2023) is an open bilingual dialog language model with 130 billion parameters and supports English and Chinese.
E.1 Experiment Settings
Following previous work (Shui et al., 2023), in the zero-shot setting, LLMs work following instructions without external law knowledge. In the fewshot setting, LLMs reason with prompts containing randomly selected (irrelevant) cases or similar cases retrieved by an information retrieval (IR) system. Fig. 10 shows the prompt template that is translated from Chinese. Since some fact descriptions are very long, we truncate them to 500 tokens. The Demo cases contain irrelevant cases or similar
cases. Specifically, for irrelevant cases, we randomly select several cases from the training dataset. For similar cases, we use the BM25 3 algorithm to measure the similarity between the query case and cases in the training dataset, and top-n cases are kept.
E.2 Anasysis and Discusion Fig. 11 shows the automatic evaluation result of LLMs.
For each LLM, few-shot baselines outperform zero-shot baselines, which conforms to our expectations. For few-shot baselines, LLMs prompting similar cases outperform LLMs prompting fixed cases, this is probably because the former import limit law knowledge compared to the the latter.
For GLM-130B, more similar cases or fixed cases in demonstrations are not always better. This
3https://pypi.org/project/rank-bm25/
is usually attributed to the noise introduced by irrelevant or false similar cases. GPT-4.0 and GPT-3.5 are more robust than GLM-130B.
It is slightly strange that LLMs perform worse than other baselines, such as Lawformer. This may be because LLMs can easily predict charges for the whole fact (LLM Ceiling in Fig. 11), but fail to align the charge for each defendant.

heading: Future Work
text: In the future, we will work on more accurate crime element extraction for interpretable charge prediction. Limitations
In this work, we aim to promote the development of LegalAI, providing a new benchmark with annotated crime elements to the community. The limitation of this work is that the proposed EJudge represents an initial exploration into incorporating crime elements for charge prediction. In EJudge, we integrate crime elements by directly concatenating implicit representations of extracted elements with fact descriptions. Although this straightforward method shows the advantage of crime elements, the potential to take full advantage of these constitutive elements is still under-explored