LJPCHECK: Functional Tests for Legal Judgment Prediction (2024)

Abstract: Legal Judgment Prediction (LJP) refers to the task of automatically predicting judgment results (e.g., charges, law articles and term of penalty) given the fact description of cases. While SOTA models have achieved high accuracy and F1 scores on public datasets, existing datasets fail to evaluate specific aspects of these models (e.g., legal fairness, which significantly impact their applications in real scenarios). Inspired by functional testing in software engineering, we introduce LJPCHECK, a suite of functional tests for LJP models, to comprehend LJP models’ behaviors and offer diagnostic insights. We illustrate the utility of LJPCHECK on five SOTA LJP models. Extensive experiments reveal vulnerabilities in these models, prompting an in-depth discussion into the underlying reasons of their shortcomings.

heading: 1 Introduction
text: Legal Judgment Prediction (LJP) aims to automatically predict judgment results given the fact description of a case. While English LJP focuses on violation prediction (Chalkidis et al., 2019; Feng et al., 2022a; Chalkidis et al., 2021), Chinese LJP focuses on charge prediction, law article prediction and term of penalty prediction (Feng et al., 2022b; Zhong et al., 2018). There also exists LJP work in other languages, such as French (Sulea et al., 2017) and Greman (Niklaus et al., 2021). Figure 1 illustrates an example of the three subtasks involved in Chinese LJP. Such LJP systems can assist legal professionals in case processing as well as offering low-cost consulting services to nonprofessionals.
How should the quality of LJP models be evaluated? Currently LJP models are evaluated using generic metrics such as F1 and accuracy on specific LJP subtasks such as charge prediction. However,
*Equal contribution. †Corresponding author. ‡Co-corresponding author.
a simple one-metric evaluation is far from sufficient for LJP since it does not provide an in-depth understanding of how a given model behaves, its weaknesses, and how to improve its performance in general. For instance, judicial fairness, which highlights that the parties involved in a case should be judged based on the case facts rather than demographic information such as gender and age, is one of the crucial considerations in the legal domain that cannot be evaluated using generic metrics. The reason is that a model that achieves a poor accuracy on, for instance, charge prediction can be fair as long as it does not make predictions based on race and gender, whereas a model that performs well on charge prediction can be unfair if it exploits race or gender information during its prediction process. While the use of one-metric evaluation is not uncommon in NLP and the shortcomings of such kind of evaluations are applicable to all NLP tasks, what is characteristic about LJP is its humancentered nature: understanding why a particular AI tool decided to charge someone with imprisonment is crucial for any real-world applications.
Motivated in part by the above observation, we propose a new approach to the analysis of LJP models that allows for targeted testing of specific aspects of LJP models, such as fairness. Our approach is inspired by functional testing in soft-
5878
ware engineering (Jin, 2009), which is a testing framework that assesses different functionalities of a given model by validating its output on sets of targeted test cases. Specifically, we introduce LJPCHECK, a suite of functional tests for LJP models that are sufficiently complex to allow better comprehension of a model’s behavior and offer diagnostic insights.
The contributions of our work are four-fold:

heading: A new approach to the analysis of LJP models.
text: To our knowledge, we are the first to employ functional testing in software engineering to augment the evaluation of LJP models to include specific aspects, such as fairness.
An easily extensible methodology. While our test suite is developed for evaluating Chinese LJP models (i.e., some of the test cases are developed with specific Chinese law articles in mind), the same methodology, which we borrow from Ribeiro et al. (2020), can naturally be applied to the development of test suites for other jurisdictions. More broadly, this methodology is task-agnostic and can therefore be applied to other NLP tasks.
A new test suite. Our test suite1, which spans over 10,800 examples, provides a valuable resource for future research into the LJP direction.
An analysis of SOTA LJP models. While the vast majority of recent work on LJP has focused on number crunching, where the goal is to improve SOTA results, we focus on understanding the strengths and weaknesses of SOTA LJP models. We believe that our work is also valuable for the broader "analysis-focused research" because it shows how to apply the methodology of Ribeiro et al. (2020) to a new domain.

heading: 2 Related Work
text: Much recent work on LJP has focused on applying deep learning and proposing different kinds of neural networks to solve the subtasks in LJP, with the primary goal of improving performance numbers. For instance, some researchers (Yue et al., 2021; Feng et al., 2022b) extract key elements from facts to improve both accuracy and interpretability, while others employ multi-task learning (Zhong et al., 2018) to simultaneously predict multiple judgments. With the development of Pretrained Language Models (PLMs), legal PLMs like
1The test suite is available at here (link).
Lawformer (Xiao et al., 2021) pretrained on legalrelated data is used to solve different legal tasks (including LJP) in a unified framework.
Nevertheless, given the significant impact of legal decisions on individuals, the evaluation of AI systems in the legal domain is getting attention (Chalkidis et al., 2023). Gumusel et al. (2022) train a Word2Vec model on a US case law dataset and discover that NLP methods make undesirable distinctions between legally equivalent entities that vary only by race, which leads to race bias. Chalkidis et al. (2022) propose a multi-lingual benchmark suite, based on four legal datasets across four jurisdictions, to test model biases in attributes like demographics and court regions. An et al. (2022) design a framework to evaluate whether existing charge prediction models are trustworthy with 3 proposed principles that they believe models should follow. Wang et al. (2021) propose a metric to measure fairness in the legal domain, highlighting that both regional and gender biases exist in the legal systems. In this paper, we provide an evaluation framework tailored to LJP systems. Unlike the above works which only focus on one or a few aspects, we give broader insights with respect to 23 different functionalities. Further, our test framework can be applied to other tasks whereas the above works are tied to their own tasks.

heading: 3 LJPCHECK
text: Functional testing in software engineering refers to the practice of testing specific functionalities of software applications by providing sample inputs and verifying whether the outputs are as expected. While prediction accuracy is a key consideration to LJP models, there are other aspects that are essential but overlooked when evaluating models. We define these considerable aspects as functionalities in this paper. For model-agnostic testing, we apply the principle of decoupling testing from implementation by treating models as black boxes, which allows different models to be tested without knowing their internal structures. Next, we detail the functionalities to be tested (e.g. Time Fairness), the test types used (e.g. Minimum Functionality Test), and how we generate tests.

heading: 3.1 Selecting Functionalities
text: To generate the list of 23 functionalities, we review LJP works and interview legal practitioners (judges and lawyers) about their expectations on LJP.
Literature review We identify challenges and weaknesses that have been mentioned in existing LJP research papers, surveys and empirical studies published in the premier publication venues in NLP, legal-tech and artificial intelligence communities (e.g. ACL, EMNLP, NAACL, COLING, IJCAI, SIGIR, AAAI and ICAIL) from 2017 to 2023. For instance, Wang et al. (2021) introduce the gender bias challenge in their paper specifying that gender differences lead to different prediction results. We thus define the functionality related to gender fairness. Papers on multiple languages (e.g., Chinese, English and French) are included when reviewing. While jurisdiction differs across languages, we aim at obtaining a broader view about the functionalities related to LJP. We also review the papers about legal theories (e.g., Wachter et al. (2021)) to include functionalities from the perspective of legal theory researchers. For instance, we include Discretionary Sentencing Factor Functionalities (e.g., surrendering voluntarily results in a reduced penalty) since they are principles established within the Chinese legal system.
Practitioner Interview By interviewing practitioners we aim to (1) verify our understanding from the literature review and (2) obtain practitioners’ opinions and expectations on LJP.
We sent invitations to legal professionals who have used LJP systems. In the end, seven agreed to join our interviews, including one judge, two corporate legal affairs, two lawyers and two law school students with experience years varying from
1 to 7. We also extended invitations to 3 nonprofessionals who have experience in legal disputes or have sought consultation through LJP systems. Interviews are conducted face-to-face or over the phone for 60-80 minutes. Each interview is composed of 3 parts: 1) collecting demographic information and work experience of the interviewees; 2) asking open questions about the challenges and weaknesses they met when using LJP systems (e.g., What errors occurred when you used LJP tools? What challenges do you think exist when applying LJP systems in real-world scenarios?); and 3) discussing their expectations on LJP (e.g., What capabilities should LJP systems possess?).
After the interviews were done, we transformed the interview findings into functionalities. For instance, given that interviewees mentioned the challenge of complex cases involving multiple charges, we define the multi-charge functionality 1.
Selecting functionalities Next, we select the functionalities from our initial list following two criteria as not all functionalities are applicable. First, we set the constraint within the scope of the Chinese legal system as we focus on testing Chinese LJP models. However, certain functionalities such as fairness are also applicable to other jurisdictions. Then, we include functionalities for which we can construct test cases with gold labels. For instance, interpretabilities are significant in the legal domain and are raised by several interviewees.
1Due to privacy concerns, we will release the interview records after getting the participants’ consent.
In the field of legal AI, model interpretability is typically demonstrated in two mechanisms: a) generating or extracting explicit "explanations" and b) aligning models with human legal reasoning and theories. Regarding the first approach, since most LJP models do not directly generate explanations and a "gold-standard" explanation cannot be determined, we leave out this aspect as it is challenging to design black-box test cases for it.

heading: 3.2 Functionalities of LJP Models
text: We ultimately identify 23 fine-grained functionalities (See Table 1), which can be categorized into four classes, as described below. All functionalities stem from the standpoint of law, embodying legal principles that must be adhered to.
Exemption from Criminal Penalty According to Article 37 of the Criminal Law of the People’s Republic of China, If the severity of crime is considered minor and does not meet the threshold for penalty, criminals may be exempted from penalty. In this case, a criminal may be convicted of a crime without being sentenced to any of the following criminal penalties: surveillance, detention, fixed-term imprisonment, life imprisonment and death penalty. For instance, criminals who are accomplices may be exempted from penalty as accomplices are subject to a reduced severity of crime. Unlike the presumption of innocence discussed in An et al. (2022), exemption from criminal penalties indicates a guilty verdict without penalty, whereas the presumption of innocence indicates that no legal crimes were committed. Thus, LJP models should possess the capability to predict nonpenalties for cases with charge conviction. (F1).
Legal Fairness Legal fairness necessitates that Everyone is treated equally before the law. Thus, we propose a set of fairness functionalities, which means that individuals, regardless of their race, gender or other features unrelated to the law, should receive equal treatment. Specifically, we test the following features: (1) demographic features, including name, gender, ethnicity, education degree and their combination (F2-F6). and (2) spatiotemporal features, including the crime time, the crime location and their combination (F7-F9).
Complex Case Handling According to legal practitioners, the practicality of an LJP model is significantly impacted by its ability to handle complex cases. One kind of complexity concerns multi-
charge cases, i.e., the defendant may violate multiple charges simultaneously. LJP systems should accurately predict multiple charges. Thus, we include the multi-charge functionality (F10). Another kind of complexity concerns the differences between felony cases and misdemeanor cases as felony cases typically involve more complex details than misdemeanor cases. Furthermore, felony cases involve more severe penalties (maybe over 10 years of imprisonment). We are more concerned with the performance of LJP systems in felony cases than in misdemeanor cases as wrong judgment predictions for felony cases can lead to more significant losses for parties involved. Thus, we define the misdemeanor case functionality (F11) and the felony case functionality (F12).
Key Element Recognition One significant judgment criterion in the Chinese legal system is to follow the legal logic that defined in the law instead of individuals’ experience. Thus, LJP systems should follow the legal logic instead of unverified experience learned from datasets. Further, it would offer some interpretability in a round-way if we can verify that LJP systems follow the legal logic, as discussed in 3.1.
The legal logic in China involves identifying key elements. Each law article stipulates specific key elements. So, when a case contains these key elements, it triggers the corresponding law articles and charges, determining the applicable penalties. LJP systems should follow this paradigm, i.e., first identifying the key elements and then predicting the corresponding judgments. Key elements can be categorized into two classes. Key Factors refer to the explicitly defined elements of a crime in the law. For instance, Article 264 of the Criminal Law of the People’s Republic of China stipulates that "Stealing public or private property constitutes the crime of theft. If the value of the stolen property is fairly large, the offender may be sentenced to a fixed-term imprisonment of not more than three years. If the value of the stolen property is large, the offender may be sentenced to a fixed-term imprisonment of not less than three years but not more than ten years". Here, stealing triggers the crime of theft, and the value of the stole property measures the final penalty. Both of them are Key Factors, which typically establish charges and the range of penalties. In contrast, Discretionary Sentencing Factors refer to flexible considerations on penalty besides Key Factors. Discretionary Sentencing Fac-
tors include voluntary surrender, mental illness, etc. For instance, if someone commits a crime with a mental illness, they may get a lighter punishment than a mentally sound person.
To test whether LJP systems can identify key elements, we derive two kinds of functionalities with respect to Key Factor and Discretionary Sentencing Factor. As Key Factors are relevant to the types of cases and it is impossible to test all types, we select six case types for testing, including Theft, Traffic Offense, Passive Bribery, Intentional Injury, Defraud and Robbery (F13-F18). These six types are chosen because there are subtle differences between them and others, which often cause models to make incorrect judgments and thereby make our tests more challenging. For instance, both illegally entering and theft can involve the fact of unlawfully entering a house. If LJP models only identify the fact of entering a house, the crime of illegally entering may be wrongly predicted. For Discretionary Sentencing Factors, we include voluntary surrender (F19), criminal attempt (F20), whether receiving forgiveness by victims (F21) and mental illness (F22) and recidivism (F23). These factors are chosen because they can be applied to any case type.

heading: 3.3 Test Types
text: We adopt four test types (i.e. Minimum Functionality test (MFT), Invariance test (INV), Directional Expectation test (DIR), and Real Case Test (RCT)) to test the 23 functionalities (Ribeiro et al., 2020). Table 2 illustrates examples of these test types with the corresponding functionalities. All functionalities and their test types can be found in Table 1.
INV and DIR are both inspired by software metamorphic tests (Segura et al., 2016): the former makes label-preserving perturbations to inputs
with the expectation that outputs remain consistent, while the latter perturbs the input and anticipates a directional change in the output. For instance, to test Time Fairness with INV, removing or replacing the time information in the fact description should make no difference to the prediction. However, in Test Case 3 in Table 2, the predicted term of penalty decreases from 65 months to 59 months while the expectation is the output should remain the same as before. INV is used to test F2-F10 as LJP models should not show any prediction change in term of fairness. As for DIR, for example, we expect that the term of penalty will decrease if we add the sentence involving voluntary surrender in the fact description (Test Case 2 in Table 2). DIR is used to test F19-F23 as Discretionary Sentencing Factors have impacts on the term of penalty and charges, and changes are expected. MFT, which is derived from unit tests in software engineering, uses simple and focused test cases to check a behavior within a functionality. Taking the functionality Key Factor as an example, we assess whether a model can make correct predictions for a theft case only with key factors (Test Case 1 in Table 2). We expect that the model will predict a term of penalty that falls within the range of 0 to 36 months. MFT is used to test F13-18 as we need to construct different test cases to determine whether LJP models can identify Key Factors. Besides, some functionalities can be tested directly using ground-truth cases. We refer to this kind of tests as RCT, which is used to test (F1 and F10-F12). For instance, we collect multi-charge cases to construct test cases for testing the Multi-charge functionality.

heading: 3.4 Generating Test Cases
text: Two methods are used to generate desired test cases: (1) starting from scratch and (2) perturbing exist-
ing data. Using the first method, we can create a small number of high quality test cases, which is typically implemented by filling the templates. The second method generates test cases based on existing data but poses a challenge in defining perturbation functions. Next, we will detail how these two methods are implemented 1.
Templates We generate test cases based on templates for testing the Key Factor functionalities (F13-F18). In Table 2, we generate "Defendant Wang sneaked into a house and stole wallets. The stolen items were worth 2,600 RMB." from the template "Defendant {Name} sneaked into a house and {Trigger} {Item}. The stolen items were worth {Value}.", where {Name}, {Trigger}, {Item} and {Value} are selected from a predefined list. We generate multiple templates for each Key Factor Functionality to improve diversity. Each template is designed to match with case types (e.g., Theft). To be specific, each template contains all the necessary key factors that are stipulated in law. For instance, theft-related templates must contain {Trigger}, {Item} and {Value}. We seek help from legal experts to derive these key factors from law articles. We then construct different contexts containing these key factors and add different confusing semantics to the templates to make the test cases challenging. As Example 1 in Table 2 shows, we add the phrase sneaked into a house to mislead LJP models to predict Illegally Entering instead of Theft, as Illegally Entering and Theft are both related to the fact of illegally entering. Here, we ask legal experts to identify the confusing semantics with respect to each case type. Regarding the potential candidate values for the key factors, we ask legal experts to identify options and establish their associations with penalty intervals (e.g., if the value of the stolen property is less than 10,000 RMB, the associated penalty is 0-36 months of prison). Gold charges are easily obtained for templates. For gold term of penalty, we identify penalty intervals given the associations with options.
Perturbing Existing Data We generate test cases for INV and DIR by perturbing existing datasets through adding, replacing or removing key information. For example, to test a model’s ability to identify the Voluntary Surrender (one of the Discretionary Sentencing Factors) functional-
1See Appendix C for details on functionalities with the corresponding test types/examples and how test cases are generated.
ity, we use regular expressions to extract and remove surrender-related words, such as "surrender", "active alarm" and "cooperation with accident investigation", from a predefined list created with the help of a legal expert. We then obtain a pair of test cases, the original case and a perturbed case without factors of voluntary surrender. The penalty term of the original case is supposed to be shorter than that of the perturbed one.
Validating Test Cases To evaluate the quality of the generated test cases, we recruit 3 legal professionals (one judge and two lawyers) and give them a one-hour tutorial on evaluation criteria. Our evaluation criteria include two aspects. The first involves whether the generated fact description is consistent with the generation method. For instance, the perturbation should successfully perturb the target words. The second involves whether the generated fact description is consistent with the expected prediction result. For instance, the generated fact description should lead to an increase of penalty. If a test case meets both criteria, it is positive; otherwise, it is negative. During evaluation, the two lawyers are required to rate test cases independently. The positive ratios are 89.5% and 91.3% respectively. Inter-annotator agreement Cohen’s Kappa score (McHugh, 2012) is 0.712, which indicates substantial agreement. Finally, each case of disagreement is resolved by the judge.
In total, we collect 10,800 test cases (See Table 1 for statistics). The original facts and gold labels are collected from China Judgment Online2.

heading: 4 Experiments
text: We test five SOTA LJP models using LJPCHECK.

heading: 4.1 Models
text: We find that SOTA models achieve different performance across papers. Thus, we conduct preliminary experiments under the same experimental setup to select reliable SOTA models. We evaluate models that are reported to achieve SOTA performance in recent years (2018-2023) on three Chinese LJP tasks using the CAIL-small dataset (Xiao et al., 2018) 3. Finally, we obtain five SOTA models, i.e., TOPJUDGE (TOP) (Zhong et al., 2018), Lawformer (LAW) (Xiao et al., 2021), NeurJudge (Neur) (Yue et al., 2021), Dependent-T5
2http://wenshu.court.gov.cn/ 3See Appendix A for preliminary experiment setup de-
tails.
(T5) (Huang et al., 2021), and Rformer (R) (Dong and Niu, 2021). Table 3 shows their performance on CAIL-small.

heading: 4.2 Functional Test Setup
text: We ensure there is no overlap between our collected test cases and CAIL-small, and test the five SOTA models after training on CAIL-small in the preliminary experiment. We test the SOTA models on charge and term of penalty prediction tasks. Model performance on functionalities is evaluated by failure rate. For MFT and RCT, a "failure" in the charge prediction task occurs when the prediction does not match the gold label. In penalty prediction, any prediction that falls outside a predefined interval is considered a "failure". For INV and DIR, a "failure" is defined by determining the variation between the predicted and expected values. See Appendix D for details on the definition of failure rate.

heading: 4.2.1 Performance Analysis
text: Results are shown in Table 4. All models exhibit weaknesses on F1. Even Rformer, which has the highest accuracy score on CAIL-small, has a failure rate of up to 78.3%. A plausible reason is that the training set of CAIL-small only contains a limited number of such samples (6.4%), leading to a bias, i.e., a conviction of a charge results in a penalty.
Multi-charge cases (F10) also pose a challenge to all models. When collecting our test cases, we select several combinations of charges that don’t exist in the training set of CAIL-small to add challenges. Dependent-T5, which may benefit from leveraging semantics of charges rather than treating them as atomic labels, gets the lowest failure rate (33.0% | 58.3%). Other models do not have any mechanism designed for multi-charge cases.
For fairness functionalities (F2-F9), NeurJudge performs the worst on penalty prediction on all fairness functionalities, while Dependent-T5 appears
to be the least sensitive to fairness data perturbation. Generally, each model’s failure rate significantly differs across fairness functionalities, making it difficult to conclude any obvious bias.
For F11 and F12, we observe that models perform better on misdemeanor cases than felony cases w.r.t. charge prediction. One reason is that felony cases are typically more complicated than demeanor cases. However, we find that models perform better on felony cases than misdemeanor cases w.r.t. term of penalty prediction. This might be because judges handle felony cases with greater caution compared to misdemeanor cases, resulting in fewer disagreements on penalties.
Rformer outperforms other models significantly in all three subtasks on CAIL-small. However, it shows high failure rates on F13-F18, suggesting weaknesses in following the legal logic. Note Rformer’s failure rate of charge prediction is the highest in identifying key factors for Passive Bribery cases (F15). An error analysis reveals that the error samples tend to be predicted as Bribery Crime of Non-state Staffs (confused with Passive Bribery). It demonstrates that while Rformer adopts strategies like global consistency graph to alleviate the problem of confusing charges, these mechanisms are still far from addressing the issue. NeurJudge and Dependent-T5, which contain confusing alleviation mechanism, also exhibit such problems.
Nevertheless, NeurJudge achieves the best performance on F19-F23, possibly due to its circumstances of crime aware fact separation (CCFS) method. This method separates the facts into parts, one of which is relevant to Discretionary Sentencing Factors.

heading: 4.2.2 Improvement Suggestions
text: We further test two Rformer variants: one replacing its backbone from bert-base-chinese to bert-largechinese (denoted as R-L), and another trained on
the entire CAIL-big dataset (denoted as R-M, M stands for more data).
As shown in Table 4, R-M outperforms others on F1 and F10, potentially benefitting from more training data, which is nearly eight times than that of Rformer and R-L. R-M learns more comprehensive and robust representations, which also explains why it performs better on F13-F18 and F2-F9 as robust representations are less sensitive to misleading semantics. However, R-M is not better at identifying Discretionary Sentencing Factors(F19-F23). In this case, simply increasing the amount of data might not be sufficient to improve these functionalities. However, R-L shows improvement as it utilizes a larger BERT. It suggests that a larger encoder is necessary to embed semantics.

heading: 4.3 Discussion
text: Building upon the insights gained from our results and analysis, we now delve into a broader discussion of various phenomena and implications.
One peculiar thing is that the performance of charge prediction and term of penalty prediction in the same functionality does not exhibit a clear correlation. The best performance in charge prediction may coincide with the worst performance in penalty prediction (e.g. Rformer on F15). Many models aim to improve charge prediction but strug-
gle to enhance penalty accuracy as term of penalty is impacted by complex factors. This also suggests that models do not effectively leverage the correlations among subtasks.
It is also noteworthy that models that perform better on DIR tests (F19-F23) show poorer performance on INV tests (F2-F9). The sensitivity to fine-grained sentencing circumstances may be unavoidably accompanied by a similar sensitivity to other unrelated information. This implies that models may not accurately identify and understand the criteria but instead engages in a fine-grained learning of all semantic information.
Overall, each model has its own strengths and weakness, and none can pass all tests. For practical applications, these weaknesses can be vital and cannot be overlooked. As a suite of blackbox tests, LJPCHECK only partially uncovers a model’s shortcomings and offers indirect source of these shortcomings. Specific reasons for these failures and improvement methods require further investigation by the users. Nonetheless, we can still surmise that a significant portion of these reasons stems from the lack of training data and deficiencies in the model structure. If it is the biased training data, models could be improved by targeted data augmentation (Gardner et al., 2020). LJPCHECK users could, for instance, generate or
select additional training cases to resemble test cases from functional tests where their model performs poorly. Note, however, that introducing additional data might introduce unforeseen biases.

heading: 5 Conclusion
text: We proposed using functional testing for a more comprehensive evaluation of LJP models. We introduced LJPCHECK, which contains 23 functionalities targeted at testing different aspects of LJP systems. Five SOTA LJP models are tested and weaknesses are revealed. In future work, we aim to explore testing model interpretability given its critical importance in LJP. We also plan to incorporate the results of law article prediction task, which were not utilized in our current tests, to enhance the completeness and reliability of our tests.
Ethical Statements
The real-world data used in this paper are all from publicly available resources and personal information (e.g. name, plate number, etc.) has been anonymized before use. Legal judgment prediction is not researched to replace judges and make final court decision solely by machine. Our motivation is to make the evaluation metrics more comprehensive rather than replace any of them. We do not intend to prove that LJP models have the potential of adjudging any cases without human verification, even if they achieve "perfect" scores on all functionalities.
Limitations
The current version of LJPCHECK covers only a small fraction of functionalities of LJP models which cannot entirely represents practitioners’ expectations. For some functionalities, we only test limited charges, leaving a large amount of other charges to be further investigated. Fortunately, LJPCHECK is extensible that users can design their own test cases following our approaches. Regarding this, the test case generation methods in this paper are not diverse enough and can possibly lead to simplification of certain cases. Besides, some functionalities are limited to Chinese and might not be applicable to other jurisdictions. However, it doesn’t mean that LJPCHECK lacks generalizability but instead, it highlights that individuals can formulate dedicated functionalities.

heading: Acknowledgments
text: We would like to thank Xilu Chen, Xiaoyu Shen and Jiwei Shang for their sharing and help at the early stage of this work. We also appreciate the valuable comments from all the reviewers. This work was supported by the Overseas Project (KFKT2023A07, KFKT2024A07) and the General Innovation Project (ZZKT2024B02) of the National Key Laboratory for Novel Software Technology in China, the National Key R&D Program of China (2016YFC0800803) and the Cooperation Fund of NJU-Jinqiao Information Joint Research Center for Smart Justice and Artificial Intelligence.

heading: A Experimental Setup
text: We cast the law article and charge prediction tasks as multi-label classification tasks instead of singlelabel classification tasks as we want to test the multi-charge functionality. We cast the term of penalty prediction as a regression task instead of a classification task as we want to measure the degrees of increases and decreases. We keep hyperparameters, such as batch size, learning rate,
training epochs and random seed, same to those in the original papers. We conduct our experiments on 4*Tesla V100 GPUs.

heading: B Functionality Description
text: All functionalities, their detailed descriptions and applicable languages are shown in Table 5.

heading: C Generation Method and Example
text: Details of generation methods and examples are shown in Table 6.

heading: D Functional Test Setup
text: When testing on the term of penalty prediction task, the definition of ’failure’ differs depending on the functionality and test type, shown in Table 7.
ID Type # Generating Method Example F1 RCT 200 Collecting real criminal
cases containing the element of ’exemption from punishment’ from China Judgment Online (CJO: wenshu.court.gov.cn) Defendant Niu XX, in November 2012, working at a school bus office, accepted a 20,000 yuan rebate from two companies during a tire purchase. Niu used this money for personal gain. After the event, Niu voluntarily surrendered and returned all 2,000 yuan of the taken money. Niu is exempt from criminal punishment for taking bribes.
F2 INV 500 Collecting real cases from CJO then remove (replace actual name with indefinite designation X) or keep the name of all parties using named entity tools. Defendant Zhang XX conspired to defraud Wang XX, a student of Fengnan No.1 Middle School. Zhang took Wang to the rented room in Tienan Building and threatened him with words. and used Wang XX’s mobile phone to call his father and ask for RMB 30,000. Expected: Invariant F3 INV 500 Collecting real cases from CJO then remove or add the gender of the defendant using regular expression.
Defendant Kou XX, female. Defendant Kou, driving a small car, collided with a motorcycle driven by He X, who was speeding in the same direction. Motorcycle rider He suffered bone marrow injuries in his neck and died after rescue efforts were unsuccessful. Expected: Invariant
F4 INV 500 Collecting real cases from CJO then remove or add the ethnicity (e.g. Han) of the defendant using regular expression. Defendant Kou XX, Han nationality. Defendant Kou, driving a small car, collided with a motorcycle driven by He X, who was speeding in the same direction. Motorcycle rider He suffered bone marrow injuries in his neck and died after rescue efforts were unsuccessful. Expected: Invariant F5 INV 500 Collecting real cases from CJO then remove or add the education level (e.g. junior high school level) of the defendant using regular expressions. Defendant Kou XX, junior high school education. Defendant Kou, driving a small car, collided with a motorcycle driven by He X, who was speeding in the same direction. Motorcycle rider He suffered bone marrow injuries in his neck and died after rescue efforts were unsuccessful. Expected: Invariant F6 INV 500 Collecting real cases from CJO then remove or add the gender, ethnicity and education level of the defendant all at once using regular expression. Defendant Kou XX, femal, Han nationality, junior high school education. Defendant Kou, driving a small car, collided with a motorcycle driven by He X, who was speeding in the same direction. Motorcycle rider He suffered bone marrow injuries in his neck and died after rescue efforts were unsuccessful. Expected: Invariant F7 INV 500 Collecting real cases from CJO then remove (i.e. replace the actual time with indefinite designation of x) or keep the time in case description using regular expression. Defendant Zhang XX conspired to defraud Wang x, a student of Fengnan No.1 Middle School, who was surfing the Internet in Baile Internet Cafe in Fengnan District at around 23:00 on January 21, 2013 XX, and took him to the rented room in Tienan Building, Fengnan District, threatened him with words, and used Wang mou’s mobile phone to call his father and ask for RMB 30,000. Expected: Invariant F8 INV 500 Collecting real cases from CJO then remove (replace the actual location with indefinite designation of ’X place’) or keep the location in fact description with named entity tools. Defendant Zhang XX conspired in advance to defraud Wang X, a student of Fengnan District No.1 Middle XX School, who was surfing the Internet in Baile XX Internet Cafe in Fengnan XX District at around 23:00 on January 21, 2013, and take him to the rented room in Tienan XX Building, Fengnan XX District, threatened him with words, and used Wang X’s mobile phone to call his father and ask for RMB 30,000. Expected: Invariant F9 INV 500 Collecting real cases from CJO then remove or keep the time and location in fact description all at once.
Defendant Zhang XX conspired to defraud Wang XX, a student of Fengnan No.1 Middle XX School, who was in Baile XX Internet Cafe at around 23:00 on January 21, 2013XX, and take him to the rented room in Tienan XX Building, threatened him with words, and used Wang’s mobile phone to call his father and ask for RMB 30,000. Expected: Invariant
F10 RCT 200 Collecting real criminal cases from CJO where the defendant is convicted of multiple charges (i.e. the charge label consists more than one item). Defendant Xu X collided with a taxi driven by Wang on the road. Later, the defendant Xu caught up with Wang, they got out of the car and quarreled and fought. During the tussle, the defendant Xu injured Wang’s nose with his fist. ... Xu was charged with the crime of traffic offense and the crime of intentional injury. F11 RCT 550 Collecting real criminal cases from CJO where the term of penalty is less than 3 years.
The defendant Fan, who was drunk and driving a small car, hit the guardrail, damaging both. After identification, the alcohol content in the blood of him is 210.2mg/100ml... The defendant Fan committed the crime of dangerous driving and was sentenced to two months of detention, with a fine of RMB 2000.
Continued on next page
ID Type # Generating Method Example F12 RCT 550 Collecting real criminal
cases from CJO where the term of penalty is more than 3 years. The defendant Gong collided with Xu’s electric bicycle while driving a truck in the rain. After, Gong drove over Xu’s body and fled the scene. Xu died the same day after unsuccessful hospital treatment... The defendant Gong committed intentional homicide and was sentenced to fifteen years in prison.
F13 MFT 500 Using multiple templates including confusing-chargeexpressions. e.g., "The defendant {NAME} sneaked in to the victim’s house {Theft-Trigger} {item}. The stolen items were worth {Value}" contains information that may lead to predicting illegally entering. The defendant Wang sneaked into the victim’s house and stole two wallets and five electric devices. The stolen items were worth 7,800 RMB. F14 MFT 500 Using multiple templates including confusing-chargeexpressions. e.g., "The defendant {NAME} was driving drunk and {AccidentTrigger}, {Result}" contains information that may lead to predicting dangerous driving charge. The defendant Ye ming was driving drunk and collided with the bicycle on the pavement ridden by the victim and crashed into the roadside, resulting in the death of one person. F15 MFT 500 Using multiple templates including confusing-chargeexpressions. e.g., "The defendant {NAME} took advantage of his job position, {Bribery-Trigger} {Value} from others and put into an official account. He later withdrew the money for personal use." contains information that may lead to predicting misappropriating public funds The defendant Ma ming took advantage of his job position, illegally accepted bribes for a total of 20,000 RMB. from others and put into an official account. He later withdrew the money for personal use. F16 MFT 500 Using multiple templates including confusing-chargeexpressions. e.g., ”The defendant {NAME} chased and insulted the victim, when the victim started to argue with him, he {InjuryTrigger}, After medical appraisal, the degree of injury to the victim is InjuryDegree." contains information that may lead to predicting creating disturbances The defendant Yuan Yihang chased and insulted the victim, when the victim started to argue with him, he picked up a stone from the roadside and hit the victim in the head. After medical appraisal, the degree of injury to the victim is minor injury of Class II. F17 MFT 500 Using multiple templates including confusing-chargeexpressions. e.g., "The defendant {NAME} used the fake patent certificate he created to attract investment from others, resulting in a total amount of {Value} of fraud." contains information that may lead to predicting counterfeiting the patent The defendant Huang Yichao used the fake patent certificate he created to attract investment from others, resulting in a total amount of 100,000 RMB of fraud. Continued on next page
ID Name Test Type Definition of Failure on the task ofCharge Prediction Term of Penalty Prediction
F1 Exemption RCT - Failure when the predicted term of penalty is not 0. F2-F9 Fairness INV Failure when
the predicted charge is not the same with the actual one.
Failure when the predicted term changes after removals or additions of certain attribute values.
F10 Multi-charge RCT Failure when the predicted charge is not the same with the actual one. Failure when the predicted term falls outside the [0.75, 1.25] range of the actual value.
F11-F12 Felony and Misdemeanor
RCT Failure when the predicted charge is not the same with the actual one. Failure when the predicted term falls outside the [0.75, 1.25] range of the actual value.
F13 Key Factors of
Theft
MFT Failure when the predicted charge is not the same with the actual one. Failure when the predicted term falls outside the sentencing range specified in the law articles. According to [Law Article 264], the prison term intervals for different severity are: {"relatively large amount": (0, 3] years}, { "huge amount": [3, 10) years}, {"especially huge amount or other especially serious circumstances": [10,+∞) years}, where the degrees like "large" and "huge" are defined with specific numerical range in the law.
F14 Key Factors of
Traffic Offence
MFT Failure when the predicted charge is not the same with the actual one. Failure when the predicted term falls outside the sentencing range specified in the law articles. According to [Law Article 133], the prison term intervals for different severity are: {"serious injuries or deaths or heavy losses of public or private property": (0, 3] years}, { "run away from the spot or other especially flagrant circumstance": [3, 7) years}, {"escape results in death of another person": [7,+∞) years}.
F15 Key Factors of
Passive Bribery
MFT Failure when the predicted charge is not the same with the actual one. Failure when the predicted term falls outside the sentencing range specified in the law articles. According to [Law Article 383], the prison term intervals for different severity are: {"not less than 5,000 yuan": (0, 2] years}, {"not less than 5,000 but less than 50,000) yuan": [1, 7) years}, {"not less than 50,000 but less than 100,000 yuan": [5,+∞) years}, {"not less than 100,000 yuan": [10,+∞) years}.
F16 Key Factors of
Intentional Injury
MFT Failure when the predicted charge is not the same with the actual one. Failure when the predicted term falls outside the sentencing range specified in the law articles. According to [Law Article 234], the prison term intervals for different severity are: {"basic": (0, 3] years}, { "severe injury": [3, 10) years}, {"death, resorting to especially cruel means, reducing the person to utter disability": [10,+∞) years}, where the degrees like "severe" are defined with specific injury level in the law.
F17 Key Factors of
Defraud
MFT Failure when the predicted charge is not the same with the actual one. Failure when the predicted term falls outside the sentencing range specified in the law articles. According to [Law Article 266], the prison term intervals for different severity are: {"relatively large amount": (0, 3] years}, { "huge amount or other serious circumstances": [3, 10) years}, {"especially huge amount or other especially serious circumstances": [10,+∞) years}, where the degrees like "large" and "huge" are defined with specific numerical range in the law.
F18 Key Factors of Rob
MFT Failure when the predicted charge is not the same with the actual one. Failure when the predicted term falls outside the sentencing range specified in the law articles. According to [Law Article 263], the prison term intervals for different severity are: {"robs public or private property by violence, coercion or other methods": [3, 10) years}, {"intruding residence; robbing on board the means of public transportation; robbing a bank or any other banking institution; repeatedly robbery or robbing a huge sum of money; causing serious injury or death, impersonating a serviceman or policeman in robbing; robbing with a gun; robbing military materials or the materials for emergency rescue, disaster relief or social relief": [10,+∞) years}.
Continued on next page

heading: Future Work
text: In future work, we aim to explore testing model interpretability given its critical importance in LJP. We also plan to incorporate the results of law article prediction task, which were not utilized in our current tests, to enhance the completeness and reliability of our tests