Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoning
Authors: Yougang Lyu
Jitai Hao
Zihan Wang
Kai Zhao
Shen Gao
Pengjie Ren
Zhumin Chen
Fang Wang
Zhaochun Ren
Venue: EMNLP 2023 Findings (2023)
Keywords: legal text mining
legal judgment prediction
legal reasoning
TL;DR: We introduce the Hierarchical Reasoning Network (HRN), a multi-defendant LJP method addressing complex case predictions, and present MultiLJP, a real-world dataset for advancing research in this field.

Abstract: Multiple defendants in a criminal fact description generally exhibit complex interactions, and cannot be well handled by existing Legal Judgment Prediction (LJP) methods which focus on predicting judgment results (e.g., law articles, charges, and terms of penalty) for single-defendant cases. To address this problem, we propose the task of multi-defendant LJP, which aims to automatically predict the judgment results for each defendant of multi-defendant cases. Two challenges arise with the task of multi-defendant LJP: (1) indistinguishable judgment results among various defendants; and (2) the lack of a real-world dataset for training and evaluation. To tackle the first challenge, we formalize the multi-defendant judgment process as hierarchical reasoning chains and introduce a multi-defendant LJP method, named Hierarchical Reasoning Network (HRN), which follows the hierarchical reasoning chains to determine criminal relationships, sentencing circumstances, law articles, charges, and terms of penalty for each defendant. To tackle the second challenge, we collect a real-world multi-defendant LJP dataset, namely MultiLJP, to accelerate the relevant research in the future. Extensive experiments on MultiLJP verify the effectiveness of our proposed HRN.

Multi-Defendant Legal Judgment Prediction via Hierarchical Reasoning
Yougang Lyu'*
Jitai Hao! Zihan Wang'
Kai Zhao?
Shen Gao
Pengjie Ren'
Zhumin Chen' Fang Wang'
Zhaochun Ren
Shandong University, Qingdao, China
Georgia State University, Atlanta, USA
Leiden University, Leiden, The Netherlands
[youganglyu, 2022151 12, 202020630) ) @ maiil.sd..du.coo
kzhao4 @ gsu.edu, shengao www...@ssuu....oo.e jay.ren @ @outlook.com (chenzhumin, wangfang226) @ sdu.edu.cn, z.ren @ leddeneiienn
Abstract
Fact description:
Defendant A trafficked drugs in room in hotel. Defendant helped Defendant
traflidking, and Defendant A trafficked 300 gramn and Defendant B traffidded grams. Defendant C illegally possessed grarre drugss Finally, Defendant C voluntarily confessed to the arime.
Multiple defendants in a criminal fact descrip- tion generally exhibit complex interactions, and cannot be well handled by existing Legal Judgment Prediction (LJP) methods which fo- cus on predicting judgment results (c.g.. law articles, charges, and terms of penalty) for single-defendant cases. To address this prob- lem, propose the task of multi-defendant LJP, which aims to automatically predict the judgment results each defendant of multi- defendant cases. Two challenges arise with the task of multi-defendant LJP: (1) indistinguish- able judgment results among various defen- dants; and (2) the lack of real-world dataset for training and evaluation. To tackle the first challenge, we formalize the multi-defendant judgment process as hierarchical reasoning chains and introduce a multi-defendant LJP method, named Hierarchical Reasoning Net- work (HRN), which follows the hierarchical reasoning chains to determine criminal relation- ships, sentencing circumstances, law articles, charges, and terms of penalty for each defen- dant. To tackle the second challenge, we col- lect a real-world multi-defendant LJP dataset, namely MultiLJP, to accelerate the relevant re- search in the future. Extensive experiments on MJ verify the effectiveness of our pro- posed HRN.
Clarify criminal relationships between defendants
Defendant B help defendant A (joint crime) Defendant unrolated others (individual crime)
Determine sentencing circumstances for each defendant
Defendant Accessory (lighter punishment) Defendent C. Confession (lighter punishment)
Make judgments for each defendant
Law articles
Charges
Terms of penaltyy
Defendant
Article 347: Crime drug trafficking
Crime of drug trafficking
15 years
Defendant Artidle 347: Crime of drug tralfiiking
Crima of drug trafficking
10 years
illgalll holding drugs
year
illegally holding drugs
Figure An illustration of multi-defendam LJP. Gener- ally, judge needs to reason on the fact description to clarify the complex interactions among different defen- dants and make accurate judgments for each defendant.
Despite these successful efforts, single- defendant LJP suffers from an inevitable restriction in practice: large number of fact descriptions with multiple defendants. According to statistics derived from published legal documents sampled from legal case information disclosure, multi- defendant cases constitute a minimum of 30% of all cases (Pan et al., 2019). As shown in Figure 1, the multi-defendant LJP task aims at predicting law articles, charges, and terms of penalty for each defendant in multi-defendant cases. Since multiple defendants are mentioned in the fact description and exhibit complex interactions, the multi-defendant LJP task requires clarifying these interactions and making accurate judgments for each defendant, which is intuitively beyond the reach of single-defendant LJP methods. Hence, there is a pressing need to extend LJP from
1 Introduction
Legal Judgment Prediction (LJP) aims at predict- ing judgment results (e.g., law articles, applicable charges, and terms of penalty) based on the fact description a given case. Existing LJP studies primarily focus on single-defendant cases, where only one defendant is involved (Luo et al., 2017; Zhong et al., 2018; Yang et al., 2019; Xu et al., 2020; Dong and Niu, 2021; Yue al., 2021; Lyu et al., 2022; Feng et al., 2022b; Zhang al., 2023a).
Equal contribution.
Corresponding author.

---

single-defendant multi-defendant scenarios. However, two main challenges arise with the task of multi-defendant LJP:
erarchical reasoning chains are divided into two levels. The first-level reasoning chain identifies the relationships between defendants and deter- mines the sentencing circumstances for each de- fendant. The second-level reasoning chain predicts and verifies the law articles, charges, and terms of penalty for each defendant, using forward predic- tion process and backward verification process, respectively. Since generative language models have shown great ability to reason (Talmor et al., 2020; Yao et al., 2021; Hase and Bansal, 2021), we convert these reasoning chains into Sequence- to- to-Sequence (Seq2Seq) generation tasks and apply the m'S5 (Xue et al.. 2021) to model them. Further- (FID) (Izacard and Grave, 2021) to process multi-defendant fact descriptions with thousands of tokens efficiently.
Indistinguishable judgment results among var- ious defendants. As complicated interactions exist among multiple defendants, fact descrip- tions of various defendants are usually mixed to- gether. Thus it is difficult to distinguish different judgment results among various defendants as to make accurate judgments for each defendant. As shown in Figure I, in order to distinguish different judgment results among various defen- dants, a judge has to clarify criminal relation- ships among defendants to determine whether defendants share same law articles and charges, and sentencing circumstances affecting terms of penalty for each defendant. Based on these inter- mediate reasoning results, the judge determines and verifies the judgment results (law articles, charges, and terms of penalty) for each defendant, following a forward and backward order. The motivation behind forward prediction and back- ward verification is rooted in the complex nature of legal reasoning, where evidence and conclu- sions can be interdependent (Zhong et al., 2018; Yang al., 2019). Overall, the multi-defendant judgment process requires simulating the judicial logic of human judges and modeling complex reasoning chains.
more, we adopt
To tackle the second challenge, we collect a real- world dataset, namely MultiLJP, with 23,717 real- world multi-defendant LJP cases. Eight profes- sional annotators are involved in manually editing law articles, charges, terms of penalty, criminal re- lationships, and sentencing circumstances for each defendant. In 89.58 percent of these cases, the defendants have different judgment results for at least one the subtasks of the multi-defendant LJP task. MultilJP requires accurate distinction of the judgment results for each defendant. This makes MultiLJJP different from the existing single- defendant LJP datasets. Our work provides the first benchmark for the multi-defendant LJP task.
Lack of real-world multi-defendant LJP
datasets. Existing datasets for LJP either only have single-defendant cases annotated for multi- ple LJP subtasks multi-defendant cases for single LJP subtask. Xiao et al. (2018) collect real-world LJP dataset CAIL, which only retains single-defendant cases. Pan et al. (2019) only annotate multi-defendant cases with the charge prediction subtask and ignore information about criminal relationships and sentencing circum- stances that can distinguish judgment results of multiple defendants in real scenarios. In order to
Using MultiL.JP, we evaluate the effectiveness of HRN for multi-defendan LJP on various subtasks. The results show that HRN can significantly out- perform all the baselines. In summary, our main contributions are:
We focus on the multi-defendant LJP task and formalize the multi-deefendnt judgment process as hierarchical reasoning chains for the multi- defendant LJP task.
We introduce HRN, a novel method that follows the hierarchical reasoning chains to distinguish the judgment results for each defendant in multi- defendant LJP.
accelerate the research on multi multi-defendant LJP, we urgently need a real-world multi-defendant LJP dataset.
We present MultiLJP, the first real-world dataset for multi-defendant LJP, which facilitates future research in this areal.
To tackle the first challenge, we formalize the multi-defendant judgment process hicrarchical
reasoning chains and propose method for multi- defendant LJP, named Hierarchical Reasoning Net-
We demonstrate the effectiveness of HRN on MultiLJP through empirical experiments.
work (HRN), which follows the hierarchical rea-
soning chains to distinguish different judgment re- sults of various defendants. Specifically, the hi-
OOur code and Multil JPP dataset are available at https: //github. www.ttttu...uueeeeeeeee COm//CURENNT/HHN

---

2 Related work
MultiJPP
Number
Training
18,968
2.1 Legal judgment prediction
Validation
2,379
Legal judgment prediction has been studied in var- ious jurisdictions (Zhong et al., 2020; Feng al., 2022a; Katz al., 2017: Chalkidis al.. 2019; Sulea al., 2017a,b; Malik et al., 2021; Paul et al.. 2020; Niklaus et al., 2021). Early stud- ies on LJP focus on rule-based methods (Kort, 1957; Nagel, 1963; Segal, 1984) and machine learn- ing algorithms (Aletras et al., 2016; Sulea et al., 2017a.b; Katz et al., 2017). Recent neural-based approaches jointly predict judgment results (law articles, charges and term of penalty) for single- defendant cases by modeling dependency between LJP subtasks (Zhong et al., 2018; Yang et al., 2019; Dong and Niu, 2021; Huang et al., 2021), 1), leverag- ing legal knowledge (Hu et al., 2018; Gan et al., 2021; Yue et al., 2021; Ma et al., 2021; Lyu et al.. 2022; Feng et al., 2022b), exploiting label infor- mation (Luo al., 2017; Wang et al., 2019; Xu al., 2020; Le et al., 2022; Liu et al., 2022; Zhang al., 2023a), or employing pre-trained language models (Chalkidis et al., 2020, 2021; Xiao al., 2021a). For multi-defendant cases, MAMD (Pan et al., 2019) utilizes multi-scale attention to dis- tinguish confusing fact descriptions of different defendants for multi-defendant charge prediction. However, existing single-defendant LJP methods neglect the complex interactions among multiple defendants. Moreover, compared with the multi- defendant LJP method MAMD (Pan et al., 2019), we follow the human judgment process to model hi- crarchical reasoning chains to distinguish different judgment results of various defendants and make accurate judgments for each defendant.
Testing
2,370
Law articles
Charges
23
Terms penalty
Criminal relationships
Sentencing circumstances
Total defendants
80,477
Average defendants
3.39
Law articles per defendant
Charges per defendant
Terms penalty per defendant
Criminal relationships per defendant
Senteneing circumstances per defendant Average length
1.19
3 704..76
Table 1: Statistics of the MultiLJP.
et al. (2021) finetune pre-trained language mod- els solve competition mathematics problems by generating multi-step solutions. Nye et (2021) train language models to predict the final outputs of programs by predicting intermediate computational results. Recently, Wei et al. (2022) propose chain of thought prompting, which feed large language models with step-by-step reasoning examples with- out fine-tuning improve model performance.
However, these methods are not designed for more realistic legal reasoning applications (Huang and Chang, 2022). In this paper, we aim to use generative language models to capture hierarchical reasoning chains for the multi-defendant LJP task.
Dataset
In this section, we describe the construction process and analyze various aspects of MultiL_JP to provide a deeper understanding of the dataset.
3.1 Dataset construction
2.2 Multi-step reasoning with language models
To the best of our knowledge, existing LJP datasets only focus single-defendant cases or charge pre- diction for multi-defendant cases. Thus, struct Multi-defendant Legal Judgment Prediction (MultiLJP) dataset from the published legal doc- uments China Judgements Online2 Instead of extracting labels using regular expressions as in ex- isting works (Xiao et al., 2018; Pan et al., 2019), we hire eight professional annotators to manually pro- duce law articles, charges, terms of penalty, crimi- nal relationships, and sentencing circumstances for each defendant in multi-defendant cases. The anno- tators native Chinese speakers who have passed
Multi-step reasoning by training fine-tuning lan- guage models generate intermediate steps has been shown to improve performance (Zaidan et al., 2007: Talmor et al., 2020; Yao et al., 2021; ; Hase and Bansal, 2021; Zhang et al., 2023b; Gu et al., 2021). Ling et al. (2017) generate natural language intermediate steps to address math word problems. Camburu al. (2018) extend the natural language inference dataset with human-annotated natural language explanations of the entailment relations, Rajani al. (2019) generate rationales that ex- plain model predictions for commonsense question- answering tasks (Talmor et al., 2019). Hendrycks
Pht /wtps///////.uuu/// courtt gov. cn/

---

China's Unified Qualification Exam for Legal Pro- fessionals. All data evaluated by two annotators repeatedly to eliminate bias. Since second-instance cases and retrial cases are too complicated, we only retain first-instance cases. Besides, we anonymize sensitive information (c.g., name, location, etc.) for multi-defendant cases to avoid potential risks of structured social biases (Pitoura et al., 2017) and protect personal privacy. After preprocessing and manual annotation, the MultiLJP consists of 23,717 multi-defendan cases. The statisties information dataset MultiLJP can be found in Table 1.
eration framework for hierarchical reasoning. Next, we introduce hicrarchical reasoning chains of multi- defendant detail. Finally, a training process with Aasion-in-Decoder for HRN is explained.
4.1 Problem formulation
We first formulate the multi-defendant LJP task. The fact description of multi-defendant case be seen word sequence x {W1, W2, Wn,,......... where n represents the number of words. Each multi-defendant case has a set defendant names E = {e), e2, .,.... e|g) where each name is a se- quence of words e = {wi, w2,,., W|/ Given the fact description x and the defendant name e of multi-defendant case, the multi-defendant task aims predict the judgment results of multiple applicable law articles, multiple charges, and term of penalty. The law article prediction and the charge prediction subtasks are multi-label classifi- cation problems, and the term of penalty prediction subtask a multi-class classification problem.
3.2 Dataset analysis
Analysis of the number of defendants. MultiLJP only contains multi-defendant cases. The num- ber of defendants per case is distributed as fol- lows: 49.40 percent of cases have two defendants, 21.41 percent have three defendants, 11.22 percent have four defendants, and 17.97 percent have more than four defendants. The MultiLJP dataset has 80,477 defendants in total. On average, each multi- defendant case has 3.39 defendants.
We introduce the criminal relationship and the sentencing circumstance as intermediate tasks to model the hierarchical reasoming chains for multi- defendant LJP and improve the prediction of the main judgment results. Criminal relationships fer the relationships between defendants, specif- ically whether one defendant assisted other co- defendants during the commission of the crime. Sentencing circumstances refer to specific behav- iors (such as confession and recidivist) or factors (like accessory and blind) that can influence the severity or leniency of penalty?. These two tasks are also multi-label classification problems. We denote the labels of law articles, charges, terms of penalty, criminal relationships, and sentencing cir- cumstances word sequences y", y y y and y respectively in this paper.
Analysis of multi-defendant judgment results. In 89.58 percent of these cases, the defendants have different judgment results for at least one of the subtasks of the multi-defendant LJP task. Specif- ically, 18.91 percent of cases apply different law articles to different defendants, 26.80 percent of cases impose different charges on different defen- dants, and 88.54 percent of cases assign different terms of penalty to different defendants.
Analysis of criminal relationships and sentenc- ing circumstances. Based on the gold labels of criminal relationships and sentencing circum- stances, idcally, a judge can distinguish between 69.73 percent of defendants with different judg- ment results (law articles, charges, and terms of penalty). Specifically, based on the criminal rela- tionship. judge can distinguish 70.28 percent of defendants with different law articles and 72.50 percent of defendants with different charges; based on the sentencing cireumstances, judge can dis- tinguish 96.28 percent defendants with different terms of penalty.
4.2 Sequence-to-sequenee generation
From the perspective of Sepuence-to-Sqqueeee (Seq2Seq) generation, each task can be modeled finding an optimal label sequence y that maximizes the conditional probability based on the fact de- scription, specific defendant name and specific task description p(y|x, xx e, d), which is calculated
4 Method
In this section, we describe the HRNN method. First, we formulate our research problem. Then, we in- troduce the -eeueenceto-SSeuenne (Seq2Seq) gen-
Details of criminal relationships and sentencing circum- stances definitions together with their explanations can he found in Appendix A and B.

---

Criminal relationships
Defendant A, Defendant Delendant
Defendamt C is unrelated to others
Specific defendant to be judged Dafendant
Sentencing circumstances
Confession
Fact description x
Hierarchical Reasoning
Law articles
Defendant trafficked druge in room in family hotel. Delendant B helped Delendant A with trafficking, and Defendant trafficked trafficked 50 grams. Dafendant illggllly possessed 16 grams drugs. Finally, Defendant C voluntarily
Article 348
grams
Charges
lllgglly holding drugs Terms of penalty year
-...--.-...____._............... First-level reasoning chain
What are criminal relationships between Defendant and others?
gi
Criminal relationships
What are sentencing circumstances for Defendant C?
Sentencing circumstances
Second-level reasoning chain
Seq2Seq Generation
What are law articles, charges, and terms of penalty for Defendant C?
lct
glct
Forward prediction
What are term penalty, charges, and law articles for Defendant C?
te
g.tel
Backward verification
Overview our proposed HRN. HRN leverages Sequence-to-Sequeee (Seq2Seq) generation framework to follow hierarchical reasoning chains to generate prediction results.
Figure 2:
follows:
As shown in Figure 2, the hierarchical reasoning chains consist of two levels:
The first-level reasoning is for intermediate tasks. The first-level reasoning chain to first identify relationships between defendants based on the fact description, the names of all defendants and the criminal relationship prediction task description d' as follows:
pyyxxe,, e, = plualii.3 32. Um 1, 1, e, d),
11
(1)
where m denotes the length of the label sequence.
and the specific task description d is semantic prompt that allows Seq2Seq generation models to execute the desired task. To accomplish the Seq2Seq generation tasks, upply the Seq2Seq language model mT5 (Xue et al., 2021) generate label sequences as follows:
= DECC(ENN((,,, E.d"))
(3)
Then, determine sentencing cireumstances for the defendant based on the fact description, name of the defendant e, prediction results of criminal relationships and the sentencing circumstance pre- diction task description d that is:
ý DEC(ENC((x,,d)
where ENC refers the encoder of the language model, DEC' denotes the decoder the language model, and y is prediction results composed of words. We use special [SEP] tokens to separate the different information to form the input of the encoder.
DEC(EECCX, e, dd)).
(4)
The second-level reasoning is for judgment prediction tasks. The second-level reasoning chain consists of forward prediction process and backward verification process. The forward pre- diction process is to predict law articles, charges, and terms of penalty (in that order) based on the fact description, the name of defendant e, first-level reasoning results, and the forward prediction task description d follows:
4.3 Hierarchical reasoning chains
To distinguish different judgment results among various defendants, our method HRN follows hier- archical reasoning chains determine each defen- dant's criminal relationships, sentencing circum- stances, law articles, charges, and terms of penalty.
glet = DEC(ENC((x, e, {")))
(5)

---

Then, the backward verification process to verify these judgment results in reverse order based on the fact description, the name of defendant e, first-level reasoning results and the backward verification task description d'd that is:
5.2 Baselines
To verify the effectiveness of our method HRN on multi-defendant LJP, we compare it with vari- ety of methods, which can be summarized in the following three groups:
y tel = DECC(ENC(,,. ,', ))
(6)
Single-defendant LJP methods, including Top- judge (Zhong et al., 2018), which is topolog- ical dependency learning framework for single- defendant LJP and formalizes the explicit depen- dencies over subtasks as directed acyelic graph; MPBFN (Yang et al., 2019), which is a single- defendant LJP method and utilizes forward and backward dependencies among multiple LJP sub- tasks; LADAN (Xu et al., 2020), which is a graph neural network based method that automatically captures subtle differences among confusing law articles; Judge (Yue et al., 2021), which utilizes the results of intermediate subtasks separate the fact statement into different circum- stances and exploits them to make the predictions of other subtasks.
4.4 Training with lusion-in-decoder
To handle multi-defendant fact descriptions whose average length exceeds the length limit of the en- coder, we adopt Fusion-in-Decodde (FID) (Izacard and Grave, 2021) to encode multiple paragraphs split from a fact description. We first split the fact description into K paragraphs containing M words. Then, combine multiple paragraph representations from the encoder, the decoder gen- erates prediction results by attending to multiple paragraph representations as follows:
y 13 DDCC b2, hg);
(7)
where h; denotes the representation of the i-th para-
Pre-trained language models, including BERT (Cui et al., 2021), which is a Transformer- based method that is pre-trained on Chinese Wikipedia documents; mT5 (Xue et al., 2021), which is multilingual model pre-trained by converting several language tasks into "text-to- text" tasks and pre-trained on Chinese datasets; Lawformer (Xiao et al., 2021b), which is a Transformer-based method that is pre-trained on large-scale Chinese legal long case documents. Multi-defendant charge prediction method, in- cluding MAMD (Pan et al., 2019), which is multi-defendan charge prediction method that leverages multi-scale attention to recognize fact descriptions for different defendants.
graph of the fact description x. Since all tasks are formulated as sequence-to-sequence generation tasks, we follow Raffel et al. (2020) to train the model by standard maximum likelihood and cal- culate the cross-entropy loss for each task. The overall loss function is formally computed as:
L A + Agh, + Lieh + Ata
(8)
where hyperparameters À determine the trade-off between all subtask losses. Lr. Ls. Liat and Lid denote the cross-entropy losses of the criminal re- lationship prediction task, the sentencing circum- stance prediction task, the forward prediction pro- cess and the backward verification process, respec- tively. At test time, we apply greedy decoding to generate forward and backward prediction results. Finally, the chain with the highest confidence is chosen for the final prediction.
We adapt single-defendant LJP methods multi- defendant LJP by concatenating defendant's name and fact description as input and training models to predict judgment results. However, we exclude some state-of of-tth--ttttttt art single-defendaat ap- proaches unsuitable for multi-defendant settings. Few-shot (Hu et al., 2018), EPM (Feng et al., 2022b), and CEEN (Lyu et al., 2022) annotate extra attributes for single-defendant datasets, not casily transferable to MultiLJP. Also, CTM (Liu et al., 2022) and CL4LJP (Zhang et al., 2023a) design specific sampling strategies for contrastive learn- ing of single-defendant cases, hard to generalize to multi multi-defendant cases.
5 Experiments
5.1 Research questions
We aim to answer the following research questions with our experiments: (RQ1) How does our pro- posed method, HRN, perform multi-defendant LJP cases? (RQ2) How do the different levels of reasoning chains affect the performances of HRN on multi-defendant IJP?

---

To accommodate the length of multi-defendant fact descriptions, we set the maximum fact length to 2304. Due to the constraints of the model input, BERT's input length is limited to 512. For training, we employed the A (Loshchilov and Hut- ter, 2019) optimizer and used a linear learning rate schedule with warmup. The warmup ratio was set Law Articles
Charges
Term Penalty
Method
Acc.
MP
MR
Fl
Acc.
MP
MR
F1
Acc.
MP
MR
F1
TopJudge
69.32
35.60
39.13
36.93
64.42
24.96
35.28
28.34
28.36
23.16
22.25
22.00
MPBFN
72.47
34.73
34,22
34.35
65.59
32.79
33.20
31.59
28.32
21.59
20.91
20.70
LADAN
54.57
38.09
22.40
26.64
46.62
20.68
32.42
24.74
27.05
24.05
23,43
23.16
NeurJudge
65.21
41.72
36.96
38.15
59.51
34.19
25.36
27.55
30.06
27.56
25.63
25.95
BERT
51.38
34.19
29.68
30.70
44.80
36,80
20.10
25.14
29.60
23.95
22.68
21.55
mT5
87.49
74,28
53.65
58.84
81.52
63.33
49.94
52.86
33.66
39.13
24.23
23.04
Lawformer
75.50
36.18
35.33
34.00
65.94
38.97
29.12
32.76
32.37
22.66
20.68
18.30
MAMD
58.73
33.00
34.15
31.60
HRN
91.46
69.87
70.95'
69.20'
89,54"
71.80"
71.83*
70.70'
42.74'
41.33'
40.20'
40.62"
gold
92.26"
75.67"
70.22*
71.58"
90.60*
78.46"
75.38*
76.27
44.44"
47.13'
42.31*
43.27'
Table 2: Judgment prediction results on MultiLJP. Significant improvements over the best baseline are marked with * (t-test, p < 0.05).
5.3
Implementation details
Compared with state-of-art single-defendant LJP methods, e.g., Topjudge, MPBEN, LADAN, and NeurJudge, our method HRN consider hierarchi- cal reasoning chains and thus achieve significant improvements. Since the single-defendant meth- ods do not consider criminal relationships and sentencing cireumstances, they can not distin- guish different judgment results among various defendants well. It indicates the importance of following the hierarchical reasoning chains predict criminal relationships and senteneing cir- cumstances for multi-defendant LJP.
0.01, and the maximum learning rate was set to 1 10 We set the batch size as 128 and adopt the gradient accumulation strategy. All models are trained for a maximum of 20 epochs. The model that performs best on the validation set is selected. For the hyperparameters, X, in the loss function, the best setting is (0.6, 0.8, 1,4, 1.2] for ()r, X Atca }. Additionally, we set the number of paragraphs K, the number of words per paragraph M, and the output length to 768, and 64, respec- tively. For performance evaluation, we employ four metrics: accuracy (Acc.), Macro-Precision (MP), Macro-Recall (MR), and Macro-F1 (F1). All ex- periments are conducted on one RTX3090.
As Table shows, our method HRN achieves considerabe performance improvements on all subtasks of multi-defendant LJP compared to pre-trained models BERT and Lawformer. This shows that modeling the hicrarchical reason- ing chains during the fine-tuning stage is cru- cial. Moreover, HRN, which combines mTT5 and the hierarchical reasoning chains, significantly outperforms m'T,, indicating that the language knowledge from pre- pre-training and the information of the hierarchical reasoning chains are comple- mentary to each other.
6 Experimental results and analysis
Compared to MAMD designed for multi- defendant charge prediction, our method HRN outperforms MAMD on charge prediction task. This shows the effectiveness and robustness of modeling reasoning chains in real-world applica- tion scenarios.
In this section, we first conduct multi-defendant legal judgment predictions and ablation studies to answer the research questions listed in the Section. 5.1. In addition, we also conducted a case study to intuitively evaluate the importance of hierarchical reasoning.
We employ gold annotations of criminal relation- ships and sentencing circumstances, rather than predicted ones, to enhance the performance of HRN. Our findings indicate a considerable im- provement in results when relying on gold anno- tations. This observation underscores the poten- tial for notable enhancement in multi-defendant
6.1 Multi-defendant judgment results (RQ1)
Table 2 shows the evaluation results on the multi- defendant LJP subtasks. Generally, HRN achieves the best performance in terms of all metries for all LJP subtasks. Based on the results, we have three main observations:

---

Law Articles
Charges
Term Penalty
Method
Acc.
MP
MR
Fl
Acc.
MP
MR
Fl
Acc.
MP
MR
Fl
HRN
91.46
69.87
70.95
69.20
89.54
71,80
71.83
70.70
42.74
41,33
40.20
40.62
w/o CR
89.14
73.71
67.85
68.25
86.84
72.81
68.61
68,95
37.83
38.20
30,30
30.69
w/o SC
88.58
73.64
61.51
65.51
85.98
77.03
64.82
68,98
32.66
37.37
25.43
24.56
w/o FP 89.05
73.61
65.61
68.17
85,97
73.24
66.09
68.29
35.83
34.35
30,47
30.00
w/o
90.91
75.60
67.58
68.50
83,45
69.34
60.86
63.73
37.41
36,90
28.42
28.88
w/o all
87.49
74.28
53.65
58.84
81.52
63.33
49,94
52.86
33.66
39.13
24.23
23.04
Table 3: Ablation studies MultiLJP
Multi defendant description:
formance:
Dafendan: A and defendont were playnng in public place of entertainment when defendant and the victim get into an orgumnet and defendant went up to see what was going on. Defendant A followed the
Removing the first-level reasoning chain. We observe that both criminal relationships and sen- tencing circumstances decrease the performance of HRN when remove the first-level reason- ing chains. Specifically, removing criminal rela- tionships (CR) negatively impacts performance, especially on law articles and charges predic- tion, which means criminal relationships are help- ful for distinguishing law articles and charges; removing sentencing circumstances (SC) nega- tively impacts performance, especially on terms of penalty, which means sentencing cireum- stances are helpful for distinguishing terms of penalty.
gathered defendant
to entrance the publc place of entertainment. and defendant B came forward and beat The injuyy constituted degree of minor Injury.
vistim
Gold labels for multiple defendants:
Defendant lew articles
293), charges (Grime picking
quarrels and provoking troubles). terms of penalty (15 months) Defendant articles (articee 293,, charges (Crime picking quarrels and provoking troubles). terms of penalty (11 months)
Prediction for multiple defendants (MAMD):
Defendant law articles (article 264 charges (Theft x), tarms penalty (11 months x)
Defendant B: aridees (article 293 v), charges (Crime of picking quarrels and provaking troubles v), terms of penalty (11 months v/
Prediction for multiple defendants (HRN):
Criminal relationships: defendant helped defendant A. Sentencing ciroumstances of defendant none.
Sentencing circumstances of defendant B: acceseory.
Defendant A: lo artickes (article 293 v). charges (Crime of picking
quarrels and provoking troubles v), terms of penalty (15 months v/)
Removing the second-level reasoning chain. We observe that the model without the second- level forward prediction process (IFP or the second-level backward verification process (BV) faces huge performance degradation in multi- defendant LJP. As a result, although the model still performs the first-level reasoning, the ab- sence of modeling forward or backward depen- dencies between LJP subtasks leads to poor LJP performances.
Defendant B: law articles (article 293 v). charges (Crime of picking quarrels and provoking troubles terms of penalty (11 months v/
Figure 3: Case study for intuitive comparisons. Red and green represent incorrect and correct judgment results, respectively. Blue denotes descriptions of criminal rela- tionships.
LJP by improving the first-level reasoning.
6.2 Ablation studies (RQ2)
Removing all reasoning chains. When remov- ing all reasoning chains from HRN, there is a substantial drop in the performances of multi- defendant LJP. Experimental results prove that hierarchical reasoning chains can be critical for multi-defendant LJP.
To analyze the effect of the different levels of rea- soning chains in HRN, we conduct an ablation study. Table 3 shows the results MultiLJP with five settings: (i) w/o CR: HRN without predicting criminal relationships. (ii) w/o SC: HRN without predicting sentencing cireumstances. (iii) w/o FP: HRN without the forward prediction process and predicts law articles, charges, and terms of penalty in reverse order. (iv) w/o BV: HRN without the backward verification process and predicts law arti- cles, charges, and terms of penalty in order. (v) w/o all: HRN degrades to mT5 with multi-task settings by removing all reasoning chains.
6.3 Case study
We also conduct case study to show how multi- defendant reasoning chains help the model dis- tinguish judgment results of different defendants and make correct predictions. Figure shows pre- diction results for two defendants, where red and green represent incorrect and correct predictions, respectively. Defendant B helped A beat the vic-
Table 3 shows that all levels of reasoning chains help HRN as removing any of them decreases per-

---

tim, but the fact description does not show di- rect crime against the victim. Without determin- ing criminal relationships and sentencing circum- stances, MAMD focuses on the tailing behavior around A and misclassifies A's law articles, charges, and terms of penalty article 264, theft, and 11 months. In contrast, by following reasoning chains to determine criminal relationships, sentencing cir- cumstances, law articles, charges, and terms of penalty, HRN distinguishes different judgment re- sults between defendants.
protect personal privacy, have anonymized sen- sitive information (e.g., name, location, etc.) for multi-defendant cases MultiLJP dataset.
Acknowledgments
This work was supported by the National Key R&D Program of China with grant No.2022YFC3303004, the Natural Science Foundation of China (62272274, 61902219, 61972234, 62102234, 62202271, 62072279, T2293773, 72371 the Natural Science Foun- dation of Shandong Province (2R2021QF129), the Key Scientific and Technological Innovation Pro- gram of Shandong Province (2019/ZZY01012)) All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.
7 Conclusions
In this paper, we studied the legal judgment pre- diction problem for multi-defendant cases. We proposed the task of multi-defendant LJP to pro- mote LJP systems from single-defendant multi- defendant. To distinguish confusing judgment re- sults of different defendants, we proposed a Hier- archical Reasoning Network (HRN) to determine criminal relationships, sentencing circumstances, law articles, charges and terms of penalty for each defendant. As there is no benchmark dataset for multi-defendaan LJP, we have collected real- world dataset MultiLJP. We conducted extensive experiments on the MultiL.IP dataset. Experimental results have verified the effectiveness of our pro- posed method and HRN outperforms all baselines.
References
Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preotiuc-Pietro, and Vasileios Lampos. 2016. Pre- dicting judicial decisions of the curopean court of human rights: a natural language processing perspec- tive. PeerJ Comput. Sci., 2:e93.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewiez, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language explana- tions. In Proceedings of NewrIPS, pages 3570-9572.
Limitations
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Ale- tras. 2019. Neural legal judgment prediction en- glish. In Proceedings of ACL, pages 4317-4323.
Although work distinguishes the judgment re- sults of multiple defendants by hierarchical rea- soning, in real life, there exist many confusing charge pairs, such as (the crime intentional in- jury, and the crime of intentional homicide). The fact descriptions of these confusing charge pairs are very similar, which makes it difficult for the multi-defendant LJP model to distinguish between confusing charge pairs We leave this challenge for future work.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: "preparing the muppets for court'". In Findings of EMNLP. pages
Ilass Chalkidis, Manos Fergadiotis, Dimitrios Tsarapat- sanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis. 2021. Paragraph-level ratio- nale extraction through regularization: case study european court of human rights cases. In Proceed- ings of NAACL, pages 226-241.
Ethics Statement
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese BERT. IEEE ACM Trans. Audio Speech Lang. Process., 29:3504-3514,
Since multi-defendant legal judgment prediction is an emerging but sensitive technology, we would like to discuss ethical concerns of our work. Our proposed method HRN is a preliminary multi- defendant work and aims to assist legal profession- als instead of replacing them. In addition, multi- defendant cases contain personal privacy informa- tion. To avoid potential risks of structured social biases (Pitoura et al., 2017; Lyu et al., 2023) and
Qian Dong and Shuzi Niu. 2021. Legal judgment pre- diction via relational learning. In Proceedings of SIGIR, pages 983-992.
Yi Feng, Chuanyi Li, and Vincent Ng. 2022a. Legal judgment prediction: A survey of the state of the art. In Proceedings of IJCAI, pages 5461-5469.

---

Yi Feng, Chuanyi Li, and Vincent Ng. 2022b. Legal judgment prediction via event extraction with con- straints. In Proceedings of ACI,, pages 648-664.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In Proceedings of ICLR.
Bingteng Luo, Yansong Feng, Jianho Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning predict charges for criminal cases with legal basis. In Pro- ceedings of EMNLP, pages 2727-2736.
Leilei Gan, Kun Kuang, Yi Yang, and Fei Wu, 2021. Judgment prediction via injecting legal knowledge into neural networks. In Proceedings of AAAI, pages
12866- 12874.
Yougang Lyu, Piji Li, Yechang Yang. Maarten de Rijke, Pengjie Ren, Yukun Zhao, Dawei Yin, and Zhaochun Ren. 2023, Feature-level debiased natural language understanding. In Proceedings of AAAI, pages 13353- 13361.
Yuling Gu, Bhavana Dalvi Mishra, and Peter Clark. 2021. DREAM: uncovering mental models behind language models. CoRR, abs/2112.08656
Peter Hase and Mohit Bansal. 2021. When can models learn from explanations? A formal framework for understanding the roles of explanation data. CoRR, abs/2102.0220)
Yougang Lyu, Zihan Wang, Zhaochun Ren, Pengjie Ren, Zhumin Chen, Xiaozhong Liu, Yujun Li, Hongsong Li, and Hongye Song. 2022. Improving legal judg- ment prediction through reinforced criminal element extraction. Inf. Process. Manag. 59(1)0007780.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Proceed- ings of NeurlPS.
Luyao Ma, Yating Zhang, Tianyi Wang. Xiaozhong Liu, Wei Ye, Changlong Sun, and Shikun Zhang. 2021. Legal judgment prediction with multi-stage case representation learning in the real court setting. In Proceedings of SIGIR, pages 993-1002.
Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-shot charge prediction with discriminative legal attributes. In Proceedings of COLING, pages 487-498.
Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripahandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. 2021. ILDC for CJPE: indian legal documents corpus for court judg- ment prediction and explanation. In Proceedings of ACL, pages
Jie Huang and Kevin Chen-Chuan Chang. 2022. To- wards reasoning large language models: survey. CoRR, abs/2212. 2. 10403.
Yunyun Huang, Xiaoyu Shen, Chuanyi Li, Jidong Ge. and Bin Luo. 2021. Dependency learning for legal judgment prediction with unified text-to-text trans- former. CoRR, abs/2112.06370.
Stuart Nagel. 1963. Applying correlation analysis case prediction. Tex. I. Rev, 42:1006.
Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. 2021. Swiss-judgment wwiis-judgment-ppeccciton: A multilin- gual legal judgment prediction benchmark. CoRR, abs/21 10.00806.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering, In Proceedings of EACL, pages 874-880.
Maxwell I. Nye, Anders Johan Andreassen, Guy Gur- Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for inter- mediate computation with language models. CORR. abs/2112.001144
Daniel Martin Katz, Michael J Bommarito, and Josh Blackman. 2017. A general approach for predicting the behavior of the supreme court of the united states. PloS one, 12(4)ee0174698
Fred Kort. 1957. Predicting supreme court decisions mathematically: A quantitative analysis of the" right to counsel" cases. The American Political Science Review, 51(1):1-12
Sicheng Pan, Tun Lu, Ning Gu, Huajuan Zhang, and Chunlin Xu. 2019. Charge prediction for multi- defendant cases with multi-scale attention. In Pro- ceedings of ChineseCSCW, volume 1042 of Com- munications in Computer and Information Science, pages 766 7777
Yuquan Le, Yuming Zhao, Meng Chen, Zhe Quan, Xi- aodong He, and Kenli Li. 2022. Legal charge predic- tion via bilinear attention network. In Proceedings of CIKM, pages 1033-1033
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of ACL, pages 157-167.
Shounak Paul, Pawan Goyal, and Saptarshi Ghosh. 2020. Automatic charge identification from facts: A few sentence-level charge annotations is all you need. In Proceedings of COLING, pages 1011-1022.
Dugang Liu, Weihao Du, Lei Li, Weike Pan, and Zhong Ming. 2022. Augmenting legal judgment prediction with contrastive case relations. In Proceedings of COLING, pages 2658-2667.
Evaggelia Pitoura, Panayiotis Tsaparas, Giorgos Flouris, Irini Fundulaki, Panagiotis Papadakos. Serge Abite- boul, and Gerhard Weikum. 2017. On measuring bias in online information. SIGMOD Rec., 16-4):

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res.,
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of NAACL, pages 483-498.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain your- self! leveraging language models for commonsense reasoning. In Proceedings of ACI, pages 4932-4942. cases
Wenmian Yang. Weijia Jia, Xiaojie Zhou, and Yutao Luo. 2019. Legal judgment prediction via multi- perspective bi-feedback network. In Proceedings of IJCAI, pages
Jeffrey A Segal. 1984. Predicting supreme probabilistically; The search and seizure cases, 1962- 1981. American Political Science Review, 78(4):891- 900.
Huihan Yao, Ying Chen, Qinyuan Ye, Xisen Jin, and Xi- ang Ren. 2021. Refining language models with com- positional explanations. In Proceedings of NeuriPS. pages 8954-8967.
Octavia-Maria Sulea, Marcos Zampieri, Shervin Mal- masi, Mihaela Vela, Liviu Dinu, and Josef van Genabith. 2017a. Exploring the use of text classifi- cation in the legal domain. In Proceedings of ICAIL, volume 2143.
Linan Yue, Qi Liu, Binbin Jin, Han Wu, Kai Zhang, Yanging An, Mingyue Cheng, Biao Yin, and Day- ong Wu, 2021. Neurjudge: A cireumstance-awan neural framework for legal judgment prediction. In Proceedings of SIGIR, pages 973-982.
Octavia-Maria Sulea. Marcos Zampieri, Mihaela Vela, and Josef van Genabith. 2017b. Predicting the law area and decisions of french supreme court cases. In Proceedings of RANLP, pages 716-722.
Omar Zaidan, Jason Eisner, and Christine D. Piatko. 2007. Using "annotator rationales" improve ma- chine learning for text categorization. In Proceedings of NAACL, pages 260-267.
Alon Talmor. Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question
Han Zhang, Zhicheng Dou, Yutao Zhu, and Ji-Rong Wen. 2023a. Contrastive learning for legal judgment prediction. ACM Transactions on Information Sys- lems.
answering challenge targeting commonsense knowl- edge. In Proceedings of NAACL, pages 4159-4158.
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Gold- berg, and Jonathan Berant, 2020. Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge. In Proceedings of NeurlPS.
Xiaoyu Zhang, Xin Xin, Dongdong Li, Wenxuan Liu, Pengjic Ren, Zhumin Chen, Jun Ma, and Zhaochun Ren. 2023b. Variational reasoning over incomplete knowledge graphs for conversational recommenda- tion. Proceedings of WSDM, pages 231-239. ACM.
Pengfei Wang, Fan, Shuzi Niu, 7e Yang, Yongfeng Zhang, and Jiafeng Guo. 2019. Hierarchical match- ing network for crime classification. In I Proceedings SIGIR, pages 325-334.
Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judg- ment prediction via topological learning. In Proceed- ings of EMNLP, pages 3540-3549. Association for Computational Linguisties.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11900
Haoxi Zhong, Yuzhong Wang, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Iter- atively questioning and answering for interpretable legal judgment prediction. In Proceedings of AAAI, pages 1250-1251
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021a. Lawformer: pre-trained language model for chinese legal long documents. A/ Open, 2:79-84.
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021b. Lawformer: A pre-trained language model for chinese legal long documents. CoRR, abs/2105.0388
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng. Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. 2018. CAIL2018: 8: A large-scale legal datasct for judgment prediction. CoRR, abs/1800 807.02248.
Nuo Xu, Pinghui Wang, Long Chen, Li Pan, Xiaoyan Wang, and Junzhou Zhao. 2020. Distinguish con- fusing law articles for legal judgment prediction. In Proceedings of ACI., pages 3086-3095.

---

Appendix
A Criminal Relationship
Criminal relationships refer to the relationships be- tween defendants, specifically whether one defen- dant assisted other co-defendants during the com- mission of the crime.
None. One defendant did not help other defen- dants to conduct criminal behaviors.
Assistance. One defendant helped other defen- dants to conduct criminal behaviors.
Sentencing Circumstance
Sentencing circumstances refer to specific behav- iors (such confession and recidivist) or factors (like accessory and blind) that can influence the severity leniency of penalty.
Old People. A person who has reached the age of 75 and conducts the crime is liable to lighter punishment.
Deaf-mute or Blind. A deaf-mute or blind per- son who conducts a crime liable to a lighter punishment.
Accessory. Accessory to the crime is liable to lighter punishment.
Attempted Crime.
person who has attempted to conduct a crime is liable to a lighter punish- ment.
Surrender. A person who has surrendered to justice is liable to lighter punishment.
Confession. A person who has confessed to of- fence is liable to a lighter punishment.
Metitorious Serviceoffset. A person who has metitorious serviceoffset liable lighter pun- ishment.
Recidivist. Recidivist is liable to heavier pun- ishment.


Reviews:
[{"review_id": "4QHQTN8YSr", "reviewer": "Reviewer_TtcG", "summary": "", "questions": "1.What is the motivation of forward and backward chains? Why use backward (tcl) in second-level reasoning chain? How about other combinations?\n\n2.In the section 4.4, how to verify and get the final predictions in test set? What is the details after ‘greedy decoding’?\n\n3.How to get the best setting of \\lambda?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer TtcG:**\n\nWe thank you for your efforts and valuable suggestions. We denote Answer, Reason to reject, Question and Typos as A, R, Q and T respectively.\n\n**A1 to R.1:** \n\nWe will clarify the variable names in the caption of Figure 2. \n\n**A2 to R.2:** \n\nTo clarify, we employ special [SEP] tokens to differentiate various text segments and concatenate them for encoder input. Outputs from the first-level reasoning chain are fed into the second-level reasoning chain. We'll provide an expanded explanation in section 4.3.\n\n**A3 to R.3:** \n\nThanks for the suggestion, we have evaluated our HRN and baselines on the single-defendant dataset CAIL-small. To ensure a fair comparison, all models are directly tested on the CAIL-small test set without fine-tuning on the CAIL-small training set. As shown in the table below, the experimental results demonstrate the effectiveness of HRN on single-defendant cases. We will include these results in the paper.\n\n|   Method  | Law  Articles |       |       |       | Charges |       |       |       | Term  of  Penalty |       |       |       |\n|:---------:|:-------------:|:-----:|:-----:|:-----:|:-------:|:-----:|:-----:|:-----:|:-----------------:|:-----:|:-----:|:-----:|\n|           |      Acc.     |   MP  |   MR  |   F1  |   Acc.  |   MP  |   MR  |   F1  |        Acc.       |   MP  |   MR  |   F1  |\n|  TopJudge |     40.71     | 21.18 | 24.09 | 21.14 |  40.31  | 26.60 | 25.02 | 21.98 |       18.61       | 11.37 | 12.40 | 11.02 |\n|   MPBFN   |     40.53     | 20.53 | 19.84 | 16.25 |  39.52  | 19.53 | 20.24 | 15.49 |       18.94       | 12.24 | 12.11 | 10.79 |\n|   LADAN   |     71.54     | 41.96 | 38.66 | 37.81 |  58.17  | 33.69 | 45.86 | 36.99 |       15.71       | 14.88 | 15.59 | 12.10 |\n| NeurJudge |     58.47     | 31.12 | 29.30 | 27.49 |  51.29  | 26.72 | 27.05 | 23.05 |       19.98       | 16.02 | 13.54 | 12.16 |\n|    BERT   |     63.10     | 25.20 | 21.36 | 21.75 |  60.66  | 24.45 | 18.95 | 20.05 |       16.02       | 12.04 | 14.49 |  6.92 |\n| Lawformer |     66.98     | 28.81 | 26.88 | 25.41 |  66.34  | 31.03 | 32.54 | 27.82 |       15.76       | 17.46 | 17.92 | 11.33 |\n|    MAMD   |       -       |   -   |   -   |   -   |  28.42  |  9.56 | 12.86 |  9.62 |         -         |   -   |   -   |   -   |\n|    HRN    |     72.04     | 54.68 | 52.02 | 50.11 |  70.51  | 54.41 | 52.38 | 50.38 |       21.93       | 18.04 | 12.11 | 10.37 |\n\n**A4 to Q.1:** \n\nThe motivation behind employing both forward and backward chains is rooted in the complex nature of legal reasoning, where evidence and conclusions can be interdependent. The backward chain (tcl) in the second-level reasoning chain serves to examine the derived conclusions and backtrace the logic to ensure consistency.  For example, the prediction of the charges can verify the rationality of predicted applicable laws. Our ablation experiments have demonstrated the effectiveness of the backward chain, and we would like to explore other combinations in future work.\n\n**A5 to Q.2:** \n\nAfter implementing 'greedy decoding', we aggregate the predictions from various reasoning chains. Specifically, the chain with the highest confidence is chosen for the final prediction. We will add these details in section 4.4.\n\n**A6 to Q.3:** \n\nWe conducted a grid search on the validation set to determine the best setting for \\lambda. We will include this in the paper.\n\n**A7 to T.1&T.2&T.3:** \n\nWe will fix these typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n1.The trackled problem is attractive to the real work. The paper systematically analyzes the challenge of multi-defendant LJP task and further releases a dataset, which is beneficial to future work.", "reasons_to_reject": "Weaknesses:\n1.The writing could be improved further. The variable naming is confused, especially in Figure 2 without captions.\n\n2.It is unclear that how to combine the inputs of each reasoning chain and how to utilize the outputs from prior reasoning chains. It is necessary to describe the detail task description in the section 4.3.\n\n3.Multi-denfendant LJP task is extended from single-denfendant LJP task. So, single defendant can be regarded as a special situation of multiple defendant. The relationship of a single defendant could be “None” or ‘Single’. But in the experiments, this work only consider the multi-denfendant cases.\n\n", "questions_to_authors": "1.What is the motivation of forward and backward chains? Why use backward (tcl) in second-level reasoning chain? How about other combinations?\n\n2.In the section 4.4, how to verify and get the final predictions in test set? What is the details after ‘greedy decoding’?\n\n3.How to get the best setting of \\lambda?", "paper_topic_and_main_contributions": "he paper extends LJP task from single-defendant to multi-defendant and proposes a multi-defendant LJP method called Hierarchical Reasoning Network (HRN). The proposed method employs Seq2Seq model and follows the hierarchical reasoning chains to generate prediction results. Moreover, this work releases a real multi-defendant LJP dataset. Results prove the effectiveness of HRN.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.line 118, ‘mt5’ -> ‘mT5’.\n2.line 367, 373 and 383, ‘e’ and ‘x’ should be marked bold.\n3.Figure 3, the MAMD result, ‘article 293’ and ‘11 months’ shoule be marked green."}, "extra_text": {}}, {"review_id": "e5C3ihue1a", "reviewer": "Reviewer_udyE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer udyE:**\n\nWe genuinely appreciate your valuable insights and the time you dedicated. We denote the Answer and Reason to reject as A and R respectively.\n\n**A1 to R.1:** \n\nThe idea of integrating the CoT capability of LLMs with our hierarchical reasoning is intriguing. We anticipate that LLMs, enriched with legal knowledge and CoT capabilities, could be integrated seamlessly into our reasoning process. While LLMs could serve as potential annotators, there's a risk of mislabeling or introducing biases. Our MultiLJP dataset emphasizes accuracy and reduces biases via expert annotations and sensitive data removal. This dataset could guide the development of legal LLMs and help in the quality annotation of legal datasets using LLM's few-shot learning. We will explore the integration of the hierarchical reasoning process with legal LLMs in future work.\n\n**A2 to R.2:** \n\nPlease note that the main purpose of our work is to assist judges in the task of multi-defendant legal judgment prediction and experimental results demonstrate the effectiveness of our method in this scenario. In daily counseling scenarios, users' descriptions are often brief and lack a complete description of the crime, making it difficult to accurately predict judgment results.  We will explore how to extend our method to daily counseling scenarios in future work.\n\n**A3 to R.3:** \n\nThanks for the advice. We have conducted experiments using gold labels of criminal relationships and sentencing circumstances as input. As shown in the table below, considerably better results can be obtained when gold criminal relationships and sentencing circumstances labels are used. These results suggest that LJP results can be substantially improved by improving first-level reasoning (criminal relationships and sentencing circumstances). We will include these results in the paper.\n\n|  Method  | Law  Articles |       |       |       | Charges |       |       |       | Term  of  Penalty |       |       |       |\n|:--------:|:-------------:|:-----:|:-----:|:-----:|:-------:|:-----:|:-----:|:-----:|:-----------------:|:-----:|:-----:|:-----:|\n|          | Acc.          | MP    | MR    |   F1  |   Acc.  | MP    | MR    |   F1  |        Acc.       | MP    | MR    |   F1  |\n|    HRN   | 76.52         | 44.61 | 43.88 | 43.20 |  70.77  | 43.77 | 38.95 | 40.15 |       33.17       | 33.22 | 30.32 | 26.32 |\n| HRN+gold | 78.52         | 45.40 | 46.30 | 45.06 |  73.04  | 53.25 | 45.90 | 46.59 |       36.15       | 39.96 | 34.05 | 29.41 |\n\n**A4 to R.4:** \n\nThanks for the suggestion. We have tested our HRN and baseline models on the single-defendant dataset CAIL-small. To ensure a fair comparison, all models are directly tested on the CAIL-small test set without fine-tuning on the CAIL-small training set. As shown in the table below, the experimental results demonstrate the effectiveness of HRN on single-defendant cases. We will include these results in the paper.\n\n|   Method  | Law  Articles |       |       |       | Charges |       |       |       | Term  of  Penalty |       |       |       |\n|:---------:|:-------------:|:-----:|:-----:|:-----:|:-------:|:-----:|:-----:|:-----:|:-----------------:|:-----:|:-----:|:-----:|\n|           |      Acc.     |   MP  |   MR  |   F1  |   Acc.  |   MP  |   MR  |   F1  |        Acc.       |   MP  |   MR  |   F1  |\n|  TopJudge |     40.71     | 21.18 | 24.09 | 21.14 |  40.31  | 26.60 | 25.02 | 21.98 |       18.61       | 11.37 | 12.40 | 11.02 |\n|   MPBFN   |     40.53     | 20.53 | 19.84 | 16.25 |  39.52  | 19.53 | 20.24 | 15.49 |       18.94       | 12.24 | 12.11 | 10.79 |\n|   LADAN   |     71.54     | 41.96 | 38.66 | 37.81 |  58.17  | 33.69 | 45.86 | 36.99 |       15.71       | 14.88 | 15.59 | 12.10 |\n| NeurJudge |     58.47     | 31.12 | 29.30 | 27.49 |  51.29  | 26.72 | 27.05 | 23.05 |       19.98       | 16.02 | 13.54 | 12.16 |\n|    BERT   |     63.10     | 25.20 | 21.36 | 21.75 |  60.66  | 24.45 | 18.95 | 20.05 |       16.02       | 12.04 | 14.49 |  6.92 |\n| Lawformer |     66.98     | 28.81 | 26.88 | 25.41 |  66.34  | 31.03 | 32.54 | 27.82 |       15.76       | 17.46 | 17.92 | 11.33 |\n|    MAMD   |       -       |   -   |   -   |   -   |  28.42  |  9.56 | 12.86 |  9.62 |         -         |   -   |   -   |   -   |\n|    HRN    |     72.04     | 54.68 | 52.02 | 50.11 |  70.51  | 54.41 | 52.38 | 50.38 |       21.93       | 18.04 | 12.11 | 10.37 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "See details in Paper Topic And Main Contributions.", "reasons_to_reject": "1. The authors propose a Hierarchical Reasoning process that uses multi-step reasoning to get the final judgment prediction. In order to implement the above method, I notice that the authors have labelled the data in detail, which is very commendable. In this regard, I have an idea that I would like to discuss with the authors. Specifically, since LLM has the CoT capability, I am curious if the CoT capability of LLM can be combined with the Hierarchical Reasoning approach in a judicial scenario. Or, whether LLM can be used as an annotator to annotate judicial data.\n\n2. The datasets used by the authors are all from the China Judgements Online, but this is not consistent with the distribution of cases in real scenarios (since judgment documents are written by judges at the conclusion of a case). I am curious whether the author's method can work in daily counselling scenarios, maybe the author can show a few cases to solve my doubts.\n\n3. If Criminal relationships and Sentencing circumstances are used as inputs (i.e., not predicted but inputted into the model along with the facts of the case), what would be the effect of the final judgement prediction?\n\n4. Whether the Hierarchical Reasoning approach proposed by the authors can work on Single-Defendant Legal Judgment Prediction, i.e., removing the Criminal relationships prediction?", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. The authors propose a real-world multi-defendant LJP dataset, which contributes to the LJP task.\n\n2. The authors propose a Hierarchical Reasoning process to solve the multi-defendant LJP problem, which is quite novel.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "z8DafFBPhf", "reviewer": "Reviewer_6vXn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer 6vXn:**\n\nWe sincerely thank you for your time and helpful feedback. We denote the Answer and Reason to reject as A and R respectively.\n\n**A1 to R.1:**\n\nThanks for the advice. \"Criminal relationships\" refer to the relationships between defendants, specifically whether one defendant assisted other co-defendants during the commission of the crime. \"Sentencing circumstances\" refer to specific behaviors (such as confession and recidivism) or factors (like deafness and blindness) that can influence the severity or leniency of a penalty. We will provide clearer definitions of these terms in the paper.\n\n**A2 to R.2:**\n\nThe motivation behind forward prediction and backward verification is rooted in the complex nature of legal reasoning, where evidence and conclusions can be interdependent. Unlike forward prediction, backward verification serves to examine the derived conclusions and backtrace the logic to ensure consistency. For example, the prediction of charges can confirm the rationality of the applicable laws. Our ablation experiments also demonstrate the effectiveness of both forward prediction and backward verification. We will further clarify the differences between forward prediction and backward verification in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors pay attention to the multi-defendant LJP tasks and provide the first real-world dataset for multi-defendant tasks.\n2. The authors propose a Hierarchical Reasoning Network (HRN) to determine criminal relationships, sentencing circumstances, law articles, charges and terms of penalty for each defendant. Experimental results have verified the effectiveness of their proposed method.", "reasons_to_reject": "1. The writing of this paper could be more clear. For example, the definition of criminal relationships and sentencing circumstances are not very clear (I can only guess the meaning from the example in Figure 1). \n2. It seems the Forward prediction and Backward verification are very similar, the only difference is the order of those tasks. So this is more of a quadratic  Forward prediction process than a Backward verification.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors propose the task of multi-defendant LJP to promote LJP systems from single-defendant to multi-defendant. Specifically, they propose a Hierarchical Reasoning Network (HRN) to determine criminal relationships, sentencing circumstances, law articles, charges and terms of penalty for each defendant. Moreover, they provide the MultiLJP, the first real-world dataset for multi-defendant LJP tasks. Experimental results have verified the effectiveness of their proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the task of of multi-defendant legal judgement prediction (LJP) task. This is an under-explored area in LJP literature. The authors introduce a new dataset resource and modelling approach. The dataset introduced in this paper contains a substantial number of instances based on real-world cases in China.  The modelling approach and baseline applies fusion in decoder architecture seq2seq model for an interesting hierarchical approach for extracting facts and relations from the case data.

There is reasonable agreement between the reviewers about the strength and excitement of this work and most questions seem to be well addressed by the authors. However, the reasons to accept listed by the reviewers seem to be fairly surface level. I think there are merits to the work, but the contributions are mostly collecting open data and applying a model architecture in an interesting way. I think the work has the potential to be used by other researchers studying LJP.
Decision: Accept-Findings