Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration
Authors: Yiquan Wu
Siying Zhou
Yifei Liu
Weiming Lu
Xiaozhong Liu
Yating Zhang
Changlong Sun
Fei Wu
Kun Kuang
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
Large Language Model
Model Collaboration

Abstract: Legal Judgment Prediction (LJP) has become an increasingly crucial task in Legal AI, i.e., predicting the judgment of the case in terms of case fact description. Precedents are the previous legal cases with similar facts, which are the basis for the judgment of the subsequent case in national legal systems. Thus, it is worthwhile to explore the utilization of precedents in the LJP. Recent advances in deep learning have enabled a variety of techniques to be used to solve the LJP task. These can be broken down into two categories: large language models (LLMs) and domain-specific models. LLMs are capable of interpreting and generating complex natural language, while domain models are efficient in learning task-specific information. In this paper, we propose the precedent-enhanced LJP framework (PLJP) – a system that leverages the strength of both LLM and domain models in the context of precedents. Specifically, the domain models are designed to provide candidate labels and find the proper precedents efficiently, and the large models will make the final prediction with an in-context precedents comprehension. Experiments on the real-world dataset demonstrate the effectiveness of our PLJP. Moreover, our work shows a promising direction for LLM and domain-model collaboration that can be generalized to other vertical domains.

Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration
Yiquan Wu', Siying Zhou', Yifei Liu', Weiming Lu Xiaozhong Liu² Yating Zhang", Changlong Sun Fei Wu' Kun Kuang"
'Zhejiang University, Hangzhou, China Worcester Polytechnic Institute, USA
3 AAlibaba Group, Hangzhou, China
(wuyiquan, zhousiying, liuyifei, luwm, kunkuang jezkkuaanngeuu..... edu.cn, en, yatingz89@gmill.. xliu148wpi.ede edu, changlong. sele..oba@a com, wufeiêcs, zju.edu... edu.cn cn
Abstract
Human Judicial Process
Judicial Process
Legal Judgment Prediction (LJP) has become an increasingly crucial task in Legal AI, i.e., predicting the judgment of the case in terms of case description. Precedents are the previ- legal cases with similar facts, which are the basis for the judgment of the subsequent case in national legal systems. Thus, worthwhile explore the utilization of precedents in the LJP. Recent advances in deep learning have en- abled variety of techniques be used solve the IJP task. These can be broken down into two categories: large language models (LLMs) and domaii- models. LLMs capable of interpreting and generating complex natural language, while domain models are efficient in learning task-specific information. In this paper, we propose the pencedent-ennanced I.IP framework (PLJP) system that leverages the strength of both L.M and domain models in the context of precedents. Specifically, the domain models are designed to provide candidate labels and find the proper precedents efficiently, and the large models will make final prediction with an in in-context precedents comprehension. Experiments on the real-world dataset demon- strate effectiveness our PLJP. Moreover, work shows promising direction for LLM and domain-modee collaboration that can be generalized other vertical domains.
Care Investigation
Fact
Judge
Prior Krowledge
Domain- Domain-Modle Model
Precedemss
Retrieval
Case Comprehensicn
L.M-enabled Case Comprehension
Case Database
Judgment
Judgment
Figure 1: An illustration of the judicial process, our motivation is promote the collahoration between the domain model and L.M (right part) for simulating the judicial process the human judge (left).
Precedents, which refer to previous cases with similar fact descriptions, hold crucial position within national legal systems (Guillaume, 2011) 1). On more macro level, precedents are known the collective body of judge-made laws in a na- tion(Gamer,, 2001). They serve the purpose of en- suring consistency in judicial decisions, providing greater legal guidance to judges and facilitating lc- gal progress and evolution to meet dynamic legal demands. In the Common Law system, the prece- dents the mandatory basis of the judgment of the subsequent case (Rigoni, 2014 4). In the Civil Law system, judge-made laws are perceived as sec- ondary legal sources while written laws are the basic legal sources(Larenz, 1992). In the contem- porary era, there is also growing trend to treat the source of "soft" law (Fon and Parisi, 2006), and judges are expected to take them into ac- count when reaching decision (Guillaume, 201 1). Thus, it is worthwhile to explore the utilization of precedents in the legal judgment prediction.
1 Introduction
Legal AI has been the subject of rescarch for sev- eral decades, with the aim of assisting individuals various legal tasks, including legal QA (Mon- roy et al., 2009), court view generation (Wu et al., 2020). legal entity recognition (Cardellino et al., 2017), and so on. As one of the most important legal tasks, legal judgment prediction (LJP) aims to predict the legal judgment of the case based on the case fact description. The legal judgment typically includes the law article, charge and prison term.
precedents
With the development of deep learning, many technologies have been adopted in the LJP task, which can be split into two categories: large lan-
"Corresponding Authors.

---

guage models (LLMs) and domain- -eenain-ppeeific mod- els (Ge et al., 2023). Owing to extensive training. LLMs are good at understanding and generating complex natural language, as well as in-context learning. On the other hand, domain-specific mod- els are designed to cater to specific tasks and offer cost-effective solutions. However, when it comes to incorporating precedents into the LJP task, both cat- egories of models face certain limitations. L.Mss, constrained by their prompt length, struggle to grasp the meaning of numerous abstract labels and accurately select the appropriate one. For domain models, though trained with label annotations, the drawback is the limited ability to comprehend and distinguish the similarities and differences between the precedents and the given case.
ensure the absence of data leakage during LLM training. The results obtained on both the original and additional test sets validate the effectiveness of the PLJP framework.
Our work shows a promising direction for LLM and domain-model collaboration that can be generalized over vertical domains. We make all the codes and data publicly available motivate other scholars investigate this novel and interesting research direction
Related Work
2.1 Legal AI
Legal Artificial Intelligence (Legal AI) aims to en- hance tasks within the legal domain through the uti- lization of artificial intelligence techniques (Zhong et al., 2020; Katz et al., 2023). Collaborative ef- forts between researchers in both law and com- puter fields have been lasting to explore the poten- tial of Legal AI and its applications across vari- ous legal tasks. These tasks encompass areas such as legal question answering (QA) (Monroy et al., 2009), legal entity recognition (Cardellino et al.. 2017), court view generation (Wu et al., 2020), legal summarization (Hachey and Grover, 2006; Bhattacharya et al., 2019), legal language under- standing( Chalkidis et al., 2022) and so In this work, focus on the task of legal judg- ment prediction, which one of the most common tasks in Legal AI.
In this paper, Fig. 1 shows, try collab- orate the LLMs with the domain-specifie models and propose a novel precedent-cnhaneed legal judg- ment prediction framework (PLJP). Specifically, domain models contribute by providing candidate labels and finding the proper precedents from the case database effectively; the LLMs will decide the final prediction through an in-context precedent comprehension.
Following the previous LJP works (Zhong et al., 2018; Yue et al., 2021; Dong and Niu, 2021), our experiments are conducted on the publicly avail- able real-world legal datasct. To prevent any poten- tial data leakage during the training of the LLMs, where the model may have already encountered the test cases, we create new test set comprising cases that occurred after 2022. This is necessary because the LLMs we utilize have been trained on corpus collected only until September 2021. By doing so, we ensure a fair evaluation of the PLJP frame- work. Remarkably, our proposed PLJP framework achieves state-of-the-art (SOTA) performance on both the original test set and the additional test set. To sum up, our main contributions are as fol- lows:
2.2 Legal Judgment Prediction
Legal judgment prediction (LJP) aims to predict judgment results based on the fact descriptions au- tomatically (Lin et al., 2012; Chalkidis et al., 2019; Yue et al., 2021; Xu et al., 2020; Niklaus et al., 2021; Malik et al., 2021: Feng et al., 2022: Lyu et al., 2022: Gan al., 2022). The LJP methods in earlier years required manually extracted features (Keown, 1980), which is simple but costly. Owing to the prosperity of machine learning (Wu et al., 2022; Shen et al., 2022; Li et al., 2022a,b; Zhang et al., 2022; Li et al., 2023; Zhang et al., 2023), researchers began to formalize the LJP problem with machine learning methods. These data-driven methods can learn the features with far less labor (e.g., only the final labels are required). Sulea et al. (2017) developed an ensemble system that aver- ages the output of multiple SVM to improve the
We address the important task of legal judg- ment prediction (LJP) by taking precedents into consideration.
We propose novel precedent-enhanced legal judgment prediction (PLJP) framework that leverages the strength of both LLM and do- main models.
We conduct extensive experiments on the real- world dataset and create an additional test set
'https:/ thub. com www/wmym////wcttc http//wyiquu puwp/PLLaa

---

performance of LJP. Luo et al. (2017) utilized an attention mechanism in the LJP. Zhong et al. (2018) considered the dependency of the sub-tasks in the LJP. Yue et al. (2021) investigated the problem by separating the representation fact description into different embedding. Liu et al. (2022) used contrastive learning in the LJP.
However, in the prediction tasks, which can in- volve dozens of abstract labels, the performance of LLMs is not as good as generation tasks, due the limited prompt length. this paper, we explore the utilization of LLMs in the LJP task with the collaboration of domain speein--peifffcc models.
However, these existing LJP methods tend to overlook the significance precedents. In this study, we propose precodent-cnhannee LJP frame- work (PLJP) that leverages the collaboration be- tween domain-specific models and large language models (LLMs) to address the L.PP task.
Problem Formulation
In this work, we focus on the problem of legal judgment prediction. We first clarify the definition of the terms as follows.
Fact Description refers to a concise narrative of the case, which typically includes the timeline of events, the actions conduct of cach party, and any other essential details that relevant the case. Here we define it a token sequence f 4
2.3 Precedent Retrieval
The precedent is the basis of judgment in the Com- mon Law system, and also an important refer- ence for decision-making in the Civil Law system. Therefore, precedent retrieval is another valuable task in Legal AI (Althammer al., 2021). There are two main precedent retrieval models: expert knowledge-basse models and natural language pro- cessing (NLP)-based models (Bench-Cappe et al., 2012). Expert knowledge-band designed sub-elements to represent the legal cases (Saravanan et al., 2009), while NLP-based models mainly convert the text into embeddings and then calculate the similarity from the embedding level (Ma al., 2021; Chalkidis al., 2020).
[ JE= where 1 is the length.
Judgment is the final decision made by a judge in legal case based on the facts and the precedents. typically consists of the law article, the charge, and the prison term. We represent the judgment case as j = (a, e, l), where a, c, refer to the labels of article. charge and prison term, respectively. Precedent is the previous case with similar fact. The judgments of the precedents references for the current case. Here, a precedent is defined as p (p). where fp its fact descrip- tion and jp is its judgment. For a given case, there can be several precedents, which can be denoted where n is the number of as P = (PL.P2, precedents.
models use the
important
Most retrieval models required additional anno- tation so can not be directly applied to the LJP task. In our paper, we use unsupervised dense retrieval model (Izacard et al., 2022) to get the precedents, which be updated by other retrieval models needed.
Then the problem be defined as:
Problem 1 (Legal Judgment Prediction). Given the fact description f, our task is to get and compre- hend the precedents P, then predict the judgment j (a, c)
2.4 Large Language Models
Large language models (LLMs), such ChatGPT, have attracted widespread attention from society (Zhao al., 2023). With pre-training over large- scale corpora, LLMs show strong capabilities in interpreting and generating complex natural lan- guage, as well as reasoning (e.g., in-context learn- ing). The technical evolution of LLMs has been making an important impact on the fields of natural language processing (Brown et al., 2020; Touvron al., 2023), computer vision (Shao al., 2023; Wu et al., 2023), and reinforcement learning (Du et al., 2023). the legal domain, LLMs can also be used for many tasks such as legal document analysis and legal document writing (Sun, 2023).
4 Eircedent-Enhanceee LJP (PLJP)
In this section, we describe our precedent-enhanced legal judgment prediction framework (PLJP), Fig. 2 shows the overall framework.
4.1 Case Database Construction
Before we use the precedents, we have to collect a large number of previous cases to construct case database. Since the fact descriptions are usually long and elaborate, it is difficult for the models get the proper precedents. To this end, we reor- ganize the fact description of these previous cases with the help of LLMs.

---

Article 263
Robbery
months
Defendant deliberately
sub: subjective motivation
followed and demanded money from B.
obj objective behavior
Predicted
P Judgment
sub
post facto circumstance
At about 2 a.m. October 2017,
law artiele charge
defendant A the west side of the road of the C City. following B who passed and asking B for $200 in cash, and was furced to hand over one mobile phone to The price certification
Defendant A forced B hand over mobile phone.
t: prison term
obj
LLM
After Arrest, returned $200 illegal gains Reorganized Fact
Case
Fact
Judgmant
(SuB. abj.
(a.
(sub. abj.
(a.
Case Database
Precedents
During the trial the case, defendant refunded S200 illogal gains.
Fact Description
Domain Model (Predictive Model)
Candidate Labels
Domain Model (Retrieval Model)
Figure 2:
The overall framework of PLJP, where the sub, obj and ex refer to the subjective motivation, objective behavior and ex post facto circumstance, respectively. The solid lines are the precedent retrieval process, while the dotted lines represent the process of the prediction.
4.1.1 Fact Reorganization
cates the subjective motivation, objective behavior, and ex post facto circumstances, respectively. Fi- nally, a previous case in the case database is stored as pair of reorganized facts and the judgment.
Given a fact description of case, we summarize it from three aspects: subjective motivation, objec- tive behavior, and post facto circumstances. The reorganization doesn't 't require human annotation and is completed by the LLMs with the following prompts: faca description can be categorized into subjective motivation, objective behavior, and ex post facto circumstances. Subjective motiva- tion refers to the psychological attitude of the per- petrator towards their harmful actions and their consequences, including intent, negligence, and purposes of the crime. Objective behavior per- tains the necessary conditions for constituting crime in terms of observable activities, including harmful conduct, harmful results, and the causal relationship between the conduct and the results. Ex post facto circumstances are various factual sit- uations considered when determining the severity of penalties. Mitigating circumsiances for lenient punishment include voluntary surrender and meri- torious conduct, while aggravating circumstances for harsher punishment include recidivism. Based on the provided information, your task is to sum- marize the following facts."
4.2 Legal Judgment Prediction
Next, we describe the collaboration of the LLM and domain models in legal judgment prediction.
4.2.1 Domain Models
The domain models are trained on specific datasets, aiming solve certain tasks. Here, we use two kinds of domain models, including the predictive model and the retrieval model.
Predictive model. The predictive model takes the fact description as the input and outputs the candidate labels of the three sub-tasks (e.g.. law article, charge, prison term). Since the fact descrip- tion {wf \w transform it into embedding sequence H yxd with an Encoder:
sequences of words, first
H' Encode(f),
(1)
The reorganization reduces the length of facts and makes the precedents easy to get and compre- hend in the PLJP.
where H³ = 11f, hb.,.., ..., "e,,...,,,, n/,, and d the dimension of the embedding.
We take max-pooling operation to obtain the pooled hidden vector h/ E Rd and then feed it into a fully-connected network with softmax activation
After the reorganization, the fact description f is translated to a triplet (sub, obj, ex), which indi-

---

obtain the label probability distribution P E R":
Fact Reorganization The fact reorganization is described in case database construction (Sec. 4.1.1), which aims to summarize the fact descrip- tion from three aspects by the LLMs. Besides the database contribution, as Fig. shows, when a new test case comes,, the LLMs will reorganize the fact description with the same prompt.
h/ = MaxPooling(H1
(2)
P Softmax(WP hf + 6%),
where WP E Rmdd and 8 R are learnable parameters. Note m varies different sub-tasks. Then, each sub-task gets its candidate labels ac- cording to the probability distribution P, and the number of candidate labels is equal to the number of precedents n.
In- Cn-CCotextt Precedent Comprehension Since LLMs are capable understanding complex nat- ural language, we stack the given case with its precedents and let the LLMs make the final pre- diction by an in-context precedent comprehension. Specifically, the prompt of law article prediction is designed as follows: "Based the facts, we select the candidate law articles by the domain models and select the following three precedents based on the candidate law articles. Please comprehend the difference among the precedents, then compare them with the facts of this case, and choose the final label."
Retrieval model. The retrieval model aims get the proper precedents of the given case based on its reorganized fact (sub, eb).
Formally, to get the similarity score of any two texts D1 and D2, we will first encode each of them independently using the same encoder:
hn, = Encoder(Di), hn, = Encoder (D2),
(3)
where hD, E IRd and hD, c dRd are the embed- ding of each, d' is the dimension. The similarity score sD11, D2) is then the cosine similarity of the hD, and hzz
Consider the topological dependencies among the three sub-tasks (Zhong et al., 2018), 3), in the pre- diction of charge. add the predicted law article in the prompt; the prediction of prison term, we add the predicted law article and charge.
hD, hDz [1pp Il [
s( s(D,, D2)
(4)
4.3 Training
In PLJP, considering the realizability, we train do- main models on legal datasets and leave the LLMs unchanged. To train predictive models, the cross- entropy loss is employed. As for retrieval models, contrastive loss is used like Izacard al. (2022).
Here we concatenate the sub, obj and ez into a whole text to calculate the similarity score of the given case and the cases in the case database. For each candidate label, we pick one as the precedent: the case that has the highest similar- ity score and has the same label. For example, if the label "Theft" is the candidate labels in the charge prediction, we will find the most similar pre- vious case with the same label as the corresponding precedent. The one-to-one relationship between the candidate label and precedent helps the LLM dis- tinguish the differences among the labels. In other words, the precedent serves supplementary explanation of the label. Finally, we get precedents P {p1, (p1.P2.-.. P..) for the given case.
Experiments
Type
CAIL2018
CJO22
# I Law Article
164
164
# Charge
42
42
# Prison Term
10
10
Sample
82138
1698
Avg. # words in Fact
288.6
461.7
Table Statistics of datasets.
5.1 Datasets
4.2.2 LLMs
Following many influential LJP works (Zhong et al., 2018; Xu et al., 2020; Yue et al., 2021; Dong and Niu, 2021), our experiment is conducted on the widely used and publicty available CAIL2018 dataset, which is Chinese dataset in the con- text of People's Republic of China (PRC). This dataset consists of worl-world cases, each of which
The large language models models with billions of parameters, which trained large-scale cor- pora, and show strong capabilities in interpreting and generating complex natural language. LLMMs contribute to PLJP by fact reorganization and in- context precedent comprehension.

---

CJO22
CAIL.2018
Method
Acc
Ma-P
Ma-R
Ma-F
Ma-P
Ma-R
Ma-F
CNN (LeCun al., 1989)
76.14
35.48
38.55
35.39
80.50
40.10
38.33
38.49
BERT (Devlin al., 2019)
82.62
45.89
47.91
45.83
82.77
36.82
35.94
35.82
Roberta (I.ju et al., 2019)
80.32
42.36
44.22
41.80
83.08
48.09
44.25
44.87
TopJudge (Zhong et al., 2018)
78.73
40.38
41.47
40.09
80.46
40.96
40.96
38.24
R-Former (Dong and Niu, 2021)
87.69
53.03
49.35
50.23
87.82
56.13
56.57
55.81
LADAN (Xu et al., 2020)
79.44
48.43
44.13
46.18
82.82
42.57
39.00
40.71
NeurJudge (Yue et al., 2021)
71,38
52.86
53.52
52.62
76.91
55.95
52.92
53.56
EPM(Feng et al., 2022)
84.19
47.21
43.79
44.39
85,80
49.08
45.76
47.32
CTM(Liu et al., 2022)
79.44
47.83
42.25
43.43
84.72
46.46
44.83
45.10
Dav003
2.10
0.82
0.17
0.26
1.02
0.30
0.08
0.13
3.Sturbo
9.13
2.54
1.61
1.53
4.08
4.95
3.64
2.30
PLJP(CNN)
87.67
55.21
$5.59
54.37
86.05
58.08
56,46
54.92
PLJP(BETT)
94.18
74.65
76.23
74,84
87.07
58.81
57.29
56.63
Table Results law article prediction, the best is bolded and the second best underlined.
CJO22
CAIL2018
Method
Acc
Ma-P
Ma-R
Ma-F
Acc
Ma-P
Ma-R
Ma-F
CNN (LeCun et al., 1989)
74,91
74.00
78.12
73.97
87.52
88.23
88.31
88.17
BERT (Devlin et ul., 2019)
80.50
80.34
81.09
78.36
89.10
90.10
89.48
89.63
Roberta (Liu et al., 2019)
79.26
78.93
81.25
78.18
90.30
91.02
90.97
90.94
TopJudge (Zhong et ul., 2018)
76.67
74.00
77.40
74.62
87.31
88.68
87.84
88.20
R- R-Former (Dong and Niu, 2021)
90.71
93.06
88.66
89.82
91.54
91.61
91.96
91.58
LADAN (Xu et al., 2020)
79.64
48.43
44.13
46.18
88.09
90.12
88.82
89.47
NeurJudge (Yue et al., 2021)
71,85
69.37
71.09
68.66
82.13
82.71
82,30
82.36
EPM(Feng al., 2022)
83,49
80.36
83.29
81.87
91.20
90.81
89.99
90.46
CTM(Liu et al., 2022)
79.33
82.39
83.12
82.81
90.28
90.34
88.08
86.30
Dav003
44.65
52.43
32.93
35,29
25.85
35.37
25.09
22.08
3,Stub
58.37
56.03
40.68
42.62
49.65
42,29
34.05
31.85
PIJP(CNN)
91.62
83.43
84.88
83,40
91.49
81,80
83.95
80.06
PIJP(BRTT)
94.18
90.25
88.67
89.05
94.99
92.12
91.10
91.33
Table 3: Results of charge prediction, the best bolded and the second best is underlined.
includes fact description accompanied by com- plete judgment encompassing three labels: law ar- ticles, charges, and prison terms?.
Ma-FF
CARB.200
106
91.33
81.00
70.82
14.84
To mitigate the potential data leakage during the training of LLMs, which were trained corpora collected until September 2021, we have compiled new dataset called CJO22. This dataset exclu- sively contains legal cases that occurred after 2022, sourced from the same origin as CAIL20183. How- ever, due to its limited size, the newly collected CJO22 dataset is inadequate for the training pur- poses of the domain models. Consequently, we utilize it solely itate meaningful comparisons, retain only the labels that common both datasets, consider- ing that the labels may not be entirely aligned.
44.35
31.98
35.43
Land Oagge Friua Tem
-B-Chap: Pore Ten
Number of Precedents
Figure 3: The Ma-F of PLJP with different number of precedents.
The previous cases in the case database are sam- pled from the training dataset, and we set the amount to 4000.
additional test set. To facil-
5.2 Baselines
For domain-specific LJP baselines, we implement the following for comparison:
Tab. shows the statistics of the processed conducted
datasets, and all the experiments the same datasets. For CAIL22018 dataset, randomly divide it into training set, validation and test set according to the ratio of I.
CNN (LeCun et al., 1989) extracts text features through convolutional operations with different ker- nels for text classification; BERT(Devlin al., 2019) a pre-trained language model and can be casily fine-tuned on the downstream tasks; Top- Judge (Zhong et al., 2018) use multi-task learn-
PPrison terms are divided into non-overlapping intervals. Phttps: http://wwnshcooceeee court. court.gov..... cn/

---

CJO22
CAIL.011
Method
Acc
Ma-P
Ma-R
Ma-F
Ma-P
Ma-R
Ma-F
CNN (LeCun al., 1989)
27.38
18.48
17.51
17.44
34.42
32.22
30.53
31.05
BERT (Devlin al., 2019)
36.80
29.83
27.50
27.03
40.00
37.53
33.66
33.58
Roberta (I.iu et al., 2019)
29.74
24.73
24.76
23.22
40.84
38.62
38.55
38,50
TopJudge (Zhong et al., 2018)
27.14
19.76
17.69
17.94
35.54
33.55
31.08
32,00
R-Former (Dong and Niu, 2021)
38.63
32.63
32.76
29.51
40.70
36.09
36.76
35.04
LADAN (Xu et al., 2020)
33,69
26.40
22.94
24.55
38.03
33.66
30.08
31.77
NeurJudge (Yue et al., 2021)
26.80
26.81
26.85
25.97
33.53
36.46
37.26
36.53
EPM(Feng et al., 2022)
36.91
30.65
31.61
30.20
40.25
37.96
37.00
37.34
CTM(Liu et al., 2022)
36.81
27.10
25.96
26.46
39.56
38.66
38.02
37.84
Dav003
0.47
5,56
0.21
0.41
0.68
10.38
0.49
0.94
3.Sturbo
1.40
1.16
1.07
1.11
1.02
2.71
1.13
1.15
PLJPICNN)
36,51
20.21
21.44
20.07
40.81
32.77
35.59
25.71
PLJP(BERT)
43,52
33.37
35.67
31.98
48.72
42.64
36.80
35.43
Table 4: Results of prison term prediction, the best bolded and the second best is underlined.
CJO22
CAIL2018
Method
Law Article
Charge
Prison Term
Law Article
Charge
Prison
Term
Ace
Ma-F
Acc
Ma-F
Ace
Ma-F
Acc
Ma-F
Acc
Ma-F
Acc
Ma-F
w/o
54,65
28.32
83.48
76.33
35.81
20.84
85.03
51.54
85.03
70.07
32.31
22.58
wfo
45.34
40.22
42.32
41.85
32.55
20.26
67.35
46.65
72.79
60.34
26.53
13.66
wfo
94.18
74.84
85.58
70.50
39.53
20.31
87.07
56.63
87.41
73.45
38.09
21.44
w/o
88.13
58.75
87.67
74.83
36.27
23.70
86.05
58.26
86.73
77.53
38.10
21.70
w/c
90.70
67.90
80.70
66.53
35.35
20.21
89.80
61.64
85.37
68.48
38.44
23.14
PLJP
94.18
74.84
94.18
89.05
43,52
31.98
87.07
56.63
94.99
91.33
48.72
35,43
Table 5: Results of ablation experiments, best bolded and the second best underlined.
ing and capture the dependencies among the three sub-task in LJP; NeurJudge (Yue et al., 2021) splits the fact description into different parts for making predictions; R-Former (Dong and Niu, 2021) formalizes LJP as node classification prob- lem over a global consistency graph and relational learning is introduced; LADAN (Xu et al., 2020) uses graph distillation to extract discriminative fea- tures of the fact Retri-BERT (Chalkidis and Ke- mentchedjhieva, 2023) retrieves similar documents augment the input document representation for multi-label text classification; EPM (Feng al., 2022) Jocates event-related information essential for judgment while utilizing cross cross-task consistency constraints among the subtasks: CTM (Liu al., 2022) establishes a LJP framework with case triple modeling from contrastive case relations.
and PLJP(BERT), The top-k accuracy of CNN and BERT is shown in the Appendix. Considering the length limit of the prompt, we set the number of precedents to 3.
We also do ablation experiments as follows: PLJPP w/o refers to the removal of precedents, and the prediction of labels is done solely based on the candidate labels using the LLM; PL.JP w/o denotes we remove the candidate labels and pre- dict the label only with the fact description and precedents; PLJP w/o means we predict the three labels independently instead of considering the dependencies among the three subtasks; PLJP w/o r denotes we find precedents based the raw fact instead of from the reorganized fact; PLJP w/ e means we let the LLMs generate the explanation of the prediction as well.
We use the LLM baselines followst: Dav003 means the text-davinci-003. 3.5turbo means the gpt-3.5-turbo. These LLMs are both from the GPT- 3.5 family, released by OpenAl and can understand and generate complex natural language
In the ablation study, PLJP means PLJP(BERT).
5.3 Experiment Settings
Here we describe the implementation of PLJP our experiments. Note all the L.Mss and domain models are replaceable in the PLJP framework. In the experiments, for the LLMs, we directly use the APIs provided by OpenAL. For the domain models, we use the unsupervised dense retrieval model (Izacard et al., 2022) precedent retrieval, which gets the precedents from the case database
For PLJP, take the CNN and BERT as the pre- dictive models, and take the text-davinci-O0) as the implementation of the LLM, named PLJP(CNN)
*We give a fixed example in the prompt help the LLMs understand tasks. 'https: //platfor.m openai. comnai.comm. cow htn//ooss/moddll

---

Given Case
On June 23, 2021, defendant leased loader from for the Z Project, agreeing monthly of $15,000, A.sioodd a w.w.a.aaaiss.. Isssing contract and subsequently sold the londer the next day for $70,000 A paid total of $32,000 in rent to D. The Price Service Center determined that the deceived loader was worth S61.167.
Around July 23, 2021, defendant A leased another loader from F for Project Z, agreeing to monthly rent of $15,000. Adia July 24 for $54,000_ On July 26, 2021, A leased Y louder from for the project, ngreeing monthly rent of $15,000. Qn the same Aay A lowed the vshisls to Eand sold_it H for 000,000 Later, F's roquest, A and F signed loader lesse contrct paid a total of $28,00 in rent to F. The Price Service Center desermined that the deceived loaders were worth $54, 167 and S46,200, respectively.
Aoters lesring the loader
E. sold to
lentu tained funds were used by to settle debis, cover living expenses, and indulge lavish spending. was also conf formeed defendant surrendered
public security authorities August 3, 2021.
Candidate labels from domain models: Contract Fraud, Froud, Extortion
Precedents
Precedeut for Contract Fraud Sub: Defendant deliberntely defruuded
Precedent for Fraud
Precedent for Extortion
Sub: Defendumt deliberately defrauded the
Sub: Defendant delibenately attacked victim. Ohj: Defendaant A entered the home of victim B with knife, beat him, and destroyed the property in B's home, causing men tal and property damage. Ex: None.
the the name of unother penson.
Vietim for the purposs of iilennl nossession.
Obj: lo the process of siening the contract,
Obj: Defsudont Adceoooded the vistim of $30000 on the groumds that he could apply for drivers license and squandered the stolen money.
the defendant fraudulently used the name
of another person to obtain 2 cars. Ex: None.
Defendant surrendered the public security.
Predicted Judgment
R-Former: Contract Fraud>X
BERT: Contract FraudXX
PLIP(BERT): Frauck
The charge prediction of given case. The green parts are useful information for prediction, while the red parts are content that can be confused by the domain models.
Figure 4:
according to the reorganized facts. For other do- main models such as TopJudge and NeurJudge, use the training settings from the original paper.
Results of ablation experiment: From Tab. 5, can conclude that: 1) The performance gap of the PLJP w/o p and PLJP demonstrates the ef- fects of the precedents. 2) The results of PLJP w/o с prove the importance of the candidate labels. 3) Considering the topological dependence of the three sub-tasks benefits the model performance as PLJP w/o d shows. 4) When we use the raw fact instead of the reorganized fact, the performance drops (e.g., the Ace of prison term in CJO22 drops from 45.32% to 36.27%). 5) If we force the LLMs to generate the explanation of the prediction, the performance also drops bit. We put cases with explanations in the Appendix.
For the metrics, we employ Accuracy (Acc), Macro-Precision (Ma-P), Macro-Recall (Ma-R) and Macro-F1 (Ma-F).
5.4 Experiment Results
We amalyze the experimental results in this section.
Result of judgment prediction: From Tab. 2, Tab. and Tab. 4, we have the following observa- tions: 1) The L.MS perform not well in the predic- tion tasks alone, especially when the label has no actual meaning (e.g., the index of the law article and prison term). 2) By applying our PLJP frame- work with the collaboration of LLMs and domain models, the simple models (e.g., CNN, BERT) gain significant improvement. 3) The model perfor- mance on CJO22 is lower than that on CAIL2018, which shows the challenge of the newly constructed test set. 4) (BER() achieves the best perfor- mance in almost all the metric evaluation metrics in both CAIL2018 8 and CJO22 test sets, which proves the effectiveness of the PLJP. 5) Compared to the prediction of the law article and charge, the pre- diction of prison term is still a more challenging task. 6) The reported results of the LJP baselines are not as good as the original papers, this may be because we keep all the low-frequency labels instead of removing them as the original papers did.
From Fig. 3, we can find that the performance of PLJP improves as the number of precedents in- creases, which also proves the effectiveness of in- jecting precedents into the LJP.
5.5 Case Study
Fig. 4 shows an intuitive comparison among the three methods in the process of charge prediction. Based on the fact description of the given case, the domain models provide candidate charges with the corresponding precedents. As the case shows, the defendant made fraud by selling the cars that were rented from other people. However, since there contains "contract" in the fact description, base- lines (e.g., R-Former and BERT) can be misled and predict the wrong charge of "Contract Fraud". Through an in-context precedent comprehension by the LLMs, PLJP(BERT) distinguishes the dif-

---

ferences among the precedents and the given case (e.g., the crime does not occur during the contract- ing process, and the contract is only a means to commit the crime), and give the right result of "Fraud""
as well as in other legal datasets (e.g.. the datasets from the Common Law system).
Acknowledgments
This work supported in part by National Key Research and Development Program of China (2022YFC3340900) National Natural Sci- ence Foundation of China (62037001, 62376243, U20A20387), the StarryNight Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU-SIAS-0010)) Young Elite Scientists Sponsorship Program by CAST (2021 22021QRCC01), Project by Shanghai Al Lab- oratory (P22KS00111), Program of Zhejiang Province Science and Technology (2022C01044), Fundamental Research Funds for the Central Universities (226-2023-00660), Key Research and Development Program of Zhejiang Province (2021C01013),
6 Conclusion and Future Work
this paper, we address the important task of le- gal judgment prediction (LJP) by taking precedents into consideration. We propose novel framework called precedent-cnhancee legal judgment predic- tion (PLJP), which combines the strength of both LLMs and domain models to better utilize (e.g.. retrieve and comprehend) the precedents. Experi- ments on the real-world dataset prove the effective- ness of the PLJP.
Based on the PLJP. in the future, we can ex- plore the following directions: 1) Develop methods to identify and mitigate any biases that could af- feet the predictions and ensure fair and equitable outcomes. 2) Validate the effectiveness of LLM and domain collaboration in other vertical domains such as medicine and education.
Finally, we would like to thank the anonymous reviewers for their helpful feedback and sugges- tions.
6.1 Ethical Discussion
References
With the increasing adoption of Legal Al in the field of legal justice, there has been growing awareness of the ethical implications involved. The potential for even minor errors or biases in Al-powered sys- tems can lead to significant consequences.
Sophia Althammer, Arian Askari, Suzan Verberne, and Allan Hanbury. 2021. Dossier@colice 2021: Leveraging dense retrieval and summarization- based re-ranking for case law retrieval. CoRR, 0S5/2108.03977.
Trevor J. M. Bench-Capon, Michal Araszkiewicz. Kevin D. Ashley, Katie Atkinson, Floris Bex, Filipe Borges, Danièle Bourcier, Paul Bourgine, Jack G. Conrad, Enrico Francesconi, Thomas Gordon, Guido Governatori, Jochen I.. Leidner, David D. Lewis, Ronald Prescott Loui, L. Thorne McCarty, Henry Prakken, Frank Schilder, Erich Schweighofer, Paul Thompson, Alex Tyrell,, Bart Verheij. Dou- glas N. Walton, and Adam Z. Wyner. 2012. A history of Al and law in 50 papers: 25 years of the interna- tional conference on Al and law, Artif. Intell. Law, 20(3):215-31.
In light of these concerns, we have to claim that our work is an algorithmic exploration and will not be directly used in court so far. Our goal is to pro- vide suggestions to judges rather than making final judgments without human intervention. In practi- cal use, human judges should be the final safeguard protect justice fairness. In the future, we plan to study how to identify and mitigate potential biases to ensure the fairness of the model.
7 Limitations
Paheli Bhattacharya, Kaustubh Hiware, Subham Raj- garia, Nilay Pochhi, Kripabandhu Ghosh, and Sap- tarshi Ghosh. 2019. A comparative study of summa- rization algorithms applied to legal case judgments. In Advances in Information Retrieval-44 -41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14-18, 2019, Proceedings, Part I, volume 11337 of Lecture Notes in Computer Science, pages 413 428. Springer.
this section, we discuss the limitations of our works as follow:
We only interact with the LLMs one round per time. The LLMs are capable of multi-round interaction (e.g., Though of Chains), which may help the LLM to better understand the LJP task.
We validate the effectiveness of LLM and do- main model collaboration in the legal domain. It's worthwhile to explore such collaboration in other vertical domains such as medicine and education,
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss

---

Gretchen Krueger, Tom Henighan. Rewon Child. Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Yuqing Du, Olivia Watkins, Zihan Wang. Cédric Co- las, Trevor Darrell, Pieter Abbeel, Abhishek Gupta. and Jacob Andreas, 2023. Guiding pretraining in reinforcement learning with large language models. CoRR, abs/2302.06662.
Yi Feng, Chuanyi Li, and Vincent Ng. 2022. Legal judgment prediction: A survey of the state of the art. In Proceedings of the First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5461-5469. ijcai.org.
Cristian Cardellino, Milagro Teruel, Laura Alonso Ale- many, and Serena Villata. 2017. Legal NERC with ontologies, wikipedia and curriculum learning. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers, pages 254-259. Asso- ciation for Computational Linguistics.
Vincy Fon and Francesco Parisi. 2006. Judicial prece- dents in civil law systems: A dynamic analysis. Inter- national Review of Law and Economics, 199 535.
Leilei Gan, Baokui Li, Kun Kuang. Yi Yang, and Fei Wu. 2022. Exploiting contrastive learning and numerical evidence for improving confusing legal judgment prediction. CoRR, abs/2211.08238.
Iliss Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras 2019. Neural legal judgment prediction in english. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Vol- ume 1: Long Papers, pages 1317-4323, Association for Computational Linguistics.
Bryan A. Garner. 2001. A dictionary of modern legal usage. Oxford University Press, New York.
Yingqiang Ge, Wenyue Hua, Jianchao Ji, Juntao Tan, Shuyuan Xu, and Yongfeng Zhang. 2023. Ope- nagi: When LLM meets domain experts. CoRR, 005/2304.04370.
This Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: the muppets straight out of' law school. CoRR, abs/2010.02559.
Gilbert Guillaume. 2011. The use of precedent by inter- national judges and arbitrators. Journal of Interna- tional Dispute Settlement, 2(1):5-23.
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael J. Bommarito II, Ion Androutsopoulos, Daniel Mar- tin Katz, and Nikolaos Aletras. 2022. Lexglue: A benchmark dataset for legal language understanding in english. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4310-4330, Association for Computational Linguistics.
Ben Hachey and Claire Grover. 2006. Extractive sum- marisation of legal texts. Artif. Intell. Law, 14(4):305- 345.
Gautier Izacard, Mathilde Caron, Lucas Hosscini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave 2022. Unsupervised dense in- formation retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022.
Iliss Chalkidis and Yova Kementchedjhieva. 2023. netrieval-augmmented multi-labed text classification. CoRR, abs/2305. 13058.
Daniel Martin Katz, Dirk Hartung, Lauritz Gerlach, Abhik Jana, and Michael J Bommarito II. 2023. Nat- ural language processing in the legal domain. arXiv preprint 2302v: 12039.
Jacob Devlin, Ming-Was Wing-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-FHILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Keown. 1980. Mathematical models for legal predic- tion. Computer/lj, 2:829.
Karl Larenz. 1992.
Methodenlehre der Rechtswis- senschaft. Springer, Berlin, Heidelberg.
Yann LeCun, Bernhard E. Boser, John S. Denker, Don- nie Henderson, Richard E. Howard, Wayne E. Hub- bard, and Lawrence D. Jackel. 1989. Backpropa- gation applied to handwritten zip code recognition. Neural Comput., 1(4):441-551
Qian Dong and Shuzi Niu. 2021. Legal judgment predic- tion via relational learning. In SIGIR 21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 983-992. ACM.
Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji, and Fei Wu. 2023. WINNER: weakly-superviser hierarchical de- composition and alignment for spatio-temporal video

---

grounding. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancou- ver, BC, Canada, June 17-24, 2023, pages 23090- 23099. IEEE
Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam. Kripabandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. 2021. ILDC for CJPE: indian legal documents corpus for court judg- ment prediction and explanation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACL/UCCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4046-4062. Associa- tion for Computational Linguistics.
Mengze Li, Tianbao Wang, Haoyu Zhang. Shengyu Zhang. Zhou Zhao. Jiaxu Miao, Wenqiao Zhang, Wenming Tan, Jin Wang, Peng Wang, Shiliang Pu, and Fei Wu. 2022a. End-to-end modeling via in- formation tree for one-shot natural language spatial video grounding. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27. 2022. pages 8707-8717. Associ- ation for Computational Linguisties.
A Alfredo Monroy, Hiram Calvo, and Alexander F. Gel- bukh. 2009. NLP for shallow question answering of legal documents using graphs. Computational Linguistics and Intelligent Text Processing. 10th In- ternational Conference, CICLing 2009, Mexico City, Mexico, March 1-7, 2009. Proceedings, volume 5449 of Lecture Notes in Computer Science, pages 498- 508. Springer.
Mengze Li, Tianbao Wang. Haoyu Zhang, Shengyu Zhang, Zhou Zhao. Wenqiao Zhang, Jiaxu Miao, Shiliang Pu, and Fei Wu. 2022b. HERO: hicrarchical spatio-temporal reasoning with contrastive action cor- respondence for end-to-end video object grounding. In MM '2: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 14, 2022, pages 3801-3810. ACM.
Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. 2021. Swiis-judgmentt-reedeeee A multilin- gual legal judgment prediction benchmark. CoRR, abs/21 10.00806.
Wan-Chen Lin, Tsung-Ting Kuo, Tung-Jia Chang, Chueh-An Yen, Chao-Ju Chen, and Shou-de Lin. 2012. Exploiting machine learning models for chi- nese legal documents labeling, case classification, and sentencing prediction. Inl. J. Comput. Linguis- tics Chin. Lang, Process., 17()..
Adam Rigoni. 2014. and analogy. Legal Theory, 20(2):133-1) 156
Common-lav judicial reasoning
M. Saravanan, Balaraman Ravindran, and S. Raman. 2009. Improving legal information retrieval using an ontological framework. Artif. Intell. Law, 17(2):101- 124.
Dugang Liu, Weihao Du, Lei Li, Weike Pan, and Zhong Ming. 2022. Augmenting legal judgment prediction with contrastive case relations. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 2658-2667. Inter- national Committce on Computational Linguistics.
Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuris- tics for knowledge-basse based visual question answering. CoRR, abs/2303.0/ 0.0900...
Kai Shen, Yichong Leng, Xu Tan, Siliang Tang, Yuan Zhang, Wenjie Liu, and Edward Lin. 2022. Mask the correct tokens: An embarrassingly simple approach for error correction. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates. December 7-11, 2022, pages 10367-10380. Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CORR, abs//197. 16692.
Bingfeng Luo,, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to predict charges for criminal cases with legal basis. In Pro- ceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2727-2736. Association for Computational Linguis- tics.
Octavia- Sulca, Marcos Zampieri. Shervin Mal- masi, Mihaela Vela, Liviu P. Dinu, and Josef van Gen- abith. 2017. Exploring the use of text classification in the legal domain. In Proceedings of the Second Work- shop on Automated Semantic Analysis of Information in Legal Texts co-located with the 16th International Conference Artificial Intelligence and Law (ICAIL 2017), London, UK, June 16, 2017, volume 2143 of CEUR Workshop Proceedings.
Yougang Lyu. Zihan Wang, Zhaochun Ren, Pengjie Ren, Zhumin Chen, Xiaozhong Liu, Yujun Li, Hongsong Li, and Hongye Song. 2022. Improving legal judg- ment prediction through reinforced criminal element extraction. Inf. Process. Manag., 59(1):
Zhongxiang Sun. 2023. A short survey of view- ing large language models in legal aspect. CoRR, abs/2303.09136.
Yixiao Ma. Yunqiu Shao, Bulou Liu, Yiqun Liu, Min Zhang, and Shaoping Ma. 2021. Retrieving legal cases from a large-scale candidate corpus. Proceed- ings of the Eighth International Competition on Legal Information Extraction/Entaai Entailment, COLJEE202).
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal

---

Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, abs/2302.13977
Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu. Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, 1ls/2303.1823..
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, ahs/2303.04671
Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judg- ment prediction via topological learning. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 3540 -3449. Association for Computational Linguisties.
Linjuan Wu, Shaojuan Wu, Xiaowang Zhang, Deyi Xiong, Shizhan Chen, Zhiqiang Zhuang, and Zhiy- ong Feng. 2022. Learning disentangled semantic representations for zero-shot cross-lingual transfer in multilingual machine reading comprehension. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), ACL. 2022, Dublin, Ireland, May 22- 27, 2022, pages 991-1000. Association for Computa- tional Linguistics.
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang. Zhiyuan Liu, and Maosong Sun. 2020. How does NLP benefit legal system: A summary of le- gal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 5218-5230, Association for Computa- tional Linguistics.
Yiquan Wu, Kum Kuang. Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. 2020. De-biased court's view generation with causality. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 763-780. Association for Computational Linguistics.
A Appendices
A.1
Top-k Accuracy
Nuo Xu, Pinghui Wang. Long Chen, Li Pan, Xiaoyan Wang, and Junzhou Zhao. 2020. Distinguish con- fusing law articles for legal judgment prediction. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 3086-3095. Associa- tion for Computational Linguistics.
Acc
CAIL2018
94.48
95.84
97.12
97.92
100.00
80.50
90.00
93.52
95..88
80.00
87.52
91.20
88.16
70.00
66.56
78.56
60.00
53.04
50.00
34.42
40.00
Linan Yue, Qi Liu, Binbin Jin, Han WU, Kai Zhang, Yanqing An. Mingyue Cheng, Biao Yin. and Day- ong Wu. 2021. Neurjudge: A circunstance-wware neural framework for legal judgment prediction. In SIGIR '21: The 44th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-5,, 2021, pages 973 982. ACM.
30.00
20.00
10.00
Law Article -8 Charge -
Prison Term
0.00
2
4
5
Top-k
Figure 5: The top-k accuracy of CNN on CAIL dataset.
Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yuet- ing Zhuang. 2023. Data-copilot: Bridging billions of data and humans with autonomous workflow. CoRR, abs/2306.07209.
Acc
CJO22
00
85.10
88.16
90.28
91.87
90.00
76. 14
Wengi Zhang, Kai Zhao, Peng Li, Xiao Zhu, Yongliang Shen, Yanna Ma. Yingfeng Chen, and Weiming Lu.
80.00
87.51
89,22
90.28
84.39
81.39
70.00
74.91
61,25
2022. A closed-loop perception, decision decision-making and reasoning mechanism for human-like naviga- tion. In Proceedings of the Thirty-Five First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4717-4724. International Joint Conferences on Artificial Intelligence Organization. Main Track.
71.97
60.00
44.88
50.00
40.00
27.38
30.00
20.00
10.00
Law Article
Churge - Prison Tern
0.00
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao
5
Top-k
Figure 6: The top-k accuracy of CNN on CJO22 dataset.

---

Acc
CAIL2018
100.00
80
95.12
96.32
97.36
97.76
90.00
95.20%..00
93.04
96.64
70.00
80.00
82.7
80.32
89.28
94.96
65.20
40.00
50,00
40.00
30.00
20.00
10.00
Law Article
Charge
4-Prison Term
2
3
Top-k
Figure
The top-k accuracy BERT on CAIL dataset.
CJO22
100.00
89.58
92.87
94.17
95.23
82.62
90.00
94.58
90.16
93. 17.17
95.
70.00
80.50
59.42
85.45
92.40
73.79
60.00
36.80
50.00
40,00
30.00
20.00
10.00
Law Article
Charge
Prison Term
0.00
2
3
4
5
Top-k
Figure 8: dataset.
The top-k accuracy of BERT CJO22
A.2
More Show Cases

---

Given Case
about 19:00 September 12, 2021, when defendant was driving neol matorsye along National Highway 106 from south north the read fiont of Hotel C, by hit nskearim B who was coasing fhe rool from east to was in frant, causing seurr houl intuy, and later B. disl aller insiiaciio Ieatosut, The D County Traffi Police Brigado determined that was primarily respons lole for the necident, After the case, defendant A immodiately reported the case and waited at the scene for the traffic police to come and deal with it. Later, defendant A reached competsation agr agreement with the victim's relativ es, compensating the victm's relitiess for oconcenic losses of $260,000 in lump and obtained the undkrstanding the victin's relatives.
Candidate labels from damain models: Causing Traffic Casualties, Dangerous Driving, Involuntary Vanslanghter
Precedents
Procedent for Causing Traffic Casullies
Precedent for Dangerous Driving
Precedent for Involuntary Manslaughter Sub: Defendant drove twn-wpheseedd meterevele
Sub: Defendint A drove a decked hookd
Sub: Defendant was driving nray Cheurakt minthus
motorgycle without
drinking
influence of alcohol collided.
along National Highway from north to south and pedestrim B.
aloohol, and failed drive sufely in accordence
Obj: Defendant A's blood alcohol content
operating specifications, resalting in the
4mg//10mm, and he was seized by the police on the spot.
Obj: Defendure knocked to the ground, then A sent
deatth of pedestrian.
the hospital, B died after ineffective rescue, and after foremsic physical csaminition, B died of severe head injury.
Obj: Defendant A druve a wheeell motorcyele under the influence of alcoltol, drove east to west along the X route C City to the Town, C City, and intuned padestrian B. walking in the same dicection and dusl afler indlisdiev poseas.
Ex: After the case, defendant A and reached
medistion agrecment on civil compensation.
Ex: Defendant A voluntarily surrendered to the Branch of the D City Public Security Bureau, and his punishment can be mitigated acconding to law.
section of Z Village
Rx: None.
Predicted Judgment
R-Former Dangerous Driving
DERT
PLIP(DERT)
Dingerous Driving
Causing Traffic Caualiie
Figure 9: More case 1.
Given Case
Around Jamary 2021, wothout obsaining. a forest baccesting pounts dedsudont & hiced petsoonel to barvest dos ttsss losatadlsssadd.. in his own rocuntaio far. oeer to. thr "Gresi Waterfill" in B. Village. Town._ County. and used the felled trees to build houses and sell them, Among them, the total profit from the sale of felled trees was about $2,000. & tocil of 1d, 149, cubic. meters of trems were identified as heing harvested, On November 9, 2021. defendant A voluntarily survendered the E Police Station of County after being notified by the police handling the case.
Candidate labels from
modek::llegal Denudation, Melaal Lumbering, Illegal Occupation of Agrricuttall Land
Precedents
Precedent for Illegal Denudation
Procedent for Illeaal Lumbering
Precedent for Illeaal Occupation o''Agriulurrll
Suh: Detendann cut down trees prodected by the
Sub: Defendant intartionally comminted the
Land
Bals Biis...wiboes the spnroval of the forestry
crime for the purpose of illegal possession.
Sub: Defendant A illegally reclaimed forest land githhout obtiing koaal formalitis
authoritias
Obj: Defendint A Falsely elaimed that 328 poplar
Obj: Defendant A. did not apply for forest hurvesting permit, and cut down troes the 'Boofu Mountain" of in B Village, C Town, with a toml area acres trees down and sotal of shmling
trees located in the northeast of X Village, Y Town. County, owned by himsef, sold the above: mentioned noplars to F. and then felled them at price $6,6000 a stooding log, accumulation. of 15.7937 yubic metar and lotal value
Obj: Defendant A illegally reclaimed 10.05 acres of forest land in the Willow River Forest Farm Application Area of the B Forestry Burean.
confession and defense evidence such the unssite investigaionn records of the Branch of the C Public Security Baresu confirmed that punistment be mitigated according to law.
Defendant
20,8702 subic
accmmolatoon
$12,068.
Ex: Defendant A volunturily sumrenders after commitiing crime and truthfully confissees his crime, which is voluntary surrender, and the punishment can be milignied according to law.
Ex: Nune.
recommended that the
Predicted Judgment
R-Formar
RERT
PLJ
Melal Denudarion
Illegal Lambering X
Illegal Denudation
Figure 10: More case 2.

---

Gives Case
on December 1, 2021. defendant doose anall ondinary bus from west east slong Wenhun Read Oueeston Disiil Baokon City 1o poni 150 meters was Intessettion Werhon Road Yinhia Rood Lhe ride of Rest Cingshan Horsh poliamaa coattig colldde the vistan
was diing stete bicslle of B on suac Aer auskent deferdam ditaa 120 emergencc number and 122, ahim nember call the poli and
lsmiss fhe nua foom nurch a2 soull noullnng as fhr woited for the arrival of the traffic pol at the scene Accuniing to the funnncic mappey ecaminutio and Aimiiiicttonn of Bacton Public Security Traffic Judicial Appnical Canter, B died of severe crun encerebcal injuyy coused by a trafte accident According the detemmationn of the rod trffic sccidens by the Qingshan Brigad ade of the Bootou Traffic Managenient also found that
eeckeent accadest libblityy issunace motor Mongolia Brunch a certain property lisurance company, and the acadent occured withis issuracce period.. $40,000 for the fimaral experees of the family member of the viatin, was further found that defandamt A volumarily propeid compersstion of $10,000 to this court during the sisl period.
Detachmnnt, the defoudant bears responsbilitty accadent purchased compuliocy motor vehicle
respusibilit the
that branch aiso found out that the asc aceaden, defendant A pail
adinary passenger
party libbility insurance Bootou Cat Center
Predicted Judgmment and Explanation
Law Artide
Change
Prison Tome
Law orticle: tnrt- 131
Cherge tnith:
Crusing tnifiic coupliie
Prison konn inth munthe
Law article prediction: Article 133, Uecause of the facts of cose enclosal by duferdant drove small 150
Charge prodiction: Causing raffis accident Since the theee of casss show that lix defenden did no dise the school asconiance
Prison tenn prediction: One to two years in peson. Since dkiur Bis del nut Iollw ba trlliic niks and lust his costect indumen, the xicim died the spot and the crine isien. impoasd so he chose fisalttm inpe inmment of one to two yorrs Precelums is also supperi the suleption of ficea-- tenm imprisommntt of one to two years this case. lhe creeeditt of of acttercel to cmppurilinc shorke see and
Road west oethnary along intersctios of Wentus Road and meters west
taifis saley regulatuns ths sutting a maini aotidon and clisal contin South
louger sontinn should
Yaighin Road and met the vietim B who was skling at deanic bicycle wrees the road from north to south Iisu poopls colliced and viclated mattls marneement comiition cbo caucing ininy dean sieam.
th: vidlim on the spot. fir stine of cusiing ltala ausien can an selastal in Uiis cae which in line with
Amtlle 133 of the "Chmital Law of the Peop le's Republic Chan"" Artilee vehicle carclessly the cirammttanss. suimord fixed-rurm imprisnmmmt three yeurs or crmmaa desention and stall alo be fined."
stipulates: "Whoever drives
all
than
concllitess crime ensing ralls. scsdlent scorlinn io Jou uu comenuuollie laodl iriels in the
more from
suntrnas whot nores f2t a lanser senrnce sbouk be adocted n. thes sace 32 acheve the diccplinary thece
sase should be Amai 133
Figure 11: Case with explanations 1.
Given
In early mocning Mf 11, 2022, defendart drove car carring B, odiars county seat, thas Rodundaut left B. Cuaud deviin smoke the doue kemmone dougkktmmencccc
st. badi star At about 10:00 on March 18, 2022, defendant A was arrested at mikk too shop in Rucheng County, and later trutiiully confessed the above facs.
Predicted Judgment and Explanation
Law Article
Charge
Prison Tonu
Law aniele truth: 354
Charge turl: Pruciling vamoos for drug usurss
Prison
monbh
Law article production: Articee 354. Comparel with ursssdarss the subiscusse sontie of the cast i do dils ofhers in take diups and the olisatiic e behan for likes duz kanminee The behivions insiuud ie billwrin others nise druse", so inoticable Issal musicuuus Eur Antick 355, the masun. S. fiat kox muslsss the zenakass for daus coteation The difference frm this case that peocodents involved crunk driving (Aride 133) muh fighing (Ariide 192), and the ciccaetonncss
Charge prediction: Providing venues: for drug users. Comparin the faree aspects of subjective motiy dllon dbjective behev and incillstal metinnn can that AsGaddant A's miisecave mole praponal,
Prison lznn preduction: Six niee months prison. The objective behas ior of fik case is clear. The vahidl drivs by defenllae A carial poisonms substines. in aiiiioo. of strinal insetion
the subs succ more line munishrent foon resoalmmis After bong arosted and brought to Jusiice, defendant A tubfully confessed his crime. Theeffore, fhe sentence is this csse was chosen as ficaliamm imprisomenn of rine monds.
oboective behavior was lo deugs, The cirumatanss onsde the incilant are diferest, In this case, defandant can be given lightr punistmest according to law after being brought to justice. From this, can he judged that the defindni this ease durged allwwing others drugs.
esaelly same, applicable lows and regulztions were differom coe
Figure 12: Case with explanations 2.

---

Given Case
In March 2019, when defendant A and victan B dated in the name of boythiend and gifffiind, fiom March to Soptemther 2019. & demendad dal money from B on the aronnds da of holaing E trnndii wurk handline mallers such ousttonn sicuus Balaliuenal tor af 228 to frouch WiChat Alpay, traufii and smunds heny hosnisbrre and decch sime the
usad April 2019, defencbent A hel vietim C 5o.0 MisChal shith use by for menural consummian Whan atked for mope A forped the moseoro maso man. in Ned ler and blacklisted Mer has On March 2020 A returnal 59,000 to the vietim C. It was alo found fuat becaae defendare A suffir al from verimee Thoss the Reschang Dier in Perple's Court of living City sentenoca deffeneent An als moothh" impe sonatent which has not yet been exeutedd It was further found Baat on Mudi 15, 2022, deferdant A poid victim B $1200000 in cash and lscued bill signed in acknow moleemmmnn of dlitt which recorlled "A voluntarily poii vietin B $129,000, and puil SI 0000000 cas fins, and owel 59,000 paid off in four mundee, $2,000 was VVcctm
and masted the
nerconal upen denne Whe sdked for
chriked
cook hele deal with the hishand Mounoo's dramn dovine and domanded monry from
name of drotern
relationsinn
the fins agree lighter puiismmer.. Defindant by Ly pleaded guitt accepted punishment, prssecution agesey. The defender his ao objection to the alloped ciimiaa fects and charges, but believes that defendant A confissed the fatts of the case before taking compulaccy museuns, which consiittice sumender and confiesiim: Al fandlleni behavior octamee the prosose of chaing, and the sujjesive viciouinces is releively small and the harn is not great: The amount suspected frand not clear. According fil pracciple of lenient critie the statatey pendlt "hage amoutt" abould not be app Rod. resommendod that the Sedendaan escrapted from punishment the
before July
indentondingg expresing face, charges and sentencing recommoonaaions
respect
Forgive public
no objection
of suncedering hansalf, activlyy compeneating tie victin obtining Borgivesess.
Predkcted Judgment and Explanation
Taw Artide
Charge
Prison Term
Law seride truth: 266
Chonge inith: Fraud
36 monbs. Prison temm prediction: Two to three years prison. From itr abuse frrse catepunees of can bt. scen fott the subissibve molue of his defendent as well is the stistiee behavvor in busically the some this
Prisum
Law prodictin:: Anticle 205, because defondant in this csee look mons fraulslently using snother parseet's aecount nunbar, fastiins fitss and concealing the tuut to obtan other people's property in lage provisions sottest of possalatio
Change prediction: Hecause prosolenis were establiched by defendant with purpose of unloo ful possession, the act taking possossion money ficstius fact and concealing the tush are fxndalent. This
and
and the actendatt roogiieed abe mum. mumionn Damt The
said urrile apely.
prosalans aunsunlk..oommm moans. suc. as voiue pands akea propets. both consinuts fund.
like
lirst cost sotimahont
aned moins ercoooeit uss fistioou
theat money.
the ansunt fraud, turifally confessed to the crme, cooperated with the invastigation and contissed the face. which cen he uool is milipeing ficken in the sentenoo, so the sunience the defimdintt this the
tie sccord caso the delendaui med the funduienn use of othher piskl's aooounts cnals canl in demand monse in thini coese the en usrd peopless fondukkn three neccoduuss all boobre dhering his
chusm be congoories of semumorr However. aqjusmentss should be made according to the actual eicunsstaress.
aeppary and the aots in the suree casce are munbs fruud. so fe prwvitton of arlele 265 us also app aplicctli fiis
Figure 13: Case with explanations 3.


Reviews:
[{"review_id": "Naalbb99yx", "reviewer": "Reviewer_9C2g", "summary": "", "questions": " This paper can be improved according to the following aspects.\n1. The authors should compare with the relevant work in 2022 to prove the effectiveness of the proposed method, such as TOPJUDGE (Feng et al., 2022, Legal Judgment Prediction via Event Extraction with Constraints) and CTM (Liu et al., 2022, Augmenting Legal Judgment Prediction with Contrastive Case Relations).\n2. Following LADAN (Xu et al., 2020) and R-Former (Dong and Niu, 2021), the authors should also verify the model using the same datasets as the previous method, such as CAIL-small and CAIL-big.\n3. The proposed method is a pipeline model, and the final prediction results of the LLMs are heavily dependent on the candidate results of the first-stage prediction model. Could the authors further analyze the reliability of the candidate results?\n4. In the ablation experiment, when the LLMs is forced to generate the explanation of the prediction, the performance of the model is greatly affected, with both improvement and decrease. Could the authors conduct further detailed analysis?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time to reviewing our paper and offering us valuable insightful feedback. We are grateful for your recognition of the novelty and potential of our idea. With regard to the concerns you raised, we have carefully considered them and would like to address them in the following response.\n\nQ1:\n\n    For the concern about \"update baselines\":\n\n- We have implemented a comparison of these two baseline models EPM[1] and CTM[2], and you can find the results displayed in the following tables.\n\n**Task: Law article prediction**\n\n#### Dataset: CJO22\n\n| Method     | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| ---------- | --------- | --------- | --------- | --------- |\n| EPM        | 84.19     | 47.21     | 43.79     | 44.39     |\n| CTM        | 79.44     | 47.83     | 42.25     | 43.43     |\n| PLJP(CNN)  | 87.67     | 55.21     | 55.59     | 54.37     |\n| PLJP(BERT) | **94.18** | **74.65** | **76.23** | **78.84** |\n\n#### Dataset: CAIL-2018\n\n| Method     | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| ---------- | --------- | --------- | --------- | --------- |\n| EPM        | 85.80     | 49.08     | 45.76     | 47.32     |\n| CTM        | 84.72     | 46.46     | 44.83     | 45.10     |\n| PLJP(CNN)  | 86.05     | 58.08     | 56.46     | 54.92     |\n| PLJP(BERT) | **87.07** | **58.81** | **57.29** | **56.63** |\n\n**Task: Charge prediction**\n\n#### Dataset: CJO22\n\n| Method     | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| ---------- | --------- | --------- | --------- | --------- |\n| EPM        | 83.49     | 80.36     | 83.29     | 81.87     |\n| CTM        | 79.33     | 82.39     | 83.12     | 82.81     |\n| PLJP(CNN)  | 91.62     | 83.43     | 84.88     | 83.40     |\n| PLJP(BERT) | **94.18** | **90.25** | **88.67** | **89.05** |\n\n#### Dataset: CAIL-2018\n\n| Method     | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| ---------- | --------- | --------- | --------- | --------- |\n| EPM        | 91.20     | 90.81     | 89.99     | 90.46     |\n| CTM        | 90.28     | 90.34     | 88.08     | 86.30     |\n| PLJP(CNN)  | 91.49     | 81.80     | 83.95     | 80.06     |\n| PLJP(BERT) | **94.99** | **92.12** | **91.10** | **91.33** |\n\n**Task: Prison term prediction**\n\n#### Dataset: CJO22\n\n| Method     | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| ---------- | --------- | --------- | --------- | --------- |\n| EPM        | 36.91     | 30.65     | 31.61     | 30.20     |\n| CTM        | 36.81     | 27.10     | 25.96     | 26.46     |\n| PLJP(CNN)  | 36.51     | 20.21     | 21.44     | 20.07     |\n| PLJP(BERT) | **43.52** | **33.37** | **35.67** | **31.98** |\n\n#### Dataset: CAIL-2018\n\n| Method     | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| ---------- | --------- | --------- | --------- | --------- |\n| EPM        | 40.25     | 37.96     | 37.00     | 37.34     |\n| CTM        | 39.56     | 38.66     | **38.02** | **37.84** |\n| PLJP(CNN)  | 40.81     | 32.77     | 35.59     | 25.71     |\n| PLJP(BERT) | **48.72** | **42.64** | 36.80     | 35.43     |\n\n- The results prove the effectiveness of our PLJP method, especially in the CJO22 dataset. We will add this result and host additional analysis in the revision to help readers better consume the experiment outcomes.\n\nQ2:\n\n    For the concern about \"datasets\":\n\n- Thanks for the suggestion. In our experiments, we utilized two datasets: CAIL-2018 and CJO22. The CAIL-2018 used in our paper is the same one referenced as CAIL-small. Additionally, the CJO22 dataset was constructed to specifically contain cases that took place post 2022. This was done to prevent data leakage from occurring during training since all LLMs were trained using a corpus collected prior to September 2021.\n\n- In addition, we are conducting experiments on the CAIL-big dataset, which is derived from the same resource as the CAIL-small. Unfortunately, due to the substantial size of the CAIL-big dataset and the limited inference speed of ChatGPT, obtaining immediate experimental results during the rebuttal period is challenging. We intend to include the results for the CAIL-big dataset in the revised version of our work.\n\n- Importantly, we will release all the data and code to ensure the reproducibility of our work.\n\nQ3:\n\n    For the concern about \"reliability of the candidate results\":\n\n- We report the top-5 accuracy of the two domain models (e.g., CNN and BERT) here:\n\n**Dataset: CAIL-2018**\n\n#### Method: CNN \n\n#### Task: Law article prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 80.50 | 40.10 | 38.33 | 38.49 |\n| 2    | 91.20  | 75.67 | 76.14 | 75.34 |\n| 3    | 93.52 | 78.08 | 79.04 | 78.30  |\n| 4    | 94.88 | 81.14 | 81.97 | 81.31 |\n| 5    | 95.28 | 81.99 | 82.25 | 81.92 |\n\n#### Task: Charge prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 87.52 | 88.23 | 88.31 | 88.17 |\n| 2    | 94.48 | 94.62 | 94.44 | 94.46 |\n| 3    | 95.84 | 95.97 | 95.79 | 95.82 |\n| 4    | 97.12 | 97.22 | 97.10  | 97.11 |\n| 5    | 97.92 | 97.97 | 97.94 | 97.93 |\n\n#### Task: Prison term prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 34.42 | 32.22 | 30.53 | 31.05 |\n| 2    | 53.04 | 51.22 | 49.55 | 49.73 |\n| 3    | 66.56 | 64.91 | 63.59 | 63.53 |\n| 4    | 78.56 | 76.94 | 75.94 | 75.86 |\n| 5    | 88.16 | 87.39 | 86.64 | 86.71 |\n\n#### Method: BERT\n\n#### Task: Law article prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 82.77 | 36.82 | 35.94 | 35.82 |\n| 2    | 93.04 | 77.21 | 76.75 | 76.55 |\n| 3    | 95.20  | 83.27 | 82.8  | 82.67 |\n| 4    | 96.00  | 85.95 | 84.71 | 85.06 |\n| 5    | 96.64 | 89.07 | 87.8  | 88.16 |\n\n#### Task: Charge prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 89.10 | 90.10 | 89.48 | 89.63 |\n| 2    | 95.12 | 95.39 | 95.00  | 95.09 |\n| 3    | 96.32 | 96.49 | 96.23 | 96.30  |\n| 4    | 97.36 | 97.51 | 97.26 | 97.33 |\n| 5    | 97.76 | 97.87 | 97.7  | 97.75 |\n\n#### Task: Prison term prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 40.00 | 37.53 | 33.66 | 33.58 |\n| 2    | 65.20  | 66.03 | 57.89 | 59.65 |\n| 3    | 80.32 | 81.37 | 73.90  | 76.60  |\n| 4    | 89.28 | 90.26 | 84.20  | 86.58 |\n| 5    | 94.96 | 94.92 | 90.67 | 92.48 |\n\n**Dataset: CJO22**\n\n#### Method: CNN \n\n#### Task: Law article prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 76.14 | 35.48 | 38.55 | 35.39 |\n| 2    | 84.39 | 42.99 | 47.24 | 43.18 |\n| 3    | 87.51 | 46.68 | 50.92 | 47.40  |\n| 4    | 89.22 | 50.38 | 55.32 | 51.50  |\n| 5    | 90.28 | 52.25 | 57.42 | 53.54 |\n\n#### Task: Charge prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 74.91 | 74.00 | 78.12 | 73.97 |\n| 2    | 85.10  | 84.16 | 87.29 | 84.20  |\n| 3    | 88.16 | 87.73 | 90.5  | 87.91 |\n| 4    | 90.28 | 90.08 | 92.49 | 90.24 |\n| 5    | 91.87 | 91.95 | 94.26 | 92.31 |\n\n#### Task: Prison term prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 27.38 | 18.48 | 17.51 | 17.44 |\n| 2    | 44.88 | 35.42 | 32.78 | 33.19 |\n| 3    | 61.25 | 51.03 | 48.92 | 49.08 |\n| 4    | 71.97 | 62.22 | 60.24 | 59.72 |\n| 5    | 81.39 | 70.73 | 68.63 | 68.54 |\n\n#### Method: BERT\n\n#### Task: Law article prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 82.62 | 45.89 | 47.91 | 45.83 |\n| 2    | 90.16 | 53.31 | 55.48 | 53.42 |\n| 3    | 93.17 | 61.03 | 63.42 | 61.65 |\n| 4    | 94.58 | 65.16 | 66.50  | 65.27 |\n| 5    | 95.23 | 66.98 | 68.59 | 67.26 |\n\n#### Task: Charge prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 80.50 | 80.34 | 81.09 | 78.36 |\n| 2    | 89.58 | 89.84 | 90.36 | 88.52 |\n| 3    | 92.87 | 93.45 | 94.06 | 92.97 |\n| 4    | 94.17 | 94.72 | 95.50  | 94.61 |\n| 5    | 95.11 | 95.7  | 96.21 | 95.57 |\n\n#### Task: Prison term prediction\n\n| topk | Acc   | Ma-P  | Ma-R  | Ma-F  |\n| ---- | ----- | ----- | ----- | ----- |\n| 1    | 36.80 | 29.83 | 27.50 | 27.03 |\n| 2    | 59.42 | 53.47 | 47.95 | 48.53 |\n| 3    | 73.79 | 65.66 | 59.86 | 61.30 |\n| 4    | 85.45 | 76.11 | 70.79 | 72.57 |\n| 5    | 92.40 | 81.95 | 78.72 | 79.94 |\n\n- Examining the results provided, it becomes evident that the improvement in accuracy from the top1 to top3 predictions is more substantial compared to the improvement from top3 to top5 predictions. Take the law article prediction of BERT in CJO22 as an example, the top1 accuracy stands at 82.62%, the accuracy for the top3 predictions increases to 93.17%, while the top5 accuracy is 95.23%.\n\n- In our PLJP framework, every candidate label necessitates a corresponding precedent. Due to the constraint of prompt length, we made the decision to select the top3 labels as the candidate labels for our experiments. We will add a detailed analysis of the above results in the revision.\n\nQ4:\n\n    For the concern about \"model performance with explanation\":\n\n- Through our ablation experiments, we find that the prediction accuracy of PLJP is affected when we force the LLM to generate the explanation of the prediction simultaneously. This may be due to a \"trade-off\" between accuracy and interpretability. \n- A straightforward approach to maintain prediction accuracy is to have the LLM generate the explanation in a separate dialogue round following the prediction, rather than do the prediction and the explanation simultaneously.\n- In the future, we will explore this interesting problem and try to enhance the predictions with explanation.\n\n\n\nMissing Reference:\n\nThanks for the reminder, we will add the missing references [1-2].\n\n[1]Dugang Liu, Weihao Du, Lei Li, Weike Pan, and Zhong Ming. 2022. Augmenting legal judgment prediction with contrastive case relations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2658–2667, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. \n\n[2]Yi Feng, Chuanyi Li, and Vincent Ng. 2022. Legal judgment prediction via event extraction with constraints. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 648–664, Dublin, Ireland. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors effectively utilize the precedents information to enhance the performance of the model.\n2. The proposed method combines the strength of both LLMs and domain models, and provides a promising direction for the collaboration of LLMs and domain-model.\n\n", "reasons_to_reject": "1. The authors should compare with the relevant work in 2022 to prove the effectiveness of the proposed method, such as TOPJUDGE (Feng et al., 2022, Legal Judgment Prediction via Event Extraction with Constraints) and CTM (Liu et al., 2022, Augmenting Legal Judgment Prediction with Contrastive Case Relations).\n2. Following LADAN (Xu et al., 2020) and R-Former (Dong and Niu, 2021), the authors should also verify the model using the same datasets as the previous method, such as CAIL-small and CAIL-big.\n3. The proposed method is a pipeline model, and the final prediction results of the LLMs are heavily dependent on the candidate results of the first-stage prediction model. Could the authors further analyze the reliability of the candidate results?\n4. In the ablation experiment, when the LLMs is forced to generate the explanation of the prediction, the performance of the model is greatly affected, with both improvement and decrease. Could the authors conduct further detailed analysis?\n", "questions_to_authors": " This paper can be improved according to the following aspects.\n1. The authors should compare with the relevant work in 2022 to prove the effectiveness of the proposed method, such as TOPJUDGE (Feng et al., 2022, Legal Judgment Prediction via Event Extraction with Constraints) and CTM (Liu et al., 2022, Augmenting Legal Judgment Prediction with Contrastive Case Relations).\n2. Following LADAN (Xu et al., 2020) and R-Former (Dong and Niu, 2021), the authors should also verify the model using the same datasets as the previous method, such as CAIL-small and CAIL-big.\n3. The proposed method is a pipeline model, and the final prediction results of the LLMs are heavily dependent on the candidate results of the first-stage prediction model. Could the authors further analyze the reliability of the candidate results?\n4. In the ablation experiment, when the LLMs is forced to generate the explanation of the prediction, the performance of the model is greatly affected, with both improvement and decrease. Could the authors conduct further detailed analysis?\n", "paper_topic_and_main_contributions": "The topic of this paper is about approaches for data- and compute efficiency.\nThe main contributions of this paper are as follows: \nIn this paper, the authors propose a novel framework to combine the strength of both LLMs and domain models to better utilize the precedents for the legal judgment prediction (LJP) task. Specifically, domain models contribute by providing candidate labels and finding the proper precedents from the case database effectively; the LLMs will decide the final prediction through an in-context precedent comprehension. Experiments on the real-world dataset demonstrate the effectiveness of the proposed model.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Dugang Liu, Weihao Du, Lei Li, Weike Pan, and Zhong Ming. 2022. Augmenting legal judgment prediction with contrastive case relations. \n    In Proceedings of the 29th International Conference on Computational Linguistics, pages 2658–2667, Gyeongju, Republic of Korea. \n    International Committee on Computational Linguistics.\n[2] Yi Feng, Chuanyi Li, and Vincent Ng. 2022. Legal judgment prediction via event extraction with constraints. In Proceedings of the 60th \n    Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 648–664, Dublin, Ireland. Association for \n    Computational Linguistics.\n"}}, {"review_id": "WH03trI1ei", "reviewer": "Reviewer_GcEk", "summary": "", "questions": "I would be interested to hear the authors' thoughts on my \"negative\" comments (reasons to reject).", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We are committed to improving our research study, so we eagerly look forward to incorporating your suggestions for further refinement. Please find our responses in the following content. \n\nQ1:\n\n    For the concern about \"retrieval-augmented BERT-based baseline\":\n\n- Thanks for the suggestion. We added a retrieval-augmented BERT-based baseline, named Retrieval BERT. We implement the Retrieval BERT referring to [1], and we set the number of retrieved documents (precedents) to 3, the results are shown below:\n\n#### Task: Law article prediction\n\n#### Dataset: CJO22\n\n| Method         | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| -------------- | --------- | --------- | --------- | --------- |\n| Retrieval BERT | 82.29     | 45.64     | 44.79     | 44.01     |\n| PLJP(CNN)      | 87.67     | 55.21     | 55.59     | 54.37     |\n| PLJP(BERT)     | **94.18** | **74.65** | **76.23** | **78.84** |\n\n#### Dataset: CAIL2018\n\n| Method         | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| -------------- | --------- | --------- | --------- | --------- |\n| Retrieval BERT | 79.00     | 48.38     | 48.28     | 47.60     |\n| PLJP(CNN)      | 86.05     | 58.08     | 56.46     | 54.92     |\n| PLJP(BERT)     | **87.07** | **58.81** | **57.29** | **56.63** |\n\n#### Task: Charge prediction\n\n#### Dataset: CJO22\n\n| Method         | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| -------------- | --------- | --------- | --------- | --------- |\n| Retrieval BERT | 82.11     | 82.89     | 80.45     | 78.00     |\n| PLJP(CNN)      | 91.62     | 83.43     | 84.88     | 83.40     |\n| PLJP(BERT)     | **94.18** | **90.25** | **88.67** | **89.05** |\n\n#### Dataset: CAIL2018\n\n| Method         | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| -------------- | --------- | --------- | --------- | --------- |\n| Retrieval BERT | 92.37     | 91.51     | 90.58     | 90.18     |\n| PLJP(CNN)      | 91.49     | 81.80     | 83.95     | 80.06     |\n| PLJP(BERT)     | **94.99** | **92.12** | **91.10** | **91.33** |\n\n#### Task: Prison term prediction\n\n#### Dataset: CJO22\n\n| Method         | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| -------------- | --------- | --------- | --------- | --------- |\n| Retrieval BERT | 34.93     | 27.36     | 25.10     | 25.89     |\n| PLJP(CNN)      | 36.51     | 20.21     | 21.44     | 20.07     |\n| PLJP(BERT)     | **43.52** | **33.37** | **35.67** | **31.98** |\n\n#### Dataset: CAIL2018\n\n| Method         | Acc       | Ma-P      | Ma-R      | Ma-F      |\n| -------------- | --------- | --------- | --------- | --------- |\n| Retrieval BERT | 39.64     | 35.74     | 34.38     | 33.66     |\n| PLJP(CNN)      | 40.81     | 32.77     | 35.59     | 25.71     |\n| PLJP(BERT)     | **48.72** | **42.64** | **36.80** | **35.43** |\n\n- From the result, we can find that PLJP outperforms the Retrieval BERT consistently in both datasets, this may be because the LLMs have a better ability to comprehend the complex precedents' facts than the BERT model.\n- We will discuss this issue in detail in the revision.\n\nQ2:\n\n    For the concern about \"BERT-based classifier is a strong baseline\":\n\n- Our method is a collaboration of LLM and domain models such as a fine-tuned BERT classifier. Each part of the collaboration holds significance. \n- By examining the results presented in Tables 2, 3, and 4, we can conclude that:\n  - The efficacy of domain models directly influences the overall performance. For instance, PLJP(BERT) outperforms PLJP(CNN), highlighting that superior domain models lead to improved outcomes.\n  - The collaboration of LLMs and domain-specific models in the PLJP yields performance enhancements. A notable example is the charge prediction accuracy in CJO22, which sees an increase from 80.5% to 94.18% with PLJP(BERT).\n\n- Therefore, the PLJP benefits from both domain models and LLMs. The collaboration between the domain models and LLMs contributes to the overall effectiveness of the approach.\n\n\n\nTypos and missing references:\n\n- Thanks for the reminder, we will make clear the language we are working on in the revision. Also, we will ensure all the missing references are included in the revision [1-5].\n\n  \n\nReference:\n\n[1]Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural Legal Judgment Prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323, Florence, Italy. Association for Computational Linguistics.\n\n[2]Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. 2021. Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 19–35, Punta Cana, Dominican Republic. Association for Computational Linguistics.\n\n[3]Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics.\n\n[4]Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripabandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. 2021. ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4046–4062, Online. Association for Computational Linguistics.\n\n[5]Ilias Chalkidis* and Yova Kementchedjhieva*. Retrieval-augmented Multi-label Text Classification. arXiv. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes an interesting retrieval-augmented approach utilizing LLMs for the task of LJP.", "reasons_to_reject": "- There are no other retrieval-augmented BERT-based (prior-art) baselines utilizing the same or similar steps with the proposed framework. Prior-art models enhanced with retrieval-augmentation to condition on precedent (summarized) cases, may perform equally well to LLMs. \n- The proposed method is heavily dependent on a fine-tuned BERT-based classifier. In other words, it can be understood as a few-shot (precedent cases) system that also considers predictions for the new input from a strong baseline.", "questions_to_authors": "I would be interested to hear the authors' thoughts on my \"negative\" comments (reasons to reject).", "paper_topic_and_main_contributions": "The authors propose a new pipelined framework for Legal Judgment Prediction (LJP) for the Chinese CAIL multi-task datasets, where a model has to predict the relevant criminal charge, law articles, and penalty term based on criminal case facts. The new framework has the following steps (modules): (a) Fact summarization using an LLM, e.g., ChatGPT, (b) Judgment prediction/classification with a domain/task-specific classifier, (c) Relevant case law (precedent) retrieval with a domain-specific neural embedder, and (d) in-context LJP by an LLM using as inputs the summarized facts of the given case, and k retrieved relevant precedent cases, and the prior prediction from step b. The authors compare the new framework with several prior-art (baselines) in the CAIL2018 dataset and a collection of new cases from 2022, dubbed CJO22. \n\nThe new framework has improved results in penalty term, and law article prediction, and comparable performance in charge prediction compared to the best prior-art models (Ar-Former, NeurJudge). The authors perform ablations un-plugging different steps (modules) from the full pipeline, e.g., w/o summarization, or w/o retrieval-augmentation, which show the importance of the selected steps (modules).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural Legal Judgment Prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323, Florence, Italy. Association for Computational Linguistics.\n- Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. 2021. Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 19–35, Punta Cana, Dominican Republic. Association for Computational Linguistics.\n- Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics.\n- Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripabandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. 2021. ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4046–4062, Online. Association for Computational Linguistics.\n- Ilias Chalkidis* and Yova Kementchedjhieva*. Retrieval-augmented Multi-label Text Classification. arXiv. 2023.", "Typos_Grammar_Style_and_Presentation_Improvements": "- line 54: \"courses\" --> sources\n- The authors should make clear and mention that they use datasets for LJP in the Chinese language in the context of People's Republic of China (PRC) jursdiction (law). There are phrases like \"The legal judgment typically includes the law article, charge and prison term.\", which sound like the LJP is a universal task across languages and countries. The Bender Rule: \"Always name the language you're working on\" is not only applicable to English."}}, {"review_id": "27NxXEPJui", "reviewer": "Reviewer_ZR8e", "summary": "", "questions": "Please see the Reasons To Reject.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your thoughtful input and dedication to this paper. Our replies can be found below.\n\nQ1-1:\n\n    For the concern about \"label in the precedent selection process\":\n\n\n- We would like to highlight that the label we’re referring to at this stage is the \"candidate label\", not the \"final label\". \n- As described in section 4.2.1, for each candidate label generated by the predictive model (Line 216), we retrieve one case that has the same label from the case database as the precedent, according to the similarity scores calculated by the retrieval model (Line 317). \n- If there are multiple candidate labels, we retrieve corresponding precedents for each of them. It is essential to note that a single case might be accompanied by a variety of actualized precedents, depending on the underlying task; for instance, in the context of law article prediction, charge prediction and prison term prediction.\n\nQ1-2:\n\n    For the concern about \"utilization of the precedent serves as a supplemental explanation of the label\":\n\n- Since the precedents are selected by the candidate labels, it is possible to view each precedent as an example of the corresponding candidate label, which contributes to the in-context learning of the LLM in the final prediction.\n\nWe will revise the writing and ensure the clarity of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper seems the first to utilize the precedents to enhance the performance of LJP tasks. \n2. This paper uses LLM reasonably.\n3. Extensive experiments demonstrate the effectiveness of the proposed method. ", "reasons_to_reject": "The writing of this paper is not clear. In section 4.2.1, the authors said that they chose the case that has the highest similarity score and has the same label as the precedents. However, since the final task is to predict the label of charge, law articles, etc., how to get the label before that? Moreover, it is not clear how they utilize the precedent \"serves as a supplemental explanation of the label\".\n\n", "questions_to_authors": "Please see the Reasons To Reject.", "paper_topic_and_main_contributions": "In this paper, the authors propose a precedent-enhanced LJP framework called PLJP, which leverages the strength of both LLM and domain models in the context of precedents. Experiments on the real-world dataset demonstrate the effectiveness of PLJP.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an interesting retrieval-augmented approach utilizing LLMs for the task of legal judgement prediction.
The proposed method combines the strength of both LLMs and domain models, and provides a promising direction for the collaboration of LLMs and domain-model.

The specific doubts of reviewer ZR8e have been addressed in the author response.
Decision: Accept-Main