Syllogistic Reasoning for Legal Judgment Analysis
Authors: Wentao Deng
Jiahuan Pei
Keyi Kong
Zhe Chen
Furu Wei
Yujun Li
Zhaochun Ren
Zhumin Chen
Pengjie Ren
Venue: EMNLP 2023 Main (2023)
Keywords: Legal Judgment Analysis
Syllogism
Syllogistic Reasoning
TL;DR: This paper proposes the resources of syllogism-based legal judgment analysis, including a dataset and benchmarks.

Abstract: Legal judgment assistants are developing fast due to impressive progress of large language models (LLMs). However, people can hardly trust the results generated by a model without reliable analysis of legal judgement. For legal practitioners, it is common practice to utilize syllogistic reasoning to select and evaluate the arguments of the parties as part of the legal decision-making process. But the development of syllogistic reasoning for legal judgment analysis is hindered by the lack of resources: (1) there is no large-scale syllogistic reasoning dataset for legal judgment analysis, and (2) there is no set of established benchmarks for legal judgment analysis. In this paper, we construct and manually correct a syllogistic reasoning dataset for legal judgment analysis. The dataset contains 11,239 criminal cases which cover 4 criminal elements, 80 charges and 124 articles. We also select a set of large language models as benchmarks, and conduct a in-depth analysis of the capacity of their legal judgment analysis.

Syllogistic Reasoning for Legal Judgment Analysis
Wentao Deng', Jiahuan Pei*, Keyi Kong', Zhe Chen', Furu Wei Yujun Li", Zhaochun Ren*, Zhumin Chen', Pengjie Ren Shandong University, Qingdao, China. "Microsoft Research Asia, Beijing, China.
$Centrum Wiskunde & Informatica, Amsterdam, The Netherlands.
*Leiden University, Leiden, The Netherlands.
(wentao. deng, luxinyayaya, cz2021 J@mall. sdu. edu. cn
[liyujun, chenzhumin, renpengjie]@sdus edu. cn
Jiahuan. Pei@cwi. nl, fuwei@microsoff com, z.ren @ liacs.
Abstract
Input: Fact
Defandan A used enafit card from Bank B May 2008 for muliciins oveddiaft consumption yina Without repoytliet..
Legal judgment assistants are developing fast due to impressive progress of large language models (LLMs). However, people can hardly trust the results generated by model with- out reliable analysis of legal judgement. For legal practitioners, it is common practice to uti- lize syllogistic reasoning to select and evaluate the arguments of the parties part of the le- gal decision- making process. But the develop- ment of syllogistic reasoning for legal judgment analysis is hindered by the lack of resources: (1) there is no large-scale syllogistic reason- ing dataset for legal judgment analysis, and (2) there is no set of established benchmarks for legal judgment analysis. In this paper, we construct and manually correct a syllogistic soning dataset for legal judgment analysis. The dataset contains 11,239 criminal cases which cover 4 criminal elements, 80 charges and 124 articles. We also select a set of large language models as benchmarks. and conduct a in-depth analysis of the capacity of their legal judgment analysis.
Task Article Retrieval
Tank 2: Criminal Flement Generation
Subject: Defendant who fully capablo person.
Geneeated Query:
A324 Credit Card Frand Crime Using Fiction of Identity or Ovendraft of Crodit Cards Fosal.
Subjective Element: Intent, Defendane knows overdratting refusess rapay the
Majer Premius
Misht Pressie
Retrived Decument: Article 196
[Crime of Cradit Card Frui))
Object: Bank B's property security
Whoever
Malicionsly overdrawns, relutively large amouni icsult
Objective Elements:
Time: May 2008.
infeimgement hank' pesperty
2. Location: Bank H.
Overdran consumpttio
more five years or detention.
ztrough credit
4. Results:
The oventaal semoun ressches 18060
Tank 3:
Article Interpnsiation Generation
Article 196: Defendans malicicusly evendrawes credit card with amountt meees the Arricl credit fraud.
Conchsion
Task Legal Aulgment Prodiction
Charges Crime Credit Fraud
Penalty Term: Imprisoment of up yoars
years
An example of criminal case of the credit card fraud. The output of task is the main premise. The output of task 2 is the minor premise. The outputs of task and 4 are the conclusion. The corresponding segments different parts marked in the same color. We mark the corresponding segments in different parts with the same color.
Figure 1:
1 Introduction
Legal judgment assistants are growing rapidly due to large demands from legal practitioners and nor- mal citizens (Cui et al., 2022) and the impressive progress of large language models (Liu et al., 2023; Li et al., 2021; Zhao et al., 2023). The core tech- nique employed in those assistants is legal judg- ment analysis, which aims to analyze the descrip- tion of a fact and produce reasonable results of a legal judgment.
process is typically accomplished through multi- classification (Xu et al., 2020; Yue et al., 202 la; Feng et al., 2022) or generation (Deroy et al., 2023) based on LLMs. However, people cannot fully trust the results predicted by LJP without analytical de- tails of legal judgments (Cui et al., 2022).
Another group of researchers explores court view generation (CVG) (Ye et al., 2018; Yue et al., 2021b; Wu et al., 2020) to enable the interpretabil- ity of the results of legal judgment. The core idea is to generate phrases sentences in the description of fact as rationales for supporting legal judgment results. However, the generated rationales might be specious (Chao et al., 2019) and can not represent the whole analysis process that leads to the final result.
There are two genres of approaches toward le- gal judgment analysis. One group of researchers is actively working on legal judgment prediction (LJP) (Feng et al., 2022). The main goal is to pre- dict the results of legal judgment (c.g., charges, lc- gal articles, and terms of penalty) given the descrip- tion of a fact (Xiao et al., 20. This prediction
*Corresponding author.

---

In practice, syllogistic reasoning routinely em- ployed by judges assess the parties' arguments for legal judgments, thereby maintaining consis- tency and fairness in their decision-man makiing process. (Hart et al., 2012; Constant, 2023). It allows judges to break down legal arguments into logical compo- nents, including major premise, minor premise and conclusion (Gardner, 2007; Constant, 2023).
Our contributions can be summarized follows: Four interrelated tasks proposed for syllogistic le- gal judgment analysis (SLJA). It is motivated by the practical application of syllogistic reasoning in legal judgments.
The first Chinese SIJA dataset, which includes detailed analysis process of syllogism. It facili- tates the frontier researches on consistency and fairness in the legal fecision-making process. Benchmarks based on the state-of-the LLMs, extensive experimental results, and in-depth anal- yses.
In this work, we seek to answer the following research questions:
RQI: How do LLMs perform on the syllogistic reasoning for legal judgment analysis?
RQ2: How does syllogistic reasoning impact the performance of LLMs in legal judgment analy-
2
Related Work
sis?
2.1
Legal Judgment Prediction
We make three main contributions to facilitate research into syllogistic legal judgment analysis (SLJA): formalization, dataset, and benchmarks. Let us expand on each of these.
Recent research focuses on exploring tailored tasks and datasets (See Table 1), and proposing effective models.
The task of LJP is originally defined to predict the results of a legal judgment given the descrip- tions of fact (Kort, 1957). The carly-stage focus was primarily theoretical and quantitative anal- ysis (Nagel, 1963; Keown, 1980), without large- scaled datasets. CAIL.2018 (Xiao et al., 2018) is the first large-scale Chinese legal dataset for judg- ment prediction. Zhong et al. (2018) also includes annotations of the topological dependencies among articles, charges, and terms of penalty. These datasets have garnered significant attention due to their predictive results encompassing charges, penalty terms, and applicable law articles (Xiao et al., 2018; Chalkidis al., 2019; Niklaus et al., 2021). Xiao et al. (2021) collect the cases with longer facts, which distribution is close to actual cases. To enable interpretability, Ge et al. (2021) provide annotations of article-fact correspondence for relevant law article recommendation. Wu et al. (2022) generate rationales for both charge and term of penalty, which are proved be useful for LJP. An al. (2022) annotate the criminal elements for each sentence in facts. However, none of these datasets can be used to comprehensively evaluate sy syllogistic legal reasoning when completing legal analysis tasks.
We formalize SLJA as four interrelated tasks (See Figure 1): (1) article retrieval (AR) selects relevant articles from article database as a major premise; (2) criminal element generation (CEG) generates the criminal elements of a fact minor premise; (3) article interpretation generation (AIG) produces the applicable articles and corresponding rationale according to major and minor premises; (4) legal judgment prediction (LJP) generates the judgments results according to the applicable arti- cles.
We construct dataset for SIJA. First, select criminal cases from the CAIL-Long dataset (Xiao et al., 2021) with the most common charges whose frequencies are larger than 40. Then, we invoke the ChatGPT API to generate the results of four tasks for the selected cases. Finally, we hire 22 work- ers with a legal background to revise the results generated by ChatGPT.
We select four state-of-the-art L.LMs as bench- marks, including ChatGLM (Zeng et al., 2022), Davinci002, Davinci003, and ChatGPT 1 We com- pare the performance of each model in each task with and without syllogistic templates (See Ap- pendix A) as prompts. Through extensive experi- ments and analyses, conclude that (1) SLJA is still challenging for state-of-the-an LLMs; (2) and syllogistic reasoning can provide effective informa- tion for LLMs to improve the performance of legal judgment analysis.
Traditional LJP models are focused on the statis- tical methods (Segal, 1984; Gardner, 1987). With the development of neural networks, RNN-based and transformer-basee models have emerged as the prevailing models for L.P. Chalkidis et al. (2019) evaluate several GRU-based and BERT-based mod- els and find that these models outperform the SVM
htttps://pltffot www.aaar.ooopp..icoommmmom commaai..om//ocs//mdel.... gpt-3-5

---

Task
Statistic
Dataset
CBG
AIG
LJP
#CE Type
Avg. #Fact
#Charge
#Term
#Artiele
#Casc
CAI1-20118 (Xiao 2018)
X
X
386
202
3
183
2,676,075
Caig.-Long (Xian el al,, 2021)
X
V
916
201
5
244
115,849
DPAM (Wang et al., 2018)
X
1,455
0
0
70
17,160
TOPJUDGE-CIO (Zhong
2018)
X
X
V
99
3
1,007,44 744
TOPJUDGE-PKII (Zhong et al., 2018)
X
64
3
175.744
TOPJUDDGE-CAI (Zhong al., 2018)
X
122
3
105
113,536
MI.MN (Ge al, 2021)
X
146
0
86
1,189
(An al., 2022)
X
RLJP (Wu et al., 2022)
402
48
95
89,768
Court-view-gen
2018)
x
219
51
0
171.981
C3VG (Yue et al., 2021b)
x
221
62
5
62,939
SLJA-SYN
1,237
80
5
133
23,913
SLIA-COR
1,379
124
11,239
Table I: Comparison among the proposed SIJAAs dataset and other datasets related to legal judgment. indicates the datasets are not publically available. SIJA-SYN is dataset generated using ChatGPT, while SLJA-COR is dataset derived from SLJA-SYN through manual correction and selection.
CEG
(2023) release the Lawyer LLaMA. which is fine-
Dataset
AIG
LIP
Objoct
Subject
Objective
Sujective
tuned with legal datasets (Chen, 2018; Zhong et al.,
SLIA-SYN
65.7
33.3
119.7
66.4
318.5
82.7
2020) based LLaMA model (Touvron al.,
SLIA-COR
66.9
40.1
119.8
66.0
156.9
36.5
2023). Therefore, also choose state-of-the-art
Table 2: Average length of each subtasks in SLJA-SYN and SIJ--COR.
LLMs as benchmarks in this work. Although these works produce the judgment results according to the facts, they do not provide the analysis process, which makes the judgment results untrustable.
with bag-of-words. Zhong et al. (2018) combine LSTM models for different LJP task in a topologi- cal graph framework. Yang al. (2019) propose the collocation attention mechanism to improve the performance of LSTM on LJP. Zhou al. (2022) propose the deep gating network (DGN) aggre- gate the features of charges. Zhao et al. (2022) and Lyu et al. (2022) propose reinforcement learning models to extract the sentences of facts that contain criminal elements. Xu et al. (2020) and Zhang et al. (2023) propose to improve the performance of LJP through distinguishing similar charges or articles. Yue et al. (2021a) propose to adopt different parts of facts conduct judgment predictions. Although these traditional models have exhibited remark- able proficiency in surpassing the majority base- lines legal tasks. The escalation towards more po- tent LLMs as state-of-the-art (SOTA) benchmarks has become an undeniable trajectory in recent re- search. More recently, LLMs have been rapidly de- Several
2.2 Court View Generation
Court view generation was first proposed to auto- matically generate rationales for judgment results given facts (Ye et al., 2018). Wu al. (2020) in- troduce counterfactual decoder to eliminate the confounding bias between a fact and its correspond- ing charges and generate rationales for charges. Li and Zhang (2021) combine two separate optimiz- ers to generate rationales for charges given facts and articles. Yue et al. (2021b) also generate fine- gained rationales for charges and terms of penalty, respectively. Although these works can improve the interpretability of judgment results, they do not demonstrate the logical relationship explicitly in the analysis process.
2.3 Legal Judgment Syllogism
Syllogism, as form of deductive reasoning, is widely used to express logical arguments (Max- einer, 201 1) and determine if specified behavior is legal
veloped and have become statc-of-the- works start explore how to design legal prompts improve the performance of LJP, for instance, Trautmann et al. (2022) introduce the legal prompt engineer for LJP, Yu et al. (2022) utilize the chain- of-thought for legal entailment task, Huang et al.
(Gold, 2018). Syllogism includes three logical components, i.e., major premise, mi- nor premise, and conclusion (d' Almeida, 2019). The major premise comprises the relevant articles, which serve as a theoretical basis. The minor

---

premise comprises the criminal clements, which provide a factual basis for the reasoning process. The conclusion involves inferring the applicable articles and corresponding judgment results based on the major and minor premises.
China Judgments Online. We filter charges that oceur more than 40 times and randomly select up to 1,000 cases for each charge. Each criminal case contains a fact and its results of the legal judgment, including the charge, term of penalty and related legal articles. In total, we collect 23,913 criminal cases. After obtaining raw data, we first elicit SLJA data using ChatGPT to create SLJA-SYN, and then sample 1,2399 cases from SIJA-SYN and man- ually correct, which is noted as SLJA-COR. The statisties of SLJA datasets are shown in Table and 2. The case distribution of SLJA-COR is shown in Figure 4 (See Appendix B).
In practice, syllogism serves as standard form of legal reasoning extensively employed by judges ensure that logical arguments are sound and indisputable (Hart al., 2012; Constant, 2023). Based on the application of syllogism in real legal scenarios, we hypothesize that completing SLJA would be beneficial for LLMs.
3 Task Definition
4.2 Syllogistic Reasoning Data Elicitation
We define the following four tasks for syllogistic lc- gal judgment analysis (SLJA) based on the concept of syllogistic reasoning:
We guide ChatGPT to generate the SLJA dataset by designing specific prompts for each task as inputs (See Appendix A).
Task 1: Article retrieval (AR) outputs applicable law articles major premises given fact and article library inputs.
The AR task is completed through two-step iteration: (1) generating the description of articles given a fact. Each fact is used as a query to re- trieve the relevant articles from the article library. (2) determining if the retrieved articles can cover the content of the fact. If the response is "No", go back to (1) to conduct retrieval again. Otherwise, the AR task is completed, and the retrieved articles are referred to the major premise.
Task 2: Criminal element generation (CEG) out- puts four criminal elements as minor premises given a fact as an input.
Subject refers to person with criminal responsi- bility that committed a crime;
Subjective Element consists of intention and neg- ligence, which refer to The psychological state of the criminal subject towards the criminal be-
For CEG task, we design four separate prompts to: (1) generate the subject, and determine whether the subject should bear criminal responsibility; (2) determine whether the subjective element is inten- tion or negligence, and explain the criminal intent; (3) generate the object according to the fact; (4) generate the objective elements including criminal time, location, actions, and consequences. These generated criminal elements are referred to the minor premise.
havior and its results;
Object refers to the social relationships protected by criminal law and infringed upon by criminal
acts;
Objective Elements refer to the concrete manifes- tations of crime, which consist of time, location, actions, and consequences.
Task 3: Article interpretation generation (AIG) outputs applicable articles and corresponding ra- tionales as part of conclusion of legal judgment given the outputs of AR and CEG as inputs.
For AIG task, we design a prompt to determine the applicable articles and generate the interpre- tation according to the outputs of AR and CEG. For LJP task, we design a prompt to generate charges and terms of penalty according to the out- puts of AIG. Based on the major premise and minor premise, the outputs of AIG and LJP serve as the conclusion of a legal judgment.
Task 4: Legal judgment prediction (LJP) outputs the charges and terms of penalty as part of con- clusion of legal judgment given the outputs of AIG.
4 Dataset Construction
We conduct the following steps to sample set of ChatGPT generated syllogistic reasoming cases from SLJA-SYN for manual correction: (1) select- ing the cases with poor performance of ChatGPT LJP task; (2) sampling data by charges. We
We construct a dataset for legal judgment analysis with syllogistic reasoning.
4.1 Raw Data Collection
We obtain criminal cases from the CAIL-Long dataset (Xiao et al., 2021), which are collected from
Pht / httpss////wshh.corrrr/ court.

---

randomly select up 500 cases for each charge.
Davinci003 is trained with RLHFF method based on the Davinci002. Davinci003 improves the capability of in-context learning (Fu et al., 2022). ChatGPT is also trained with RLHF method based on the Davinci002. ChatGPT improves the capabilities of in-context learning and dialogue context modeling (Fu et al., 2022).
4.3 Manual Data Correction
We hire 22 workers with a legal background to work the correction process. For each task in SLJA, we define three phrase-level operations for correction, including insertion, deletion, and rewrit- ing. We construct an online correction system and provide a guideline for workers. The guideline in- troduces the definition of each task, the detailed system operations, and an example of data correc- tion. Each worker corrects one case per round, and the system records the corrected positions in the sentences, the corresponding operations, the cor- rection results, and questions during this process. Each worker responsible for correcting the cases involving to 4 charges.
We also evaluated Yuat (Xuanwei Zhang and Zhao, 2022), BELLE-GLAN (Yunjic Ji and Li, 2023), and Vicuna (Chiang al., 2023), but they cannot understand the concepts of four criminal elements, so they cannot follow instructions to con- duct syllogistic reasoning.
In the experiments. we use the prompt templates as shown in Table 6 (See Appendix A), and to en- the reproducibility, we set the hyper-parameter temperature=0 control the randomness of gener- ation.
4.4 Data Quality Assurance
5.2 Evaluation Metrics
To ensure the quality of data correction, pro- vide: (1) Correction feedback mechanism. We col- lect questions during the correction process, con- sult with experts for solutions, and share the ques- tion-solution pairs with all workers. (2) Automatic quality estimation. For each corrected data, we compare the corrected charges with ground-trum charges, if the corrected charges do not contain the ground-truth charges, then we request another worker to correct it again. (3) Manual sampling inspection. For each charge, we randomly sample 10% data and request another worker check the miss-corrected and grammatical issues. there are errors in the corrected data, we require the original worker to re-correct the data.
We use R@ @@k for applicable article prediction and charge prediction, instead of classification met- rics (Gan al., 2022), the reason is that the LLMs produce more than one possible correct answer for each task.
We add penalty factor in R@k metric for term of penalty prediction as shown in Eq. 1,
N Σ1(t E T:l:
kJ
R
N Σ
1| E Ti
(1)
t E imprisonment
manr( (mgr(1,
0:
otherwise
5 Experiments
where 0j is the penalty factor, which aims to pe- nalize the ambiguous range of predicted impris- onment with a lower recall score. The greater the ambiguity of the predicted imprisonment, the lower the penalty factor, resulting in decreased recall score. indicates the type of term of penalty, N indicates the data size, I indicates the scope imprisonment, T; indicates the ground-truit -ruth and [: k] indicates top-k results in predicted terms of penalty. When the scope is less than 12 months, we do not punish the results, otherwise, the penalty factor decreases as the scope increases.
5.1 Benchmark Models
We employ four LLMs as decoders to generate task outputs, which serve as benchmark models with strong performance in Chinese.
ChatGLM (Du et al., 2022) adopts the tradi- tional feed-forward network with model param- eters of 6.2 billion. It conducts supervised fine- tuning and reinforcement learning from human feedback (RLHF) training mixed Chinese and English dataset.
Davinci002 adopts transformer-base structure with model parameters of 175 billion, conducts supervised instruction tuning on a multilingual corpus (Fu et al., 2022).
For CEG task, we extract criminal elements from the generated results by regular expressions and evaluate these elements with specific metrics, re- spectively. Precision (P), Recall (R), and F1 score

---

Charge Prediction
Davin00022
Davinc0003
ChaGPr
ChaKiLM
Syllagsem
55.02
40
47.07
65.22
-1.24
16.65
42.51
41.34
46.00
42.61
21.28
32.96
31.31
31:33
20
23.62
20.28
23.82
23.52
11.68
11,29
11.29
RG@1
R@33
Rig
Term of Penalty Prediction
65.1
15.94
60
42.61
60 18.18
23,24
41.86
38.81
4PA30
$3.99
35.61
25.31
20
9,20
10:91
10.93
Regg
R@3
R
Article Interpretation Generation
4L.0
44.35
44 55
23.27
26:42
29.18 27.02
31,81
33.15
30,00
27.1
33.79
16:12
15.50
15.98
15,95
R
R@3
R@85
Figure 2: Main performance, including the results charge prediction, term of penalty prediction and article
interpretation generation.
Subjective Element
Subject
Objective Elements
Object
Model
((S)
R (%%
F ((6)
Match (%)
ROUGE (%)
ROUGE (%)
Match (%)
ROUGE (%)
CharGLM
76.96
53.32
59.95
79.41
39.90
52.51
16.56
27.15
Davinci002
94.96
51.64
51.60
94.18
33.92
61.51
9.40
23.71
Davinci003
92.22
69.25
75.95
93.07
48.45
66.32
17.85
34.23
CharGPT
82.25
95.66
89.60
99.61
77.76
96.04
48.26
86.41
Table 3: The performance of criminal element generation.
used to evaluate the performance of subjective element prediction, which is binary classification task determine if intention negligence. Match score evaluates the exact matches between gener- ated items and human-corrected ones. ROUGE score evaluates the word overlap between gener- ated subject, objective elements, and object and manually corrected ones.
reaches 55.02/62.87/63.22 R (/1/@///Ree @5,
which are 0..70/16.2//66 superior then plain- text, and in the term of penalty prediction, Chat- GPT with syllogism reaches 15.22465588/665 on
R @ 1/R @ 3/R @5, which are 25.23/37.48/37.. 3..61 su- perior then plaintext. This indicates that the tasks of AR, CEG and AIG are of great benefit for the LJP task.
5.3 Main Performance
Second, for charge prediction, ChatGPT outper- forms Davinci003 model. This is because the mod- els generate charges based applicable articles and Davinci003 model performs worse on the AIG task. As we can see the Article Interpretation Generation, ChatGPT reaches 36.07/44.85/45.87 on @@ 1/R @ 3/R@5, which 9.60/17.83/18.76 points higher than Davinci003.
We present the judgment performance of all the LLMs, both with syllogism and without syllogism (i.c., plaintext) the SLJA-SYN dataset. (Sce Figure 2).
First, for charge prediction and term of penalty prediction, models perform better with syllo- gism than without syllogism. For example in the charge prediction, ChatGPT with syllogism
Thirdly, for penalty term prediction, both

---

Charge
Article
Term of Penalty
Setting
R@3
R@5
R@3
R@5
ChatGPT
55.02
62.87
63.22
36.07
44.85
45.87
53.24
65.81
65.94
-bbjcc
54.27
62.31
63.00
35.88
44.10
45.27
49.89
59.68
60.08
-bjective
49.98
58.65
59.06
33.33
42.81
43.58
43.94
60.58
60.86
-subject
54.91
62.62
62.92
35,96
44.04
45.14
52.42
64.54
64.72
-sujjective
54,30
61.91
62.31
35.62
43.72
44.66
53.32
65.93
66.03
-ciiiial elements
52.30
60.00
60.63
35.26
42.25
43.83
46.20
54.26
54.73
Table 4: The ablation study of criminal elements, including removing criminal element and replacing all criminal elements with fact (denoted as -criminal elements). Bold and underlined fonts indicate leading and compared results in each setting.
Charge
Article
Term of Penalty
#lteration
R@1
R@3
R@5
R@D
R@3
R@5
R@D
R@3
R@5
3
55.02
62.87
63.22
36.07
44.85
45.87
53.24
65.81
65.94
2
54.24
63.06
63.78
35,50
43.61
45.30
50.52
64.06
64.36
52.04
58.17
58.44
33.19
39.31
40.34
49.09
61.74
61.83
0
39.86
41.55
41.57
25.13
37.86
37.86
Table 5: The ablation study of maximum iteration number in article retrieval, 0 indicates removing the AR task, and only uses minor premises as the imput of LJP.
ChatGLM and Davinci002 demonstrate sig- nificantly better performance when utilizing syllogism compared to plaintext. As can see in the term of penalty prediction, Davinci002 and ChatGLM with syllogism reach 35.61/37.51/37.5 and 49.00/58.81/58.99 respectively on @@ 1/R @ which are superior
Distribution of terms
20590
179338
Davincilt6
CharGPT
15000
10000
6068
5000
2677
4333
966
0-
Light
Middle
Severe
Figure The distribution of Terms of Penalty. Light refers to the light penalty, including imprisonment with less than three years, detention and surveillance. Mid refers the middle penalty, including imprisonment with more than three years and less than ten years. Se- vere refers to the severe penalty, including imprisonment with more than ten years, life penalty and death penalty.
30.01/31.75/31.75 and 39.71/47.90/488 to plaintext. The reason is that the Davinci002 and ChatGLM without syllogism always output meaningless output, such "there is not enough information to determine the sentence". This indicates that the syllogism method can provide enough useful information to help the model generate meaningful terms of penalty.
RLHF training method is beneficial for the syllo- gism method. The reason may be that syllogism is more close to the human thinking mode.
Fourth, in the prediction of terms penalty, metrics.
Davinci003 outperforms ChatGPT As we can see in the Term of Penalty. Davinci003 reaches 60.38/72.40/72.43. on R 1/R 3/R@) 55 which outperforms ChatGPT by 7.14/6.5//649, The reason is that ChatGPT tends to predict more middle severe terms of penalty, the distribution is shown in Figure
5.4 Criminal Element Generation Analysis
We report the results of CEG task on corrected SLJA-COR dataset in Table 3.
First, ChatGPT outperforms all the other models all metrics except P in the subjective element. Specifically, objective elements and the subject are contained in the facts which can be extracted by LLMs, and the subjective element and the object are not contained in the facts which need to be inferred from the facts. The reason that Chat-
Fifth. for article interpretation generation, Davinci002 with syllogism performs worth than plaintext on metrics, while Davinci003 with syllogism outperforms plaintext on R@ and per- forms close on other metrics. This indicates that

---

GPT has stronger capabilities of understanding and deduction of fact description.
de impact on decision prediction. The reason is that in most cases, subject is natural person with full capacity for the conduct, while in reality, the subjective element of most cases are intentional, so these two elements can only bring a minor im- provement.
Second, for the subject, Davinci003 performs worse than Davinci002 on Match metric. Specif- ically, Davinci003 is 14.53 superior on ROUGE metric but 1.11 inferior Match metric. This indi- cates that both the Davinci002 and Davinci003 can accurately extract the subject, but the analysis part of Davinci003 is closer to the manual corrected one. This reason may be the RLHF training in Davinci003 can align the model with humans.
5.6 Ablation of Major Premises
In Table 5, we report the ablation study of the max- imum number of article retrieval.
First, As the maximum number of searches in- creases, the performance of all metrics continu- ously improves except R @@3 and @@5 in charge prediction. The reason is that during each iteration, ChatGPT will determine whether the currently re- trieved methods meet the requirements, so it is pos- sible to supplement the correct articles that were not retrieved before. This indicates that repeatedly asking ChatGPT can correct the retrieved articles. Second, without article retrieval, the perfor- mance of charge and term of penalty prediction has significant decrease. This indicates the crim- inal elements need to be combined with retrieved articles can obtain reasonable judgment results.
Third, for the object, other models outperform Davinci002 on Match and ROUGE metries. As we can see the object of Table 3, Davinci002 is 7.16, 8.45 and 38.86 inferior on Match metrics than Ch Davinci003 and ChatGPT respectively. The reason is that Davinci002 cannot understand the concept of the object, it refers to the object as somebody or something that has been violated by criminal, rather than social relationship.
5.5 Effects of Criminal Elements
In Table 4, we report the ablation study of criminal elements.
Third, for the prediction of charge and article, the performance of maximum iteration number of close to 2, as we can see in Table 5, the improvements of R @ 1/R @ 3/R @5 on the predic- tion of charge and article are 0.78/-0.1//-0.56 and 0.57/1. 122/0.57 respectively. This indicates that ChatGPT conducts only one check can achieve good results.
First, the objective is much helpful for le- gal judgment. As we can see in Table 4, re- moving the objective elements makes the perfor- mance of R @ I/R @ 3/R @5 5 drops 5.04/4.22/4.J 1,, 2.74/2.04/2.29 and 9.30/5.23/5.08 in the prediction of charge, article, and term of penalty respectively. The reason is that the objective elements describe the characteristics of the crime (time, location, and behavior) and the consequences of the crime, which can be referred to as the summary of a fact.
Conclusion and Future Work
Second, using fact the minor premise is inferior on all metrics to criminal ele- ments. The performances of -criminal ele- ments on R@ 1/R @5 are 2.72/2.87/2.59. 0.81/2.60/2.04, and 7.04/1 1.55/11.21 inferior in the prediction of charge, article and the terms of a penalty than full prompt respectively. The reason is that the four criminal elements can provide more comprchensive information for the model than only using facts.
In this paper, we introduce the resources of syllo- gistic reasoning for legal judgment analysis: the task, dataset and benchmarks. We define the SLJA according to the concept of syllogistic reasoning, including four tasks: article retrieval (AR), crim- inal element generation (CEG), article interpre- tation generation (AIG) and legal judgment pre- diction (LJPP. We release the first Chinese SIJAA dataset. The dataset is constructed by correcting the outputs of each task, which are generated by Chat- GPT. In order to ensure the quality of the dataset, we hire workers with legal background for data correction. Each case in SLJA dataset includes a detailed analysis process of syllogistic reasoning and we expect the dataset will facilitate the develop- ment of trustworthy legal judgment assistants. Be- sides, we also collect four ofate-of-the theh-rrt LLMs for
Third, the object has a significant impact on the term of penalty prediction. Specifically in Ta- ble 4, the performance without the object (-object) is 3.35/6. 13/5.84 86 inferior to the full prompt. The reason is that the object can determine the severity of the crime, leading different terms of penalty, Fourth, subject and subjective elements have lit-

---

extensive experiments. We design specific prompts for each task as inputs of LLMs. We evaluate each model with and without syllogism and conduct an in-depth analysis. The experimental results indi- cate that: (1) SLJA is still challenging for L.MSs. (2) Syllogistic reasoning can provide effective in- formation for LLMs to improve LJP. (3) RLHF training is beneficial for syllogistic reasoning. (4) The major premise and minor premise in syllogism are both crucial for the final judgment results which can provide more detailed interpretation to make the legal judgment assistant trustable.
the future.
Ethics Statements
We acknowledge the potential risks in the research of legal judgment assistants, and thus, it is impera- tive to address ethical issues in the legal decision- making process. ment results while also taking into account ethical concerns such as safeguarding personal privacy and respecting the authority of the law articles. All raw data collected for this study are sourced from the publicly accessible website and have undergone ethical considerations by the website administra- tors and dataset publishers, in which majority of cases have undergone anonymization to safeguard sensitive information.
crucial to predict legal judg-
As to future work, on the one hand, we will extend SLJA dataset with more legal types and theories, such as civil law and two-stage theory. On the other hand, this work is currently releas- ing test set for LLM evaluation. In the future, we will construct more data to train and evaluate traditional models and conduct a human study to enhance the practicality and interpretability of our approach. Legal LLMs (e.g.. Lawyer LLaMA) that have emerged during the same period as this work will be evaluated in future research. Last but not least, we call for studies to improve the benchmark performance, well as conduct underexplored re- search, such as legal judgment analysis for rare charges under fow-resource settings.
Acknowledgement
We would like to thank the editors and viewers for their helpful comments. This search was supported by the National Key R&D Program of China (No.2022YFC330000 No. No.2020YFB (0022000011400704)) the Natural Science Foun- dation of China (62102234, 62272274, 62202271, 61902219, 61972234, 62072279), the Key Sci- entific and Technological Innovation Program of Shandong Province (20191Z2Y01112),, the Fun- damental Research Funds of Shandong Univer- sity, and VOXReality (European Union Grant, No. 101070521). All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or spon- sors.
7 Reproducibility
To promote the development of legal analysis tasks and facilitate the reproducibility of the re- sults reported in this paper, our released datasets and codes are available at https: / /gi thub. com/ dengwentao99//LJJ
Limitations
References
Note that while syllogistic reasoning serves as guide for analyzing legal arguments, it is not rigidly applied to overturn legal decisions in prac- tice. Judges also take into consideration other fac- tors, such as statutory law, case law, public policy, and the specific facts of the case. Instead, it is a use- ful and practical tool to assist judges in analyzing legal arguments and assessing whether the law has been correctly applied to the parties' arguments. This approach helps maintain consistency and fair- ness in the decision-mann makinn process. Additionally, our proposed dataset will be extended to evaluate traditional models. Evaluation for the L.LMs fine- tuned by legal datasets that have emerged during the same period as this work will be conducted in
Zhenwei An, Quzhe Huang, Cong Jiang, Yansong Feng, and Dongyan Zhao. 2022. Do charge prediction mod- els learn legal theory? Findings of the Association for Computational Linguistics: EMNLP.
Ilis Chalkidis, Ion Androutsopoulos, and Nikolaos Ale- tras. 2019. Neural legal judgment prediction in en- glish. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317-4323.
Wenhan Chao, Xin Jiang. Zhunchen Luo, Yakun Hu, and Wenjia Ma. 2019. Interpretable charge predic- tion for criminal cases with dynamic rationale at- tention. Journal of Artificial Intelligence Research. 66:743-764.
Feida Chen. 2018 The legal consultation data and cor- pus of the thesis from china law network.replicttion

---

data for: Design and research of legal consultation text classification system.
Herbert Lionel Adolphus Hart, Herbert Lionel Adol- phus Hart, Joseph Raz, and Leslie Green. 2012. The concept of law. oxford university press.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Erie Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Quzhe Huang, Mingxu Tao. Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yan- song Feng. 2023. Lawyer LLaMA technical report. ArXiv:2305.1500 15062..
R. Keown. 1980. Mathematical models for legal predic- tion. Computer/lj. 2:829.
Axel Constant. 2023. A bayesian model of legal syl- logistic reasoning. Artificial Intelligence and Law, pages 1-22.
Fred Kort. 1957. Predicting supreme court decisions mathematically: A quantitative analysis of the "right to counsel" cases. American Political Science Re- view, 51(1):1-12.
Junyun Cui, Xiaoyu Shen, Feiping Nie, Zheng Wang. Jinglong Wang. and Yulong Chen. 2022. A survey on legal judgment prediction: Datasets, metries, models and challenges. ArXiv:2204.04859
Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Pretrained language models for text gen- eration: A survey. Proceedings of the Thirtieth International Joint Conference on Artificial Intelli- gence (IJCAI).
Aniket Deroy Kripabandhu Ghosh, and Saptarshi Ghosh. 2023. How ready are pre-trained abstractive models and lims for legal case judgement summariza- tion? arXiv preprint arXiv:2306.0124
Quanzhi Li and Qiong Zhang. 2021. Court opinion
generation from case fact description with legal basis. In Proceedings of the AAAI Conference on A Artificial Intelligence, volume 35, pages 14840-1488 14848.
Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GLMM: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics, pages 320-335.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, 2023. Pre- train, prompt. and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
Luís Duarte d' Almeida. 2019. on the legal syllo- gism. Dimensions of Normativity: New Essays on Metaethics and Jurisprudence. page 335.
Yougang Lyu, Zihan Wang, Zhaochun Ren, Pengjie Ren, Zhumin Chen, Xiaozhong Liu, Yujun Li, Hongsong Li, and Hongye Song. 2022. Improving legal judg- ment prediction through reinforced criminal element extraction. Information Processing & Management, 59(1):002780.
Yi Feng. Chuanyi Li, and Vincent Ng. 2022. Legal judgment prediction: A survey of the state of the art. In Proceedings of the Thirty-Firsi International Joint Conference on Artificial Intelligence, 1JCAI-22, pages 5461-5469.
James R Maxeiner. 2011. Failures of American civil justice in international perspective. Cambridge Uni- versity Press.
Yao Fu, Hao Peng, and Tushar Khot. 2022. How does GPT obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu's 's Notion.
Stuart S Nagel. 1963. Applying correlation analysis to case prediction. Tex. L. Rev. 42:1006.
Leilei Gan, Baokui Li, Kun Kuang, Yi Yang, and Fei Wu. 2022. Exploiting contrastive learning and numerical evidence for improving confusing legal judgment prediction. 2rXiv:2211.082555
Joel Niklaus, Ilias Chalkidis. and Matthias Stürmer. 2021. Swiss-judgment tatssjuuugmmentppeccitiooo A multilingual legal judgment prediction benchmark. Proceedings of the Natural Legal Language Processing Workshop.
Anne von der Lieth Gardner. 1987, An artificial intelli- gence approach to legal reasoning. MIT press.
Jeffrey A Segal. 1984. Predicting supreme court cases probabilistically: The search and seizure cases. 1962- 1981. American Political Science Review, 78(4):891- 900.
James Gardner. 2007. Legal argument: the structure and language of effective advocacy. LexisNexis.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foun- dation language models. 2302:2302.13971
Jidong Ge. Yunyun Huang, Xiaoyu Shen, Chuanyi Li, and Wei Hu. 2021. Learning fine-grained fact-article correspondence in legal cases. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 29:3694-3706.
Dietrich Trautmann, Alina Petrova, and Frank Schilder. 2022. Legal prompt engineering for multilingual legal judgement prediction. ArXh::221..0219,
Michael Evan Gold. 2018. Primer on Legal Reason- ing. Cornell University Press.

---

Pengfei Wang, Ze Yang. Shuzi Niu, Yongfeng Zhang. Lei Zhang, and ShaoZhang Niu. 2018. Modeling dynamic pairwise attention for crime classification over legal articles. In the 41st international ACM SIGIR conference on research & development in in- formation retrieval, pages 485-494.
Linan Yue, Qi Liu, Han Wu, Yanqing An, Li Wang. Senchao Yuan, and Dayong Wu, 2021b. Circum- stances enhanced criminal court view generation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 1855-1859.
Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang. Luo Si, and Fei Wu. 2020. De-biased court's view generation with causality. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 763-780.
Yan Gong Yiping Peng Qiang Niu Baochang Ma Yun- jie Ji, Yong Deng and Xiangang Li. 2023. BELLE: Be everyone's s large language model engine.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. GLM-130B: An open bilingual pre-trained model. arXiv preprint arXiv:2210.0244
Yiquan Wu, Yifei Liu, Weiming Lu, Yating Zhang. Jun Feng. Changlong Sun, Fei Wu, and Kun Kuang. 2022. Towards interactivity and interpretability: A rationale-based legal judgment prediction framework. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 4799..
Han Zhang, Zhicheng Dou, Yutao Zhu, and Ji-Rong Wen. 2023, Contrastive learning for legal judgment prediction. ACM Transactions on Information Sys- tems.
Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021. Lawformer: A tre-trained language model for chinese legal long documents, Al Open, 2:79-84.
Jie Zhao, Ziyu Guan, Cai Xu, Wei Zhao, and Enze Chen. 2022. Charge prediction by constitutive elements matching of crimes. In Proceedings of the Thirry- First International Joint Conference on Artificial In- telligence, IJCAI, volume 22, pages 4517-4523.
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. CAIL2001 8: A large-scot scale legal dataset for judgment prediction.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.1822
ArXiv:1807.0m 827.02478.
Nuo Xu, Pinghui Wang, Long Chen, Li Pan, Xiaoyan Wang, and Junzhou Zhao. 2020. Distinguish confus- ing law articles for legal judgment prediction. Pro- ceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics.
Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judg- ment prediction via topological learning. Pro- ceedings of the conference on empirical methods in natural language processing, pages 3349.
Liang Xu Xuanwei Zhang and Kangkang Zhao. 2022. Yuan large language model for dialogue in chinese and english.
Haoxi Zhong, Chaojun Xiao, Cunchao Tu. Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. JEC- QA: legal-domain question answering dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9701-9708.
Wenmian Yang. Weijia Jia, Xiaojie Zhou, and Yutao Luo. 2019. Legal judgment prediction via multi- perspective bi-feedback network. Proceedings of the 28th International Joint Conference on Artificial Intelligence.
Siying Zhou, Yifei Liu, Yiquan Wu, Kun Kuang, Chun- yan Zheng, and Fei Wu. 2022. Similar case based prison term prediction. In Artificial Intelligence: Sec- ond CAAI International Conference. CICAI 2022, Beijing, China, August 27-28, 2022, Revised Selected Papers, Part III, pages 284-297.
Hai Ye, Xin Jiang, Zhunchen Luo, and Wenhan Chao. 2018. Interpretable charge predictions for criminal cases: Learning to generate court views from fact descriptions. Proceedings of the Conference of the North American Chapter of the Association for Com- putational Linguistics.
A Prompt templates
Fangyi Yu, Lee Quartey, and Frank Schilder. 2022. Le- gal prompting: Teaching language model to think like lawyer. ArXiv:2212.0132.
The Prompt templates are shown in Table 6.
B Case distribution
Linan Yue, Qi Liu, Binbin Jin, Han WU, Kai Zhang, Yanqing An. Mingyue Cheng, Biao Yin, and Day- ong Wu. 2021a. NeurJudge: A circumstance-awee neural framework for legal judgment prediction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 973-982.
Figure demonstrates the distribution of cases, which shows the charges with the number of cases exceeding 50.

---

Task
Prompt template
Generation
AR
Please list the relevant articles based on the fact.
Determination
Please determine if the retrieved articles can the content of the fact, Please "Yes" "No" and explain
Subject
Please extract subject
fact analyze the constituent elements subject.
Constituent elements subject includes reaching criminal age and heing a fully capable person.
CEG
# H fu H
Subjective Element
Please extract subjective element from the fact and describe the criminal intents. The subjective element includes: intention and negligence.
* 1E P2 ;
Object
Please extract the object from the fact and explain it. Object: The social relationship protected by criminal law and infringed upon by the crime.
# PY 21
T2 [i] TH 32 10
Objective Elements
Please extract the objective elements from the fact. Objective elements include: criminal time, location, actions and consequences.
3 Jf
AIG
Please determine the applicable articles from relevant articles according to criminal elements.
List applicable articles and explain them.
1. 2.101 Please generate the judgment results according to the applicable articles. judgment results include; charges; 2. terms penalty.
# 13 # A6 ti A #
LIP
Table 6: The prompt templates for each task.

---

Fread
Allowing Others to Use Drugs
Obstruution of Official Duty
Contract Fraad
Drny, Smouzgiing Trallicking, Transportnng and Manuaassarng
Degal Possession of Drugs
Rublwry
Illegd Poseession, Conccolment of Fircarms, and Ammunition
Illegal Operation
'I heft
Gambling
Extartion
Rinboodkmen
Unlswfll Detention
Crafit Corl Frord
Rockless Driving
Manslsnghter
Dribery
Production Sole Couneerfess Drugs
Falatiication Valtue Added Tix Expoct Refands Deductions
Asomlt
Megal Logging
Illegdl Fundraising
Vandalsmn
Non- Poo-payment of Labor Compencation
Trafic Accidents
Public Braw!
Murber
Arson
Rape
Misappoopriationn of Pusds
Accepting Dribes
Pongery Allerstion, Sale of Official Docoments Certifiioles and Sedl
Timber Thet
Sale of Connerrfei Registoeed Trademack Goods
Illaal Cultivation of Drog-Pruddcing Plonts
Emberekeme of Publlic Funds
Envirommental Pollution
Major Liahility Aocidents
Opersting a Gamiling Establishment
Unauthorized Occupation Agricultuzal Land
Illegal Mlanfauttree Purduase, Stkt Transportation, Mailing, and Storage of Frecrmss Ammunition, and Explocives
Beibing
Production, Duplicatioon Publication Sale, and Distribation of Ohacene Materidk Profit
Forgeey of Corporate, Enterprise, Institution. and People's
Organization Seals
Production and Sale of
Products
Swiscllin
Extortion Blackmail
Coneadling and Conccoling Oriminal Procueds and Criminal Inooese
Number of Cusees
Figure 4: The distribution of cases in SLIA-COR dataset, in which the number of cases for each charge is larger than 50.


Reviews:
[{"review_id": "Ntf6QhI8Ms", "reviewer": "Reviewer_JXs1", "summary": "", "questions": "A. Why add a penalty factor in Eq 1 on Line 360? What is the meaning of that factor? I.e. what does the penalty stand for, what behavior of the model does it penalize?\n\nB. Figure 2: there is no baseline or previous SOTA to compare to. What score would a majrity baseline get?\n\nC. The ethics statement says that sensitive information has been anonymized by the authors (Line 590). How was that done?\n\nD. How were the LLMs prompted? Was there any randomness in the generation? This is important for reproducibility.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment:**\n\nIt is not clear whether the proposed framework is actually interpretable for humans; it's possible that output produced by models following this framework are not useful in practice. A human study could help answer that question.\n\n**Response:**\n\n**Our proposed framework is designed to offer actual interpretability and is rooted in practical utility.**\n\nWe hired 22 workers with a legal background to build the dataset. We asked their advice and confirmed that, in the real judgment practice, syllogism is a commonly used method of reasoning in legal judgment analysis to analyze cases and arrive at judgment results [1-3]. This serves as a strong basis for the interpretability of our framework in a real-world legal context.\n\nFurthermore, past research has demonstrated that the reasoning framework applied to LLM can emulate human-like reasoning processes, lending credence to its practical relevance [3].\n\nWhile we acknowledge your concern about the practical utility of our framework, we are confident that it can yield valuable insights and meaningful results. We appreciate your suggestion of incorporating a human study to further demonstrate the effectiveness of our proposed framework.  In our future work, we will certainly explore the possibility of conducting a human study to enhance the practicality and interpretability of our approach. Thank you for your constructive comments, which will undoubtedly contribute to the improvement of our work. \n\n&nbsp;\n\n\n**Question A:**\n\nWhy add a penalty factor in Eq 1 on Line 360? What is the meaning of that factor? I.e. what does the penalty stand for, what behavior of the model does it penalize?\n\n**Response:**\n\n**The penalty factor aims to penalize the ambiguous range of predicted imprisonment with a lower recall score.**\n\nThe greater the ambiguity of the predicted imprisonment, the lower the penalty factor, resulting in a decreased recall score. For example, compared to \"imprisonment for more than 2 but less than 3 years,\" \"imprisonment for more than 1 but less than 3 years'' would incur a lower penalty factor. We hope that the predicted imprisonment can be sufficiently precise on the range. Our aspiration is that the projected duration of imprisonment remains highly accurate within its intended range.\nWithout the penalty factor, Eq. 1 in line 360 will degrade into a classical Recall score, which can only evaluate whether the predicted penalty type is correct or not, but unable to evaluate if the predicted imprisonment is ambiguous. \n\n&nbsp;\n\n**Question B:**\n\n Figure 2: there is no baseline or previous SOTA to compare to. What score would a majority baseline get? \n\n**Response:**\n\n**Our goal is to comprehensively assess the influence of syllogism on State-of-the-Art (SOTA) LLMs for legal tasks.**\n\nAs demonstrated in many recent papers (e.g., [5-7]), BERT-based models have exhibited a remarkable proficiency in surpassing majority baselines when tackling legal tasks within the scope of legal tasks. This pronounced advantage primarily stems from their robust language modeling capabilities. The escalation towards more potent Large Language Models (LLMs) as SOTA benchmarks have become an undeniable trajectory in recent research. In alignment with this trajectory, we have meticulously assembled an exhaustive compilation of cutting-edge LLMs, drawing from our best available knowledge. These curated models will serve as the cornerstone for our benchmarking standards. This strategic approach effectively eliminates the need for relying on conventional majority baselines, paving the way for the integration of advanced models into the framework.\n\n&nbsp;\n\n**Question C:**\n\nThe ethics statement says that sensitive information has been anonymized by the authors (Line 590). How was that done?\n\n**Response:**\n\nOur proposed benchmark dataset is constructed based on CAIL-Long [8], in which a majority of cases have undergone anonymization to safeguard sensitive information.\n\n&nbsp;\n\n**Question D:**\n\nHow were the LLMs prompted? Was there any randomness in the generation? This is important for reproducibility.\n\n**Response:**\n\n**We design specific prompts for each subtask.**\n\nTable 5 in the Appendix A demonstrates the prompt templates. The facts serve as input for the AR and CEG tasks, while AIG utilizes the outputs from AR and CEG as its input. Subsequently, LJP takes the output of AIG as its input.\n\n**There is no randomness in the generation.**\n\nWe set the hyper-parameter temperature=0 to control the randomness of generation. The temperature parameter is used to adjust the probabilities of the predicted words in the softmax output layer of the model to control the randomness. Lower temperature indicates less random outputs. \n\n&nbsp;\n\n**Typos Grammar Style And Presentation Improvements:**\n\n1. The colors in Figures 2 and 3 could be more color-blind friendly. See https://www.extensis.com/hs-fs/hubfs/Colorblind-Color-Palette.jpg?width=15003&name=Colorblind-Color-Palette.jpg for ideas.\n2. The related work around Lines 147-163 enumerates papers and summarizes them. It would be more helpful to provide an overview, e.g. saying that many papers have addressed the task of LJP, and then cite a string of papers without going into too much detail.\n3. Line 363 \"scope of imprisonment\" Do you mean \"duration of imprisonment\"?\n4. \"plaintext\": this word is used multiple times, for example on Line 390. From the paper, I'm guessing that it means \"without syllogism\", i.e. prompting a model to predict charges without prompting it for the intermediate tasks. To help the reader understand, it would be good to clarify this, for example with a footnote.\n\n**Response:**\n\nThanks for providing a lot of valuable suggestions for the details of the paper writing.\n\n1. Thanks for your suggestion, we will change the colors of Figure 2 and 3 to be more color-blind friendly. \n2. We will revise the description of related works as follows: With the development of neural networks and transformer-based models, there are some works use the neural networks to predict judgment results, such as, RNN-based models [9,10], BERT [9], reinforcement learning [11] and contrastive learning methods [12]. \n3. The “scope of imprisonment” means the range of possible imprisonment, for example, the scope of imprisonment is more than two but less than three years. While “duration of imprisonment” indicates the specific length of imprisonment.  \n4. Thanks for your suggestion, we will clarify that “plaintext” means “without syllogism” in the footnote. \n\n&nbsp;\n\n**References:**\n\n[1] Herbert Lionel Adolphus Hart et al. 2012. The concept of law.\n\n[2] Mingxuan Gao. 2009. Criminal Law. \n\n[3] Cong Jiang, and Xiaolei Yang, 2023. Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction. ICAIL. \n\n[4] Jason Wei et al. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS. \n\n[5] Junyun Cui et al. 2022. A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges. Arxiv. \n\n[6] Gil Semo et al. 2022. ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US. ACL. \n\n[7] Ashutosh Modi. 2023. SemEval 2023 Task 6: LegalEval - Understanding Legal Texts. ACL. \n\n[8] Chaojun Xiao et al. 2021. Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents. Arxiv. \n\n[9] Ilias Chalkidis et al. 2019. Neural Legal Judgment Prediction in English. ACL. \n\n[10] Wenmian Yang et al. 2019. Legal judgment prediction via multi-perspective bi-feedback network. IJCAI. \n\n[11] Jie Zhao et al. 2022. Charge prediction by constitutive elements matching of crimes. IJCAI. \n\n[12] Han Zhang et al. 2023. Contrastive learning for legal judgment prediction. TOIS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a principled way of introducing interpretability into legal AI decisions.\n- This paper introduces a benchmark dataset, and the annotation process is sound.", "reasons_to_reject": "- It is not clear whether the proposed framework is actually interpretable for humans; it's possible that output produced by models following this framework are not useful in practice. A human study could help answer that question.", "questions_to_authors": "A. Why add a penalty factor in Eq 1 on Line 360? What is the meaning of that factor? I.e. what does the penalty stand for, what behavior of the model does it penalize?\n\nB. Figure 2: there is no baseline or previous SOTA to compare to. What score would a majrity baseline get?\n\nC. The ethics statement says that sensitive information has been anonymized by the authors (Line 590). How was that done?\n\nD. How were the LLMs prompted? Was there any randomness in the generation? This is important for reproducibility.", "paper_topic_and_main_contributions": "This paper breaks down Legal Judgment Prediction as a sequence of tasks. The point is to introduce interpretability and explainability. The paper describes how a set of cases from Chinese courts were annotated for each task. The annotation was done as a prediction made by ChatGPT followed by post-editing by crowdworkers. Finally, the paper compares various LLMs on this new dataset.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The colors in Figures 2 and 3 could be more color-blind friendly. See https://www.extensis.com/hs-fs/hubfs/Colorblind-Color-Palette.jpg?width=15003&name=Colorblind-Color-Palette.jpg for ideas.\n- The related work around Lines 147-163 enumerates papers and summarizes them. It would be more helpful to provide an overview, e.g. saying that many papers have addressed the task of LJP, and then cite a string of papers without going into too much detail.\n- Line 363 \"scope of imprisonment\" Do you mean \"duration of imprisonment\"?\n- \"plaintext\": this word is used multiple times, for example on Line 390. From the paper, I'm guessing that it means \"without syllogism\", i.e. prompting a model to predict charges without prompting it for the intermediate tasks. To help the reader understand, it would be good to clarify this, for example with a footnote."}}, {"review_id": "J3H9IzMtHk", "reviewer": "Reviewer_mGTi", "summary": "", "questions": "Question A. Why use LLM to do the prediction?\n\nQuestion B. The abstract say there are 11,239 cases, is it all processed by the Manual Data Correction? \n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1:**\n\nLack of experiments. The authors only use LLMs in the experiments, and such experiment settings are not sufficient enough to prove the value of the dataset, since the new content is only used as in-context learning but not in the model training.\n\n**Response:**\n\n**Our experiments focus on assessing the influence of syllogism on State-of-the-Art (SOTA) LLMs for legal judgment analysis.**\n\nWe aim to provide a benchmark dataset for assessing the influence of syllogism on LLMs. \nThe syllogism reasoning legal judgment analysis benchmark dataset contains all subtasks in syllogism, syllogism is a commonly used method for reasoning in legal judgment to analyze cases and arrive at judgment results in real practice[1,2].  Although BERT-based models excel in legal tasks [1-3], legal research on Large Language Models (LLMs) is a frontier trend [1], which is the basis of forming legal task benchmarks.  In the experiments, we employ the prompt learning approach which is suitable for inference in large models [4,5], instead of the training methods used in the traditional LJP models. \n\n&nbsp;\n\n**Comment 2:**\n\nBesides, why use LLM to do the prediction? Especially in LJP task, there are so many labels to predict, and it’s difficult for LLMs. Moreover, the inference speed of LLMs are much slower than the small predictive models, and the prediction results of LLMs are also un-controlable (one irrelevant word can change the whole result). In fact, the sota LJP models already achieve much better performance (e.g., LADAN[1] or NeurJudge[2]).\n\n[1] Distinguish Confusing Law Articles for Legal Judgment Prediction\n\n[2] NeurJudge: A Circumstance-aware Neural Framework for Legal Judgment Prediction\n\n**Response:**\n\n**Compared to the traditional LJP model, LLMs can offer a more extensive reasoning process to assist judges in judgment decisions.**\n\nAlthough traditional BERT-based models [1-3] have exhibited a remarkable proficiency in surpassing majority baselines when tackling legal tasks within the scope of legal tasks. This pronounced advantage primarily stems from their robust language modeling capabilities. The escalation towards more potent Large Language Models (LLMs) as SOTA benchmarks have become an undeniable trajectory in recent research. In alignment with this trajectory, we have meticulously assembled an exhaustive compilation of cutting-edge LLMs, drawing from our best available knowledge. These curated models will serve as the cornerstone for our benchmarking standards. This strategic approach effectively eliminates the need for relying on conventional majority baselines, paving the way for the integration of advanced models into the framework. \n\nAlthough generating complete reasoning steps takes more time compared to predicting a single judgment result, it can offer more explanations to foster judges' trust in the model. \n\nSome previous works have demonstrated that varying the prompts with minor irrelevant content does not alter the output results[4].\n\n&nbsp;\n\n**Comment 3:**\n\nSome details of dataset are not clear, at least there should be a data analysis, including the distribution of the cases, the average length of each tasks and so on.\n\n**Response:**\n\nProviding a comprehensive dataset analysis is indeed valuable, and we acknowledge your point. We appreciate your suggestion, and we have incorporated the Case distribution of the top-10 charges in Table 1. Additionally, we've included the average length of each task in Table 2 to enhance the clarity of our dataset's characteristics. We believe the supplementary information we have added below contributes positively to our work. \n\n**Table 1.** Case distribution of top-10 charges. \n|    &nbsp;  |    &nbsp;   |    &nbsp;    |    &nbsp;    |    &nbsp;    |    &nbsp;   |\n|    :-:   |    :-:   |    :-:    |    :-:    |    :-:    |    :-:    |\n|    **Charge**      | Fraud |  Harboring Drug Users    | Obstruction of Official Duties | Contract Fraud | Smuggling, Trafficking, Transporting, and Manufacturing Drugs | \n|   **Proportion**  |    4.85%   |    4.84%    |   4.40%    |    3.91%    |   3.56%    |\n| **Charge**      | Illegal Possession of Drugs | Robbery | IIllegal Possession and Concealment of Firearms and Ammunition | Illegal Business Operation | Theft | \n|   **Proportion**   |    3.52%  |   3.52%    |    3.51%   |   3.21%    |    3.11%   |\n\n &nbsp; \n &nbsp; \n &nbsp; \n\n**Table 2.** The average length of each task.\n\n|    &nbsp;  |    &nbsp;   |    &nbsp;    |    &nbsp;    |    &nbsp;    |    &nbsp;   |    &nbsp;   |    &nbsp;   |\n|    :----:   |    :----:   |    :----:    |    :----:    |    :----:    |    :----:    |    :----:    |    :----:    |\n| **Subtask** |     AR    | Object | Objective | Subject | Subjective |  AIG |   LJP  |\n| **Avg. #Task**  |    1217.3   | 66.8 | 119.8 | 40.2 | 65.9 |  121.3 |   36.5 |\n\n&nbsp;\n\n**Question A. Why use LLM to do the prediction?**\n\n**Response:**\n\nAlthough traditional BERT-based models [1-3] have exhibited a remarkable proficiency in surpassing majority baselines when tackling legal tasks within the scope of legal tasks. This pronounced advantage primarily stems from their robust language modeling capabilities. The escalation towards more potent Large Language Models (LLMs) as SOTA benchmarks have become an undeniable trajectory in recent research. In alignment with this trajectory, we have meticulously assembled an exhaustive compilation of cutting-edge LLMs, drawing from our best available knowledge. These curated models will serve as the cornerstone for our benchmarking standards. \n\n&nbsp;\n\n**Question B. The abstract say there are 11,239 cases, is it all processed by the Manual Data Correction?**\n\n**Response:**\n\nYes, you are right, all 11,239 cases are processed by manual data correction. We hired 22 workers with a legal background to work on the correction process.\n\n&nbsp;\n\n**References:**\n\n[1] Cui Junyun et al. 2022. A survey on legal judgment prediction: Datasets, metrics, models and challenges. Arxiv. \n\n[2] Gil Semo et al. 2022. ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US. ACL. \n\n[3] Ashutosh Modi. 2023. SemEval 2023 Task 6: LegalEval - Understanding Legal Texts. ACL. \n\n[4] Trautmann et al. 2022. Legal prompt engineering for multilingual legal judgment prediction. Arxiv. \n\n[5] Blair-Stanek et al. 2023. Can GPT-3 perform statutory reasoning?. Arxiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dateset is valuable with the human correction.\n\nThis writing is clear and the paper is easy to follow.\n", "reasons_to_reject": "Lack of experiments. The authors only use LLMs in the experiments, and such experiment settings are not sufficient enough to prove the value of the dataset, since the new content is only used as in-context learning but not in the model training.\n\nBesides, why use LLM to do the prediction? Especially in LJP task, there are so many labels to predict, and it’s difficult for LLMs. Moreover, the inference speed of LLMs are much slower than the small predictive models, and the prediction results of LLMs are also un-controlable (one irrelevant word can change the whole result). In fact, the sota LJP models already achieve much better performance (e.g., LADAN[1] or NeurJudge[2]).\n\nSome details of dataset are not clear, at least there should be a data analysis, including the distribution of the cases, the average length of each tasks and so on.\n\n[1] Distinguish Confusing Law Articles for Legal Judgment Prediction\n\n[2] NeurJudge: A Circumstance-aware Neural Framework for Legal Judgment Prediction\n", "questions_to_authors": "Question A. Why use LLM to do the prediction?\n\nQuestion B. The abstract say there are 11,239 cases, is it all processed by the Manual Data Correction? \n", "paper_topic_and_main_contributions": "This paper mainly constructs a new dataset called SLJA, according to the concept of syllogistic reasoning, including four tasks: article retrieval (AR), criminal element generation (CEG), article interpretation generation (AIG) and legal judgment prediction (LJP). Besides, the authors evaluate the performance of four popular LLMs on the SLJA dataset.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vVkQhgryRd", "reviewer": "Reviewer_vyvp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comment 1:**\n\nWhether the authors have tried to experiment with LLMs that have been fine-tuned using legal data, e.g., Lawyer LLaMA [1].\n\n**Response:**\n\n**Our work and Lawyer Llama emerged during the same period**, both our submission date and the release date of the Lawyer Llama model fall in the same month of June. Your suggestion holds significant value, and we greatly appreciate it. In forthcoming endeavors, we intend to include legal fine-tuning models such as Lawyer LLaMA for comparison, enhancing the comprehensiveness of our research.\n\n&nbsp;\n\n**Comment 2:**\n\nAuthors propose a four-criminal element approach to analyse legal judgment is feasible, but at present, the four-element system receives a lot of controversy, whereas the two-stage system is currently the dominant theory for analysing legal judgment, which the authors could explore further [2].\n\nReferences\n\n[1] Quzhe Huang et al. 2023. Lawyer LLaMA Technical Report\n\n[2] Mingkai Zhang et al. 2003. Criminal Law\n\n**Response:**\n\n**In the Chinese legal system, the Four Elements theory is commonly employed as a theoretical foundation.**\n\nThe Four Elements theory features a rigorous logic and aligns with cognitive principles. It can distinctly delineate the boundaries between guilt and innocence. Therefore, compared to the Two-stage system Theory, the Four Elements theory is more in line with China's requirements for legal analysis [1-3]. \n\n&nbsp;\n\n**Comment 3:**\n\nThe authors can compare some classical methods to show the improvement of LLMs.\n\n**Response:**\n\n**Our goal is to comprehensively assess the influence of syllogism on State-of-the-Art (SOTA) LLMs for legal tasks, instead of outperforming classical methods.**\n\nAlthough classical BERT-based models [1-3] have exhibited a remarkable proficiency in surpassing majority baselines when tackling legal tasks within the scope of legal tasks. This pronounced advantage primarily stems from their robust language modeling capabilities. The escalation towards more potent Large Language Models (LLMs) as SOTA benchmarks has become an undeniable trajectory in recent research. In alignment with this trajectory, we have meticulously assembled an exhaustive compilation of cutting-edge LLMs, drawing from our best available knowledge. These curated models will serve as the cornerstone for our benchmarking standards. We focus on investigating the influence of syllogism on LLMs and this approach opens pathways for integrating advanced models into the framework instead of comparing them to classical methods. \n\n&nbsp;\n\n**Comment 4:**\n\nThe authors can discuss the feasibility in civil law, not only limited to criminal law.\n\n**Response:**\n\n**Our proposed framework is firmly grounded in practical utility, while we recognize the value of discussing its feasibility in civil law contexts.**\n\nIt remains pertinent to acknowledge that the theory of elements of a crime, rooted in the evolution of criminal law [1], holds no applicability within the domain of civil law. While our framework's context primarily engages with criminal elements as minor premises, acknowledging and comparing these distinctions remains constructive. Adding such a comparative discussion below ensures our work's accessibility across domains, fostering comprehensive understanding among readers from diverse backgrounds.\n\nCivil law and criminal law exhibit substantial disparities in their initiation, resolution processes, and overarching objectives. In criminal law, cases address offenses against society, resulting in punitive measures against wrongdoers [4]. Conversely, civil law governs disputes between individuals or entities, emphasizing restitution for aggrieved parties [5]. Criminal cases may entail penalties like imprisonment, while civil cases aim to redress losses [6,7]. This stark contrast underscores the profound divergence in purpose and consequences between the two legal realms.\n\n&nbsp;\n\n**Comment 5:**\n\nMore cases are needed.\n\n**Response:**\n\nAs shown in Table 1, our proposed benchmark dataset encompasses the largest count of cases involving complete subtasks of syllogistic reasoning. Furthermore, our approach to data annotation is adaptable, allowing for potential expansion of the dataset in the future.\n\n&nbsp;\n\n**References:**\n\n[1] Mingxuan Gao. 2009. Criminal Law. \n\n[2] Zhao, J., et al. 2022. Charge prediction by constitutive elements matching of crimes. IJCAI. \n\n[3] Yougang Lyu et al. 2022. Improving legal judgment prediction through reinforced criminal element extraction. IPM. \n\n[4] P.J.A. Von Feuerbach. 1847. Lehrbuch des gemeinen in Deutschland gültigen peinlichen Rechts. Georg Friedrich Heyer. \n\n[5] Yuan Zhi-jie. 2020. The Civil Basis of State liability. Journal of Political Science and Law. \n\n[6] Civil Code of the People's Republic of China. 2020.\n\n[7] Criminal Procedure Law of the People's Republic of China. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.It is necessary to use LLMs for legal judgement analysis.\n\n2.Authors design a Syllogistic Reasoning approach for Legal Judgment Analysis and demonstrate the effectiveness of multiple LLMs on legal judgment tasks.", "reasons_to_reject": "1.Whether the authors have tried to experiment with LLMs that have been fine-tuned using legal data, e.g., Lawyer LLaMA [1].\n\n2.Authors propose a four-criminal element approach to analyse legal judgment is feasible, but at present, the four-element system receives a lot of controversy, whereas the two-stage system is currently the dominant theory for analysing legal judgment, which the authors could explore further [2].\n\n3.The authors can compare some classical methods to show the improvement of LLMs.\n\n4.The authors can discuss the feasibility in civil law, not only limited to criminal law.\n\n5.More cases are needed.\n\n\nReferences\n\n[1] Quzhe Huang et al. 2023. Lawyer LLaMA Technical Report\n\n[2] Mingkai Zhang et al. 2003. Criminal Law\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, authors construct a syllogistic reasoning dataset for legal judgment analysis. Besides the authors compare a lot of LLMs, conducing an in-depth analysis of the capacity of their legal judgement analysis.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a structured approach to Legal Judgment Prediction, emphasizing interpretability and explainability. It introduces a new dataset, SLJA, based on the concept of syllogistic reasoning. This dataset encompasses four tasks: article retrieval (AR), criminal element generation (CEG), article interpretation generation (AIG), and legal judgment prediction (LJP). The data was annotated using predictions from ChatGPT, which were then post-edited by crowdworkers. The paper evaluates the performance of various Large Language Models (LLMs) on this dataset, aiming to provide an in-depth analysis of their capabilities in legal judgment tasks.

The paper offers a systematic method to incorporate interpretability into AI-driven legal decisions. The introduction of the SLJA benchmark dataset is a significant contribution, and the annotation process employed is robust. The dataset's value is further enhanced by human corrections, ensuring its reliability. The paper is well-written, making it easily comprehensible. The use of LLMs for legal judgment analysis is timely and necessary, and the paper's syllogistic reasoning approach provides a novel perspective on legal judgment analysis. The evaluation of multiple LLMs on legal tasks further underscores the paper's significance.

The experimental setup is limited, relying solely on LLMs, which may not fully demonstrate the dataset's value. The use of LLMs, especially for tasks with multiple labels like LJP, raises concerns due to their unpredictability and slower inference speeds. Existing state-of-the-art LJP models have reportedly achieved better performance, making the choice of LLMs questionable. The paper could benefit from exploring other dominant theories for analyzing legal judgment, like the two-stage system, and expanding its scope beyond criminal law. Comparisons with classical methods and LLMs fine-tuned with legal data could provide a more comprehensive evaluation.
Decision: Accept-Main