A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction
Authors: Ruihao Shui
Yixin Cao
Xiang Wang
Tat-Seng Chua
Venue: EMNLP 2023 Findings (2023)
Keywords: Language Model
Law
Legal Judgment Prediction
Large Language Model
Language Model Evaluation

Abstract: Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain. However, recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. To systematically investigate their competency in the law, we design practical baseline solutions based on LLMs and test on the task of legal judgment prediction. In our solutions, LLMs can work alone to answer open questions or coordinate with an information retrieval (IR) system to learn from similar cases or solve simplified multi-choice questions.
We show that similar cases and multi-choice options, namely label candidates, included in prompts can help LLMs recall domain knowledge that is critical for expertise legal reasoning.
We additionally present an intriguing paradox wherein an IR system surpasses the performance of LLM+IR due to limited gains acquired by weaker LLMs from powerful IR systems. In such case, the role of LLMs becomes redundant. Our evaluation pipeline can be easily extended into other tasks to facilitate evaluations in other domains.
Code is available at https://github.com/srhthu/LM-CompEval-Legal

A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction
Ruihao Shui
Yixin Cao
National University of Singapore
Singapore Management University
ruihaoshui@u. nus. edu
yxcaoθsmu. edu. sg
Tat-Seng Chua University of Science and Technology of China National University of Singapore dcsctsênus edu. sg
Wang Xiang"
xiangwang (atpnnggg222@mmmallmm comm
Abstract
Promot for
Legal Judgment Prediction
Public Cases
simllr colse
Deternine the charge
Large language models (LLMMs) have demon- strated great potential for domain-sppp speciic ap- plications, such the law domain. However, recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. To systematically investigate their competeney in the law, we design practical baseline solutions based on LLMs and test on the task of legal judgment prediction. In our solutions, LLMs can work alone to answer open questions or coordinate with an information retrieval (IR) system to learn from similar cases or solve simplified multi-choice questions. We show that similar cases and multi-choice options, namely lahel candidates, included in prompts can help L.Mss recall domain knowledge that is critical for ex- pertise legal reasoning. We additionally present an intriguing paradox wherein an IR system surpasses the performance of LLM+IR due to limited gains acquired by weaker LLMs from powerful IR systems. In such cases, the role of LLMs becomes redundant. Our evaluation pipeline can be easily extended into other tasks facilitate evaluations in other domains. Code is available at httpss https: //i thub. com/ /or//rhtu// -o---EEvvallle
defendant based on case facts.
snatched the vittm' shoulder
Charge: Foraible seizure
IR (yStytem
sim case
(label cod
aboth
Open QN, zs
Open QN, FS
Multicchoce,, ZS
Multi-chaiae, FS
+instuction>
<latmaation
A:
A
Scenario
Charge
Figure 1: The task of Legal Judgment Prediction and the evaluation settings. Different colors refer to different charges. For similar cases, "T" refers to true similar cases with the same charges the query cases, while "F" refers to false similar cases. For task settings, "ZS" the abbreviation for zero-shot and "FS" for few-shot.
This raises interesting question: How exactly LLMs perform in various real-world legal tasks?
In this paper, we design practical bascline solu- tions based on LLMs and systematically investigate their competency in the law, to shed light on other domains as well. We attribute the main issues of the previous benchmark as follows. First, UBE is too general and not subject any legal jurisdiction (Martínez, 2023). Second, UBE contains multi- choice questions and open-ended questions that require human experts to evaluate. To avoid human evaluation, some datasets (Hendrycks et al., 2020) replace open-ended questions with multi-choice questions. However, in real-world applications, there are not only multi-choice but also open ques- tions. Using multi-choice questions only may not be comprehensive enough. Third, specifically in but not limited to common law (Shulayeva et al., 2017: Xiao et al., 2019), similar cases are always introduced as evidence to support expertise legal reasoning (Zhong et al., 2020b), which are not fully
1 Introduction
Large language models have achieved great suc- cess various Natural Language Processing (NLP)
tasks (Brown et al., 2020; Touvron et al., 2023),
while there are still some disputes over the potential for domain-speeific applications (Martínez, 2023). Focusing on the law domain, the leading LLM, GPT-4 (OpenAI, 2023), was claimed to pass the Uniform Bar Exam (UBE) with 90th percentile score. Although inspiring, however, this result was pointed out be overestimated (Martínez, 2023).
"Xiang Wang also affiliated with Institute of Artificial Intelligence, Institute of Dataspace, Hefei Comprehensive National Science Center.

---

studied in previous benchmark (Hendrycks al., 2020).
LLMs recall domain knowledge that is critical for expertise legal reasoning.
For the first issue, we choose legal judgment pre-
Label candidates result in more consistent out- puts, indicating LLMs gain greater confidence in their domain knowledge (Jiang et al., 2021).
diction (LJP) (Xiao et al., 2018; Chalkidis et al.. 2019; Zhong et al., 2020a) as the example task for investigation. It is real-world problem to de- termine the charges committed by the defendants under a juridical system, as shown in Figure 1. LJP is typically formulated a classification task to predict the most possible one from list of pre- defined charges. Then, for the second and third issues, we design four settings derived from two work scenarios of LLMs to cover open and multi- choice questions and the usage of similar cases. In the first scenario, LLMs work alone without ex- plicit knowledge in prompts, assuming all domain knowledge is implicitly stored in parameters. In the second scenario, LLMs coordinate with an information retrieval (IR) system that enriches prompts with similar demonstrations and label can- didates to benefit expertise reasoning. Specifically, demonstrations consist of pairs of similar cases and their charges, which are retrieved by the IR system based on similarity of case facts. Labels of the retrieved cases form label candidates, shown as circles of different colors in Figure 1, to hint LLM with label information and narrow down label space (Ma et al., 2023).
3. Irrelevant demonstrations formed by fixed cases hardly improve performance. This ex- cludes their effect task illustration.
4. Paradox: An IR system outperform LLM+IR since weaker LLMs acquire limited gains from informative documents retrieved by a powerful IR system. Thus, it is critical to adapte LLMs to generate with retrieved docu- ments.
5. More similar cases introduce more knowledge and noise simultaneously, whose final out- come depends LLMs.
The main contributions are summarized in three aspects:
We investigate the law competency of LLMs on the task of legal judgment prediction.
We propose practical baseline solutions for LLMs that tackle two scenarios: working alone or in coordination with an IR system.
We evaluate five LLMs and conduct compre- hensive analysis to demystify their character- istics of expertise reasoning.
The four evaluation settings in Figure 1 can be categorized based on the presence of two elements in prompts: demonstrations (similar cases) and la- bel candidates. Demonstrations convert the setting from zero-shot to few-shot prompting, while label candidates simplify the task from open questions to multi-choice questions'. The first scenario cor- responds to the first setting, where neither element is present, while the second scenario encompasses the remaining three settings. We evaluate five up- to-date LLMs of the close-source GPT-3 (Brown et al., 2020) family, ChatGPT and GPT-4 (OpenAl, 2023), and open-source LLMs including Vicuna (Chiang et al., 2023), ChaIGLM (Du al., 2022) and BLOOMZ (Muennighoff et al., 2022). The evaluation is conducted on Chinese LJP dataset, namely CAIL (Xiao et al., 2018), which contains cases of 112 criminal law charges?.
Baseline Method
The goal of legal judgment prediction is to deter- mine the committed charges given case facts. To harness LLMs for LJP, we adopt in-context learn- ing (Brown al., 2020) and use LLMs to generate the charges conditioned on prompts (Section 2. 1). To enhance LLMs, we incorporate label candidates and demonstrations consisting of similar cases into prompts, which are acquired by an IR system (Sec- tion 2.2). This derives four settings of baseline so- lutions, namely zero-shot open questions, few-shot open questions, zero-shot multi-choice questions, and few-shot multi-choice questions. The multi- choice settings employ label candidates while few- shot settings include demonstrations, shown in Figure 1. Finally, we introduce how simulate IR systems with different capabilities to understand their effects (Section 2.3).
We highlight our key findings as follows: 1. Similar cases and label candidates can help
'It is not strict multi-choice questions. LLMs can generute correct answers even though ground-truth labels are absent candidates.
2.1 LLM Prompting
Prompt Design. A prompt begins with an in- struction to illustrate the task followed by label
"After filtering less frequent (article, charge) pairs

---

candidates and task demonstrations in the form of input-output pairs. The templates of prompts are displayed in Appendix A.1.
Realistic Simulation. We prioritize the return- ing of true similar cases for easy query cases, rather than the returning a random manner. The query difficulty is measured by the Precision@ 10 of the B225 retriever described in Section 2.2. The mo- tivation is that queries with shadow linguistic fea- tures are more possible to get relevant retrieval results than complex or obscure queries. For a specific value (e.g., a%) of Precision @ be sim- ulated, the top a% of easy test cases are assured have a true similar case, while the rest are assigned false similar cases.
Parsing. We adopt one automatic parsing func- tion for all LLMs to map LLM outputs to pre- defined charge labels. No ad hoc heuristics are employed for fair comparison. Specifically, the BM25 algorithm² measure text similar- ity between outputs and pre-defined charges and predict the most similar charges. BM25 is robust and yields comparable performances to neural sim- ilarity methods like text2vec* in our pilot experi- ments.
Inference. Sampling is enabled during genera- tion for consistent results, inspired by Wang et al. (2022). Five outputs are sampled for each prompt with the temperature of 0.8. Their similarity scores of pre-defined labels are averaged.
Experimental Setup
3.1 Models
Below is a concise introduction to the five LLMs to be evaluated.
GPT-4 (OpenAI, 2023) and ChatGPT are available from OpenAI API and the versions of gpt-4-0314 and gpt-3. 5-turbo-0301 are used. For technological details, ChatGPT is claimed to be a sibling model to InstructGPT (Ouyang et al., 2022) that is trained to follow instructions and align to human preferences with the RLHF algorithm (Christiano et al., 2017).
2.2 IR System for Knowledge Incorporation
IR systems are utilized to retrieve similar cases, commonly referenced by lawyers and judges, to inform their judgments. In addition to providing demonstrations, these similar cases can also aid in generating potential labels by incorporating the la- bels from the top similar cases. By employing these smaller sets of predefined charges, namely label candidates, complex open questions be simpli- fied into multiple-choie questions. This upproach is effective enhancing LM prompting (Ma et al., 2023), as including hundreds of charges directly in prompts is impractical.
Vicuna- (Chiang et al., 2023) LLaMA model (Touvron et al., 2023) fine-tuned on 70K public user-shared conversations with ChatGPT. It can be viewed to learn distilled knowledge (Hinton et al., 2015) of ChatGPT.
ChatGLM-6B' is a dialog language model based on the GLM (Du et al., 2022) architecture and supports English and Chinese.
Implementation of IR System. We use the BM25 algorithm to measure the semantic similar- ity between cases. Similar cases are retrieved from the training dataset. To guarantee that the demon- strations exemplify one of the multi-choice options, we exclude demonstrations with labels that are not among the candidate o
BLOOMZ (Muennighoff et al., 2022) is an in- struction fine-tuned BLOOM (Scao et al., 2022), multilingual language model. We use the bloomz-7b1-mt version that is tuned for multilin- gual prompts. Except for BLOOMZ, Vieuna and ChatGLM are mainly fine-tuned on conversational data.
2.3 Simulation of IR Systems
To investigate the effects of IR capabilities, we simulate a series of IR systems of different capa- bilities as measured by Precision@ 1 Then the top retrieved cases are used as demonstrations. We consider cases with identical charges to the query cases true similar cases and vice versa.
3.2 Dataset and Pre-processing
The Chinese LJP dataset, CAIL (Xiao et al., 2018). is used in our experiments. Each sample consists of the case facts and the committed charge as the la- bel. As the original dataset is very large (-100K for training and ~20K for test), we randomly sample balanced small test set from the original test set. Five cases sampled for each charge, accounting
"hups://ypiiorg///// org/project/rank bbm25/ 'hups://github. com//rownpkuuexxxevee
'This condition not violated for the top four similar cases
without filtering-
'The accuracy of the top one retrieved case.

---

Tokenizer Median
<=500
<=1000
Performance on Four Settings
ChatGPT 396.5
68.75
92.32
GPT4 CharGPT
BLOOMZ CharGLM
Vicuna
Vicuna 496.0
50.89
86.96
ChatGLM
206.5
91.07
98.57
0.7
BLOOMZ 210.5
90.54
98.93
0.6
Accuracy
Table I: Statistics of the number tokens across tok- enizers. The last two columns present the ratios of test samples with token counts below the specified values.
0.3
Model Average
0.2
Zero-shot
Label
Sim Case
+ Label
for 560 test cases in total for 112 charges. Simi- larly, we also sample the training and validation sets with 10 cases per charge. The training set used to retrieve similar cases (Section 2.3), while the validation set is used to determine the optimal k of the kNN algorithm.
Open QN
Sim Case
Figure The macro comparison between the four set- tings. "+Label" refers to zero-shot multi-choice ques- tions; "+Sim Case" refers to few-shot open questions and "+Label +Sim Case" refers to few-shot multi-choice questions. More than one points of model in the last two settings refer to runs with different number of demonstrations.
Truncation. Since some cases have very long descriptions, we truncate the case facts demon- strations to 500 tokens and those of test samples to 1000 tokens. is worth noting that the text is tok- enized by the tokenizer of each model before trun- cation for fair comparison. Recently, Petrov et al. (2023) address the issue that tokenizer lead different performances of different languages. This suggests that the performance on a particular language can also influenced by tokenizers from various models with varying language encoding efficiency.
Zero-shot
Few-shot
Gpen question
Open question
Multi choice
Multi choice
0.7
Accuracy
0.5
0.3
Table shows the statistics of the number of to- kens processed by different tokenizers The most efficient tokenizers for Chinese are those of Chat- GLM and BLOOMZ, indicated by the medians of token numbers. In contrast, the tokenizer of ChatGPT produces 2x tokens and that of Vicuna produces 2.5x tokens. The truncation length is proper to accommodate most samples.
0.
GPT-4
ChaiGPT
BLOOMZ
Chetauu M
Viaun
GPT4
ChaIGPT
BLOOME
ChaIGLM
Viaua
Figure 3: Compare the models under each setting. Few- shot performances are averaged among I-shot to 4-shot.
4.1 Overall Results
The macro comparison between the four settings is shown in Figure 2, where each point represents the performance of one specific run of one model. Significance of label candidates and similar cases. In comparison to the zero-shot open ques- tion setting where LLMs work alone, the inclusion of label candidates, similar cases, or both demon- strates noteworthy enhancements. This highlights the effectiveness of our baseline solutions that lever- age IR systems to expand the capabilities of LLMs in legal domains. These findings align with previ- ous research that has also recognized the signifi- cance of the two components (Ma et al., 2023; Liu et al., 2021).
4 LLM vs. LLM with IR System
We initially present the overall results, highlight- ing the importance of label candidates and similar cases, and conduct comparative analysis of the models. Subsequently, we investigate the relation- ship between label candidates and self-consistency unveil their actual effects on expertise reason- ing. Additionally, we perform an ablation study by replacing similar cases with fixed cases as demon- strations to further understand their impact.
*GPT-4 and ChatGPT have the same results. Following OpenAl's guidance, we use the python package tiktoken for tokenization
The effects of label candidates and similar cases differ slightly in terms of performance mean and

---

variance. Label candidates contribute higher mean performance, while similar cases introduce greater variance. Examining the model perfor- mances in the third setting (+Sim Case) displayed in Figure 2, GPT-4 and ChatGPT exhibit more significant improvements from similar cases com- pared to their smaller counterparts. They also gain more benefit from similar cases than from label candidates. This observation the varying difficulty levels of knowledge utiliza- tion. While the knowledge within label candidates is readily accessible and straightforward, leverag- ing similar cases requires stronger language under- standing and few-shot learning abilities.
Zero-shot
One-shol
questoo
07-
0.6
Nuti chokes
Accurasy
0.5
GPT4
Chetier
0.4
0.3
be attributed to
0.3
Self-consistenov
Figure 4: Changes of performance and self-consistency after adding label candidates. The change of each model is illustrated by an arrow pointing from the open ques- tion setting to the multi-choice setting.
Furthermore, the coexistence of label candidates and similar cases further enhances the performance of GPT-4 and ChatGPT, but it diminishes the per- formance of Vicuna, ChatGLM, and BLOOMZ. This suggests that smaller LLMs may encounter challenges effectively managing knowledge in multiple forms simultaneously, leading to confu- sion.
4.2 Label Candidates Enhance Self-consistency and Confidence
To further understand the effect of label candidates, propose metric to measure the self-consistency of LLMs that is calculated as the number of the majority prediction'. Consistent outputs indicate a high level of confidence in LLMs, which is often associated with a better grasp of knowledge (Jiang et al., 2021, 2023).
Model comparison. The performances of the models under zero-shot and few-shot prompting is shown in Figure 3, where few-shot performances are averaged among 1-shot to 4-shot.
The changes in performance and self- consistency after introducing label candidates are shown in Figure 4 the arrows. We observe that the incorporation of label candidates leads more consistent outputs (8 of 10 cases) and higher confidence in LLMs except zero-shot GPT-4 with a slight decrease and few-shot BLOOMZ. In the zero-shot setting, label candidates significantly boost LLM performances. We postulate that label candidates help by eliciting pre-stored domain knowledge with concise charge names. Besides, the self-consistency also correlates with model performances of 10 cases). Such correlation is also observed in other tasks like question answering (Jiang et al., 2021). It is worth noting that label candidates decrease both self consistency and performance of few-shot prompted BLOOMZ, which also aligns with the correlation.
The zero-shot setting emphasizes the ability to understand instructions. When only instructions are available, BLOOMZ performs better than Chat- GPT, indicating a superior multilingual instruc- tion following ability. This result reasonable as BLOOMZ is the only smaller LLM that is fine-tuned on multilingual instructions. Once pro- vided with explicit domain knowledge, ChatGPT outperforms all smaller LLMs. The case is the same for BLOOMZ and ChatGLM, where Chat- GLM overtakes BLOOMZ. with knowledge of la- bel candidates. BLOOMZ performs worst when prompted with two forms of knowledge, indicat- ing that BLOOMZ is not very robust to prompts. Among the three smaller LLMs, ChatGLM is the most robust to various forms of knowledge.
4.3 Domain Knowledge Is More Critical Than Task Illustration
The significant effects of label candidates and similar cases can be explained as they activate LLM's memory of relevant domain knowledge. This view can be supported by two pieces of ev- idence about the relationship between label can- didates and self-consistency (Section 4.2) and the negligible effect of irrelevant cases as fixed demon- strations (Section& 4.3).
There is possible argument that similar demon- strations can help LLMs understand instructions and tasks. To disentangle their effects task il- lustration and provision of domain knowledge, we
FFor example, if the five sampled outputs are mapped to labels of (a.a.a.b.c), the consistency score is 3.

---

Fixed Demonstratios
Performance over simulated IR systems
0.7
Similar
100
0.6
Accuracy
Accurscy
0.5
Mean
0.8
ChatGPT
GPT-
ChalGPT
real
BLOOMZ
ChatGLM
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.2
Vicuna
IR Precision@11
Fixed n=2
Fixed n=1
Similar
Simlar n=2
Open
n=1
Figure 6: The performance of ChatGPT coordinated with series of simulated IR systems with varying ca- pabilities as measured by Precision@) @ 1. The vertical blue line represents the threshold of IR capability at which IR systems overtake ChatGPT. The performance of ChatGPT in the real setting (1 (1-shot open questions) is indicated by the red plus sign.
Figure 5: The effects of fixed (irrelevant) and similar cases as demonstrations. Divided by the baseline setting zero-shot open questions, the left part refers fixed demonstrations with increasing numbers of demonstra- tions, while the right part refers to similar demonstra- tions. The shadow area represents the range standard deviation.
ity demonstrations. However, we raise an intuitive question: Do LLMs gain substantial improvement from IR systems compared to the kNN baseline that harnesses IR systems for classification tasks? The question is inspired by our observation that the BM25 retriever achieves 48.03% of Precision @ and 57.68% prediction accuracy by majority vote of top k = 17 retrieved similar cases.
experiment with irrelevant demonstrations fixed for all test samples. We manually select common cases with frequent charges in the original dataset as the fixed demonstrations. The 1-shot perfor- mance was averaged on the two demonstrations.
We compare the effects of fixed and similar demonstrations with the baseline setting of zero- shot open questions in Figure 5. The change of per- formance from center to left demonstrates that fixed demonstrations hardly benefit LLMs and some- times harm the performance (e.g., ChaGLLM). This indicates that LLMS can basically understand in- structions and do not need general demonstrations for task clarification, implying that the main chal- lenge of expertise reasoning is to recall domain knowledge instead of understanding specific task.
This observation suggests paradoxical sce- nario wherein an IR system outperforms the com- bination of LLM and IR, with the LLM taking the leading role and the IR serving as supporting role. In such a scenario, the LLM becomes redun- dant due its failure to fully utilize the informative retrieved documents.
To investigate the paradox, instead of experi- menting with diffcrent IR systems, we manipulate the BM25 retriever to simulate a series of IR sys- tems with different capabilities measured by Preci- sion @ 1 described by Section 2.3. We take case study of ChatGPT, whose 1-shot performance un- der different IR systems (denoted as Precision @ I) is shown in Figure 6.
We inspect the notable performance drop of ChatGLM resulting from fixed demonstrations. We find that ChatGLM tends to analyze the cases of both demonstrations and test samples and then an- swer with both of their charges. Its wordy style seems result from the fine-tuning dialog corpus where an assistant LLM is supposed to provide rich information. In contrast, similar cases seem en- courage more concise outputs following the format demonstrations.
Results Although the performance of ChatGPT enhanced by IR systems improves with IR capabil- ity, it will eventually underperform the IR system once the IR capability surpasses certain threshold. In the ideal situation where true similar cases are always retrieved, ChatGPT is unable to attain 100% accuracy and lags significantly behind the optimal IR system. According to Appendix A.4, all smaller LLMs are not comparable to the BM25 retriever. Discussion The findings demonstrate that LLMs face challenges in effectively leveraging informa- is identical to the precision of &NN with k: = =1
5 Paradox of Information Retrieval System
The significance of similar demonstrations illus- trated in Section 4.3 has motivated research fo- cusing on prompting-orient IR systems (Rubin et al., 2021; Sun et al., 2023) to retrieve high qual-

---

Open Question
Mult Choice
the inclusion of new demonstration with (T) or false (F) similar cases, while the columns indi- cate the combinations of existing demonstrations. Take the second heat map as an example. The cell in the column of (F, T) and the row of (T) displays the performance variation between 2-shot of (F, T) demonstrations and 3-shot of (F, T, T) demonstra- tions. Purple represents performance improvement, while green represents performance decline.
07-
0.-
0.-
0.4
0.3
GPT-4
CrwGGU
Charcue Viaua
0.3
ChotGPT
Vitura
02
ChatGPT BLOOMZ
BLOOMZ
02
3
0.1
2
Demo
For ChatGPT and BLOOMZ, the second rows of their three heat maps are mainly in purple, indi- cating significant enhancements resulting from the inclusion of true similar cases. However, the first lines of BLOOMZ. display deeper green color than those of ChatGPT, suggesting that BLOOMZ experiences greater degree of performance declines caused by the inclusion of false similar cases. These findings indicate different sensitivity to false similar demonstrations. Powerful language mod- els like GPT-4 and ChatGPT exhibit robustmess to noise in false similar cases, allowing them to main focused on relevant information in true simi- lar cases, In contrast, weaker LLMs are susceptible to the influence of such noise. Overall, ChatGPT performs better when provided with more similar demonstrations, whereas BLOOMZ demonstrates the opposite, as shown in Figure
Figure 7: Performance vs. the number of similar demon- strations of the five LLMs.
tive retrieved documents. This underscores the need for significant research efforts to enhance the synergy between auto-regressive language models and retrieval by conditioning model outputs more retrieved documents. Previous work has ex- plored the augmentation of LLMs with retrieval both the pre-training and fine-tuning stages (Borgeaud et al., 2022; Wang et al., 2023). More- over, the marginal and inadequate improvement with retrieval indicates the limited legal reasoning ability of existing general LLMs. There is need for future efforts to enhance domain-specific rea- soning abilities of pre-trained foundation models.
6 Ablation Study
The conclusion is that increased numbers of demonstrations have both positive and negative im- plications for expertise reasoning. However, LLMs could potentially gain from additional demonstra- tions in tasks that requires clear task illustration.
6.1 More Demonstrations Are Not Always Better
The impact of the number of similar demonstra- tions (n) is depicted in Figure 7. It is evident that GPT-4 and ChatGPT demonstrate proficiency in handling larger numbers of demonstrations, leading to enhanced performance, whereas Vicuna, Chat- GLM and BLOOZ experience varying degrees of performance degradation with increasing numbers. Notably, ChatGLM displays the least sensitivity to Furthermore, even ChatGPT's performance de- clines when increased from three to four. The performance improvement resulting from larger values n can be attributed to the increased recall of true similar cases. Conversely, the decline in per- formance can be attributed to the noise introduced by more false similar cases.
6.2
The Impact of Absent Ground Truth Labels
We manually incorporate ground-truth labels into label candidates in cases where they are absent, which may occur due to the limited recall capability of the IR system described in Section 2.2. The test samples are categorized into two groups, namely "Easy" and "Hard", based on the retrieval of their ground truth labels by the IR system. The original performance of the two groups and the performance of the "Hard" group with modified prompts to in- clude ground truth labels, namely "Hard+GT": are displayed in Figure 9.
Performance variations. The change of perfor- mance after including an additional demonstration are visualized using heat maps in Figure 8. For each model, the three heat maps stand for the varia- tions from shohot to sshot, which are denoted below. For each heat map, the two rows indicate
The performance gaps between the "Easy" and "Hard+GT" groups suggest that challenging sam- ples for IR systems are also difficult for LLMss However, this gap is insignificant for the power- ful GPT-4 who perceives them as equal challeng-

---

FF T.F
FFF
F.F.
RTF
F.T
TFF
T,F
T.F
T.T.T
CharGLM
40
BLOOMZ
20
2
2 - 3
-
Figure 8: Heat maps of performance variations resulting from the inclusion of addition demonstration. "T" corresponds to demonstrations with true similar cases, while "IF" represents those with false similar cases. Each row represents the included new demonstration, while each column indicate the status of existing demonstrations.
Effect of missing ground-truth labels
Discussion
Easy
GT Hard
We compare the LLMs with supervised baselines. We fine-tune BERT (Devlin al., 2018) on the same training set and achieve a comparable accu- racy of 68% to ChatGPT but lower than GPT-4. Since LLMs are not fine-tuned on the specific LJP task, this result highlights the remarkable superior- ity of LLMs in acquiring significant knowledge and leveraging transfer learning Raffel et al. (2020).
+0.42
0.6
+0.30
+0.28
0.2
+0.11
+0.03
GPT4
ChailPT
BLOOMZ
ChatGLM
Vicuna
Figure 9: The performance of "Easy" and "Hard" sam- ples under the setting of zero-shot multi-choice ques- tions. "Hard+GT" refers to improvement of including the absent ground truth labels in label candidates.
However, observe that BERT's performance improves 89% when trained with the original training set (-10K). We find that certain knowledge is present shadow features, which can be easily learned with supervision. These superficial features can result in biased supervised models. Fortunately, unsupervised pre-training objectives, make LLMs more robust and less vulnerable to this issue. This depicts promising future for NLP applications in various domains.
ing. The improvement of "Hard+GT" compared to "Hard" is notable in GPT-4, ChatGPT and Chat- GLM but inconspicuous in Vicuna with inferior competency in the law. Considering the relatively small size of the "Hard" group (79/560), the ab- sence ground truth labels does not have signifi- cant impact, especially for weaker LLMs.
Conclusion
To address the deficiency in evaluating the compe- tency of LLMs in the field of law, we focused on the task of legal judgment prediction and devised four settings to facilitate thorough evaluation that encompassed both open and multiple-choice ques- tions and incorporated similar cases to aid the decision-makee process.
6.3 Incorporation of Law Articles
We examine the effect of incorporating legal arti- cles that explicitly define the charges into prompts. For each charge retrieved by the IR system Chat- GPT required determine whether the defen- dant guilty for the particular charge by answering with a yes or no. We find that 94.46% the ground truth charges are accurately detected, while only 27.31% % of the detected charges are correct. The high recall and low precision indicate substantial difference between ChatGPT and legal experts in the ability distinguish charges and make precise judgments.
The evaluation results revealed different behav- iors among the prominent LLMs, namely GPT- 4 and ChatGPT, compared to their smaller coun- terparts. Both GPT-4 and ChatGPT exhibited markable proficiency in effectively leveraging do- main knowledge in various formats. Among the smaller LLMs, ChatGLM displayed greater robust- ness, while BLOOMZ showcased superior zero- shot ability.
Il we also include the ground truth charge

---

We presented an intriguing paradox wherein LLMs could become abundant in the presence of powerful IR system. When improving IR systems to benefit LLMs, it is crucial for researchers to ac- knowledge this paradoxical scenario and prevent great disparity between LLMs and IR systems.
References
Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, lean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240. PMIR.
Limitations
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901.
One limitation of this paper is the use of the close-source GPT-4 and ChatGPT whose avail- ability depends on the commercial company OpenAI. According to OpenAl, the ChatGPT and GPT-4 versions used in this paper, namely gpt-3. 3. 5-turbo-0330 and gpt-4-0314, 14, will be deprecated and not available after September I 13th, 2023.
Ilias Chalkidis, Ion Androutsopoulos. and Nikolaos Ale- tras. 2019. Neural legal judgment prediction in en- glish. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317-4323.
for
Another limitation pertains to the selection of LLMs. Due to the rapid emergence of new L.Mss, we are not able to include all of them with the con- straint of limited time. Instead of more models, we focus more on designing comprehensive evaluation settings and conducting insightful analyses to shed light on other domains.
Wei-Linn Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Ad- vances in neural information processing systems, 30.
Ethics Statement
The task of legal judgment prediction is used to evaluate LLM's competency in the law. The pri- mary objective of this task is to assist judges and lawyers in comprehending lengthy legal documents by offering them a supplementary tool. It is im- portant to note that this task does not seek to re- place the roles of judges and lawyers, nor does it aim to determine the guilt or charges of defen- dants through machine learning algorithms. Addi- tionally, there is research focused on interpreting LJP models, aiming to enhance the transparency of black-box models for improved utilization by legal practitioners. The paper utilizes a public and anonymized dataset to exclude the potential issue of personal information lcakage.
Jacob Devlin, Ming- Ming-Wei Wei Chang. Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint arXiv: T00::8810.000 810004805.
Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding. Jiezhong Qiu, Zhilin Yang. and Jie Tang. 2022. Glm: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 320-335.
Dan Hendrycks, Collin Burs,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Mcasuring massive multitask language under- standing. arXiv preprint N0X:22009003300
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network. arXiv preprint 1503005003
Acknowledgements
Zhengbao Jiang. Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Associa- tion for Computational Linguistics, 19622997.
We thank all reviewers for their constructive com- ments. This research is supported by NExT Re- search Center, the National Natural Science Foun- dation of China (9227010114) and the University Synergy Innovation Program of Anhui Province (GX)
Zhengbao Jiang, Frank F Xu. Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Ya. Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Ac- tive retrieval augmented generation. arXiv preprint arXiv:2305.06688.

---

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.00 068044
Faisal Azhar, et 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13937.17977.
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, et al. 2023. Shall pretrain autoregressive language models with retrieval? a comprehensive study. arXiv preprint arXiv:2304.06762.
Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023. Large language model is not good few-shot information extractor, but good reranker for hard samples! arXiv preprint arXiv:2303.08559
Eric Martínez. 2023. performance. Available at SSRN 4441311.
Re-evaluating gpt-4's bar exam
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-cousistency improves chain of thought reasoning in language models. arXiv preprint 2rXhv:2203.1177.
Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Seao, Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint MrXiv:221.001780
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng. Xianpei Han, Zhen Hu, Heng Wang. et al. 2018. Cail201 & A large-scale legal dataset for judgment prediction. arXiv preprint 1rXiiv1807.000 000.022788
OpenAl. 2023, Gpt-4 technical report. arXiv preprint arXiv:2303.08774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances Neural Information Processing Systems, 35:27730-27741
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao
Tu, Zhiyuan Liu, Maosong Sun, Tianyang Zhang, Xianpei Han, Zhen Hu, Heng Wang, et al. 2019. Cail2019-semn dataset similar case matching legal domain. arXiv preprint arXiv:1911.08962
Haoxi Zhong, Yuzhong Wang. Cunchao Tu, Tianyang Zhang. Zhiyuan Liu, and Maosong Sun. 2020a. Itera- tively questioning and answering for interpretable le- gal judgment prediction. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 1250-1257.
Aleksandar Petrov, Emanuele La Malfa, Philip HS Torr, and Adel Bibi. 2023. Language model tokeniz- ers introduce unfairness between languages. arXiv preprint arXiv:2305.1m 200223555552555 15425.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena. Yanqi Zhou, Wei Li, Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, (1(1):5485-5555
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang. Zhiyuan Liu, and Maosong Sun. 2020b. How does nlp benefit legal system: A summary of legal artificial intelligence. arXiv preprint 2rXiv:2004.12158
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arxiv:21/2.08633.
Teven Le Scao, Angela Fan, Christopher Akiki, BI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05/00
Olga Shulayeva, Advaith Siddharthan, and Adam Wyner. 2017. Recognizing cited facts and principles in legal judgements. Artificial Intelligence and Law, 1571):107-126
Xiaofei Sun, Linfeng Dong, Xiaoya Li, Zhen Wan, Shuhe Wang, Tianwei Zhang, Jiwei Li, Fei Cheng, Lingjuan Lyu, Fei Wu, and Guoyin Wang. 2023. Pushing the limits of chatgpt nlp tasks.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro,

---

A Appendix
The SFTT models are evaluated on the same evalua- tion dataset described in Section 3.2. The smaller training set aims to compare the few-shot perfor- mance of SFT baselines and LLMs in low data scenario.
A.1 Prompt Templates
The prompt template shown in Figure 10. The translation of the original Chinese prompt is dis- played using orange text. The setting of zero-shot open questions use a longer instruction that ap- pends "Output the charge name directly" to the instruction in Figure A.1.
The results of SFT models are shown in Figure 3. Considering the highest accuracy of GPT-4 being 74.46% (multi-choice, 4shot), GPT-4 can outper- form supervised baselines in low data scenario. If there abundant training data, supervised base- lines are still better than GPT-4 by 15%.
Instruction
HU (1 PU 11
Baped on the Chinese cominal law, determing the charges comonited the difenddant.
SE A. 8-
<label 1>
Model BERT 89.64
CAIL
CAIL_ few
Label
<label 2>
candidates
68.04
<label
XI.MRRoERTa
88.75
66.43
DeBERTa 88.57
30.89
###4 St: <demo facts>
Case facts: facts>
31 6): <demo charge>
Charge: salemo charges
Query
#//f 3t: <query facis>
Case facts: squary facts>
Table 3:
Prediction accuracy of SFT models fine-tuned on two training datasets of different sizes.
18 8::
Charge:
Figure 10: The prompt template in Chinese and English.
A.4 Detailed Results
The specific values of performances displayed in Figure 2 are presented in Table 4. Besides, we also provide the performance of the F1 score in Table 5.
A.2
Robust to Fixed Demonstrations
Model
1shot
2shot
GPT-4
49.59 / 48.84
50.69
ChatGPT
47.01 46.57
47.55
Vicuna- 13B
22.74 / 29.38
28.37
ChatGLM-6B
22.39 / 25.14
21.36
BLOOMZ-7B
36.65 / 43.94
42.24
Table 2: The classification accuracy scores with prompts consisting fixed cases.
We examine the effects of the two fixed cases mentioned Section 4.3 in Table We find that GPT-4 and ChatGPT are robust to the selection the fixed demonstration in I-shot setting, while Vicuna, ChatGLM and BLOOMZ are less robust.
A.3 Comparison with Supervised Baselines
To understand the performance of supervised fine- tuning (SFT) baselines on LJP,, we experiment on three models: 12 XLM- RoMMRRa 13 and De- 14 These models are fine-tuned on two datasets of different sizes: the original CAIL dataset (~100k samples) and the sampled training set (1120 samples) that is used retrieval corpus described in Section 3.2, denoted CAIL_ few.
">bert-base-chinsse
18 "xx_m-roberta-base
14 "mmicrrsoftt/ddderraaa----

---

Open Questions
Multiple-choice Questions
Model
Oshot
Ishot
2shot
3shot
4shot
Oshot
Ishot
2shot
3shot
4shot
GPT-4
55.18
64.82
69.11
69.82
71.96
63.93
71.25
72.50
73.75
74.46
ChatGPT
46.61
60.00
62.86
64.82
66.96
61.61
64.46
66.96
70.36
67.14
Vicuna- 13B
28.21
50.36
49.64
51.79
35.89
47.86
44.82
43.39
35.71
19.46
ChatGLM-6B
41.43
51.79
50.00
50.36
50.54
55.71
50.54
49.64
49,46
47.32
BLOOMZ-7B
49.82 19.82
54.82
52.68
52.50
51.25
53.39
31.96
31.07
27.32
26.61
Table 4: The classification accuracy scores of all models under the four settings.
Open Questions
Multiple-choice Questions
Model
Oshot
Ishot
2shot
3shot
4shot
Oshot
Ishot
2shot
3shot
4shot
GPT-4
50.52
62.72
67.54
68.61
71.02
62.31
70.42
71.81
73.24
74.00
ChatGPT
43.14
58.42
61.86
64.40
66.16
60.67
63.51
66.85
69.59
66.62
Vicuna- 13B
25.50
48.85
47.64
49.49
39.82
44.70
41.73
41.48
35.03
21.61
ChatGLM-6B
41.89
50.30
47.76
48.59
48.67
53.74
49.26
47.56
47.61
45.32
BLOOMZ-7B
46.90
53.28
51.06
50.90
49.26
50.68
29.25
27.92
25.27
23.37
Table 5: The classification Fl scores of all models under the four settings.


Reviews:
[{"review_id": "coaLlvxJnW", "reviewer": "Reviewer_gyVE", "summary": "", "questions": "Question A: Why LJP baselines e.g. Neural Legal Judgment Prediction in English, Chalkidis et al. ACL 2019; Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction, Zhong et al. AAAI 2020; ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation, Malik et al. ACL 2021., are not considered for comparison?\nQuestion B: How are the case facts truncated to 500 tokens? Are they done randomly? Or are the first/last 500 tokens considered?\nQuestion C: Are the results fair w.r.t the under-represented groups? E.g. are females or blacks discriminated by the recommendations of LLMs?\nQuestion D: Are the results acceptable to legal experts? Have legal experts qualitatively/quantitatively analyzed the results? In the midst of \ninstances where judges are using ChatGPTs for ruling (see https://www.theguardian.com/technology/2023/feb/03/colombia-judge-chatgpt-ruling), apart from accurate prediction, it is absolutely essential to highlight the underlying risks of using such models.\nQuestion E: How about hallucinations by LLMs? Nothing is discussed about the hallucinations of the LLMs.\nQuestion F: What about the explainability of the results?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts on reviewing our paper. We are grateful for the detailed comments and clear suggestions. As our work is one of the pioneering efforts in LLM legal benchmarks and considering that LLMs are emerging and have been significantly reshaping the field, it is inevitable that our paper is not clear enough to present its motivation and significance.\n\nYour concerns may stem from the dissimilarity between our work and previous Legal NLP work, e.g., judgment prediction. For example, as mentioned in the Reasons to Reject, we do not compare LLMs to supervised baselines and the classification task is tackled with a retrieval system. This paper is not to propose new methods on the LJP tasks; instead, we use the LJP task to measure the legal ability of LLMs. Perhaps our title and the description in the 2nd paragraph that “we design practical baseline solutions based on LLMs” make you confused about the contribution of the paper.\n\nThe aim of our work is to propose a benchmark to evaluate the legal ability of LLMs and to give a comprehensive analysis to provide some insights and shield light on future work, especially the retrieval-based LLM applications. Our results, e.g., ablation study of the number of similar cases and the paradox that LLM+IR underperform IR, indicate the importance of the quality of retrieval results, and point out two research directions: \n- how to pretrain LLMs to work with retrieval noise (Wang et al. 2023)\n- how to improve the capability of retrieval systems to provide useful information for LLMs, e.g., searching from structured data like knowledge graph.\n\nAs LLM evaluation is subtle and vulnerable to prompting and inference implementations (https://huggingface.co/blog/evaluating-mmlu-leaderboard), we try to focus the paper on LLMs and we cannot include more aspects due to space limits. However, we do have some additional experiments and we will discuss them later.\n\n### Response to Rejecting Reason “Lack of Baselines”\nWe experiment on three baseline models: BERT, XLM-RoBERTa and DeBERTa. We train these models with different sizes of training data: the original CAIL dataset (~100k samples) and the sampled training set (1120 samples), denoted as CAIL_few, described in Section 3.2. The smaller training set is for fair comparison with LLMs, as LLMs are good at few-shot and zero-shot learning and our retrieval system only use the sampled smaller training set. The sampled training set estimates the ability of supervised baselines under low data.\n\nThe prediction accuracy is shown below:\n| model | CAIL | CAIL_few|\n| - | - | - |\n| BERT | 89.64 | 68.04 |\n| XLM-RoBERTa | 88.75 | 66.43 |\n| DeBERTa| 88.57 | 30.89 |\n\nConsidering the highest accuracy of GPT-4 being 74.46% (multi-choice, 4shot), GPT-4 can outperform supervised baselines in low data scenario. If there is abundant training data, supervised baselines are still better than GPT-4 at 15%.\n\n### Response to Rejecting Reason “Framing LJP as IR”\nEnhancing LLMs with retrieval systems is an emerging research direction and has been drawing increasing attention (ACL 2023 tutorial: Retrieval-based Language Models and Applications). IR can provide explicit knowledge and help mitigate the hallucination of LLMs (Jiang et al. 2023). Besides, similar demonstrations can also improve in-context learning results (Liu et al. 2022), which are obtained by IR systems. We adopt this common practice and incorporate the IR system to retrieve similar cases. Such similar cases, also known as precedents in legal practice, are valuable for legal judgments.\n\n### Response to Rejecting Reason about “Fairness, Expert Evaluation, Explainability, and Hallucination”\nThese points are critical topics of LLMs and we are also interested in them. However, as they are all hard problems and they are independent research directions, we cannot discuss them in our paper due to the limitation of space and less relevance of our work. \n\nWe also notice that there are some work discussing these topics, like Santosh et al. 2022, Wu et al. 2022. We are very pleased to explore these important aspects in our future work.\n\n### Response to Question A\nOur paper is more to evaluate LLM with LJP than to solve LJP with LLM, so we replace the comparison of baselines with more analysis to give some insights of LLMs. We are pleased to include these comparison in the Appendix.\n\n### Response to Question B\nWe truncate similar cases to preserve the first 500 tokens. We also notice that some other work chooses to truncate beginning tokens. Our truncation setting is based on the heuristics on the CAIL dataset that important case facts usually appear at the beginning followed by some details.\n\n### Response to Question C, D, E, F\nThese comments are discussed in the *Response to Rejecting Reason about “Fairness, Expert Evaluation, Explainability, and Hallucination”*. As they are not relevant to this work, we leave them to be explored in future work.\n\n\n### References\n- Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study. Wang et al. 2023.\n- Active Retrieval Augmented Generation. Jiang et al. 2023.\n- What Makes Good In-Context Examples for GPT-3?. Liu et al. 2022\n- Deconfounding Legal Judgment Prediction for European Court of Human Rights Cases Towards Better Alignment with Experts. Santosh et al. EMNLP 2022\n- Towards Interactivity and Interpretability: A Rationale-based Legal Judgment Prediction Framework. Wu et al. EMNLP 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper shows many state-of-the-art LLMs (GPT, Vicuna, ChatGLM, BLOOM) for the legal judgment prediction (LJP) task. \n\nThe detailed and comparative performance analysis of LLMs is rigorous.\n\nThe interplay of LLMs with IR models esp. identification and analysis of cases where IR outperforms LLMs is detailed.", "reasons_to_reject": "A baseline of any LJP task should be an existing state-of-the-art LJP method. It would have been interesting to see how LLMs (GPT, BLOOM etc.) presented in this paper live up to much stronger baselines for this task. It is also not clear why the task is framed as an IR task when LJP is usually a supervised problem e.g. as in Neural Legal Judgment Prediction in English, Chalkidis et al. ACL 2019; Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction, Zhong et al. AAAI 2020; ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation, Malik et al. ACL 2021. A comparison of LLMs in this paper with such strong baselines would have been worthy of investigation w.r.t answering questions like \"Are GPTs, BLOOM etc. capable of outperforming BERT-based baselines in LJP tasks?\"\n\n\n\nThe paper does highlight the usefulness of LLMs in a legal downstream task. However, it ignores the following key questions:\n* Are the results fair w.r.t the under-represented groups? E.g. are females or blacks discriminated by the recommendations of LLMs?\n* Are the results acceptable to legal experts? Have legal experts qualitatively/quantitatively analyzed the results? In the midst of instances where judges are using ChatGPTs for ruling (see https://www.theguardian.com/technology/2023/feb/03/colombia-judge-chatgpt-ruling), apart from accurate prediction, it is absolutely essential to highlight the underlying risks of using such models.\n* How about hallucinations by LLMs? Nothing is discussed about the hallucinations of the LLMs.\n* What about the explainability of the results?", "questions_to_authors": "Question A: Why LJP baselines e.g. Neural Legal Judgment Prediction in English, Chalkidis et al. ACL 2019; Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction, Zhong et al. AAAI 2020; ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation, Malik et al. ACL 2021., are not considered for comparison?\nQuestion B: How are the case facts truncated to 500 tokens? Are they done randomly? Or are the first/last 500 tokens considered?\nQuestion C: Are the results fair w.r.t the under-represented groups? E.g. are females or blacks discriminated by the recommendations of LLMs?\nQuestion D: Are the results acceptable to legal experts? Have legal experts qualitatively/quantitatively analyzed the results? In the midst of \ninstances where judges are using ChatGPTs for ruling (see https://www.theguardian.com/technology/2023/feb/03/colombia-judge-chatgpt-ruling), apart from accurate prediction, it is absolutely essential to highlight the underlying risks of using such models.\nQuestion E: How about hallucinations by LLMs? Nothing is discussed about the hallucinations of the LLMs.\nQuestion F: What about the explainability of the results?", "paper_topic_and_main_contributions": "The paper \"A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction\" addresses the Legal Judgment Prediction (LJP) problem with regard to LLMs coupled with an IR framework, on CAIL (Xiao et al., 2018) dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "@inproceedings{DBLP:conf/acl/ChalkidisAA19,\n  author       = {Ilias Chalkidis and\n                  Ion Androutsopoulos and\n                  Nikolaos Aletras},\n  editor       = {Anna Korhonen and\n                  David R. Traum and\n                  Llu{\\'{\\i}}s M{\\`{a}}rquez},\n  title        = {Neural Legal Judgment Prediction in English},\n  booktitle    = {Proceedings of the 57th Conference of the Association for Computational\n                  Linguistics, {ACL} 2019, Florence, Italy, July 28- August 2, 2019,\n                  Volume 1: Long Papers},\n  pages        = {4317--4323},\n  publisher    = {Association for Computational Linguistics},\n  year         = {2019},\n  url          = {https://doi.org/10.18653/v1/p19-1424},\n  doi          = {10.18653/v1/p19-1424},\n  timestamp    = {Fri, 06 Aug 2021 00:41:04 +0200},\n  biburl       = {https://dblp.org/rec/conf/acl/ChalkidisAA19.bib},\n  bibsource    = {dblp computer science bibliography, https://dblp.org}\n}\n\n@inproceedings{DBLP:conf/aaai/ZhongWTZ0S20,\n  author       = {Haoxi Zhong and\n                  Yuzhong Wang and\n                  Cunchao Tu and\n                  Tianyang Zhang and\n                  Zhiyuan Liu and\n                  Maosong Sun},\n  title        = {Iteratively Questioning and Answering for Interpretable Legal Judgment\n                  Prediction},\n  booktitle    = {The Thirty-Fourth {AAAI} Conference on Artificial Intelligence, {AAAI}\n                  2020, The Thirty-Second Innovative Applications of Artificial Intelligence\n                  Conference, {IAAI} 2020, The Tenth {AAAI} Symposium on Educational\n                  Advances in Artificial Intelligence, {EAAI} 2020, New York, NY, USA,\n                  February 7-12, 2020},\n  pages        = {1250--1257},\n  publisher    = {{AAAI} Press},\n  year         = {2020},\n  url          = {https://ojs.aaai.org/index.php/AAAI/article/view/5479},\n  timestamp    = {Mon, 07 Mar 2022 16:57:39 +0100},\n  biburl       = {https://dblp.org/rec/conf/aaai/ZhongWTZ0S20.bib},\n  bibsource    = {dblp computer science bibliography, https://dblp.org}\n}"}}, {"review_id": "XRma8NVUTx", "reviewer": "Reviewer_JuZH", "summary": "", "questions": "A:  Perhaps I am not familiar enough with this dataset.  I would appreciate more detail on the labels that are used.  In particular, is there a natural metric space where some labels are close.  Eg., Theft vs Grand Theft.  Murder vs homicide.\n\nB: Are the model errors (or corrects) correlated across models?  Is GPT-4 better in the sense that it gets the same things correct as other models and then additional corrects as well? \n\nC: I found Figure 8 hard to interpret. What should be my focus when I look at that picture?  Is it the relative color (performance) in the same position across the models?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for your insightful feedback and constructive comments and thanks for your efforts on reviewing our paper. We are encouraged by your appreciation on our “clear, fair and straightforward” evaluation benchmark. We are going to public the toolkit to evaluate LLMs by one-line command and maintain a leaderboard of legal LLMs.\n\nBesides the merits, we are also pleased to hear your questions and concerns that are helpful for improving our paper. We hope the response bellow can provide a clearer explanation of our work.\n\n### Res to Quest. A:\nThe labels (charges) covered by the dataset are diverse while there are also close and confusing ones. We provide some statistics and analyze the ChatGPT results to showcase confusion labels.\n\nThe labels are the most common **112 charges** of the criminal law of China, which are categorized into **10 chapters**. Below are the most common chapters, number of included charges and charge examples:\n- *Crime of disrupting social order* (35), e.g., Affray, Environmental Pollution, and Illegal Possession of Drugs\n- *Crime of Disrupting the Socialist Market Economy Order* (31), e.g., Producing or Selling Food Products that Do Not Meet Safety Standards and Contract Fraud.\n- *Crime of Violating Personal and Democratic Rights* (17), e.g., Intentional Injury\n- *Crime of Endangering Public Safety* (15), e.g., Dangerous Driving and Arson.\n- *Crimes Against Property* (9), e.g., Fraud and Forcible Seizure\n\nThe confusing charges are determined by the confusion matrix of ChatGPT. We build the graph of charges (nodes) and link two charges if the counts of misclassification between the two charges exceed the threshold 7, based on the results described in Section 6.3. The charges in a *maximal connected subgraph* are recognized as confusing charges. Below are some confusing charge groups:\n- Producing or Selling Food Products that Do Not Meet Safety Standards; Producing or Selling Toxic or Harmful Food; Producing or Selling Counterfeit Drugs\n- Affray; Illegal Detention; Intentional Injury\n- Negligently Causing Serious Accident; Negligence Causing Death; Dangerous Driving; Traffic Accident\n\n### Res to Quest. B\nTo investigate the correlations of two models and the randomness of results, we propose two metrics\n-  *Agree*: to measure the ratio of **agreeing** cases where two models output same predictions\n- *Gain*: to measure the ratio of cases that the first model is correct if two model **disagrees**\n\n We divide the samples into four groups based on the correctness of two models, denoted as {T, F}. For example, when comparing GPT-4 and BLOOMZ, the *TF* group means that GPT-4 gets a correct result while BLOOMZ fails. Then the two metrics are calculated as \n$$Agree=\\frac{TT+FF}{TT+TF+FT+TT}$$\n$$Gain=\\frac{TF}{TF+FT}$$\n\nThe open question results of zero shot and few-shot are (compare GPT-4 to other models):\n| zero-shot | ChatGPT| ChatGLM | BLOOMZ| Vicuna|\n|-|-|-|-|-|\n| Agree | 78.93 | 70.18 | 75.00 | 63.04 |\n| Gain | 70.34 | 73.05 | 60.71 | 86.47 | \n\n| 2-shot | ChatGPT| ChatGLM | BLOOMZ| Vicuna|\n|-|-|-|-|-|\n| Agree | 83.75 | 68.75 | 70.71 | 71.61 |\n| Gain | 69.23 | 80.57 | 78.05 | 84.28 | \n\nAs GPT-4 benefits more from similar cases (Section 4.1, Line 301-303), the *Gain* under 2-shot setting over ChatGLM and BLOOMZ increases to around 80%, which means that when two model disagrees, GPT-4 can get correct predictions four fifths of the time, and the smaller counterparts is only correct one fifths of the time. This significant evidence shows that GPT-4 can do better on samples that other models fail on.\n\nThen we take a close look at the numbers of samples grouped by GPT-4 and ChatGLM results of 2-shot setting, which is {'TT': 246, 'TF': 141, 'FT': 34, 'FF': 139}. It shows that compared to ChatGLM, GPT-4 get additional 141(25.2%) samples correct and 34(6.1%) samples wrong, resulting in a 19.1% performance improvement (consistent to results in Table 2 in Appendix).\n\nAs there may be some uncertainty in LM generation and result parsing, the above results show clear improvements of GPT-4 over smaller counterparts.\n\n### Res to Quest. C\nWe have tried several layouts of Figure 8 and we recognize that it may be challenging to comprehend. The focus should be the comparison of the **two horizon blocks (Green vs. Purple)**, denoted as “F” and “T” at the left, of **each model**. The “F” line represents the performance changes after including a false similar case into a prompt, and its “*Green*” color means that the performance decreases due to the new false similar case. In contract, the “T” line featured by the “*Purple*” color indicates a performance enhancement due to a new true similar case in prompts. The implication of Figure 8 will be explained later to address the Reject Reason 2.\n\n### Res to Reject Reason 1\nLegal judgment prediction is a common task in Legal NLP, which is usually formulated as a classification task (Zhong et al. EMNLP 2018, Chalkidis et al. ACL 2019). LJP can serve as a fundamental task for other applications such as legal consultation, and the research of interpretable LJP can benefit both legal practitioners and laymen to comprehend lengthy legal documents (Wu et al. EMNLP 2022, Zhong et al. AAAI 2020).\n\nWe select this task for evaluating LLMs as LJP requires multiple level of legal abilities, including the knowledge of laws and legal terms and legal expertise reasoning. Besides, the ground truth labels of LJP make it easy to evaluate the outputs of LLMs. As shown in the response to Question A, LJP remains challenging in discriminating between confusing charges, an issue that that legal experts strive to address.\n\nThere are also other possible tasks to evaluate legal LLM, such as bar exam and legal document writing (e.g., case report, court debate and defense) as you suggested. The legal writing tasks are more challenging for requiring sophisticated logic and coherence. However, the main issue lies in the difficulty in content evaluation. The automatic evaluation metrics, e.g., BLEU and ROGUE, mainly highlight the text similarity and thus are not an appropriate measurement of knowledge and logic intense content. The alternative is human evaluation that was adopted by Open AI to evaluate the writing part in the bar exam of GPT-4, but it is costly, time-consuming and subject, hindering the reproduction of evaluation results.\n\nWe acknowledge your viewpoint “*LLMs have to do well on this simpler task before they can be useful in more complex tasks*” as both objective and insightful, which motivates us to think deeply and systematically investigate this question in our future work.\n\n### Res to Reject Reason 2\nThanks for pointing out the unclear writing of the conclusion. The paradox and the ablation study of the number of similar cases indicate the importance of retrieval systems for LLMs. Recently, retrieval-based (augmented) LMs are drawing increasing attention (ACL 2023 Tutorial https://2023.aclweb.org/program/tutorials/).\n\nOur results point out two research directions: 1) how to pretrain LLMs to work with retrieval noise (Wang et al. 2023) and 2) how to improve the capability of retrieval systems to provide useful information for LLMs, e.g., searching from structured data like knowledge graph.\n\n### Conclusion\nFinally, **thanks again for your review and the writing suggestions** that are very helpful to improve our paper. Limited by our abilities and time, we only choose one task to build the benchmark, but we try to make it a comprehensive one by including multiple fine-grained settings to showcase the two abilities of LLMs: using domain knowledge (multi-choice) and learning from previous cases (few-shot). As one of the pioneering legal and domain-specific LLM benchmarks, our framework and analyses can facilitate the proposal of more legal LLM benchmarks, and the settings tailored for LMs can also be transferred to other domains, such as medicine and finance.\n\n### Reference\n- Legal Judgment Prediction via Topological Learning. Zhong et al. EMNLP 2018\n- Neural legal judgment prediction in English. Chalkidis et al. ACL 2019\n- Towards Interactivity and Interpretability: A Rationale-based Legal Judgment Prediction Framework. Wu et al. EMNLP 2022\n- Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction. Zhong et al. AAAI 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very clear/clean, setting to test LLMs.  The task uses clear and well-labeled data. \n- The experiments are straightforward, so it is easy (and fair) to compare the performance of different model", "reasons_to_reject": "- While this is a clear/clean setting for evaluation, I could not tell from the paper if this a practical and important use case for law.  Is correctly labeling situations an important goal?  Or, is the argument that LLMs have to do well on this simpler task before they can be useful in more complex tasks (like framing the charges or defense)?\n\n- The \"paradox\" that an information retrieval system may or may not help performance is interesting, but the implication for research is unclear.  The conclusion: \"The conclusion is that increased numbers of demonstrations have both positive and negative implications for expertise reasoning.\"  This does not seem to illustrate why or what to do next.", "questions_to_authors": "A:  Perhaps I am not familiar enough with this dataset.  I would appreciate more detail on the labels that are used.  In particular, is there a natural metric space where some labels are close.  Eg., Theft vs Grand Theft.  Murder vs homicide.\n\nB: Are the model errors (or corrects) correlated across models?  Is GPT-4 better in the sense that it gets the same things correct as other models and then additional corrects as well? \n\nC: I found Figure 8 hard to interpret. What should be my focus when I look at that picture?  Is it the relative color (performance) in the same position across the models?", "paper_topic_and_main_contributions": "This paper evaluates several LLM on a specific task in the legal domain.  The specific task is to map a fact description to a specific legal charge.  (The example in Fig 1 is helpful:  \"A purse was stolen\" -> \"theft\").  Two types of experiments are run. One uses just LLM.  The second uses LLM's and an Information Retrieval system to provide the LLM with a similar case.  The results rank the tested LLMs based on accuracy with GPT-4 ranking best.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "As a general suggestion, shorter sentences might make things easier for the reader.  If an example would help:\n\nfrom the paper:\n\"Given that label candidates only hint LLMs with limited information of charge names, it is more possible that the significant improvement is attributed to the recall of pre-stored domain knowledge, which is elicited by label candidates.\"  \n\nAs two sentences:\nThe label candidates help LLM performance.  We postulate that the label candidate helps by eliciting pre-stored domain knowledge."}}, {"review_id": "peXLZ6P2pI", "reviewer": "Reviewer_9huf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition of our work. We are pleased to receive your critical comments. Our evaluation benchmark considers the scenario of LLM + retrieval system, which is a recent hot topic of retrieval augmented LLMs (ACL 2023 Tutorial: Retrieval-based Language Models and Applications). As a legal benchmark, we showcase the effects of two types of legal knowledge: charge names and similar cases (precedents), on legal judgment prediction. Besides, the observed “paradox” indicates the importance of retrieval quality and suggests two future directions of LLM + IR: 1) how to pretrain LLMs to ignore retrieval noise (Wang et al. 2023) and 2) how to enhance retrieval systems to deliver user-friendly information to LLMs.\n\nDue to the limitation of space, some results are not included in the paper. We would like to explain with more experiment results, examples, and statistics and hope they can answer your constructive questions to some extent.\n\n## Res to Reject Reason 1\nWe are interested into how in-context learning works and have conducted some ablation study on ChatGPT. Besides, we also manually check the LLM outputs to identify the characteristics of challenging cases.\n### Corrupted Demonstrations in In-context Learning\nTo explore to what extent LLMs can learn from in-context demos, we corrupt and build artificial demos by two strategies. In the first strategy, the demo input text (i.e., case description) is chosen from a random case, while the output (i.e., charge label) is set to be identical to the query case, denoted as **random input+ground label**. In the second strategy, we choose a true similar case as input text while set the output label to be random, demoted as **misleading**. We experiment on ChatGPT and compare the two artificial demo strategies to the **oracle**, where the demos are true similar cases without any corruption (i.e., cases retrieved by a 100% accurate retrieval system). The results are:\n| Demo Strategy | Acc |\n| - | - |\n| Oracle | 73.39 |\n| Random input + Ground label | 65.89 |\n| Misleading | 51.61 | \n\nThe results show that even the input-output correlation is broken, the model’s performance only drops slightly and it can still provide correct answers in many cases. It suggests that the model do not totally rely on the context to make predictions and mostly rely on its **prior knowledge** acquired during pretraining. Previous work of demystifying in-context learning also has similar observations (Min et al. 2022, Wei et al. 2023(a), Wei et al. 2023(b)). \n\nBesides, the “Ground label” setting yields a much higher accuracy than the “Misleading” setting. This suggests that the ground truth labels existing in prompts can provide strong hints for LLMs to reach the correct predictions. We conjecture that ground truth labels, or *label candidates* described in the paper, can activate LLM’s memory of relevant domain knowledge.\n\n### Case study\nBesides the confusing charges, LJP is challenging to both LLMs and supervised models as there could be **multiple criminal actions/facts in one case**. For example, one defendant may rob the handbag of the victim, and use the victim’s credit card to withdraw money. The lexical features can be correlated to the labels of both robbery and credit card fraud, while legal experts can tell that the proper charge should be robbery. \n\nAnother case is that when multiple laws are violated, the ultimate judgment should be based on the charge with the most stringent conditions and harsher penalties. For example, “Affray” could escalate to the graver offense of “Intentional Injury” if the injuries of the victims are deemed severe.\n\nWe have observed that small LMs, e.g., BERT, Roberta and DeBERTa, finetuned on the task specific data would **fail** these examples due to weaker reasoning ability and relying on superficial features. LLMs with standard prompting methods can also make mistakes when handling these cases, but they are promising to be competent if prompted properly. There have been increasing attempts to harness LLMs to solve complex reasoning tasks, e.g., task decomposition, improving consistency, embodied agents, etc.\n\n\n## Res to Reject Reason 2\nYour question also interests us a lot. The main effect of tokenizers seems to be the different tokenization efficiency for different language, which results from the distribution of pretraining corpus. The resulting different context length may pose the challenge for LMs to capture long term dependency. We are not experts of this domain, so we try to give some empirical statistics.\n\nFor each model, we aim to display the relationship between the prompt length and prediction accuracy. We sort all samples based on the numbers of prompt tokens and equally split them into 5 groups. That is, the token number upper limits of these groups are the 20, 40, 60, 80, 100 percentiles. The performances (accuracy) of the samples of the groups are averaged among 1-shot to 4-shot settings, and we calculate the performance drop of the longest group compared to the shortest group. Below are the results:\n| model | 1(short) | 2 | 3 | 4 | 5(long) | drop | \n| - | - | - | - | - | - | - |\n| gpt4 | 75.67 | 78.84 | 71.72 | 61.52 | 56.83 | -18.84 |\n| chatgpt | 71.43 | 72.16 | 68.61 | 55.27 | 50.80 | -20.63 |\n| bloomz | 63.18 | 61.88 | 54.99 | 46.32 | 37.57 | -25.61 |\n| chatglm | 61.48 | 60.81 | 53.15 | 41.74 | 36.10 | -25.38 |\n| vicuna | 61.83 | 61.24 | 54.34 | 32.69 | 24.33 |  -37.5 |\n\nThe results indicate that all models experience significant decrease of performance on longer prompts. There could be another confounder that longer samples are inherently difficult. However, the varying degrees of performance drop suggest LLMs different abilities to handle long inputs (around 2000 tokens in our experiments). If we take GPT-4 as a baseline and attribute its performance drop to the inherent difficulty of long samples, the additional ~8% drop of Bloomz and ChatGLM can be attributed to their inferior ability to model long sequences.\n\nIn our experiment settings, we truncate the query case to 1000 tokens that can preserve most of the information (see Table 1). We also check this by feeding Vicuna’s prompts (truncated by the least efficient tokenizer) to ChatGLM and Bloomz, and we do not observe significant performance changes compared to prompts truncated by their own tokenizers. This indicates that in our experiments, where we prioritize the truncation of demonstrations and the maximum shot is set to 4, useful information is well maintained and the tokenizer efficiency is not the bottleneck of model performance.\n\n**Many thanks for your insightful feedback and your time on our rebuttal.**\n\n### References\n- Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study. Wang et al. 2023.\n- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Min et al. 2022 EMNLP\n- Larger language models do in-context learning differently. Wei et al., 2023 (a)\n- Symbol tuning improves in-context learning in language models. Wei et al., 2023 (b)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper did a pretty detailed and comprehensive study on LLM's ability of legal domain using various prompting methods.\n2. The paper discovers the interesting observation that well-performing IR can perform better than LLM + IR.\n3. very nice graphs and clear demonstrations", "reasons_to_reject": "1. The paper investigates a lot of the effects of prompting methods. But I hope the authors can compare the result with general domain prompting results, as (1) many conclusions seem to overlap with general domain prompting results and (2) authors should highlight the specific/new difficulty met in the legal domain, thus rendering the study less interesting. \n2. The section 3.2, the paper mentions the difference in tokenizers, which is very interesting. I would like to see how the tokenizer difference may explain/correspond to some experiment phenomena, but unfortunately, it is not discussed.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is centered on assessing the performance and utility of large language models (LLMs) in the legal domain, specifically focusing on legal judgment prediction. This paper provides a comprehensive evaluation of LLMs in legal judgment prediction tasks. This evaluation involves designing and comparing various models or approaches, including GPT-4, in terms of their performance on legal judgment prediction. This can help to clarify the strengths and limitations of using LLMs in the legal domain.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 430: the investigate the paradox --> to investigate the paradox"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a comprehensive benchmark for evaluating large language models on legal judgment prediction using the CAIL dataset. The work designs prompt-based solutions leveraging LLMs to map case facts to charges. An information retrieval system is incorporated to provide similar cases as context. The experiments analyze model performance under various settings like open vs multiple choice questions and zero-shot vs few-shot learning.  

Reviewer 1 highlighted the rigorous comparative analysis of multiple state-of-the-art LLMs as a strength. However, they felt using supervised learning baselines from existing legal judgment prediction literature would make the evaluation more compelling. They also questioned the lack of analysis around critical issues like fairness, expert evaluation, explainability and hallucination risks. In the rebuttal, the authors provide additional experiments with BERT-based models as baselines and acknowledge the importance of the highlighted issues for future work.

Reviewer 2 found the clean experimental design and straightforward performance comparison between models to be merits. A concern raised was whether the charge labeling task evaluated is a practical legal application. The authors clarify in the rebuttal that legal judgment prediction serves as a proxy task requiring multifaceted legal abilities, though other complex applications like case report writing could also be considered. The reviewer also sought more clarity on the implications of the "paradox" finding. The authors explain this points to the importance of retrieval quality and future research directions.

Reviewer 3 recognized the comprehensive analysis of prompting strategies as a strength. They suggested comparing results to general domain prompting and highlighting legal-specific challenges. The rebuttal provides examples of legal reasoning difficulties like charges with overlapping facts and penalties. The reviewer also asked about tokenizer differences which the authors analyze by correlating sequence length and performance.

I agree with the reviewers that the paper is beset with  some of these limitations:
- Lack of comparison to supervised baselines on LJP (Review 1). The authors provide additional experiments with BERT-based models in the rebuttal.
- Unclear implications/applications of the "paradox" finding (Review 2). The authors clarify the takeaways for future work.
- Overlap with general domain prompting effects without highlighting legal-specific challenges (Review 3). The rebuttal provides more legal case analysis.


Summarily, the reviewers recognize this work provides one of the first comprehensive LLM benchmarks tailored for the legal domain, which is a novel contribution. While comparisons to supervised learning baselines are lacking, the authors provide additional experiments in the rebuttal indicating LLMs can outperform BERT-based models under low resource settings. They also acknowledge the importance of analyzing issues like fairness and plan to investigate them in future work.

The remarks about practical applications and implications of findings help highlight areas where the paper could be strengthened, but the core utility of extensive LLM analysis for an under-explored domain remains. The authors have made efforts in their responses to provide more legal case insights and clarify the takeaways.

Considering the authors' effort in addressing some of these issues in the rebuttal, I would encourage them to carry out thorough revisions to enrich the legal analysis, highlight unique challenges, and crystallize the implications of findings that could make this a stronger contribution.
Decision: Accept-Findings