LLMs -- the Good, the Bad or the Indispensable?: A Use Case on Legal Statute Prediction and Legal Judgment Prediction on Indian Court Cases
Authors: Shaurya Vats
Atharva Zope
Somsubhra De
Anurag Sharma
Upal Bhattacharya
Shubham Kumar Nigam
Shouvik Kumar Guha
Koustav Rudra
Kripabandhu Ghosh
Venue: EMNLP 2023 Findings (2023)
Keywords: Legal judgement prediction
Legal Statute prediction
LLMs
explainability
bias
fairness
ethics

Abstract: The Large Language Models (LLMs) have impacted many real-life tasks. To examine the efficacy of LLMs in a high-stake domain like law, we have applied state-of-the-art LLMs for two popular tasks: Statute Prediction and Judgment Prediction, on Indian Supreme Court cases. We see that while LLMs exhibit excellent predictive performance in Statute Prediction, their performance dips in Judgment Prediction when compared with many standard models. The explanations generated by LLMs (along with prediction) are of moderate to decent quality. We also see evidence of gender and religious bias in the LLM-predicted results. In addition, we present a note from a senior legal expert on the ethical concerns of deploying LLMs in these critical legal tasks.

LLMs - the Good, the Bad or the Indispensable?: A Use Case on Legal Statute Prediction and Legal Judgment Prediction on Indian Court Cases
Shaurya Vats
Atharva Zope ISSER Kolkata
Somsubhra De IIT Madras
Anurag Sharma IISER Kolkata
IIT Kharagpur
Upal Bhattacharya
Shubham Kumar Nigam
Shouvik Kumar Guha
IISR Kolkata
IIT Kanpur
NUIS
Koustav Rudra
Kripabandhu Ghosh
IIT Kharagpur Abstract
IISRR Kolkata
text. Recently, (Charlotin, 2023) presented de- tailed take on the future of law in the light of the large Language Models.
The Large Language Models (LLMs) have im- pacted many real-life tasks. To examine the ef- ficacy of LLMs in a high-stake domain like law, we have applied state-of-the I.IMs for two popular tasks: Statute Prediction and Judgment Prediction, on Indian Supreme Court cases. We see that while LLMs exhibit excellent predic- tive performance in Statute Prediction, their performance dips in Judgment Prediction when compared with many standard models. The explanations generated by LLMs (along with prediction) are of moderate to decent quality. We also see evidence of gender and religious bias in the LLM- -reeddpddcteed results. In addition. we present note from a senior legal expert the ethical concerns of deploying LLMs in these critical legal tasks.
However, the acceptability of LLMs in a high- stake domain like law is still under scrutiny. To this end, we address the following in this paper: (i) We attempt two legal-specific tasks statute prediction (Section 2) and judgment prediction (Section 3), ing LLMs in the light of state-of-the-at baselines. on Indian Supreme Court cases. (ii) We generate explanations for each of these tasks using LLMs and compare them with annotated explanations by legal experts, providing detailed quantitative and qualitative analysis of the same (Sections 2 and 3). We also identify errors and hallucinations in the generated explanations (Tables 11 and 12). (iii) We also discuss the presence of bias in the outputs generated by LLMs and ethical considerations for LLMs law (Section 4). (iv) We also release a segallexpert-explaaatioon -nnootated statute predic- tion dataset. To our knowledge, this is among the first works so far on Indian cases. In recent work, (Deroy et al., 2023) applied LLMSs to legal court summarization. Also, to our knowledge, no prior work has addressed these issues these le- gal problems esp. for Indian cases. The data and code are available here: https: //gi thub, com/ Mmmpubbrre4LLLLLLLga_PPPPeeL...
1 Introduction
The Large Language Models (in short, the LLMs) have touched upon many Al-based tasks. There has been a recent surge in the application of LLMs c.g. GPT (Brown et al., 2020), BLOOM (blo), FLAN-T5((5), LLaMA(Ila) which has heralded a new area in data science. Along with many other areas, L.Ms have been applied to the legal domain. Blair-Stanek et al. (2023) has applied GPT-3 for statutory reasoning U.S. statutes with moderate success. Similarly, (Yu al., 2022)Nguuenn al., 2023) applied GPT-3 for the legal reasoning task statute detection on a Japanese Bar exam dataset effectively. Katz. et al. (2023) shows that GPT-4 is able to pass the Uniform Bar Examination (UBE), often outperforming humans. However, (Choi et al., 2023) reported that ChatGPT performs moderately real exams the University of Minnesota Law School. (Hamilton, 2023) has designed multi- agent system for judgment prediction using GPT-2 for U.S. Supreme Court with moderate accuracy. As a different line of work, Savelka (2023) lever- aged GPT for sentence-level annotation of legal
Statute Prediction
Given a fact text (Bhattacharya al., 2019b) of case, we consider two types of tasks (i) prediction of statutes (statute predn) and (ii) prediction of statutes along with the generation of explanation for the prediction (statute predn expln).
2.1 Datasets
statute predn expln test dataset: We have an- notated the FIRE AILA 2019 (Bhattacharya et al., 2019a) Task (Statute Retrieval task) dataset with explanations (see example in Table 4). This dataset

---

has 45 fact texts from Indian Supreme Court cases and the explanation annotations are done by law students of a reputed law university in India. We plan to release this dataset upon the acceptance of the paper.
Prediction results: The results of these experi- ments are reported in Table 1. We report Macro Precision, Macro Recall and Macro Fl on statute predn test dataset and statute predn + expln test dataset. Note that, Template 2 prompt is was applied on statute predn + expln test dataset (without the explanation part) for predie- tion (shown as "Prediction only (45 cases)") and also while generating explanations for the predic- tion (shown as "Prediction with explanation (45 cases)"). We observe that when the LLM model is prompted to generate explanations along with the prediction, its overall performance (Macro FI) increases. We see that LLM outperforms all the baselines in the prediction task and it is statisti- cally significantly better (μ-value 0.05) than the best-perfone pest-performing baseline (XLNet) by two-sided t- test (tte) at 95% confidence level. Template 2 in Ta- ble 6 provides the structure of the prompts given the LLM for statute prediction which provides, the titles and descriptions of the relevant statutes. This may have been an advantage of the LLM model over the baselines which does not consider the text of statutes. More baseline results are reported in Appendix G.
statute predn test dataset: We randomly select 100 cases (excluding statute predn + expln test dataset cases) from the Indian Supreme Court col- lection. In addition, select 100 more cases from Indian Supreme Court (excluding already consid- ered 145 cases) containing gender-specific statutes and religion-speciffe statutes from the Indian law (See Table 13 for a list of these statutes curated after consultation with legal experts) for analysis of religion and gender bias in the statute prediction task (Section 4). These 245 (45+100+ 100+ cases form the test dataset for statute predn.
statute predn train dataset: This is created by randomly selecting 18021 cases excluding the statute predn test dataset cases from Indian Supreme Court collection.
2.2 Experimental Setup and Results
LLM models used: We applied several models like GPT-3.5 turbo, text-davinci-003, text-davinci- 002, davinci (ope), flan-t5-large (t5), bloom- 560m (blo), gpt2 (gpt) etc. We observed that the best results our dataset are provided by text- davinci-003 (paid version) on random set of 5 cases (not a part of the train-test setup) and all results reported in this paper are produced by this model.
Explanation results: We have also evaluated the explanations generated by LLM by comparing them with gold standard explanations annotated by legal experts. To this end, we use several eval- uation measures. We use the n-gram or word- overlap based measures like ROUGE-I, ROUGE-L (Lin, 2004), METEOR (Banerjee and Lavie, 2005) and BLEU (Papineni et al., 2002). In addition, 2020) that captures the semantic similarity between the gold standard and LLM LLM-generated explana- tions. We have also used a recent measure, viz. BLANC (Vasilyev et al., 2020) to estimate the qual- ity of the generated explanations without the gold standard. The results are shown in Table 2. We see that the BERTScore values are better which possi- bly indicates good semantic match not captured by verbatim match. Table shows some decent expla- nations generated by LLM in comparison with the expert-labeled annotations.
We use Template 2 (Table 6 in appendix) to which, in zero-shot setup, texts of ap- plicable statutes with the ids were shown (which does not use statute predn train dataset) and for test, given fact texts, the LLMs were asked to gener- ate the applicable statutes (from the already shown ones). For explanation generation, additionally, an appropriate request was done. In addition, we have used Chain of Thought Prompting (Wei et al., 2022) as described in Appendix F. We have also reported additional experiments in the Appendix H.
Prompts used:
BERTScore (Zhang et al.,
Baselines: We deployed InLegaBERRT (Paul et al., 2023), XLNet (Yang et al., 2019) and LEGAL- BERT (Chalkidis et al., 2020) which were fine- tuned for 30 epochs with a learning rate of 3e- using Adam (Kingma and Ba. 2014) optimization on statute predn train dataset. It is to be noted, that it a multi-label classification problem on 80 statutes.
Expert opinion on L-M-ggeerated explanations: We randomly selected 13 cases from statute predn test dataset (which do not have annotated explanations) and sent the explanations generated by LLMs these to senior legal expert for eval- uation. The expert asked to comment on the

---

quality of explanations and rate them on Likert scale of 0-10 (0 completely unsatisfactory, 10 perfect). According to the expert, these explana- tions are of moderate quality with an average score of 4.27 out of 10.
Evaluation measure
Statute Pre- diction
Judgement
Prediction
cases)
(54 cases)
ROUGE-1
0.2522
0.5490
ROUGE-2
0.1934
0,4217
ROUGE-L
0.2445
0.5268
METEOR
0,2362
0.4685
Hallucinations: Tables 11 and 12 show some com- mon hallucinations found in LLM predictions and generated explanations. The LLM tends to incor- porate statutes that appear more frequently in the contextual examples, even if they are not relevant or applicable to the given context and attempts to provide explanations for the fact statement by uti- lizing words from the description of the statute provided in the prompt. Furthermore, it priori- tizes including the statutes applicable to the fact statement provided in the prompt rather than those relevant to the test case. In order utilize all the available tokens in the completion, the model aims include as many statutes as possible, resulting higher number of statutes being included in its response. The performance of the model tends to decline when presented with a large test input. As the model progresses down the list of statutes provided in the prompt, its performance tends to decline. This suggests that the model's ability to generate accurate and relevant responses is affected by the order in which the statutes are presented, with the later statutes being less effectively utilized in generating appropriate outputs.
BLEU 0.1043
0.2745
ScRTe 0.3986
0.8970
BLANC
0.2024
0.3394
Table 2: Explanation performance (F1 (FI-score) for Statute Prediction and Judgement Prediction.
ation of explanation for the prediction (judgment predn expln). We follow the setup of Malik et al. (2021).
3.1 Datasets
judgment predn expln test dataset: We have used the 54 (out of 56) annotated cases (ILDC_expert) of Malik et al. (2021) as the remain- ing two cases will used for in-context training of the LLM.
judgment predn test dataset: We randomly se- lect 100 cases (excluding judgment predn expln test dataset) from ILDC_singlt of Malik et al. (2021). In addition, we select 100 more cases ILDCCsingle of Malik al. (2021) (exclud- ing already considered 100 cases and 56 cases of ILDC 2cD__xeeett containing gender-specific sec- tions and religion-speciie law (See Table 13) for analysis of religion and gender bias in the judgment prediction task (Sec- tion 4). In addition, we consider all 56 annotated cases (ILDC_ed _eC_exeet) of (Malik et al., 2021). These 256 (100+100+50) cases form the test dataset for judgment predn.
statutes from the Indian
Praliction only
Macholl
Marro Periision
Miemo Recall
Maemi
LTM
0,1629
6,7914
IL2566
Inl ayalERTT
1082
0,1519
0220
LEGAL-BERT
0,99
0,1469
0.1133
0.1052
0.1545
0.1215
Prodiction with explanation (45 cress)
Method
Marco Piccisina
Macro Recail
Macro Fl
0.2794
0.4004
0.2811
InLogalBLKK
0.1119
LEGAL-BERT
0,00
0.1469
8.1855
XLNet
0,2
0.1543
0.1215
judgment predn train dataset: This the entire training set of (Malik et al., 2021) (see appendix Section B for details) for prediction. Two randomly chosen cases from this training set are used to train LLMs for judgment predn task. In addition, two annotated cases (out of 56) (ILDC_expeel were used for training of judgment predn expln task for LLM.
Prediction only
Piccssoon
Mocro
LLM
0.4343
6,5378
0L3871
Int mLagalBUK
0,3120
0,3892
0.1131
LEGAL BEKT
0.897
03343
Table 1: Statute Prediction results. The best values are
shown in bold.
3 Judgement Prediction
3.2 Experimental Setup and Results
Given whole case (except the final decision), consider two types of tasks (i) prediction of judg- ment, viz. allowed (the appellant won the case; label 1) or dismissed (the appellant lost the case; label 0) (we call this task judgment predn) and (ii) prediction of judgment along with the gener-
As the complete judgments are too long to fit in LLM or non-hierarchical models, in these exper- iments, we have used the last 512 tokens of the judgments as prescribed by Malik al. (2021). LLM models used: As mentioned in Section 2.2, we tried many LLM models and finally chose GPT-

---

3.5 turbo for judgment prediction task it pro- duced the best results.
ROUGE-2, ROUGE-L, METEOR and BLEU scores of Table 2 are comparable with those re- ported in Table (Machine explanations v/s Expert explanations) by (Malik et al., 2021), which possi- bly indicates that the quality of explanations gener- ated by LLMs is not yet of satisfactory level when the verbatim match is considered. However, we see that the BERTScore values are considerably better which possibly indicates that the explanations have good semantic alignment with the gold standard explanations. Table (in appendix) shows some decent explanations generated by LLM in compar- ison with the expert-labele's annotations and the generated explanations for judgment prediction, in general, are well-aligned with expert-annotated ex- planations.
Prompts used: We used Template (Table 9 in ap- pendix) for judgment predn expln task. This prompt, for in-context training in few-shot setup, shows a case-description, gold standard prediction label (1/0), and gold standard explanation text, and for test, ask the LLM to generate both the predic- tion and explanations. For only prediction, we have used Template (Table 10 in appendix) which is similar to Template 1 except that it does not have the explanation component in prediction. We also used zero-shot prompting which did not work well. Baselines: We deployed InCaseLaw, InLT (Paul et al., 2023), XLNet (large), XLNet (base) (Yang et al., 2019) and LEGAL-BERT (Chalkidis et al., 2020). We set the batch size to 16, us- ing Adam (Kingma and Ba, 2014) optimizer, con- ducted training for 3 epochs on judgment predn train dataset, and employed a learning rate 2e 6. were set to their default values as provided by the HuggingFace library. It is to be noted, that it is binary classifi- cation problem.
Expert opinion LLM-generated explanations: On randomly selected 10 cases from judgment predn test dataset (which do not have annotated explanations) the legal expert assigned an average score of 7.2 out of 10. For some of the cases, the scores are as high as 9.5 out of 10. This possibly also accounts for high BERTScore for judgment predn + expln as reported in Table 2.
The remaining
Prediction results: The results of these experi- ments reported in Table 3. We report Macro Pre- cision, Macro Racall and Macro Fl on judgment predn test dataset and judgment predn expln test dataset. Note that, Template 2 prompt runs on judgment predn + expln test dataset with- out generating explanation (shown as "Prediction only (54 cases)"" and Template 1 prompt while generating explanations for the prediction (shown "Prediction with explanation (54 cases)"). We note that when the LLM model is prompted to gen- erate explanations along with the prediction, its performance improves than without explanation. On 54 cases, the best performance is by XLNet while LLM performance is comparable with the other models (in Macro F1). However, LLM lags behind the baselines noticeably in 256 cases. As op- posed to statute prediction, context is provided about the judgment decision. As result, the LLM inferred the decision from few in-context train- ing examples. In addition, the fine-tuned baseline models possibly benefited from having simpler bi- nary classification task than the multi-label statute prediction objective and were, therefore, able to perform better.
Prodiction only cascs)
Mathod
Macro Procisioo
Macro Recali
Macco Pi
L.M
0,5922
0,5493
0.5500
IoCuseLaw
0.7208
XLNet (lagg))
0.736
NL.Nt (base)
0.0922
0.623
U5786
InLegalSSEii
0,7522
U.MEY
0.603
LEGAL BLKT
0,738
013000
explatation
Methol
MacTo Phecision
Macro Recall
Macro PI
LTM
0.6144
D.451
InCaeeLaw
0,7708
0.907
03288
(angg)
0,726
NLNNt (hase)
I13766
Tnl #galFFRT
0.7036
0.3833
TEGAL-BUR
0,714
113910
Proliction
Mathval LIM
Precisum
Maamn Rocall
0.5703
0.5888
(.564)
InC
0,7131
Ne Clange)
0,7336
(base)
0.7171
0.6786
Inl-culBEER
0.7436
0,6579
066822
LEGAL-BERT
0,7321
0.6898
IX6755
Table 3: Judgement Prediction results. The best results are shown in bold.
4 Bias, Fairness and Ethies
In this section, we perform tests for the presence of gender and religious bias in the LLM predictions. Statute Prediction for religion and gender- specific sections: On statute predn + expln test dataset, the LLM performance deteriorates for cases containing religion-speeific sections (R) (See Table 14 in appendix) than that in neutral cases. statute predn test dataset, the performance of
The results are shown in Table 2 under "Judgment Prediction".. ROUGE-
Explanation results:

---

LLMs deteriorates for cases containing religion- specific (R) or gender-specific (G) cases. Note that we already chose cases containing genderfreligion- specific statutes earlier (see Sections 2 and 3).
ments on larger number of cases. Also, legal ex- pert annotations are expensive and time-consuming to obtain (though, to our knowledge, we have used some of the biggest legal expert annotated datasets available for Indian cases, in this paper). Conse- quently, the findings reported in this paper regard- ing LLMs may not hold for the whole Supreme Court collection or for entire judicial system. We plan to do more rigorous experiments big- ger datasets, as and when available, to ensure the statistical validity of research findings on the appli- cation of LLMs in high-stake domains like law. We have not used a baseline model that can jointly pre- dict (statute/judgmeent and generate explanations. which could have been an interesting comparison with the LLM that has simultaneously predicted and generated explanations in this paper. We see that statute prediction using LLMs has done bet- ter than baselines that do not consider the textual content of the statutes. We have not compared with (Paul et al., 2022) and (Paul et al., 2020) which leverage the text of statutes due to their re- quirement for additional citation graph information, handcrafted legal hierarchies and their specificity (focusing criminal law).
Gender frequency disparity: We have analyzed the number of gender-definition words (Zhao et al., 2018) in the gold-standard and LLM-generated explanations. Figure 3 (in appendix) shows for statute prediction (SP) there is radical increase in Male:Female ratio in the LLM generated expla- nations (M/F ratio: 2.62) than that in the expert- annotated explanations (M/F ratio: 1.80). This disparity is less pronounced for judgment predic- tion (JP) though there is slight increase. We do not report this for religion due to the lack of good religion-definition wordlist.
Group disparity: We report the Worst Class In- fluence (WCI) score per group (Chalkidis al.. 2022) note group disparity for the under- represented groups: Female, Muslim in India. Ta- ble 15 shows higher values and hence higher dis- parity for females in statute prediction and for Mus- lims in judgment prediction.
(please see Appendix Section D for more details) Ethical concerns: We have obtained note on the ethical concerns of using LLMs in tasks like Statute Prediction and Judgement Prediction from a senior legal expert. For the paucity of space, we have included it in the Appendix Section E.
Ethics Statement
The law students who did annotations (upon their consent) were treated fairly for the datasets used in this paper, have been fairly compensated and they are not authors of this paper. Two hundred documents (100 for statute prediction, 100 for judgment prediction) were chosen such that they contained gender-related or religion-relater statutes (see Table 13 for these statutes) to ensure that we have cases to examine existing gender or religion bias and not to incorporate any bias con- sciously. We have duly paid subscription fees to OpenAl for using their paid GPT models used in this paper.
5 Conclusion
In this paper, we applied state-of-the-art LLMs for Statute Prediction and Judgment Prediction. We see that LLMs (i) outperform standard baselines in statute prediction, (ii) underperform similar base- lines for judgment prediction, (iii) for subset of cases, generate explanations moderate for statute prediction, but often quite good for judgment pre- diction and (iv) exhibits evidence of gender and religion bias in the predictions and generated expla- nations. The commendable performance in com- plex task like multi-label statute prediction (also with explanations) with no explicit training in the legal domain, possibly shows promise in the appli- cation of LLMs in future legal tasks. At the same time, judicious monitoring of bias, fairness and ethics in LLMs is a desideratum.
Acknowledgement
This work is supported in part by the Science and Engineering Research Board, Department of Science and Technology, Government of In- dia, under Project SRG/2022/001548 and Mi- crosoft Academic Partnership Grant 2023 Agree- ment No. 7581365. Koustav Rudra is cipient of the DST-INSPIRE Faculty Fellowship [DSTIINSPIRE//442000//0000 in the year 2021 under Engincering Sciences.
Limitations
Due to the token limitation and high subscription charges of paid LLMs, we could not do experi-

---

References
law school. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 2898- 2904, Online. Association for Computational Lin- guistics.
model-
Ilama-meta-ai/.
Ilias Chalkidis, Tommaso Pasini. Sheng Zhang. Letizia Tomada, Sebastian Schwemer, and Anders Søgaard. 2022. FairLex: A multilingual benchmark for evalu- ating fairness in legal text processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1389-44066, Dublin, Ireland. Association for Computational Linguistics.
htpps//uugggnngaaccccooooo/////s
hmpss//hugginggffee.coo/gg.
httpss//openai.com//
https:///oces scipy https:///oss..cccoyoooooo//o//// denerated/scipy.steeee stats. ttess
Damien Charlotin. 2023, Large language models and the future of law.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop
Jonathan H. Choi, Kristin E. Hickman, Amy Monahan, and Daniel Schwarez. 2023, Chatgpt goes law school.
Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. 2023. How ready are pre-trained abstractive models and Iims for legal case judgement summariza- tion?
Paheli Bhattacharya, Kripabandhu Ghosh, Saptarshi Ghosh, Arindam Pal, Parth Mehta, Arnab Bhat- tacharya, and Prasenjit Majumder. 2019a. Overview of the FIRE 2019 AILA track: Artificial intelligence for legal assistance. In Working Notes of FIRE 2019- Forum for Information Retrieval Evaluation, Kolkata, India, December 12-15, 2019, volume 2517 of CEUR Workshop Proceedings. pages 1-12.
Sil Hamilton. 2023. Blind judgement: Agent-based supreme court modelling with gpt.
Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2023. Gpt-4 passes the bar exam.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6900..
Paheli Bhattacharya, Shounak Paul, Kripabandhu Ghosh. Saptarshi Ghosh, and Adam Wyner. 2019b. Identification of rhetorical roles of sentences in in- dian legal judgments. In Legal Knowledge and In- formation Systems JURIX 2019: The Thirty-second Annual Conference, Madrid, Spain, December 11-13, 2019, volume 322 Frontiers in Artificial Intelli- gence and Applications, pages 3- IOS Press.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain, Association for Computational Linguistics.
Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripahandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. 2021. ILDC for CJPE: Indian legal documents corpus for court judg- ment prediction and explanation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 4046-4062, Online. Association for Computational Linguistics.
Andrew Blair-Stanek, Nils Holzenberger, and Ben- jamin Van Durme. 2023. Can gpt-3 perform statutory
reasoning?
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Adilya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Ha- Ta-Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas Stathis, and Ken Satoh. 2023. How well do sota legal reasoning models support abductive rea- soning?
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Tias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LERAL-BERT: The muppets straight out of

---

Shounak Paul, Pawan Goyal, and Saptarshi Ghosh. 2020. Automatic charge identification from facts: A few sentence-level charge annotations is all you need. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1011-1022,
Jieyu Zhao, Yichao Zhou, Zeyu Li. Wei Wang. and Kai- Wei Chang. 2018. Learning cender-neutral word em- beddings. In Proceedings of the 2018 Conference Empirical Methods Natural Language Processing, pages 4847-4853, Brussels, Belgium. Association for Computational Linguisties.
Barcelona, Spain (Online). International Committee Computational Linguisties.
Shounak Paul, Pawan Goyal, and Saptarshi Ghosh. 2022. Lesicin: A heterogeneous graph-based approach for automatic legal statute identification from indian le- gal documents. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 36, pages 1119-11116...1
Shounak Paul, Arpan Mandal, Pawan Goyal, and Sap- tarshi Ghosh, 2023, Pre-trained language models for the legal domain: A case study on indian law. In Proceedings of 19th International Conference on Artificial Intelligence and Law ICAII. 2023.
Jaromir Savelka. 2023. Unlocking practical applica- tions in legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts.
Oleg Vasilyev, Vedant Dharnidharka, and John hannon. 2020. Fill in the BLANC: human-free quality estimation of document summaries. CoRR, abs/2002.09836.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurlPS.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of- the-art natural language processing, arXiv preprint 0rXiv:1910.03777
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime G. Car- bonell, Ruslan Salakhutdinov, and Quoe Le. 2019. Xinet: Generalized autoregressive pretraining for lan- guage understanding. CoRR, abs/1906.00 190.08337.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He. Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Associa- tion for Computational Linguistics.
Fangyi Yu, Lee Quartey, and Frank Schilder. 2022. Le- gal prompting: Teaching language model to think like lawyer.
Tianyi Zhang, Varsha Kishore, Felix Wu. Kilian Q Weinberger, Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenRe- veew.net.

---

A Hallucinations
models followed a consistent approach. We the batch size using Adam (Kingma and Ba, 2014) optimizer, conducted training for epochs, and employed a learning rate of 2e 6. The remaining hyper-parameters were set to their default values as provided by the HuggingFace library.
B Judgment Prediction Dataset
(Malik et al., 2021) presents a meticulously cu- rated collection of case proceedings from the Supreme Court of India (SCT) in the English lan- guage. Within the SCI's decision-making process, judge or bench evaluates claims filed by the ap- pellan//petitioner against the respondent, ultimately determining whether these claims should be "ac- cepted" or "rejected." The ILDC dataset encom- passes these case proceedings, providing valuable labeled data with the original decisions rendered by the SCI judges, serving as gold labels for analysis and evaluation.
By leveraging these transformer architectures and fine-tuning techniques, we aimed to develop accurate case prediction models capable effec- tively analyzing the ILDC_multi dataset.
D Bias and Fairness
Statute Prediction for religion and gender- specifie sections: We analyzed the performance of LLM the religion-specific sections (Table 13), where it performed best in 11 cases with- out explanations. We also observed the perfor- mance of LLM the gender-specific sections 85 cases without explanation. Overall, LLM per- formed best for the sections which contained no gender or religion-specifie sections in 144 cases without explanation. On statute predn + expln test dataset, the LLM performance deteriorates for cases containing religion-speciff sections (R) (Table 13) than that in neutral cases. In statute predn test dataset, the performance of LLMs dete- riorates for cases containing edigion-speccfie (R) or gender-specific (G) cases (see Table 14).
The ILDC involved publicly available SCI case proceedings spanning from 1947 to April 2020. The ILDC is subdivided into three distinct sets. The ILDC_single subset comprises cases with ei- ther a single petition or multiple petitions result- ing in the same decision. On the other hand, the ILDCCmulti subset encompasses cases involving multiple appeals that lead to differing decisions. Lastly, the ILDC_expel _eppeet subset consists of anno- tated case documents where legal experts have pro- vided explanations elucidating the rationale behind the decisions. The experts predict the judgment out- comes and mark specific sentences they consider as explanations, assigning ranks to indicate the rel- ative importance of each sentence in shaping the final judgment. This subset allows for evaluating prediction algorithms for the explainability aspect and highlights the differences between ML-based explainability methods and expert explanations.
Gender frequency disparity: We have analyzed the number of gender-definitio words (Zhao et al., 2018) in the gold-standard and LLM-generated ex- planations as, think, the gold-standard explana- tions annotated by legal experts serve as a focussed summary aligned towards the topic (governing the prediction of statutes or decisions) the cases.
C Judgment Prediction Model Details and Hyper-parameters
Figure 3 contains overall word distribution plots for both tasks. 45 fact texts from Indian Supreme Court cases used for the statute prediction task. In the annotated explanations, the ratio of male fe- male definition words in the annotated explanation was 1.80, and the generated explanation was 2.62. For 54 annotated cases _I_DC_exeert) used for the judgment prediction task, the ratio in male to female definition words in the annotated explana- tion was 6.21, and in the generated explanation was 6.87. For the statute prediction task, the male to fe- male words ratio in annotated explanation is lower than in generated explanation. Whereas, for the judgment prediction task, the male-to-female ratio in annotated explanation is higher than in generated explanation. Figure shows for statute prediction
We conducted experiments using previously men- tioned on (Malik et al., 2021) state-of-the-art mod- els such XLNet (Yang et al., 2019) and trans- formers trained on legal corpora, namely Legal BERT (Chalkidis al., 2020), InL_egaBERTT (Paul et al., 2023), and InCaseLaw (Paul et al., 2023). To fine-tune these models, we utilized the Hugging- Face library (Wolf et al., 2019) and trained them on the ILDCCmulti dataset, considering only the last 512 tokens of each document. This choice was based experimental results, which indicated that using the last 512 tokens achieved the best perfor- mance.
The fine-tuning process for all transformer-based

---

coming from Fact Text
Appellants call in question legality of the judgment rendered by the High Court upholding conviction of the appellants (hereinafter referred the accused:) and sentence as imposed by the trial Court which had sentenced each to undergo rigorous imprisonment for three months, two years and seven years respectively with separate fines for each of the alleged offences with default stipulations. Background facts leading to the trial of the accused appellants are as follows: The case was registered on the basis of information lodged by PI (PW-6), which was recorded on 10.11.1989 at about 2.00 a.m. According to the informant, he and his son P2:s wife P3 (PW- 7) were sitting in the courtyard of the house of P2 (hereinafter referred to the deceased:). It was about 1.00 a.m. 9.1 191999 when deceased the village after purchasing vegetables. When he reached near the house of P4, son of P5, P6 (A-1) armed with Gandasi and P7 (A-2) armed with a lathi were present there. P7 made an obscene gesture. At this P7 and the deceased exchanged hot words and abused each other. P6 gave a Gandasi blow on the right hand of the deceased, which caused a grievous injury. P7 gave a lathi blow on the left foot of the deceased and also gave thrust blow of lathi on the left side of his head. Deceased fell down the ground. The occurrence was witnessed by P1 (PW-6) and P3 (PW-7). Applicable statutes
IPC 34: Acts done by several persons in furtherance of common intention
IPC 325: Punishment for voluntarily causing grievous hurt
Table 4: Example case from statute statute predn + expln test dataset. The colors show the explanation annotations and corresponding statutes are shown in same colours. asked law students a reputed law university to perform these annotations for 45 cases of FIRE AILA 2019 dataset (Task 2: Statute retrieval) (Bhat- tacharya et al., 2019a). There were two students who independently did the annotations and a senior faculty of law resolved disagreements, if any.
genden-efiition exclanations (Change in verdict
Treqancc gendendetivitrn worth (No char wirck))
40-
12-
gridke Mitiader
Figure I: Fele: ratios explanations for accu- rate Judgment Prediction LLM
Figure 2: Mele:Female ratios explanations for wrong
Judgment Prediction by LLM

---

(SP) there is radical increase in male:female ra- tio in the LLM generated explanations (M/F ratio: 2.62) than that in the expert-anootated explanations (M/F ratio: 1.80). This disparity is less pronounced for judgment prediction (JP) though there is a slight increase. For particular case, for statute predic- tion. the expert-annotated explanation read: "both the deceased persons were killed by the appellant by inflicting dagger blows, P3 and P9 died at the spot." " while the corresponding
general and the legal domain in particular. From search engine, voice assistance, preservation of languages and debugging codes to providing re- search assistance, judgment and statute prediction, document summarisation and trend analysis, the breakthroughs that have been made by LLMs can certainly open up new vista of possibilities in front of legal professionals as well as the subjects governed and affected by law. At the same time, usage of such technology also carried significant social and ethical challenges and the implications of such challenges may assume particular gravity in the legal context. The emergent capability of LLMs of performing a range of downstream tasks, including tasks requiring an increasing level of intu- itive association, processing and decision-making. unless coupled with adequate oversight, guidance, responsible design and operation with humans-in- the-loop, may not only remain relegated to a super- ficial stage with tremendous potential of spreading misinformation and harmful content at an unprece- dented scale and scope, but also have a deep impact on the legal and justice system as whole. Such im- pact can be felt via concerns such as distribution of harmful content and solutions generated by LLMs trained on biased and discriminatory data, exac- erbating reputational, financial and legal risks of proprietary solutions being used in an unauthorised manner to generate data and strategy options for the users, dislcosure of sensitive information, compro- mising data privacy and data security, amplification of existing bias or inaccuracy by multiple times due to large scale usage, data hallucination leading to unwanted and harmful outcomes, workforee dis- placement and other issues. For instance, lawyers having used LLMs generate supporting case- laws for their arguments have already fallen victim to fake citations and cases being provided. Simi- larly, if an LLM makes wrong predictions about ju- dicial orders, applies incorrect statutory provisions or fails to apply correct provisions to fact situations while generating an opinion, passes off proprictary or even worse, imaginary solutions as innovative, then the potential harm caused to the legal sector and the justice system can be catastrophic. Having said that, a focus on explainability, sector-specific training, responsible design, and multi- multi-layered au- dits with adequate human supervision on the design and dissemination stage of LLM technology, on the post-training but pre-release stage of such models and on the usage stage for applications based
-eneereedd ex- planation read: "asked the appellant as to why he was stabbing P3. P7 stated that she was the root of all troubles so the appellant started stabbing P9 at her abdomen, neck and other parts of her body." In the L.LM-generteed explanation, apart from factual inaccuracies, we see the injection of one male and three female words into the text. In addition, see that in cases where there LLM wrongly predicts a verdict, there are no female words (see Figure 2); however there are female words, in lower propor- tion than male words, for the cases where LLM makes a correct prediction (see Figure 1).
Group disparity: The unequal performance across statutes and judgments affects the macro-averager performance across gender groups (male and fe- male) and religious groups (Hindu and Muslim). We report the Worst Class Influence (WCI) score per group (Chalkidis et al., 2022). We observed the under--epresenttd group (Female, Muslim); in the statute prediction task, the Female group is the group with the worst performance (higher values), while in the judgment prediction task, the Mus- lim group performed worst (See Table 15). The cases that contained male and female definition words (Zhao al., 2018) in the annotated explana- tion for the statute prediction or judgment predic- tion were considered the test cases for those groups. The cases that contained Hindu and Muslim names, as either appellant or respondent of the cases, were considered the test cases for those groups The Hindu and Muslim names were identified using a surname word list curated by us (which we will release with the paper).
E Ethical Concerns about LLMs in legal domain: a legal expert perspective
The Large Language Models (LLMs), from what has been seen of them and their work so far, do have tremednous potential and an increasing de- gree of sophistication when it comes to the manner in which they may be used in the the society in

---

Frequency gender explenations
250
forra vert
200
100
52
SP shoe
FtAcuati
PYPvokted)
Figure 3: Number of male and female words in the gold-standen explanations and the LLM L.M-generated explanations for Statute Prediction and Judgment Prediction.
LLMs, can actually go long way to assuage most these ethical concerns. For instance, an LLM is used to predict the verdict of the court by going through the rest of the judgment and at the same time, it is also used to generate explanations about which parts of the judgment it has relied upon for such prediction, then the legal expert supervisor will easily be able identify any poten- tial bias or inaccuracy involved in the prediction. fact, there a possibility that such use of LLMs may actually deliver better results than the current practices, because while even human experts will have bias of their own, their conclusions are not always supported with explanations every level, something which the LLMs provide owing to their greater processing power and performance speed. In the course of this very research, the legal expert has found the accuracy and explainability of the LLM in judgment prediction to have reached a high level of performance, based on the sampled data review, while the same explainability assisted the expert to identify the drawbacks of the LLM in statute prediction. Filtered data use for training and test purposes coupled with explainability can be used to counter instances of racial, gender, caste related bias. Similarly, the governance audits referred above can be perceived or shaped as training received by even human experts legal
and professional ethics. The data hallucination as- peet can be checked the model-level audit. The workforce displacement issue can be countered by considering the LLM technology as an augmenta- tive and not substitutive force for human effort in knowledge retrieval, making decisions and similar cognitive and intuitive processes. Eventually, if such an approach is adopted, the LLMs can evolve into efficient and trustworthy assistive technology for legal information and expertise management, without compromising on the ethical scale.
human
F
Chain Of Thought prompting
Yu et al., 2022 (Yu et al., 2022) used zero-shot Chain-Of-Thouggh (CoT) prompting for statute pre- diction, where they first generated explanation and then used it for prediction. However, this did not produce the best results (than without explanations) and also they did not present any analysis of the quality of explanations. We, however, have inte- grated prediction and explanation as a part of the prompts (in both templates and 2, for statute prediction, by stating additionally "extract words or lines from the case due to which the predicted statutes applicable".
For statute prediction, adhere to the more conventional few-shot CoT prompting (Wei et al., 2022) where the predictions with explanations are

---

a part of few-shot prompting training, for the rebut- tal, we have designed few-shot prompt for statute prediction as below:
cial leave to appeal from any judgment, decree. determination, sentence or order in any cause or matter passed or made by any court or tribunal the territory of India (2) Nothing in clause (1) shall apply to any judgment, determination, sentence or order passed or made by any court or tribunal con- stituted by or under any law relating to the Armed Forces.
<start of prompt>
prompt = ff """"Given a fact statement as a Ques- tion delimited by triple backticks (*") and multiple choice options representing the names of statutes, predict the applicable statutes from the Statute Op- tions, and the explanation for each of the applicable statute containing lines from the Question delim- ited by triple backtickss"").. Example is given for your reference. The lecture provides a description of each statute, and your task is to select the statutes from Statute Options that best align with the words from the given Question. Choose the most rele- vant statutes and lines that match the given con- text. Statute options (Answer must always be from these) and Lecture (description of statute) are same for each Question. Answer must always be from Statute Options.
Answer: ['Indian Penal Code, 1860_321', 'In- dian Penal Code, 1860_3261 2__23_22" 'Indian Penal Code, 1860_302 'Indian Penal Code, 1860_34']
Explanation (words from the fact statement for each of applicable statute):
Statute: "Indian Penal Code. 1860_34" Lines: "the appellants felt offended and when the mem- bers of the complainant party came forward and obstructed the appellant from doing the work and restrained them from pulling out the pipe."
Statute: "Indian Penal Code, 1860_3211 " Lines "ttacked the complainant party. the first injury to be an incised wound. Second and third were abrasions on the left shoulder and neck. The fourth injury was a lacerated wound on the right parietal area of scalp. On the post-mortem conducted on P4, an incised wound was found on the parietal area of the scalp"
Example:
Fact statement: "These appeals are directed against the judgment of High Court whereby an appeal and a criminal revision were disposed of. The appellants were found guilty and sentenced to undergo various terms of sentences. The Criminal Appeal was filed by three appellants questioning the conviction and sentence as recorded. Com- plainant filed revision petition stating that she was entitled to compensation. Background facts giving rise to the trial are essentially as follows: .." Statute options (Answer must always be from these):
Statute: "Indian Penal Code, 1860_324" Lines: " P4 was attacked a.ttakkeeed the complainant party.
Format of Response: Answer containing a list of applicable statutes from the Statute Options above and Explanation (containing lines for each of the applicable statutes from the Question) for the Fact Statement delimited by triple backticks (""). Note: Response must not include Description from Lec- ture (useceeciiptioo of statute) as Explanation. Do not include statute options in your response. Lines must be from the Question delimited by triple back- ticks ("").
1. "Constitution 226" 2. "Constitution 136".....
Lecture (description of statute):
1. L."Consitution__ _26": Title: Power of High Courts to issue certain writs Description: (1) Notwithstanding anything in Article 32 every High Court shall have powers, throughout the territories in relation to which it exercise jurisdiction, to issue to any person authority, including in appropri- ate cases, any Government, within those territories directions, orders or writs, including writs in the nature of habeas corpus, mandamus, prohibitions, quo warranto and certiorari, or any of them, for the enforcement of any of the rights conferred by Part III and for any other purpose
Question: "' [[Fact_Statement
<end of prompt>>
Please note that above prompt provides not only the fact text and statute text, but also the extractive explanations from the fact text to train the LLM. For this experiment, cases were used for training the few-shot prompt, while the remaining 37 were used test cases. This produced Macro-Precision: 0.41, Mecro-Recall: 0.42 and Macro-F1: 0.36, which outperforms the LLM performance (Tem-
2. "Constitution_1366 Title: Special leave to appeal by the Supreme Court Description: (1)
Notwithstanding anything in this Chapter, the
Supreme Court may, in its discretion, grant spe-

---

plate 2) that recorded Macro-Precision: 0.17, Macro-Recall 0.68 and Macro-Fl: 0.27 and In- LegalBERT (the best-performing baseline) that recorded Macro-Precision: 0.18, Macro-Recall: 0.18 and Macro-F1: 0.17, on the same test set. However, the quality of explanations generated by this CoT prompt is considerably inferior to those produced by LLM (Template 2). This possibly leads us to the conclusion that few shot CoT-style prompting, although being trained with explana- tions, does not necessarily produce high-quality explanations for the test set.
text of the statutes (in Template 2). To ensure this for the baselines, we have designed transformer, viz. BERT base model (uncased) model that takes into account both the fact text and statute (label) text. In the context of the paired sentence task, the 512 token-limit was shared equally between the facts and statutes, with each segment being al- lowed maximum of 256 tokens. The two texts were separated by the [SEP] token is recognized by BERT for such tasks. Taking up to 150 positive pairs and an equal number of negative pairs for each statute yielded around 22k training samples. For the test set, the pairs were created between a fact text and each statute text. The framework leveraged the BERT base model (uncased) for ef- fectuating binary classificcation, discerning between the fact text and individual statute texts. The model underwent training over 100 epochs, with the most optimal outcomes observed at the 29th epoch. Note that the binary results for each test fact, statute pair were converted to a multi-label setup (used in our experiments as reported in Table 1), where for each test fact we have the prediction for each of the candidate statutes
Kindly note that for judgement prediction and explanation we have already adhered to the more conventional few-shot CoT prompting (Wei et al., 2022) where the predictions with explanations are part of few-shot prompting training. Kindly refer to Table 9 in the appendix to note that for a given case, the explanation along with the actual prediction is provided in the few-shot prompt training. To our knowledge, few-shot CoT prompting has not been applied to any legal task yet.
For judgement prediction task, we reversed the order of the prediction label and the explanation the training prompt (that first explanation and then prediction label) in the few-shot training and requested the corresponding output from the LLM. We have drawn inspiration from the paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei al. 2022 (Wei al., 2022), Section 3.3 Ablation Study, Chain of thought after answer, in order to analyse the sequential reasoning embodied in the explanation and its dependennce/conscqueee on the prediction ability of the LLM model beyond just extracting knowledge from the input text.
The model produced Macro-Precision 0.07, Macro-Recall: 0.28, Macro-fl: 0.10 on 245 test cases. On 45 test cases (used for prediction and ex- planation), the results are Macro-Precision: 0.02, Recro-Recall: 0.04, Macro-F1: 0.03. Clearly, these results are inferior to the LLM results in Ta- ble 1. This can possibly be attributed to the partial context captured by BERT model due to the token limitation. To alleviate this issue further, we re- placed BERT (512 token limit) with Longformer* which has token limit of 4096. After running for 26 epochs (72+ hours), the model produced a Macro-Precision: 0.05, Macro-Recall. 0.20, Macro-F1: 0.06 on 245 test cases (that is worse than the reported LLM results and also worse than BERT). It is to be noted that Longformer is ex- tremely memory intensive and takes long time to train.
The output explanations were obtained for 80% and were logically reasoned. But, still, only 20% was predicted correctly by the model. This there- fore led to a deterioration in the performance of our previous prompt style where prediction preceded explanation.
Therefore, from these sets of experimentations, we can safely say that the prompting style used by us previously in the paper was already in alignment with the state-of-the-art style (like CoT).
We also used a Hierarchical Attention Network (HAN) (Yang et al., 2016) which supposed consider the complete context of the text and which produced Macro-precision 0.05 Macro Recall = 0..31 and Macro-F1 = 0.06, on 245 cases. For both models, we observed steady training and no signs of overfitting.
More baselines
A possible reason for the apparent underperfor- mance of the non-LLM models in statute prediction can be ascribed to the fact that the LLMs use con-
htttps://hugginggace. to/bert-bbae-uncased https://hugginggface co/ fetocs//trasfoommmettt model_ Meee_ddocllonformmr

---

This possibly shows that the paired-text task is challenging for multi label-setup with 80 classes and also upholds the efficacy of the LLM prompts used in the paper.
Additional experiments
For template 2, we have also used statute numbers like "Indian Penal Code Section 1860_5", "Special Courts Act, 1979_57 "Constitution Article 136" etc. instead of statute ids like S1, S2 etc. However, the prediction results for 45 cases (Macro precision: 0.11. Macro recall: 0,56, Macro Fl: 0.18), were worse than the original Template 2 results and so was the explanation quality. So, we can possibly infer that providing the actual statute numbers does not necessarily help the task.
Data Contamination
We checked the paper Language Models are Few- Shot Learners, Brown et al. 2020 (Brown et al., 2020)), and found on pages 8-9 (Table 2.2) that the datasets which GPT-3 is trained are Common Crawl (filtered), WebText2, Wikipedia etc. and found no documentation (here or in other sources like httpss//heelp.ppppaa...cc//////r
stating whether Indian Supreme Court Judgements (on which we have reported our results) belong to this training set. So, we have not identified any evidence of data contamination, i.e., whether GPT has already seen the test cases which the tasks are reported in this paper.

---

Template (only prediction)
Task: Given examples of Supreme Court case and the statutes applied in that case, your objective is to make accurate predictions of the specific charge or statute that is most likely to be applied within the context of the case delimited by triple backticks (**), ensuring exact predictions and learning from the provided examples. You should only include the statutes it is most confident about. The response format should include the statutes applied as the context. You should showcase creativity and knowledge to enhance the accuracy of statute predictions based on the given fact statement.
Training:
Fact Statement: Statutes:
###
Fact Statement: Statutes:
Response and instructions:
Format your response as follows: Statutes applied: [List of applicable statutes]
Learn from the examples provided in the context to understand the task of charge or statute prediction.
Your response should be focused on providing the exact statute or charge that aligns with the legal principles and precedents applicable to the given facts. In your response, include only the statutes you are most confident about. Ensure that the statutes generated as responses are valid and recognized legal statutes. Avoid generating fabricated or invalid statutes. The model's performance will be evaluated based on its ability predict the correct statute, include only confident statutes, and showcase creativity in its predictions.
Template (prediction explanation)
(For Explanations these lines were included in the prompt:)
and extract words or lines from the case due to which the predicted statutes are applicable,
You should only include the statutes and words or lines from the fact statement you are most confident about.
Format your response as follows:
Statutes applicd: [List applicable statutes]
Explanation: [ (oods or lines from the fact statement due to which the statute is applicable]]
Note: Words or lines must be from Fact Statement delimited by triple back- ticks(")).
Table 5: Prompts in Statute Prediction (Template 1)

---

Template (only prediction)
Task:
You are given a fact statement delimited by triple backticks (")) and statutes with their title and description. Your task is identify the statutes applicable the fact statement from the given statutes that you are most confident apply to the fact statement. Each statute consists of title and a description of its scope and provisions. Include only those statute in your response which description logically matches with some parts of the fact statement. Training:
Statutes:
Statute Id:
Title:
Deseription:
###
Statute Id:
Title:
Description:
Response and Instructions:
Format of response:[Pythonn list of applicable statutes from Statutes]
Your response should include the statutes applicable to the fact statement. The applicable statute must be mentioned exactly as appears in Statutes provided. Include only those statutes which you are very sure about. Template (prediction explanation)
For Explanations these lines were added:
along with the words lines from the fact statement due which the statute is applicable for each of the statute applicable.
The applicable statute should be mentioned exactly as it appears in the list, and the words should be from the fact statement.
Explanation [words/lines from the fact statement due to statute is applicable].
Table Prompts Statute Prediction (Template 2). Temperature = 0. max tokens between 40 and 500 was used.

---

Annotated explanation
Generated explanation
In the fight, the accused persons started inflicting injuries to the complainant, and his sons P5 (A-2) gave a pharsi blow on the head of P2, P4 (A-1) caused injury to PI (PW-5) with two jaily blows. Addition- ally, P7 and P6 attacked the complainant with lathi blows on shoulder and left cl- bow respectively and caused several other injuries to the complainant party.
In the fight, the accused persons started inflicting injuries to the complainant, and his sons P5 (A-2) gave a pharsi blow on the head of P2, P4 (A-1) caused injury to P1 (PW-5) with two jaily blows. Additionally, P7 and P6 attacked the complainant with lathi blows shoulder
..the first injury to be an incised wound. Second and third were abrasions on the left shoulder and neck. The fourth injury was lacerated wound on the right parietal area of scalp. On the post-mortem conducted P4, incised wound was found on the parietal area of the scalp.
Punishment for voluntarily causing hurt "On examining P6, he found the first injury to be an incised wound. Second and third were abrasions on the left shoulder and neck. The fourth injury wound on the right parietal area of scalp."
lacerated
...they they conspired with stamp vendors and document writers and other staff to gain monetary benefit and resorted to manipula- tion of registers and got the registration of the documents with old value of the prop- erties, resulting in wrongful gain to them- selves and loss the Government, and thereby cheated the public and the Govern- ment.
The main allegation against the respon- dents was that while they were working as Sub-Registrars in various offices in the State, they conspired with stamp vendors and document writers and other staff to gain monetary benefit and resorted to ma- nipulation of registers and got the regis- tration of the documents with old value of the properties, resulting in wrongful gain to themselves and loss to the Gov- ernment, and thereby cheated the public and the Government.
Table 7: Competent explanations (with statute prediction) generated by LLM. We see that the L.M--genrrted explanations are very close to legal-expen annotated explanations.

---

Annotated explanation
Generated explanation
On full review of the facts is clear that this transaction was voluntarily entered into in order indirectly to facilitate the car- rying on of the business of the company and was made on the ground of commer- cial expediency. It therefore falls within the purview of section 12(2) of the Income- tax Act, 1922 before its amendment in 1939. This being an investment company, it borrowed money and utilised the same for its investments on which it earned in- come, the interest paid by it on the loans will clearly be a permissible deduction un- der section 12( of the Income- tax Act. Whether the loan is taken on an overdraft, debenture
On full review of the facts it clear that this transaction was voluntarily entered into in order indirectly to facilitate the car- rying on of the business of the company and was made on the ground of commer- cial expediency. It therefore falls within the purview of section 12(2) of the Income- tax Act, 1922 before its amendment in 1939. This being an investment company, if borrowed money and utilised the same for its investments on which it earned in- come, the interest paid by it on the loans will clearly be a permissible deduction un- der section 12(2) of the Income-tax Act. Whether the loan is taken on an overdraft, debenture
is fixed deposit makes no difference in law.
is fixed deposit or makes difference in law. The only argu- ment urged against allowing this deduction to be made is that the person who took the debentures was the party who sold the or- dinary shares. It cannot he disputed that if the debentures were held by a third party, the interest payable on the same would be allowable deduction in calculating the total income of the assessee company. What difference does it make if the holder of the debentures a shareholder.
we feel no doubt or difficulty in reaching the conclusion that the intervention of the Industrial Tribunals was intended to be excluded and the matter was intended to be kept within the discretion of the Cen- tral Government far the payment of bonus by the insurers is concerned. Hav- ing regard to the unqualified and absolute prohibition contained in s. 31A(1)(c) it seems to us difficult to hold that the pay- ment of bonus to the employees of Insur- ance Companies is not absolutely condi- tioned by proviso (vii). In the absence of the said provision bonus could have been claimed by Insurance employees, and the effect of the said provision must be to limit the said right to the conditions pre- scribed by it.
The intervention of the Industrial Tri- bunals was intended to be excluded and the matter was intended to be kept within the discretion of the Central Government so far as the payment of bonus by the in- surers is concerned. Having regard to the unqualified and absolute prohibition con- tained in s. A1A())() it seems to us dif- ficult to hold that the payment of bonus to the employees of Insurance Companies is not absolutely conditioned by proviso (vii). In the absence of the said provi- sion no bonus could have been claimed by Insurance employees, and the effect of the said provision must be limit the said right to the conditions prescribed by it. That is why we think that the Tribunal was right in coming to the conclusion that the reference made by the Central Govern- ment is invalid.
Table 8: Competent explanations (with judgment prediction) generated by LLM. We see that the ALM-generated explanations are very close to legal-expert annotated explanations.

---

Template 1 (prediction explanation)
prompt = Task: Given Supreme Court of India case proceeding enclosed in angle brackets 2, your task is to predict the decision of the case (with respect the appelant) and provide an explaination for the decision.
Prediction: Given case proceeding, the task to predict the decision 0 or 1, where the label corresponds to the acceptance of the appeal/petition of the appel- lant/petitioner and the label O corresponds to the rejection of the appeal/petition of the appellan//petitioneer, Explanation: The task is to explain how you arrived at the decision by predicting important sentences that lead to the decision.
Context: Answer in consistent style shown in the following two examples: case_ ans__proceeding # case pesc___nnedding example
Prediction: example prediction
Explanation: # example explanation
case aase_proceeding # case caaseprooceeding example 2
Prediction: example 2 prediction
Explanation: # example explanation
Instructions: Learn from the above given two examples and perform the task for the following case proceeding.
case_proceeding: < (case_proceeding)
Format your output in list format: [prediction, explanation]"""
Table 9: Prompts in Judgment Prediction (Template I: prediction explanation). Temperature used.
Template (prediction only)
prompt Task: Given a Supreme Court of India case proceeding enclosed in angle brackets >, your task is to predict the decision of the case (with respect to the appellant).
Prediction: Given case proceeding, the task is predict the decision 0 where the label corresponds to the acceptance of the appeal/petition of the appel- lant/petitioner and the label corresponds to the rejection the appeal/petition of the appellann/petitioner
Context: Answer a consistent style as shown in the following two examples: case proceeding: # case caas__pooceedig example
Prediction: # example 1 prediction
case case_ coiss__rcceeing # case_prpoee coss___rocedingg example 2
Prediction: # example 2 prediction
Instructions: Learn from the above given two examples and perform the task for the following case proceeding.
case case_prooees uoee_poccediigg < cccas_procceddngg
Give the output predicted case decision as either or 1,!!
Table 10: Prompts in Judgment Prediction (Template 2: prediction only). Temperature = was used.

---

The LLM model tends incorporate statures that not relevant the context; e.g. the case Case description
Actual statutes/explamation
Predictions with Halluci- nations
Observation
Sexual assault case
Constitution 226',
"Constitution_22 226', 'Con-
"Constitution 14',
stitution_ 136'. IIdiian
"Constitution_2 21", 'Con- stitution_32 'Indian Penal Code, 1860_376
Penal Code 1860_376
'Indian Penal Code
murder and the section for murder like 'Indian Penal Code, 1860 302' has been pre- dicted (the correct predic- tions in blue)
"36A,, 'Indian 498A'.
Penal Code 1860
'Indian Penal Code, 1860_302', 'Indian Penal Code 1860_323', 'Indian Penal Code 1860_504 'Indian Penal Code 1860_506', IIndian Penal Code 1860 34', 'Indian Penal Code 1860 307' 'Indian Penal Code 1860_467; 'Indian Penal Code 1860_471', 'Indian Penal Code 1860_400 4066,, 'Indian Penal Code 1860_468' Indian Penal Code 1860_2.. 'Indian Penal Code 1860 409', 'Code of Criminal Proce- dure 1973 482'
Case on appeal for special leave, an excerpt is given as follows: "the respon- dent that the imported gonds fell within this item and were liable to be charged with duty at that Tate.The Custom author- ities, however, considered that the consignment fell within the description ar- ticles plated with gold silver being item which duty was payable per cent., adjudicated the duty this latter ba- sis and thereafter the spondent having filed appeal to [ORG], the levy upheld by order dated ... learned Judge fur- ther held that the interpre- tation that he placed upon item in context of the other entries the Tar- ifT can only reasonably possible for any person take contrary view..."
'Constitution_24 226', 'Con- stitution__13 136'
'Customs Tariff 1975_25
Act,
The model tends to con- sider any statute men- tioned in the fact state- ment as the applicable statute in its response, regardless of its actual relevance or applicability. E.g. here the presence of the terms custom, tar- iff led to wrongly sug- gest 'Customs Tariff Act, 1975 25' the applica- ble statute
and
Table 11: Examples of ballucination in Statute Prediction

---

Case description
Actual
Predictions Halluci-
Observation
statutes/explanation
nations
The following excerpt from a case contains
LLM statute pre-
The LLM predictions and explanations are based on superficial fea- tures rather than in-depth issues the case. E.g. fails to identify that is dowry death case.
Section the Dowry
dictions:
"Constitu-
Prohibition Act, 1961',
expert annotations
tion 226'.
'Constitu-
marked: The ceased PI was daughter PW1. P2 resident village Sanjkheda and she was married to appellant no. P3 son P4 about two and half years prior to the date of incident which took place on 15 September 1991. ...For about six months Pl was treated well but thereafter the accused started asking her to bring Rs. 1,000-1,200 from her purents to meet the household expenses and also for purchasing manure. Whenever P1 went her parental home, she used tell her parents that her husband and mother-in-law (ac- cused appellants) harassing her and used occastonally beat her. Her father PW.1 P2 along with of his relatives the accused house of tried persuade them ill-treat Pl. Thereafter, the accused treated properly but after about four months they again started harassing her. A few days before Nag Pan- chami festival P1 came to her parental home complained accused were not giving her proper food. clothing and even footwear. PW.1 then immediately went to the house of the accused along with some of his relatives. There he saw that Pl was lying dead and froth was coming out of her mouth which indicated that she had consumed some poisonous substance. The Police Patil of the village PW.3 P6 lodged accidental death report 9.00 on 15 September 1991 at the police station....
Scction 30BB in The In-
tion tion_322 hon__2", 'Indian Penal Code, 1860_149", 'Indian Penal Code, 1860_300 307'.
dian Penal Code (Dowry death
'Indian Penal Code, 1860 20003311____ Constitu- tion_32: "The right to move the Supreme Court by uppropriate proceed- ings for the enforcement of the rights conferred by this Part is guaranteed." LLM explanations for the predictions: Consti- tution_226: "This appeal, by special leave, has been preferred against the judgment and order dated 23 Fehruary 2005 of the High Court (Aurangabad Bench)"
Indian Penal Code, 1860_149: "If an offence is committed by any member of unlawful assembly in prosecution of the common object of that assembly". Indian Penal Code, 1860 1860_307: Attempt murder "Whoever does any with such inten- tion knowledge, under such circumstances that, he by act caused death, he would he guilty or murder" Indian Penal Code, 1860_321 "Whoever, except the case pro- vided for by section 334, voluntarily causes hurt, shall be punished with imprisoument of either description for a term which may extend to one year"
went
the
Table 12: Examples of hallucination in Statute Prediction (contd.)

---

Religion specific sections
Statutes
Description
IPC Sections 295 -298
Of Offences relating to Religion
IPC Section 153A
Promoting enmity between different groups on grounds of religion, race, place of birth, residence, language, etc., and doing acts prejudicial to mainte- harmony.
Tentatively religion-specific sections
Statules Act 2000
Description
Provention of Terrorism Act, 2002
Arms act, 1959
Act of the Parliament of India to consolidate and
amend the law relating to arms and ammunition in Criminal conspiracy
order to curb illegal weapons and violence stem-
ming from them
IPC Sections 120A and 120B
IPC Sections 121-130
Of Offences against the State
IPC Scctions 141 153
Of Offences against the Puhlic Tranquillity
Gender-specific sections
Statutes
Description
IPC Sections 375-377
Sexual Offences including Rape and Sodomy
IPC Sections 493 498
Of Offences related to marriage
IPC: Sections 498A
OF Crueity by Husband or Relatives of Husband
IPC Section 304B
Dowry death
Table 13: Statutes related to religion and gender issues. Note that the "Tentatively religion-specific sections" are not directly related religion, but often associated with crimes committed by members of certain religion.
Data
Precision
Recall
F1
No of cases
predietion+explanation on statute predn +
0.0722
0.0629
0.0570
11
expln test dataset (45
cases) (R)
prediction
only statute predn
0.1055
0.2222
0.1353
11
expln test dataset (45 cases) (R)
prediction+explanation on statute predn
0.2804
0.3922
0.2913
34
expln test dataset cases) (N)
prediction
only statute predn
0.1874
0.7622
0.2858
34
expln test dataset (45
cases) (N)
prediction
only statute predn test dataset (245 cases) (R)
0.0439
0.0612
0.0465
50
prediction only
0.1116
0.1327
0.1182
85
statute predn test dataset (245 cases) (G)
prediction
only statute predn test dataset (245 cases) (N)
0.4185
0.5598
0.3860
144
Table 14: Statute Prediction results LLM for cases containing religion and gender-specific sections. Cases with Religion-specift sections, G: Cases with Gender-specifie sections, N: Cases with Neutral sections. The LLMs on N cases outperform those on R and G cases.
Group
Statute Prediction
Judgment Prediction
Male
0.06
0.60
Female
0.11
0.33
Hindu
0l.44
Muslim
0.75
Table 15; Worst Class Influence (WCI) score per group. The higher the value, the more the disparity. There are no values for Hindu/Muslim in Statute Prediction because there were no religion-specific statutes in the statute predn expln test dataset.

---

Table 16: Top 5 cases from Explanation Generation Analysis (on 45 cases)
Case Num-
Actual Statute Involved
Type of actual statute
Avg of Eval Metrie Scores
ber
34
[Constitution 226",
Constitution 21']
constitutional night of HC to writs,
0.7982645277
protection life and personal liberty
[Indian Penal Code, 1860 120')
criminal conspiracy
0.756938956
3
('Indian Penal Code, 1860 321'.
voluntarily causing hurt with weapons, actions done with intention, attempt murder
0.7340507586
dian Penal Code, 1860_34] "Indian
Penal Code, 1860_307', "Indian Penal Code, 1860_324", "Indian Penal Code,
1860_325'|
18
['Constitution_22 __6", "Constitution_1" 14']
constitutional right of HC to issue writs, equal protection of laws rioting. armed with deadly weapons, murder
0.6652548341
16
('Indian Penal Code, 1860_1116600000 148'. 'Indian Penal Code, 1860 147", 'Indian Penal
0.6506418727
Code, 18600_199 "Indian Penal Code,
1860 ]02"]
Table 17: Worst 5 cases from Explanation Generation Analysis (on 45 cases)
Case Num-
Actual Statute Involved
Type of actual statute
Avg Eval Metric Scores
41
'Indian Penal Code, 1860 302"]
punishment for committing murder punishment of criminal conspiracy, caus- ing hurt intentionally
0.08537873928
12
('Indian Penal Code, 1860 120', "Indian Penal Code, 1860_34', 'Indian Penal
0.08882598817
Code, 1860_302']
26
['Constitution_1] 21', 'Constitution_1e 14',
protection of life and personal liberty. equality hefore law, punishment for mur- der
0.10759701
'Indian Penal Code, 1860_302']
['Indian Penal Code, 1860_307', CConstitution 136', 'Indian Penal Code, 1860_302", 'Indian Penal Code, 1860_34'
attempt murder, criminal done several person furtherance of the com-
0.1193747665
mon intention
32
I'Constitution 14', "Constitution 21']
protection of life and personal liberty, cquality before
0. 1.9773117706
Table 18: Top Performing Statutes from Prediction Analysis (on 245 cases)
Actual Statute
Type of actual statute
Statute wise F1 Score
Indian Penal Code, 1860_302
offences affecting life including murder, culpable homicide kidnapping or abducting in order mur- der kidnapping, abducting or inducing woman to compel her for marriage, etc.
0.8253741981
Indian Penal Code, 1860_364
0.7958333333
Indian Penal Code, 1860_366
0.7914893617
Indian Penal Code, 1860_306
abetment of suicide sexual offences including rape, etc.
0.7826086957
Indian Penal Code, 1860_376
0.7402409639

---

Table 19: Worst 5 Performing Statutes from Prediction Analysis (on 245 cases)
Actual Statute
Type of actual statute
Statute wise F1 Score
Indian Penal Code, 1860_467
offences relating documents- forgery of a valuable security, will, etc. offences against property-criminal Breach of Trust
0.4524265645
Indian Penal Code, 1860_400
0.4556451613
Indian Penal Code, 1860_2
introduction punishment of offences committed within India
introduction-pumiihnert
0.4738867955
Constitution 142
article providing unique power to the Supreme Court, to do complete justice between the parties, where, at times, the law or statute may not provide remedy. punishment for possession or carrying of prohibited arms or prohibited ammu- nition
0.4739076155
Arms Act, 1959_22 25
0,4776119403


Reviews:
[{"review_id": "fROWruo1sQ", "reviewer": "Reviewer_Je4i", "summary": "", "questions": "1. For LLM prompting, do you use zero-shot prompting or few-shot prompting? What type of statute information is include into the prompts, e.g., statute number, name or content? Do you use similar cases for prediction statute?\n2. What is the meaning of the “Prediction with explanation” setting for supervised baselines? These baselines, e.g., XLNet, cannot generate explanations.\n3. Why judgment prediction is a binary classification task? It differs from other literature, e.g., \"Legal Judgment Prediction via Topological Learning\" and \"Neural Legal Judgment Prediction in English\"", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We have tried our best to address them and hope the reviewer will now have a better understanding of the work.\n\n\nQ. For LLM prompting, do you use zero-shot prompting or few-shot prompting? \n\nResponse: We have tried both zero-shot (statute prediction) and few-shot (for judgment prediction). The prompts used in this paper are shown in Tables 5, 6, 9 and 10 in the appendix. For the rebuttal, we have also tried few-shot Chain-of-Thought (CoT) prompting    (described later).\n\n\nQ. What type of statute information is include into the prompts, e.g., statute number, name or content? \n\nResponse: Statute ids (unique identifier), title (brief overview of the statute) and description (more detailed description of the statute) are provided in the prompt. Kindly refer to Tables 5 and 6 in the appendix.\n\nQ. Do you use similar cases for prediction statute?\n\nResponse: We have tried training cases, which may not be similar to a current test case, on one of the prompts (Template 1, which did NOT produce better results than Template 2, whose performance is reported in the paper). Kindly refer to Tables 5 and 6 in the appendix.\n\n\nQ. What is the meaning of the “Prediction with explanation” setting for supervised baselines? These baselines, e.g., XLNet, cannot generate explanations.\n\nResponse: \n\nWe agree with the reviewer. However, we draw attention to the fact that generating explanations along with predictions is a challenging task and to our knowledge, this has not been tried in Law-AI applications. The setup will possibly be similar to that of Task 2 as reported by Lamm et al. in QED: A Framework and Dataset for Explanations in Question Answering (https://arxiv.org/abs/2009.06354). However, a major challenge in the legal domain is building large enough datasets (with explanations manually annotated) for such complex tasks. We look forward to implementing such a model in the future after expanding the current dataset. Nevertheless, with our explanation annotations, we have been able to apply CoT-style prompting on our dataset (described later) to generate explanations with the prediction.\n\n\nQ. Why judgment prediction is a binary classification task? It differs from other literature, e.g., \"Legal Judgment Prediction via Topological Learning\" and \"Neural Legal Judgment Prediction in English\"\n\nResponse: We have used the CJPE setup by Malik et al. 2021 where they have treated judgement prediction as a binary classification task.\n\nAdditional comments:\n\n1. About the lack of novelty:\n\nTo our knowledge, the prompts used in this paper are novel as they have not been used before on the two tasks presented in the paper (particularly for Indian datasets). Yu et al., 2022 (cited in the paper) used zero-shot Chain-Of-Thought (CoT) prompting for statute prediction, where they first generated explanation and then used it for prediction; this did not produce the best results (than without explanations) and also they did not present any analysis of the quality of explanations. We, however, have integrated prediction and explanation as a part of the prompts (in both the templates 1 and 2 by stating additionally “extract words or lines from the case due to which the predicted statutes are applicable”; kindly see Tables 5 and 6 in the appendix). \n\nFor statute prediction, to adhere to the more conventional few-shot CoT prompting (Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. 2022) where the predictions with explanations are a part of few-shot prompting training, for the rebuttal, we have designed a few-shot prompt for statute prediction as below:\n\n<start of prompt>\n\nprompt = f\"\"\"Given a fact statement as a Question delimited by triple backticks (```) and multiple choice options representing the names of statutes, predict the applicable statutes from the Statute Options, and the explanation for each of the applicable statute containing lines from the Question delimited by triple backticks(```). Example is given for your reference. The lecture provides a description of each statute, and your task is to select the statutes from Statute Options that best align with the words from the given Question. Choose the most relevant statutes and lines that match the given context. Statute options (Answer must always be from these) and Lecture (description of statute) are same for each Question. Answer must always be from Statute Options.\n\nExample :\n\nFact statement: \"These appeals are directed against the judgment of a High Court whereby an appeal and a criminal revision were disposed of. The appellants were found guilty and sentenced to undergo various terms of sentences. The Criminal Appeal was filed by three appellants questioning the conviction and sentence as recorded. Complainant filed a revision petition stating that she was entitled to compensation. Background facts giving rise to the trial are essentially as follows: ..\"\n\nStatute options (Answer must always be from these):\n\n1. \"Constitution_226\"\n2. \"Constitution_136\"\n...\n\n\nLecture (description of statute):\n\n1.\"Constitution_226\":\nTitle: Power of High Courts to issue certain writs\nDescription: (1) Notwithstanding anything in Article 32 every High Court shall have powers, throughout the territories in relation to which it exercise jurisdiction, to issue to any person or authority, including in appropriate cases, any Government, within those territories directions, orders or writs, including writs in the nature of habeas corpus, mandamus, prohibitions, quo warranto and certiorari, or any of them, for the enforcement of any of the rights conferred by Part III and for any other purpose\n\n2. \"Constitution_136\":\nTitle: Special leave to appeal by the Supreme Court\nDescription: (1) Notwithstanding anything in this Chapter, the Supreme Court may, in its discretion, grant special leave to appeal from any judgment, decree, determination, sentence or order in any cause or matter passed or made by any court or tribunal in the territory of India. (2) Nothing in clause (1) shall apply to any judgment, determination, sentence or order passed or made by any court or tribunal constituted by or under any law relating to the Armed Forces.\n\n...\n\nAnswer : ['Indian Penal Code, 1860_321', 'Indian Penal Code, 1860_324', 'Indian Penal Code, 1860_302', 'Indian Penal Code, 1860_34']\n\nExplanation (words from the fact statement for each of applicable statute):\n\nStatute: \"Indian Penal Code, 1860_34\"\nLines : \"the appellants felt offended and  when the members of the complainant party came forward and obstructed the appellant from doing the work and restrained them from pulling out the pipe.\"\n\nStatute: \"Indian Penal Code, 1860_321\"\nLines : \" attacked the complainant party.  the first injury to be an incised wound. Second and third were abrasions on the left shoulder and neck. The fourth injury was a lacerated wound on the right parietal area of scalp. On the post-mortem conducted on P4, an incised wound was found on the parietal area of the scalp\"\n\nStatute: \"Indian Penal Code, 1860_324\"\nLines : \" P4 was attacked ...attacked the complainant party. \"\n\n#####\n\nFormat of Response : Answer containing a list of applicable statutes from the Statute Options above ,and Explanation (containing lines for each of the applicable statutes from the Question) for the Fact Statement delimited by triple backticks (```).\nNote: Response must not include Description from Lecture(description of statute) as Explanation. Do not include statute options in your response.  Lines must be from the Question delimited by triple backticks (```).\n\nQuestion: ```{Fact_Statement}```\n\n\"\"\"\n<end of prompt>\n\nPlease note that above prompt provides not only the fact text and statute text, but also the extractive explanations from the fact text to train the LLM. For this experiment, 8 cases were used for training the few-shot prompt, while the remaining 37 were used as test cases. *This produced Macro-Precision: 0.41, Macro-Recall:  0.42 and Macro-F1: 0.36, which outperforms the LLM (Template 2) that recorded Macro-Precision: 0.17, Macro-Recall:  0.68 and Macro-F1: 0.27 and InLegalBERT (the best performing baseline) that recorded Macro-Precision: 0.18, Macro-Recall:  0.18 and Macro-F1: 0.17, on the same test set.* However, the quality of explanations generated by this CoT prompt is considerably inferior to those produced by LLM (Template 2). This possibly leads us to the conclusion that few-shot CoT-style prompting, although being trained with explanations, does not necessarily produce high-quality explanations for the test set.\n\nKindly note that, for judgement prediction and explanation we have already adhered to the more conventional few-shot CoT prompting (Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. 2022) where the predictions with explanations are a part of few-shot prompting training. Kindly refer to Table 9 in the appendix to note that for a given case, the explanation along with the actual prediction is provided in the few-shot prompt training. To our knowledge, few-shot CoT prompting has been under-explored for legal tasks. On the whole, we feel, our proposed prompts are novel with regard to the tasks presented in the paper.\n\n\n2. About the size of test dataset being small:\n\nWe acknowledge the concern of the reviewer. As already mentioned in the limitations, due to the token limitation and high subscription charges of paid LLMs (the better results were produced by the paid models), we could not do experiments on bigger test datasets. Also, legal expert annotations are expensive and time-consuming to obtain. However, to our knowledge, we have used some of the biggest legal expert annotated datasets available for Indian cases, in this paper. We also will release an expert annotated dataset with this paper, upon acceptance.\n\n\n3. About the paper reporting the performance of one LLM model for each task:\n\nWe acknowledge the concern of the reviewer. Since we used subscribed version of GPT, we could not perform extensive experiments with all the models like text-davinci-003 due to high cost. We did some initial experiments with randomly chosen sets of 5 cases and tried them on various models and chose the one that produced the best results. The paid GPT models handsomely outperformed the free models like BLOOM, FLAN-T5 and LLaMA. Among paid GPT models, we chose the numerically better performing model though the performances were not necessarily significantly different.\n\n4.  About the results being doubtful: \n\nFor statute prediction, the pretrained models (InLegalBERT, LEGAL-BERT and XLNet) were actually fine-tuned (and not trained) on 18021 cases (lines 111-115 in the paper). Kindly also note that InLegalBERT is pre-trained on Indian legal documents and hence is expected to have the necessary context to act as a strong baseline for both our tasks.\n\n5.  About the  prompt design being not well illustrated: \n\nWe could not extensively explain the prompts in the main paper due to the lack of space. However, we have tried our best to motivate our prompts in the main paper and provided the exact prompts used in the appendix (kindly see Tables 5, 6, 9 and 10 in the appendix). We have also used a new prompt on few-shot Chain-Of-Thought in the rebuttal with the performance (already discussed), which we will add to the camera-ready version, if accepted.\n\n6. About Typos Grammar Style And Presentation Improvements: We thank the reviewer for pointing these out. We will correct this in the camera-ready version, if accepted.\n\nAdditional experiments:\n\nFor judgement prediction task, we reversed the order of the prediction label and the explanation in the training prompt (that is first explanation and then prediction label) in the few-shot training and requested the corresponding output from the LLM. We have drawn inspiration from the paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. 2022, Section 3.3 Ablation Study, Chain of thought after answer, in order to analyse the sequential reasoning embodied in the explanation and its dependence/consequence on the prediction ability of the LLM model beyond just extracting knowledge from the input text.\nThe output explanations were obtained for 80% and were logically reasoned. But, still, only 20% was predicted correctly by the model. This therefore led to a deterioration in the performance when compared with our previous prompt style where prediction *preceded* explanation.\n\nTherefore, from these sets of experimentations, we can safely say that the prompting style used by us previously in the paper was already in alignment with the state-of-the art style (like CoT).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work focuses on the legal tasks of India, which is not extensively studied.\n2. This work analyzes the quality of LLM generated explanations and compares them with expert annotations.\n", "reasons_to_reject": "1. There is not novelty in methodology. \n2. The size of test dataset is small.\n3. The paper claims to evaluate several LLMs, but only report the performance of one model for each task.\n4. The results are doubtful. For statute prediction, the paper indicates that davinci is better than GPT-3.5 turbo (Line 102), while for judgment prediction, GPT-3.5 is better (Line 220-222). Besides, the baseline performances in Table 1 are extremely low (macro prec around 0.1), while these baselines are claimed to be trained on 18021 samples. There could be huge distribution shifts between the training and test data. However, this is not discussed at all.\n5. The prompt design is not well illustrated.\n", "questions_to_authors": "1. For LLM prompting, do you use zero-shot prompting or few-shot prompting? What type of statute information is include into the prompts, e.g., statute number, name or content? Do you use similar cases for prediction statute?\n2. What is the meaning of the “Prediction with explanation” setting for supervised baselines? These baselines, e.g., XLNet, cannot generate explanations.\n3. Why judgment prediction is a binary classification task? It differs from other literature, e.g., \"Legal Judgment Prediction via Topological Learning\" and \"Neural Legal Judgment Prediction in English\"", "paper_topic_and_main_contributions": "This paper evaluates LLMs and supervised baselines on two India legal tasks: statute prediction and judgment prediction.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 28: wrong citation format. Remove the parentheses.  \nLine 123-124: “is was” -> “is”\n"}, "extra_text": {}}, {"review_id": "eAOGLiGyM1", "reviewer": "Reviewer_ZNGP", "summary": "", "questions": "Question A:\n\nIt is interesting to see that, in Table 3, LLM's performance on the judegement prediction without explanation is considerablely lower than prediction with explanation. Did you try prediction with explanation for all 256 cases? would it have the same trend (better performance than prediction only (256 cases))?\n\nQuestion B:\n\nHave you tried SOTA extractive QA systems for the explanation task since the task seems comparable?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We have tried our best to address them and hope that the reviewer will now have a better understanding of the paper.\n\nQuestion A:\nIt is interesting to see that, in Table 3, LLM's performance on the judgment prediction without explanation is considerablely lower than prediction with explanation. Did you try prediction with explanation for all 256 cases? would it have the same trend (better performance than prediction only (256 cases))?\n\n\nResponse: No, we tried it in only 56 cases as we have expert annotations for only 56 cases. In the future, we look forward to expanding the annotated set and performing the experiments as suggested by the reviewer.\n\n\nQuestion B:\nHave you tried SOTA extractive QA systems for the explanation task since the task seems comparable?\n\nResponse: \n\nTo our knowledge, the prompts used in this paper are novel as they have not been used before on the two tasks presented in the paper. Yu et al., 2022 (cited in the paper) used zero-shot Chain-Of-Thought (CoT) prompting for statute prediction, where they first generate explanation and then used it for prediction; this did not produce the best results (than without explanations) and also they did not present any analysis of the quality of explanations. We, however, have integrated prediction and explanation as a part of the prompts (in both the templates 1 and 2 by stating additionally “extract words or lines from the case due to which the predicted statutes are applicable”; kindly see Tables 5 and 6 in the appendix). \n\nFor statute prediction, to adhere to the more conventional few-shot CoT prompting (Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. 2022) where the predictions with explanations are a part of few-shot prompting training, for the rebuttal, we have designed a few-shot prompt for statute prediction as below:\n\n<start of prompt>\n\nprompt = f\"\"\"Given a fact statement as a Question delimited by triple backticks (```) and multiple choice options representing the names of statutes, predict the applicable statutes from the Statute Options, and the explanation for each of the applicable statute containing lines from the Question delimited by triple backticks(```). Example is given for your reference. The lecture provides a description of each statute, and your task is to select the statutes from Statute Options that best align with the words from the given Question. Choose the most relevant statutes and lines that match the given context. Statute options (Answer must always be from these) and Lecture (description of statute) are same for each Question. Answer must always be from Statute Options.\n\nExample :\n\nFact statement: \"These appeals are directed against the judgment of a High Court whereby an appeal and a criminal revision were disposed of. The appellants were found guilty and sentenced to undergo various terms of sentences. The Criminal Appeal was filed by three appellants questioning the conviction and sentence as recorded. Complainant filed a revision petition stating that she was entitled to compensation. Background facts giving rise to the trial are essentially as follows: ..\"\n\nStatute options (Answer must always be from these):\n\n1. \"Constitution_226\"\n2. \"Constitution_136\"\n...\n\n\nLecture (description of statute):\n\n1.\"Constitution_226\":\nTitle: Power of High Courts to issue certain writs\nDescription: (1) Notwithstanding anything in Article 32 every High Court shall have powers, throughout the territories in relation to which it exercise jurisdiction, to issue to any person or authority, including in appropriate cases, any Government, within those territories directions, orders or writs, including writs in the nature of habeas corpus, mandamus, prohibitions, quo warranto and certiorari, or any of them, for the enforcement of any of the rights conferred by Part III and for any other purpose\n\n2. \"Constitution_136\":\nTitle: Special leave to appeal by the Supreme Court\nDescription: (1) Notwithstanding anything in this Chapter, the Supreme Court may, in its discretion, grant special leave to appeal from any judgment, decree, determination, sentence or order in any cause or matter passed or made by any court or tribunal in the territory of India. (2) Nothing in clause (1) shall apply to any judgment, determination, sentence or order passed or made by any court or tribunal constituted by or under any law relating to the Armed Forces.\n\n...\n\nAnswer : ['Indian Penal Code, 1860_321', 'Indian Penal Code, 1860_324', 'Indian Penal Code, 1860_302', 'Indian Penal Code, 1860_34']\n\nExplanation (words from the fact statement for each of applicable statute):\n\nStatute: \"Indian Penal Code, 1860_34\"\nLines : \"the appellants felt offended and  when the members of the complainant party came forward and obstructed the appellant from doing the work and restrained them from pulling out the pipe.\"\n\nStatute: \"Indian Penal Code, 1860_321\"\nLines : \" attacked the complainant party.  the first injury to be an incised wound. Second and third were abrasions on the left shoulder and neck. The fourth injury was a lacerated wound on the right parietal area of scalp. On the post-mortem conducted on P4, an incised wound was found on the parietal area of the scalp\"\n\nStatute: \"Indian Penal Code, 1860_324\"\nLines : \" P4 was attacked ...attacked the complainant party. \"\n\n#####\n\nFormat of Response : Answer containing a list of applicable statutes from the Statute Options above ,and Explanation (containing lines for each of the applicable statutes from the Question) for the Fact Statement delimited by triple backticks (```).\nNote: Response must not include Description from Lecture(description of statute) as Explanation. Do not include statute options in your response.  Lines must be from the Question delimited by triple backticks (```).\n\nQuestion: ```{Fact_Statement}```\n\n\"\"\"\n<end of prompt>\n\nPlease note that above prompt provides not only the fact text and statute text, but also the extractive explanations from the fact text to train the LLM. For this experiment, 8 cases were used for training the few-shot prompt, while the remaining 37 were used as test cases. *This produced Macro-Precision: 0.41, Macro-Recall:  0.42 and Macro-F1: 0.36, which outperforms the LLM (Template 2) that recorded Macro-Precision: 0.17, Macro-Recall:  0.68 and Macro-F1: 0.27 and InLegalBERT (the best performing baseline) that recorded Macro-Precision: 0.18, Macro-Recall:  0.18 and Macro-F1: 0.17, on the same test set*. However, the quality of explanations generated by this CoT prompt is considerably inferior to those produced by LLM (Template 2). This possibly leads us to the conclusion that few shot CoT-style prompting, although being trained with explanations, does not necessarily produce high-quality explanations for the test set.\n\n\nKindly note that, for judgement prediction and explanation have already adhered to the more conventional few-shot CoT prompting (Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. 2022) where the predictions with explanations are a part of few-shot prompting training. Kindly refer to Table 9 in the appendix to note that for a given case, the explanation along with the actual prediction is provided in the few-shot prompt training. To our knowledge, few-shot CoT prompting has been under-explored for legal tasks. On the whole, we feel, our proposed prompts are novel with regard to the tasks presented in the paper.\n\nAdditional comments:\n\nRegarding the paper being not easy to follow: We will try our best to improve the paper in the camera-ready version, if accepted. We are apologetic for keeping the bias and fairness section (Section 4) short due to the paucity of space in the main paper. However, we have tried to connect the discussion in the main paper with the supporting results in the appendix (kindly refer to Sections D and E in the appendix). To our knowledge, this is among the first works that discuss bias, fairness and ethics w.r.t. statute prediction and judgement prediction esp. on Indian cases.\n\nRegarding the paper lacking technical clarity: We have mostly kept the hyper-parameters to their default values. Since we used subscribed version of GPT, we could not perform extensive experiments with all the models like text-davinci-003 due to high cost. We did some initial experiments with randomly chosen sets of 5 cases and tried them on various models and chose the one that produced the best results. The paid GPT models handsomely outperformed the free models like BLOOM, FLAN-T5 and LLaMA. Among paid GPT models, we chose the numerically better performing models though the performances were not necessarily significantly different.\n\nAbout the related work being insufficient and lack of comparison with SOTA (LLM?) models: Due to the paucity of space in the main paper, we have restricted to citing only the directly relevant works. We will try to add a more detailed literature study in the camera-ready version, if accepted. In this rebuttal, we have added a qualitative comparison of a close prior work. \n\nAbout Typos Grammar Style And Presentation Improvements: We thank the reviewer for pointing these out. We will correct this in the camera-ready version, if accepted.\n\nAdditional experiments:\n\nFor judgement prediction task, we reversed the order of the prediction label and the explanation in the training prompt (that is first explanation and then prediction label) in the few-shot training and requested the corresponding output from the LLM. We have drawn inspiration from from the paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. 2022, Section 3.3 Ablation Study, Chain of thought after answer, in order to analyse the sequential reasoning embodied in the explanation and its dependence/consequence on the prediction ability of the LLM model beyond just extracting knowledge from the input text.\nThe output explanations were obtained for 80% and were logically reasoned. But, still, only 20% was predicted correctly by the model. This therefore led to a deterioration in the performance when compared with our previous prompt style where prediction *preceded* explanation.\n\nTherefore, from these sets of experimentations, we can safely say that the prompting style used by us previously in the paper was already in alignment with the state-of-the art style (like CoT).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper evaluates the recent GPT 3.5 models on the legal domain tasks.\n\n2. The annotated explanations, though small, could be helpful for other researchers who are interested in Indian legal cases in the community.\n\n3. The analysis of generated explanations by GPT 3.5 compared to annotated explanations by legal experts, provides insights into the potential bias in the LLMs generations.", "reasons_to_reject": "1. The paper is not easy to follow. For example, the introduction of the dataset is not self-contained. And the analysis of bias also deserves more discussion.\n\n2. The paper lacks clarity in the technical part as well, e.g., how do you select the hyper-parameters since there are no development set? Line 100, \"We observed that the best results on our dataset are provided by text-davinci-003 (paid version) and so all results reported in this paper are produced by this model.\" How does text-davinci-003 compare to other models? Does it significantly outperform others or do they have comparable results?\n\n3. The authors do not include a good brief of related work in the legal domain, or a comparison with previous SOTA models.", "questions_to_authors": "Question A:\n\nIt is interesting to see that, in Table 3, LLM's performance on the judegement prediction without explanation is considerablely lower than prediction with explanation. Did you try prediction with explanation for all 256 cases? would it have the same trend (better performance than prediction only (256 cases))?\n\nQuestion B:\n\nHave you tried SOTA extractive QA systems for the explanation task since the task seems comparable?", "paper_topic_and_main_contributions": "This paper examines the performance of recent large language models, specifically GPT 3.5 series from OpenAI, on the legal domain tasks. They evaluate the GPT 3.5 for two tasks, statute prediction and judegement prediction. For the multi-label classification task of statute prediction, they evaluated text-davinci-003 on 45 texts from FIRE AILA 2019 (Bhattacharya et al., 2019a) and 200 random cases from Indian Supreme Court. For the binary classification task of judegement prediction (allowed or dismissed), they evaluate GPT 3.5 turbo on 256 cases from Malik et al. (2021). They also manually annotated a small number of cases to find the explanations in the context. The authors compare the performance of GPT 3.5 with several fine-tuned language models such as BERT for the classification tasks. The experimental results show that GPT 3.5 performs better than fine-tuned models on statute prediction, but worse on judgement prediction. The authors also discuss the evidence of gender and region bias in the GPT predictions and generated explanations, and further convey ethical concerns of deploying LLMs in the legal domain.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Some spacing issues, e.g., spacing above \"5 Conclusion\" is too small."}, "extra_text": {}}, {"review_id": "uIMgMmAhja", "reviewer": "Reviewer_8zfw", "summary": "", "questions": "A.  I assume in all places training LLM refers to few-shot examples provided to LLM when prompting. Please clarify this terminology and refine the terminology. (Lien 104, 191, 209)\n\nB. Statute prediction : Line 136-141: In template 2, do you provide any few-shot examples  (Examples of fact statement and statutes?) It only provides statutes and their description ? In that case, why are they called as training in the prompt ? \n\nC. Statute prediction : If LLM for statute prediction are provided with the information about text of statutes, why not employ statute based baselines to compare, rather than fact text only bases? Better baselines would be to consider label text during training to obtain label-aware attention sort of, for fair comparison with LLM's. It would be also interesting to observe what LLM produces if just given statute id without description. It can test whether LLM's already have knowledge of Indian legal statutes.\n\nD. Statute prediction Explanation results - Why not prompt LLM to generate explanation which are extractive in nature rather than abstractive? This might have made evaluation more easier and to draw better conclusions. Though BERT Scores have been shown in Legal summarization works (Shukla et al. 2022) not to reflect true score given they are not trained on legal text specifically.\n\nE. Judgement Prediction: Why are few-shot examples provided in this setting ? (which is mentioned as training and I assume them to be few-shot examples) Did you try with zero-shot way and they did not perform well ? Any specific reason to consider few-shot in this case and not in statute prediction case?\n\nF. Line 248-250: Do you mean judgement prediction instead of statute prediction ?\n\nG. I assume Malik et al. 2021 Dataset of judgement prediction has extractive explanations. Why not prompt LLM for extractive explanations ?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We have tried our best to address them and hope that now the reviewer will have a better understanding of the paper.\n\nA. I assume in all places training LLM refers to few-shot examples provided to LLM when prompting. Please clarify this terminology and refine the terminology. (Lien 104, 191, 209)\n\n\nResponse: We have indeed used few-shot where some examples were shown to the LLM, and based on which a prediction was sought for a test case. We will clarify this in the camera-ready version, if accepted.\n\nB. Statute prediction : Line 136-141: In template 2, do you provide any few-shot examples (Examples of fact statement and statutes?) It only provides statutes and their description ? In that case, why are they called as training in the prompt ?\n\nResponse: We apologize for the confusion. In fact, for template 2 there was no training (training was for Template 1). We will correct this in the camera ready, if accepted.\n\n\nC. Statute prediction : If LLM for statute prediction are provided with the information about text of statutes, why not employ statute based baselines to compare, rather than fact text only bases? Better baselines would be to consider label text during training to obtain label-aware attention sort of, for fair comparison with LLM's. \n\nResponse: \n\nWe thank the reviewer for the comment. For the rebuttal, we have now covered this by designing a transformer, viz. BERT base model (uncased) model that takes into account both the fact text and statute (label) text. In the context of the paired sentence task, the 512 token limit was shared equally between the facts and statutes, with each segment being allowed a maximum of 256 tokens. The two texts were separated by the [SEP] token as is recognized by BERT for such tasks. Taking upto 150 positive pairs and an equal number of negative pairs for each statute yielded around 22k training samples. For the test set, the pairs were created between a fact text and each statute text. The framework leveraged the BERT base model (uncased) for effectuating binary classification, discerning between the fact text and individual statute texts. The model underwent training over 100 epochs, with the most optimal outcomes observed at the 29th epoch. Note that the binary results for each test fact, statute pair was converted to a multi-label setup (used in our experiments as reported in Table 1), where for each test fact we have the prediction for each of the candidate statutes. \n\nThe model produced a *macro-precision: 0.07,  macro-recall:  0.28, macro-f1: 0.10 on 245 test cases. On 45 test cases (used for prediction and explanation), the results are macro-precision: 0.02, macro-recall: 0.04,  macro-f1: 0.03*. Clearly, these results are inferior to the LLM results in Table 1. This can possibly ascribed to the partial context captured by a BERT model due to the token limitation. In the future, we will use more advanced models for the fact-statute paired text prediction task.\n\nC: It would be also interesting to observe what LLM produces if just given statute id without description. It can test whether LLM's already have knowledge of Indian legal statutes.\n\n\nResponse: The statute ids are usually like S1, S2 etc. which do not contain any context of the actual statute used (e.g. Indian Penal Code 302). That is why using the textual description of the statute (as used in Template 2) proved to be an effective strategy.\n\n\nD. Statute prediction Explanation results - Why not prompt LLM to generate explanation which are extractive in nature rather than abstractive? This might have made evaluation more easier and to draw better conclusions. Though BERT Scores have been shown in Legal summarization works (Shukla et al. 2022) not to reflect true score given they are not trained on legal text specifically.\n\n\nResponse: \n\nThough we did not explicitly state the LLM to produce extractive summary (we said in the prompt “Your response should include…….along with the words or lines from the fact statement due to which the statute is applicable for each of the statute applicable”), the LLM did produce extractive explanations from within the fact text on most of the occasions. Kindly refer to Table 7 in the appendix where some annotated and LLM-generated explanations are shown. E.g. \n\nwhile the following was the annotated explanation:\n\n“In the fight, the accused persons started inflicting injuries to the complainant, and his sons P5 (A-2) gave a pharsi blow on the head of P2, P4 (A-1) caused injury to P1 (PW-5) with two jaily blows. Additionally, P7 and P6 attacked the complainant with lathi blows on shoulder and left elbow respectively and caused several other injuries to the complainant party.”\n\nthe generated explanation was:\n\n“In the fight, the accused persons started inflicting injuries to the complainant, and his sons P5 (A-2) gave a pharsi blow on the head of P2, P4 (A-1) caused injury to P1 (PW-5) with two jaily blows. Additionally, P7 and P6 attacked the complainant with lathi blows on shoulder”\n\nFor statute prediction, to adhere to the more conventional few-shot CoT prompting (Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. 2022) where the predictions with explanations are a part of few-shot prompting training, for the rebuttal, we have designed a few-shot prompt for statute prediction as below. Here we have directed the LLM in the prompt to specifically extract explanations from the fact text. This did not necessarily produce better results as although the explanations were always from the fact text, they did not match well with the gold standard annotations.\n\n<start of prompt>\n\nprompt = f\"\"\"Given a fact statement as a Question delimited by triple backticks (```) and multiple choice options representing the names of statutes, predict the applicable statutes from the Statute Options, and the explanation for each of the applicable statute containing lines from the Question delimited by triple backticks(```). Example is given for your reference. The lecture provides a description of each statute, and your task is to select the statutes from Statute Options that best align with the words from the given Question. Choose the most relevant statutes and lines that match the given context. Statute options (Answer must always be from these) and Lecture (description of statute) are same for each Question. Answer must always be from Statute Options.\n\nExample :\n\nFact statement: \"These appeals are directed against the judgment of a High Court whereby an appeal and a criminal revision were disposed of. The appellants were found guilty and sentenced to undergo various terms of sentences. The Criminal Appeal was filed by three appellants questioning the conviction and sentence as recorded. Complainant filed a revision petition stating that she was entitled to compensation. Background facts giving rise to the trial are essentially as follows: ..\"\n\nStatute options (Answer must always be from these):\n\n1. \"Constitution_226\"\n2. \"Constitution_136\"\n...\n\n\nLecture (description of statute):\n\n1.\"Constitution_226\":\nTitle: Power of High Courts to issue certain writs\nDescription: (1) Notwithstanding anything in Article 32 every High Court shall have powers, throughout the territories in relation to which it exercise jurisdiction, to issue to any person or authority, including in appropriate cases, any Government, within those territories directions, orders or writs, including writs in the nature of habeas corpus, mandamus, prohibitions, quo warranto and certiorari, or any of them, for the enforcement of any of the rights conferred by Part III and for any other purpose\n\n2. \"Constitution_136\":\nTitle: Special leave to appeal by the Supreme Court\nDescription: (1) Notwithstanding anything in this Chapter, the Supreme Court may, in its discretion, grant special leave to appeal from any judgment, decree, determination, sentence or order in any cause or matter passed or made by any court or tribunal in the territory of India. (2) Nothing in clause (1) shall apply to any judgment, determination, sentence or order passed or made by any court or tribunal constituted by or under any law relating to the Armed Forces.\n\n...\n\nAnswer : ['Indian Penal Code, 1860_321', 'Indian Penal Code, 1860_324', 'Indian Penal Code, 1860_302', 'Indian Penal Code, 1860_34']\n\nExplanation (words from the fact statement for each of applicable statute):\n\nStatute: \"Indian Penal Code, 1860_34\"\nLines : \"the appellants felt offended and  when the members of the complainant party came forward and obstructed the appellant from doing the work and restrained them from pulling out the pipe.\"\n\nStatute: \"Indian Penal Code, 1860_321\"\nLines : \" attacked the complainant party.  the first injury to be an incised wound. Second and third were abrasions on the left shoulder and neck. The fourth injury was a lacerated wound on the right parietal area of scalp. On the post-mortem conducted on P4, an incised wound was found on the parietal area of the scalp\"\n\nStatute: \"Indian Penal Code, 1860_324\"\nLines : \" P4 was attacked ...attacked the complainant party. \"\n\n#####\n\nFormat of Response : Answer containing a list of applicable statutes from the Statute Options above ,and Explanation (containing lines for each of the applicable statutes from the Question) for the Fact Statement delimited by triple backticks (```).\nNote: Response must not include Description from Lecture(description of statute) as Explanation. Do not include statute options in your response.  Lines must be from the Question delimited by triple backticks (```).\n\nQuestion: ```{Fact_Statement}```\n\n\"\"\"\n<end of prompt>\n\nPlease note that above prompt provides not only the fact text and statute text, but also the extractive explanations from the fact text to train the LLM. For this experiment, 8 cases were used for training the few-shot prompt, while the remaining 37 were used as test cases. *This produced Macro-Precision: 0.41, Macro-Recall:  0.42 and Macro-F1: 0.36, which outperforms the LLM (Template 2) that recorded Macro-Precision: 0.17, Macro-Recall:  0.68 and Macro-F1: 0.27 and InLegalBERT (the best performing baseline) that recorded Macro-Precision: 0.18, Macro-Recall:  0.18 and Macro-F1: 0.17, on the same test set.* However, the quality of explanations generated by this CoT prompt is considerably inferior to those produced by LLM (Template 2). This possibly leads us to the conclusion that few-shot CoT-style prompting, although being trained with explanations, does not necessarily produce high-quality explanations for the test set.\n\n\nE. Judgement Prediction: Why are few-shot examples provided in this setting ? (which is mentioned as training and I assume them to be few-shot examples) Did you try with zero-shot way and they did not perform well ? Any specific reason to consider few-shot in this case and not in statute prediction case?\n\nResponse: Yes, we did try zero-shot and noted that it did not perform well. We will add this to the camera-ready version, if accepted.\n\n\n\nF. Line 248-250: Do you mean judgement prediction instead of statute prediction?\n\nResponse: Yes. We deeply apologize for the mistake and we will correct it in the camera-ready version, if accepted.\n\n\n\nG. I assume Malik et al. 2021 Dataset of judgement prediction has extractive explanations. Why not prompt LLM for extractive explanations ?\n\nResponse: Though the prompt was not directed at producing extractive explanations, it did produce extractive explanations on most of the occasions. Kindly refer to Table 8 in the appendix which shows examples of annotated and generated explanations. E.g. the following is an expert-annotated explanation:\n\n“On a full review of the facts it is clear that this transaction was voluntarily entered into in order indirectly to facilitate the carrying on of the business of the company and was made on the ground of commercial expediency. It therefore falls within the purview of section 12(2) of the Incometax Act, 1922 before its amendment in 1939. This being an investment company, if it borrowed money and utilised the same for its investments on which it earned income, the interest paid by it on the loans will clearly be a permissible deduction under section 12(2) of the Income- tax Act. Whether the loan is taken on an overdraft, or is a fixed deposit or on a debenture makes no difference in law.”\n\nwhile the corresponding LLM-generated explanation is:\n\n“On a full review of the facts it is clear that this transaction was voluntarily entered into in order indirectly to facilitate the carrying on of the business of the company and was made on the ground of commercial expediency. It therefore falls within the purview of section 12(2) of the Incometax Act, 1922 before its amendment in 1939. This being an investment company, if it borrowed money and utilised the same for its investments on which it earned income, the interest paid by it on the loans will clearly be a permissible deduction under section 12(2) of the Income-tax Act. Whether the loan is taken on an overdraft, or is a fixed deposit or on a debenture makes no difference in law. The only argument urged against allowing this deduction to be made is that the person who took the debentures was the party who sold the ordinary shares. It cannot be disputed that if the debentures were held by a third party, the interest payable on the same would be an allowable deduction in calculating the total income of the assessee company. What difference does it make if the holder of the debentures is a shareholder”\n\nNevertheless, for the rebuttal, we created another prompt stating in the prompt:\n“Extract the explanation output explicitly from the case proceeding text and not from outside.”\n\nIn this process, however, the model's prediction ability was heavily compromised with 60 %  wrong predictions even though the sentences in the explanations matched with the Gold standard ones (not necessarily better than the earlier results).\n\n\nAdditional comments:\n\nAbout the small datasize: \n\nWe acknowledge the concern of the reviewer. As already mentioned in the limitations, due to the token limitation and high subscription charges of paid LLMs, we could not do experiments on bigger test datasets. Also, legal expert annotations are expensive and time-consuming to obtain. However, to our knowledge, we have used some of the biggest legal expert annotated datasets available for Indian cases, in this paper. We also will release an expert annotated dataset with this paper, upon acceptance.\n\nAbout data contamination issue if the LLM's have already seen the cases:\n\nKindly note that the training set of cases (for few-shot) is always disjoint from the test set of cases. So, there should not be any data contamination.\n\nAbout analysis on generated explanations:\n\nIn the rebuttal we have provided some analysis of the generated explanations esp. w.r.t. the extractive nature of the same. Kindly refer to Tables 7 and 8 in the appendix where we have shown some examples of generated explanations along with the annotated explanations. We will try to add more insights on the explanations in the camera-ready version, if accepted.\n\nAbout study of hallucinations:\n\nWe request the reviewer to kindly refer to Tables 11 and 12 in the appendix where we have tried to capture the prominent examples of hallucinations observed for the statute prediction and explanation task. The hallucinations mostly comprise: generation of wrong statutes based on superficial features like words found in the fact text and not always judging the applicability of the same for the case. We could not provide a more detailed analysis of hallucinations, which we identified in the course of our experiments, in the main paper due to the lack of space.\n\nAbout Typos Grammar Style And Presentation Improvements: We thank the reviewer for pointing these out. We will correct this in the camera-ready version, if accepted.\n\n\nAdditional experiments:\n\nFor judgement prediction task, we reversed the order of the prediction label and the explanation in the training prompt (that is first explanation and then prediction label) in the few-shot training and requested the corresponding output from the LLM. We have drawn inspiration from the paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Wei et al. 2022, Section 3.3 Ablation Study, Chain of thought after answer, in order to analyse the sequential reasoning embodied in the explanation and its dependence/consequence on the prediction ability of the LLM model beyond just extracting knowledge from the input text.\nThe output explanations were obtained for 80% and were logically reasoned. But, still, only 20% was predicted correctly by the model. This therefore led to a deterioration in the performance when compared with our previous prompt style where prediction *preceded* explanation.\n\nTherefore, from these sets of experimentations, we can safely say that the prompting style used by us previously in the paper was already in alignment with the state-of-the art style (like CoT).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is a interesting direction to explore capabilities of LLM in two specific legal tasks related to Indian Supreme Court cases and establish certain evidences on their performance.", "reasons_to_reject": "On the whole, some more analysis on generated explanations or different types of hallucinations etc can make it more of an interesting read. I am skeptical if the findings hold true given data contamination issue if the LLM's have already seen the cases etc (unless some probing is carried out to back these claims).", "questions_to_authors": "A.  I assume in all places training LLM refers to few-shot examples provided to LLM when prompting. Please clarify this terminology and refine the terminology. (Lien 104, 191, 209)\n\nB. Statute prediction : Line 136-141: In template 2, do you provide any few-shot examples  (Examples of fact statement and statutes?) It only provides statutes and their description ? In that case, why are they called as training in the prompt ? \n\nC. Statute prediction : If LLM for statute prediction are provided with the information about text of statutes, why not employ statute based baselines to compare, rather than fact text only bases? Better baselines would be to consider label text during training to obtain label-aware attention sort of, for fair comparison with LLM's. It would be also interesting to observe what LLM produces if just given statute id without description. It can test whether LLM's already have knowledge of Indian legal statutes.\n\nD. Statute prediction Explanation results - Why not prompt LLM to generate explanation which are extractive in nature rather than abstractive? This might have made evaluation more easier and to draw better conclusions. Though BERT Scores have been shown in Legal summarization works (Shukla et al. 2022) not to reflect true score given they are not trained on legal text specifically.\n\nE. Judgement Prediction: Why are few-shot examples provided in this setting ? (which is mentioned as training and I assume them to be few-shot examples) Did you try with zero-shot way and they did not perform well ? Any specific reason to consider few-shot in this case and not in statute prediction case?\n\nF. Line 248-250: Do you mean judgement prediction instead of statute prediction ?\n\nG. I assume Malik et al. 2021 Dataset of judgement prediction has extractive explanations. Why not prompt LLM for extractive explanations ?", "paper_topic_and_main_contributions": "The paper explores LLM's ability on two tasks namely statute prediction and judgement prediction of Indian Supreme Court cases. They also evaluate LLM's from explainability perspective on those tasks and compare them with expert generated rationales for both the tasks. While judgement prediction + explanation dataset is derived prior work of Malik et al. 2021, they enhanced statute dataset of Paheli et al. 2019 for statute prediction with a legal-expert-explanations. They also obtain separate set of cases dealing with religion and gender specific cases and observed disparities in performance and bias exhibited by LLM's. Their findings reveal that LLMs  outperform in statute prediction and underperform in judgment prediction. LLM's explanation are moderate for statute prediction and often good for judgment prediction. Finally, LLM display gender and religion bias explanations and prediction performance disparities.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 28-29: Please use \\citealt for better formatting (Yu et al., 2022)(Nguyen et al.,2023) \n\nLine 70-77 : Please specifically mention how annotations are obtained? phrase-level or sentence level ? in-text annotations or free text explanations ? (Table 4 in appendix shows them, please qualify them in writing)"}}]

Author rebuttal:

Meta-review: This paper evaluates the performance of large language models (LLMs) on two legal tasks: statute prediction and judgment prediction. The reviewers appreciate the focus on the Indian legal system, which is not extensively studied, and the comparison of LLM-generated explanations with expert annotations. However, they raise concerns about the lack of novelty in methodology, the small size of the test dataset, and the clarity of the paper. The authors' rebuttal addresses these concerns, providing detailed explanations and additional experiments. The reviewers also suggest improvements in the presentation and request further clarification on certain aspects of the study. Overall, the paper is seen as a valuable contribution to the field, despite some weaknesses.
Decision: Accept-Findings