Legal Judgment Reimagined: PredEx and the Rise of Intelligent AI Interpretation in Indian Courts (2024)

Abstract: In the era of Large Language Models (LLMs), predicting judicial outcomes poses significant challenges due to the complexity of legal proceedings and the scarcity of expertannotated datasets. Addressing this, we introduce Prediction with Explanation (PredEx), the largest expert-annotated dataset for legal judgment prediction and explanation in the Indian context, featuring over 15,000 annotations. This groundbreaking corpus significantly enhances the training and evaluation of AI models in legal analysis, with innovations including the application of instruction tuning to LLMs. This method has markedly improved the predictive accuracy and explanatory depth of these models for legal judgments. We employed various transformer-based models, tailored for both general and Indian legal contexts. Through rigorous lexical, semantic, and expert assessments, our models effectively leverage PredEx to provide precise predictions and meaningful explanations, establishing it as a valuable benchmark for both the legal profession and the NLP community.

heading: 1 Introduction
text: In the evolving landscape of legal technology, the integration of Artificial Intelligence (AI) into the judicial system has emerged as a frontier of immense potential and challenge. The Indian judiciary, characterized by a significant backlog of cases1, stands to benefit substantially from advancements in AIassisted legal decision-making. This paper introduces a novel approach to facilitating the legal decision-making process, specifically focusing on the Indian context, in conjunction with explanations for the same. Our work builds upon two foundational studies: (Malik et al., 2021) and (Vats
∗These authors contributed equally to this work †Corresponding author 1https://www.nytimes.com/2024/01/13/world/
asia/india-judicial-backlog.html
et al., 2023). Our objective is to develop an advanced system capable of predicting judicial outcomes and providing cogent explanations for these predictions. This system leverages a newly compiled dataset, PredEx, of approximately 15,000 annotated legal documents, considerably larger than those used in previous research, particularly in terms of its volume and depth of annotations. Table 1 compares PredEx with other popularly used corpora for legal judgment prediction, highlighting the uniqueness of our dataset in terms of its size and focus on providing explanations. Unlike previous works that predominantly focused on predicting legal outcomes, PredEx introduces the largest annotated dataset for judgment prediction and explanation in the Indian legal context, addressing a critical gap in legal AI research. This dataset enables us to train and refine sophisticated machine learning models, particularly focusing on instruction tuning, to achieve unprecedented accuracy and relevancy in legal judgment prediction.
Our work is distinguished by several key contributions that mark significant advancements in the field of legal AI:
1. Publication of a New Annotated Dataset (PredEx): We introduce the largest annotated dataset to date for judgment prediction and explanation in the Indian legal context. This dataset surpasses previous efforts in both scope and depth, providing a more robust foundation for training AI models in legal judgment prediction.
2. Exploration of Instruction-Tuning on Large Language Models (LLMs): Our work goes beyond the traditional methods of fine-tuning conventional transformers. We delve into instruction tuning on LLMs, an approach not extensively explored in previous research, to enhance prediction accuracy.
4296
3. Expert Evaluation and Validation: We employ a rigorous evaluation process, utilizing a Likert score scale to assess the efficacy of our system. This evaluation, conducted on a sample of 50 documents, provides critical insights into the performance of our AI models compared to human expert standards.
Our research aims to provide a comprehensive and sophisticated AI-based system for legal judgment prediction and explanation, specifically tailored for the Indian judiciary. This system is not only a technological advancement but also a step towards addressing the pressing challenge of case backlog in India. We believe our contributions will not only enhance the efficiency and transparency of the legal process but also pave the way for further research and development in AI-assisted legal technology. For the sake of reproducibility, we have made the PredEx dataset and the code for our prediction and explanation models accessible via a GitHub link2. Additionally, for convenience, we have uploaded the data3 and models4 to Huggingface.

heading: 2 Related Work
text: The field of Legal Natural Language Processing (NLP) has witnessed significant advancements, with researchers exploring a variety of complex tasks within the legal domain. A prominent area of
2https://github.com/ShubhamKumarNigam/PredEx 3huggingface.co/collections/L-NLProc/predex-models 4huggingface.co/collections/L-NLProc/predex-datasets
focus has been Legal Judgment Prediction (LJP), where the goal is to predict the outcomes of legal cases based on their facts and contexts. Seminal works in this area include the contributions of (Zhong et al., 2020), (Malik et al., 2021), (Aletras et al., 2016), (Chen et al., 2019) (Long et al., 2019), (Xu et al., 2020) (Yang et al., 2019a), and (Chalkidis et al., 2019). These studies have laid the groundwork for understanding the nuances involved in automating legal decision-making processes.
Another key area of research has been the application of Large Language Models (LLMs) in the legal field. The versatility of models such as GPT, BLOOM, FLAN-T5, and LlaMA has been demonstrated in various studies, including those by (Vats et al., 2023) (Blair-Stanek et al., 2023) and (Katz et al., 2023), highlighting their potential in tasks ranging from statutory reasoning to judgment prediction. However, challenges remain in terms of the acceptability and reliability of LLMs in highstakes legal contexts. The LegalEval (Modi et al., 2023) workshop further exemplifies the diversity and complexity of legal NLP research, especially on legal judgment prediction and explanation.
Our research utilizes advanced Large Language Models and a comprehensive dataset to create a system that predicts and explains judicial outcomes, enhancing legal text processing and transparency. This work supports legal practitioners and the public, especially in complex systems like India’s, and sets the stage for future AI advancements in legal technology.

heading: 3 Task Description
text: Our research project aims to advance the Court Judgment Prediction and Explanation (CJPE) task, incorporating insights and methodologies from both (Malik et al., 2021) and (Vats et al., 2023). The CJPE task involves two key sub-tasks: Prediction and Explanation. These tasks are performed sequentially, addressing the critical need to predict legal judgments and provide explanations for these predictions. To provide a visual representation of our task framework, Figure 1 illustrates the overall process of Court Judgment Prediction and Explanation (CJPE) as employed in our study. This figure includes the sequential steps of prediction and explanation. To demonstrate an Indian case structure in Table 8 and how explanations are derived from judicial judgments, refer to Figure 2 in the Appendix. For context, you can view the original case text on which this annotation is based.5 This example showcases the detailed process by which our annotators have identified and extracted these critical parts, reflecting the essence of judicial reasoning in each case. Similarly, an Example of an Indian Case Structure Table 8.
Prediction Task: The core of the CJPE task is to predict the outcome of a legal case based on the case proceedings. Given a document D that includes the case proceedings from the Supreme Court of India (SCI), the task is to predict the decision y ∈ {0, 1}, where ‘1’ signifies the acceptance of the appeal or petition by the appellant or petitioner, and ‘0’ indicates its rejection.
Explanation Task: The second part of the CJPE task involves explaining the predicted decision. Our approach is two-fold, integrating methodolo-
5https://indiankanoon.org/doc/97694707/
gies from both referenced papers:
1. Identifying Key Sentences (ILDC for CJPE approach): Similar to the approach in (Malik et al., 2021), we focus on identifying and highlighting key sentences or segments within the case proceedings that significantly contributed to the predicted outcome. This method relies on extracting specific parts of the text that are directly related to the decision, providing an evidence-based explanation.
2. Generating Abstract Reasoning (LLMs approach): Drawing from the approach in (Vats et al., 2023), we attempt to generate more abstract reasoning for the prediction. This involves providing zero and few-shot examples to the LLMs to guide them in generating explanations that are not just tied to specific text excerpts but also encompass broader reasoning and legal principles.
Additionally, we introduce a novel aspect to this task by training the LLMs specifically for both prediction and explanation. This training is tailored to enable the models to understand and process legal texts more effectively, improving their capability to predict outcomes and generate relevant explanations.

heading: 4 Dataset
text: In our research, we introduce “PredEx”, significantly differentiating itself from existing datasets in Legal Natural Language Processing (L-NLP), particularly in the context of the Indian judiciary. This dataset is designed to address the limitations of previous datasets, which primarily focused on prediction tasks and offered limited annotations for explanations.

heading: 4.1 Dataset Compilation
text: In the Data Compilation process, we initially gathered a substantial corpus of about 20,000 court judgments randomly from the Supreme Court of India and various High Courts, utilizing the IndianKanoon website6, a legal search engine widely recognized for its comprehensive database of Indian legal documents. The corpus underwent a meticulous annotation process, where our team of legal experts focused on annotating explanations for the judgments. These annotations involved identifying and highlighting key sentences or segments within the case proceedings that significantly influenced the predicted outcomes, as well as providing reasoning for the judgments. Through this process, the original corpus was distilled to approximately 16,000 case files, each richly annotated with expert legal explanations.
Our approach in compiling the PredEx dataset was to randomly select cases, ensuring a broad representation across various types of judgments and legal decisions. This method was deliberately chosen to avoid bias towards any specific temporal aspect or domain. By adopting a random selection process, we aimed to capture the diverse nature of legal cases in the Indian judiciary system. This diversity was crucial to cover various aspects of law and legal decision-making, thereby enhancing the representativeness and applicability of our dataset for training AI models in legal judgment prediction and explanation.
Subsequent to the annotation phase, we undertook a preprocessing step to refine the dataset further. This preprocessing involved the removal of cases that were either too brief or where the final decision segments were challenging to discern. Such preprocessing is crucial for ensuring the quality and consistency of the data, particularly for training robust and reliable AI models; otherwise, it could introduce noise or bias into the model training. As a result of this preprocessing, the total number of case files in our dataset was reduced to 15,222 and is further divided into training and testing sets. We adopted an 80-20 split ratio for this purpose, ensuring a substantial volume of data for model training while still retaining a robust set for testing. Specifically, the training set consists of 12,178 documents, and the test set comprises 3,044 documents.
In terms of balancing the test set, special atten6https://indiankanoon.org/
tion was given to ensure fairness and representativeness in model evaluation. We carefully curated the test set to include a diverse range of case outcomes, such as different types of judgments and legal decisions. This diversity was not just in terms of the nature of cases but also in terms of the outcomes - for instance, balancing cases where appeals were accepted versus those that were dismissed. Such a balanced composition is crucial in avoiding biases towards any particular type of judgment and ensures that our AI models are tested against a wide spectrum of legal scenarios. This balanced nature of the test set is particularly important for maintaining the validity of our experiments and for ensuring the reliability and generalizability of our model’s performance. These carefully processed and curated case files now form the core of our PredEx dataset, offering a rich resource for the Court Judgment Prediction and Explanation (CJPE) task. Detailed statistics of the final dataset, post-preprocessing, are presented in the following Table 2.

heading: 4.2 Annotation Process

heading: 4.2.1 Expert Involvement
text: We engaged a team of 10 legal experts, primarily law students in their 3rd and 4th years, from various Indian law colleges. These experts were selected based on their academic standing and understanding of legal processes, ensuring high-quality annotations.

heading: 4.2.2 Annotation Timeline
text: The annotation process spanned from April 1, 2022, to October 30, 2023. This extensive period allowed for meticulous and thorough annotation, considering the complexity and detail required in legal document analysis.

heading: 4.2.3 Work Allocation
text: In our annotation process, each student was assigned around 30 judgment documents weekly,
striking a balance between efficiency and the need for thorough, accurate annotations. This workload allocation enabled students to devote adequate time to each document, fostering precise and insightful annotations.

heading: 4.2.4 Role of Student Annotators
text: The role of our student annotators was to meticulously identify and extract specific segments from the judgments that were pivotal to the judge’s reasoning, rather than interpreting or analyzing these segments with their own legal reasoning. Their task was to pinpoint these key sections accurately, ensuring that the extracts faithfully represented the judicial reasoning as stated in the case documents. This extractive approach was critical to maintain the integrity and authenticity of the annotations, allowing the dataset to accurately reflect the content of the original legal texts without the introduction of subjective interpretations by the annotators.

heading: 4.2.5 Annotation Quality Control Mechanism
text: To guarantee the accuracy and consistency of the annotations, we implemented a comprehensive quality control system. Initially, each document was reviewed by a single annotator to ensure a consistent interpretation of the judicial content. Recognizing the complexity of legal texts and the potential for subjective interpretation, we established several layers of review to enhance the reliability of our annotations:
• Senior Expert Review: Any disagreements or uncertainties in annotations were promptly escalated to a specialized review panel led by senior legal experts. These experts not only provided additional scrutiny but also mediated discrepancies among the initial annotations. Their extensive experience in legal practice and education enabled them to provide decisive and informed resolutions to any contentious or ambiguous annotations.
• Regular Training and Meetings: To further ensure consistency across annotations, regular training sessions and review meetings were conducted. These sessions served to align annotators on the legal framework and annotation criteria, reducing variability and enhancing the uniformity of the annotation process. Training included detailed discussions on identifying key legal arguments and the rationale within the judgments, which are
critical for both the prediction and explanation aspects of our dataset.
This rigorous quality control process has ensured that our dataset meets high standards of reliability and validity. The annotations not only reflect the factual content of the legal decisions but also the detailed reasoning behind these judgments, making our dataset a robust resource for training and evaluating AI models in legal judgment prediction and explanation.

heading: 4.2.6 Focus on Prediction and Explanations
text: Diverging from previous datasets that primarily concentrate on the task of prediction, our PredEx dataset spans both prediction and explanations. The annotations in our dataset serve a dual purpose. Firstly, they identify the outcomes of the cases, fulfilling the prediction aspect. More importantly, they go a step further by providing detailed explanations behind these outcomes. These explanations elucidate the rationale or the legal reasoning that underpins the judgments. This dual emphasis on prediction and explanations fills a significant void in existing legal datasets. Typically, in other datasets, the aspect of explanation is either absent or not explored in depth. By contrast, PredEx enriches the field of legal AI with comprehensive annotations that shed light not just on what the judicial decisions are, but crucially, why these decisions were made. This focus on explanations is particularly vital, as it contributes to a more transparent and understandable AI-driven legal decision-making process.

heading: 4.2.7 Largest Explainable Dataset
text: As a result of this extensive and detailed annotation process, we are releasing what is arguably the largest annotated dataset for legal judgment prediction and explanation in the Indian context. The size and comprehensiveness of this dataset set it apart from existing datasets in the field.
Our dataset represents a significant advancement in legal NLP, particularly for research and applications pertaining to the Indian judiciary. By providing a large-scale, richly annotated dataset that encompasses both prediction and explanation, we aim to facilitate more nuanced and sophisticated AI models capable of understanding and interpreting legal texts in a manner akin to human legal experts. This dataset is not only a resource for advancing AI technology in the legal domain but also a step
towards enhancing transparency and accountability in AI-assisted legal decision-making.

heading: 5 Methodology
text: This section outlines the methodology employed in our research for the tasks of Judgment Prediction and Judgment Prediction with Explanation.

heading: 5.1 Judgment Prediction

heading: 5.1.1 Language Model based
text: In our approach, we utilized several language models including InCaseLaw, InLegalBERT (Paul et al., 2023), XLNet (large) (Yang et al., 2019b), and Roberta (large) (Liu et al., 2019) as baselines for binary classification. Due to the length constraints of complete judgments, which exceed the token capacity of these models, we adopted a chunking strategy. Each document was divided into 512- token chunks using a moving window approach with a 100-token overlap to preserve textual context. For model training, we used a batch size of 16, the Adam optimizer (Kingma and Ba, 2014), and a learning rate of 2e-6. The training was conducted over 5 epochs on the PredEx train dataset. The remaining hyperparameters were set to their default values as provided by the HuggingFace library.

heading: 5.1.2 Large Language Model based
text: For utilizing LLMs in prediction, we employed two strategies: one involving prediction instructions only, and the other combining prediction with explanation instructions. Various models like Zephyr (Tunstall et al., 2023), Gemini 1.0 Pro (Team et al., 2023), Llama-2-7B (Touvron et al., 2023), and Llama-2-7B with instruction-tuning were used. We followed the prompts and instruction-tuning approaches published by (Vats et al., 2023) in a fewshot setup, and used the PredEx training data for instruction-tuning.

heading: 5.1.3 Prompts Used
text: For inference, we utilized prompts published by (Vats et al., 2023) and employed Template 2 in a zero and few-shot setup exclusively for prediction tasks, as detailed in Table 9 in the Appendix. These prompts provide a case description alongside a gold standard prediction label, directing the LLM to generate the judicial decision. For instruction tuning, we employed our custom prompts for prediction tasks, which are listed in Table 10 in the Appendix of this paper.

heading: 5.1.4 Instruction-Set
text: We developed 16 instruction sets using ChatGPT4 (DALL-E), validated by legal experts and then used for PredEx training data for instruction tuning. Given the token limit of 4096 in LLMs, we selected the last 1000 words from each document to fit within this constraint. This choice is supported by findings from (Malik et al., 2021) who achieved optimal results using the last 512 tokens of judgments. The input comprised the case proceedings and case decision and a random selection of instructions, with the output being the case outcome prediction. For a comprehensive understanding of our methodology and the full range of instructions used, we have included the complete list of all 16 instruction sets in Table 11 located in the Appendix of this paper.

heading: 5.2 Judgment Prediction with Explanation
text: For this task, we employed the same LLMs with settings similar to the Judgment Prediction task, but with different instructions focusing on both prediction and explanation.

heading: 5.2.1 Prompts used
text: In our approach, we also adopted prompts from (Vats et al., 2023), specifically utilizing Template 1 for the combined task of judgment prediction and explanation. This template, detailed in Table 9 in the Appendix, expands on the prediction-only format of Template 2 by including an explanation component. Here also, for instruction tuning, we employed our custom prompts for prediction with explanation tasks, which are listed in Table 10 in the Appendix of this paper. In this setup, the LLM is instructed not only to predict the outcome of a case but also to articulate the reasoning behind the decision. The precise formatting and examples of how predictions and explanations are structured and solicited from the LLM can be viewed in the referenced table.

heading: 5.2.2 Instruction-Set
text: For judgment prediction with explanation, we created 16 instruction sets using ChatGPT4 (DALLE), also validated by legal experts. This time, the input included case proceedings, decisions, and reasoning, with randomly chosen instructions, and the output being the case outcome prediction with explanation. For a comprehensive view of all 16 instruction sets, we have included the full list in Table 11 in the Appendix of this paper.

heading: 6 Evaluation Metrics
text: In our study, We report Macro Precision, Macro Racall, Macro F1, and Accuracy on the PredEx judgment prediction test dataset and employ a multifaceted approach to evaluate the performance of our models on the PredEx judgment explanation test dataset. Our evaluation metrics encompass both quantitative and qualitative methods, ensuring a thorough assessment of the model’s capabilities in both prediction and explanation tasks.
1. Lexical Based Evaluation: We utilized lexical similarity metrics such as Rouge scores (Rouge-1, Rouge-2, and Rouge-L) (Lin, 2004), BLEU (Papineni et al., 2002), and METEOR (Banerjee and Lavie, 2005). These metrics assess the similarity between the generated explanations and the reference texts based on word overlap and order, providing an insight into the lexical accuracy of the model outputs.
2. Semantic Similarity Based Method: To capture the semantic essence of the generated explanation, we employed BERTScore (Zhang et al., 2020), which measures the semantic similarity between the generated and ground truth explanations. Additionally, we used BLANC (Vasilyev et al., 2020) to estimate the quality of generated explanations in the absence of a gold standard, offering a perspective on the model’s ability to generate semantically rich and contextually relevant text.
3. Expert Evaluation: Human evaluation played a crucial role in our assessment. Legal experts reviewed the explanations generated by the models and rated them on a 1–5 Likert scale based on their accuracy, relevance, and completeness. The criteria for the rating scale were as follows:
1. The explanation is entirely incorrect or fails to provide any relevant information.
2. The model’s response is irrelevant or shows misunderstanding of the case judgment.
3. The explanation is partially accurate but misses critical details.
4. The response is comparable and relevant to the ground truth.
5. The explanation is completely accurate, relevant, and potentially superior to the expert’s explanation.

heading: 7 Results and Analysis

heading: 7.1 Judgment Prediction
text: Our experiments, as detailed in Table 3, reveal interesting insights into the performance of various models on the PredEx test data. Notably, Roberta emerges as the top performer, outstripping even the Large Language Models (LLMs). This suggests that traditional language models might be more adept at analyzing and predicting outcomes in legal documents compared to generative-based models. Even among the generative models, the few-shot Llama-2-7B model surpassed the finetuned Zephyr model, which is surprising given Zephyr’s supervised fine-tuning (SFT) approach and its reinforcement learning training on general corpora. It appears that the Llama-2-7B models, both instruction-tuned for prediction and prediction with explanation tasks, show promising results in this domain.

heading: 7.2 Judgment Prediction with Explanation
text: The results, as shown in Table 4, provide valuable insights into the performance of machinegenerated explanations versus expert explanations
across a range of models. These assessments include lexical-based, semantic, and expert evaluations on the PredEx test data. To augment our evaluation process, we also incorporated a comparison with the instruction-tuned models on the 54 ILDC_expert (Malik et al., 2021) dataset. This dataset, to our knowledge, represents the largest collection of legal expert-annotated data available for Indian cases, offering a valuable benchmark for assessing the performance of our models. This multi-faceted evaluation offers a comprehensive understanding of the models’ capabilities in generating explanations.
Given the expense and time required to obtain legal expert annotations, we carefully sampled 50 cases from our dataset for Likert score evaluations by legal experts. This sampling strategy was chosen to provide a representative and manageable subset of cases for in-depth expert analysis, while also considering the practical constraints associated with expert-driven evaluations.

heading: 7.3 Lexical Based Evaluation
text: In the lexical-based evaluation, the performance of LLMs in generating explanations shows that verbatim matches are not at a satisfactory level. However, it’s important to note that these metrics, while valuable, do not fully encapsulate the models’ proficiency in analyzing cases, predicting outcomes, and generating reasoning. Thus, we turn to Semantic Similarity-Based Evaluation and Expert Score Evaluation for a more thorough assessment.

heading: 7.4 Semantic Evaluation
text: Semantic evaluation, particularly the BERTScore, indicates better alignment of the explanations with the gold standard, suggesting a good semantic understanding in the generated explanations. The Llama-2-7B model with instruction-tuning for prediction and explanation tasks excels in semantic similarity. Nevertheless, lower scores in opensource models point to challenges in accurately generating case analysis, predictions, and reasoning. It’s crucial to recognize that generative models may exhibit hallucination issues, not entirely captured by this metric, necessitating manual evaluation by legal experts for a more complete assessment.

heading: 7.5 Expert Evaluation
text: Evaluating generative models in the legal judgment prediction task with explanation requires domainspecific expertise. The expert evaluation, detailed in Table 5, shows that the Llama-2-7B model with instruction-tuning performs notably well, although it sometimes produces truncated or repetitive responses. Despite these limitations, the instructiontuned model demonstrates fewer non-factual responses and better overall explanation quality compared to other pre-trained models. Interestingly, models with well-designed prompts for explanation generation displayed enhanced performance without instances of hallucination.
The expert ratings, as reflected in Table 5, further underscore the efficacy of our instruction-
tuned model, which even surpasses the quality of explanations provided by legal professionals (achieving a rating score of 4). This underlines the potential of generative models, particularly those leveraging our instruction-tuning approach, in generating accurate and relevant legal explanations. The average expert rating scores, presented in Table 4, corroborate the superiority of our generative models over other approaches.

heading: 7.6 Hallucination
text: We address the issue of hallucinations in modelgenerated text, a common challenge in using large language models for generating legal judgments. Hallucinations refer to instances where the model generates false or irrelevant information that is not supported by the input data. To combat this, we have implemented a fine-tuning strategy that significantly reduces these errors. A detailed comparative analysis in the Appendix B showcases these strategies and their effectiveness. This analysis demonstrates how fine-tuning and instructiontuning specifically tailored to the legal domain can mitigate hallucinations, providing clearer, more accurate, and legally coherent outputs.

heading: 8 Conclusions and Future Work
text: We introduced PredEx, the largest dataset for legal judgment prediction and explanation in this context, marking a significant advancement over previous datasets. Our research explored instruction tuning on Large Language Models (LLMs), showing promise in improving prediction accuracy and explanatory depth.
Looking ahead, our focus will be on training Indian Legal domain-specific Large Language Models. This approach will ensure that the models are
ingrained with domain-specific knowledge, crucial for tasks like legal judgment prediction with explanations. Furthermore, we plan to undertake Supervised Fine-Tuning (SFT) on various downstream tasks, including the judgment prediction with explanation task. Another key objective will be to incorporate contextual understanding into the models to mitigate issues like hallucinated responses, a common challenge with generative models.
The question remains as we advance in this field: How ready is the State-of-the-Art to aid in explainable judgment prediction? Our future efforts aim to answer this question by refining the capabilities of AI in legal applications, making a significant contribution to the evolving field of AI-assisted legal judgment prediction and explanation. The ultimate goal is to develop AI tools that can not only alleviate the backlog in the Indian judiciary but also deliver justice efficiently and transparently. To further enhance the accuracy and reliability of our system, we plan to implement a Reinforcement Learning from Human Feedback (RLHF) pipeline. This pipeline aims to refine the model’s predictions and explanations based on human feedback, ensuring that the outputs align more closely with expert legal understanding and reasoning. The inclusion of RLHF represents a significant advancement in developing AI systems for legal judgment prediction and explanation, as it allows for continuous improvement and adaptation based on real-world feedback and interactions.
Limitations
Our study faced several significant limitations that impacted our approach and findings. A primary constraint was the token limitation and high subscription charges for paid cloud services, which restricted our ability to perform inference and finetuning on larger parametric models, particularly those with 70B or 40B parameters. This limitation likely curtailed our exploration of the full capabilities of these advanced models, which could have provided deeper insights or enhanced performance.
Another critical limitation was the resourceintensive nature of obtaining legal expert annotations. Due to the high costs and extensive time required for this process, it was not feasible for us to obtain expert evaluations for the entire PredEx test dataset. Consequently, we opted to sample 50 random documents for expert review and Likert score evaluations. While necessary, this ap-
proach potentially limits the breadth and depth of our expert-based evaluation, as it does not encompass the entire dataset.
In terms of the effectiveness of Large Language Models (LLMs) in the legal domain, our findings suggest that while these models are proficient in conversational contexts, their applicability in logic or knowledge-intensive tasks like legal judgment prediction and explanation is less convincing. Analyzing lengthy legal documents and generating predictions with explanations poses a significant challenge for generative-based models. This is particularly true in cases where the models need to process and understand complex legal reasoning and contexts.
Furthermore, the performance of the opensource baseline model, which was intended to jointly predict and generate explanations, did not meet our expectations. This underperformance could be attributed to the token limitations imposed during our study. By only using the last 1000 tokens of documents for fine-tuning, there is a possibility that the model did not fully grasp the entire context of the cases. Moreover, our fine-tuned models frequently produced truncated responses due to the 512-token limit set for generation. This limitation may have hindered the models’ ability to generate comprehensive and nuanced explanations.
Lastly, the pre-trained models used in our study inherently lacked detailed knowledge specific to Indian legal cases. Even after undergoing tuning processes, these models struggled to generate explanations that paralleled the depth and specificity of human-like legal reasoning. This shortfall highlights the challenge of adapting general AI models to specialized domains such as law, where domainspecific knowledge and reasoning are crucial.
These limitations underscore the challenges in applying LLMs to complex and specialized tasks like legal judgment prediction and explanation. They also highlight the necessity for continued research and development efforts aimed at enhancing the capabilities of AI models in interpreting and understanding legal documents and contexts.
Ethics Statement
Ethical conduct was a cornerstone in our research, especially considering the sensitive nature of the data and the methodologies involved. In collecting and annotating the PredEx dataset, we ensured that the law students involved in the annotation process
were treated fairly and compensated appropriately. Their consent was obtained for all participation, and while they made significant contributions to the dataset, they are not listed as authors of this paper. This distinction is made to acknowledge their contribution while also maintaining the academic integrity of the publication process.
Significantly, the senior legal expert who played a pivotal role in mentoring the annotation process, as well as providing guidance on the Likert rating system and evaluating the generated explanations for both the PredEx and ILDC datasets, is credited as one of the authors of this paper. This inclusion reflects the expert’s substantial intellectual contribution to the research, in line with ethical norms and authorship guidelines in academic publishing.
Moreover, for the computational resources used in this study, we adhered to ethical standards by duly paying the subscription fees for Google Colab Pro. This payment ensured legitimate access to the necessary paid cloud services, which were instrumental in the development and testing of our AI models. We believe in supporting the services and platforms that enable research like ours, and this includes the responsible financial support of technology providers.
In summary, our approach to ethics encompassed not only the respectful and fair treatment of all individuals involved but also the adherence to legal and financial obligations. This comprehensive ethical stance underscores our commitment to conducting research that is not only innovative and impactful but also responsible and respectful of all parties involved.

heading: A Experimental Setup and Hyper-parameters
text: Our experimental setup was designed to optimize the performance of instruction fine-tuning on LLMs and to accurately assess their capabilities in legal judgment prediction and explanation tasks. We utilized two cores of NVIDIA A100-PCIE40GB with 126GB RAM of 32 cores for instruction fine-tuning, ensuring powerful computational resources for processing and model training. In addition to the dedicated hardware, we employed a Google Colab Pro subscription having A100 Hardware accelerator for conducting inference and other experiments. This platform provided us with the necessary flexibility and scalability for our extensive experimentation.
Regarding the model training specifics, we finetuned the LLMs for 5 epochs. This duration was chosen to balance between adequately training the models on our PredEx dataset and preventing overfitting. During our experiments, we encountered a common issue with generative models – the tendency to hallucinate and repeat sentences. To address this, we implemented a post-processing step after inference. This step involved selecting the first occurrences of the decision and explanation parts from the model outputs and omitting any subsequent repetitions. This approach helped us refine the output quality, ensuring the results to be coherent and concise.
However, it is important to note that certain LLMs did not yield inference results in some cases. In such instances, we excluded those cases from our evaluation process. This decision was made to maintain the integrity and accuracy of our experimental findings, as including non-inferential results could have skewed our overall assessment of the models’ performance.
Overall, our experimental setup was carefully crafted to provide a robust and reliable framework for evaluating the efficacy of instruction-tuned LLMs in the context of legal judgment prediction and explanation.

heading: B Hallucination Examples
text: B.1 Pre-trained vs Fine-tuned
In the Appendix, we conduct a thorough comparison between pre-trained and fine-tuned models to demonstrate the reduction of hallucinations through our fine-tuning methods. Table 7 in the
Appendix presents this comparative analysis, illustrating the impact of instruction-tuning on legal judgment prediction with explanation tasks in our PredEx dataset. This section specifically focuses on how instruction-tuning can mitigate the issues of inaccuracies often found in the responses generated by pre-trained Large Language Models (LLMs).
B.2 Zephyr-7b-beta and Gemini pro For a practical illustration of hallucination issues, Table 6 in the Appendix provides examples from the Zephyr-7b-beta and Gemini Pro models. Notably, Zephyr-7b-beta sometimes produced contradictory outcomes (‘accepted’ and ‘rejected’) for the same case with identical explanations. Additionally, an instance was observed where Gemini Pro produced a decision in Chinese, despite the prompt and the rest of the proceedings being in English. These examples highlight the critical need for model training and tuning specific to the dataset and task to ensure the reliability and accuracy of AI-generated legal predictions and explanations.
Monday, 1 April 2024 9:31 PM

heading: Future Work
text: The question remains as we advance in this field: How ready is the State-of-the-Art to aid in explainable judgment prediction? Our future efforts aim to answer this question by refining the capabilities of AI in legal applications, making a significant contribution to the evolving field of AI-assisted legal judgment prediction and explanation. The ultimate goal is to develop AI tools that can not only alleviate the backlog in the Indian judiciary but also deliver justice efficiently and transparently. To further enhance the accuracy and reliability of our system, we plan to implement a Reinforcement Learning from Human Feedback (RLHF) pipeline. This pipeline aims to refine the model’s predictions and explanations based on human feedback, ensuring that the outputs align more closely with expert legal understanding and reasoning. The inclusion of RLHF represents a significant advancement in developing AI systems for legal judgment prediction and explanation, as it allows for continuous improvement and adaptation based on real-world feedback and interactions. Limitations
Our study faced several significant limitations that impacted our approach and findings. A primary constraint was the token limitation and high subscription charges for paid cloud services, which restricted our ability to perform inference and finetuning on larger parametric models, particularly those with 70B or 40B parameters. This limitation likely curtailed our exploration of the full capabilities of these advanced models, which could have provided deeper insights or enhanced performance. Another critical limitation was the resourceintensive nature of obtaining legal expert annotations. Due to the high costs and extensive time required for this process, it was not feasible for us to obtain expert evaluations for the entire PredEx test dataset. Consequently, we opted to sample 50 random documents for expert review and Likert score evaluations. While necessary, this ap-
proach potentially limits the breadth and depth of our expert-based evaluation, as it does not encompass the entire dataset. In terms of the effectiveness of Large Language Models (LLMs) in the legal domain, our findings suggest that while these models are proficient in conversational contexts, their applicability in logic or knowledge-intensive tasks like legal judgment prediction and explanation is less convincing. Analyzing lengthy legal documents and generating predictions with explanations poses a significant challenge for generative-based models. This is particularly true in cases where the models need to process and understand complex legal reasoning and contexts. Furthermore, the performance of the opensource baseline model, which was intended to jointly predict and generate explanations, did not meet our expectations. This underperformance could be attributed to the token limitations imposed during our study. By only using the last 1000 tokens of documents for fine-tuning, there is a possibility that the model did not fully grasp the entire context of the cases. Moreover, our fine-tuned models frequently produced truncated responses due to the 512-token limit set for generation. This limitation may have hindered the models’ ability to generate comprehensive and nuanced explanations. Lastly, the pre-trained models used in our study inherently lacked detailed knowledge specific to Indian legal cases. Even after undergoing tuning processes, these models struggled to generate explanations that paralleled the depth and specificity of human-like legal reasoning. This shortfall highlights the challenge of adapting general AI models to specialized domains such as law, where domainspecific knowledge and reasoning are crucial. These limitations underscore the challenges in applying LLMs to complex and specialized tasks like legal judgment prediction and explanation. They also highlight the necessity for continued research and development efforts aimed at enhancing the capabilities of AI models in interpreting and understanding legal documents and contexts