ELLA: Empowering LLMs for Interpretable, Accurate and Informative Legal Advice (2024)

Abstract: Despite remarkable performance in legal consultation exhibited by legal Large Language Models(LLMs) combined with legal article retrieval components, there are still cases when the advice given is incorrect or baseless. To alleviate these problems, we propose ELLA, a tool for Empowering LLMs for interpretable, accurate, and informative Legal Advice. ELLA visually presents the correlation between legal articles and LLM’s response by calculating their similarities, providing users with an intuitive legal basis for the responses. Besides, based on the users’ queries, ELLA retrieves relevant legal articles and displays them to users. Users can interactively select legal articles for LLM to generate more accurate responses. ELLA also retrieves relevant legal cases for user reference. Our user study shows that presenting the legal basis for the response helps users understand better. The accuracy of LLM’s responses also improves when users intervene in selecting legal articles for LLM. Providing relevant legal cases also aids individuals in obtaining comprehensive information. Our github repo is: https: //github.com/Huyt00/ELLA1.

heading: 1 Introduction
text: Large Language Models (LLMs), such as LLAMA (Touvron et al., 2023), ChatGLM (Zeng et al., 2023) and GPT4 (OpenAI et al., 2024), have shown impressive performance in various tasks, showing great potential for specific domains, such as law (Lai et al., 2023) and finance (Wu et al., 2023; Yang et al., 2023). In the legal domain, many attempts have been made(Colombo et al., 2024; Huang et al., 2023; Yue et al., 2023; Nguyen, 2023; Cui et al., 2023), which acquire legal knowledge
∗ Equal Contribution. † Corresponding author.
1Video demonstration is available at: https://youtu.be/ V8iaIXSJ2i8
through continual training and performing a supervised fine-tuning stage with a large-scale legal dataset. These models can offer various services including legal consultations, explaining legal terminology, analyzing legal cases, and preparing legal documents.
Despite the remarkable performance of LLMs within the legal domain, they are not exempt from the occurrence of hallucination (Ji et al., 2023). To alleviate this, previous studies (Huang et al., 2023; Yue et al., 2023; Cui et al., 2023) have proposed retrieval-augmented generation(RAG) (Lewis et al., 2021) frameworks to retrieve legal articles from an external datastore. By leveraging retrieved legal articles, hallucination is reduced and LLMs can generate more faithful answers.
In the legal domain, LLMs’ responses are required to have high accuracy and be supported by reasonable legal bases. Therefore, the retrieval component plays an important role as it provides correct and related legal articles for LLMs. While LLMs could be augmented with retrieved legal articles to generate faithful responses, when irrelevant ones are retrieved, they inevitably bring noise to LLMs, leading LLMs to produce responses with incomplete, incorrect or inconsistent information.
For instance, as shown in Figure 1, when a user asks Q1, the legal article retrieval model retrieves articles 1098, 1101, and 1105 of the Civil Code 2 according to the query, while fails to retrieve another three relevant ones: article 1100, 1102 and 1093 of the Civil Code. Therefore, LLM only suggests that adopters need to meet the conditions c1, c2 and c3 mentioned in the retrieved legal articles, resulting in incomplete suggestions. Then the user continues to ask Q2. Although the related article is retrieved, the irrelevant ones are also retrieved. Such irrelevant articles bring noise to LLM, leading to the incorrect response R2(In fact, the case
2https://www.gov.cn/xinwen/2020-06/01/content_ 5516649.htm
374
mentioned in Q2 should not be considered as automatic divorce). Besides, LLMs may be sensible to the input perturbation (Zhu et al., 2023; Dong et al., 2023). Responses can be contradictory when inputs only differ slightly. For example, when the user begins a new chat and asks Q3, which is identical to Q2, the response R3 is contradictory to R2. This inconsistency can potentially bring confusion to users, resulting in a lower-quality consultation.
When LLMs fail to produce coherent and complete responses, relevant legal cases can offer users more in-depth reference information (Su et al., 2024). However, a legal case retrieval module has rarely been integrated into the existing legal domain LLMs in civil law systems. Additionally, legal terminology may sometimes be embedded in the responses lacking sufficient explanations, posing potential understanding difficulties for users without domain knowledge (Savelka et al., 2023).
To address the issues mentioned above, we propose ELLA, a tool Empowering LLMs for interpretable, accurate, and informative Legal Advice.
Firstly, we fine-tune BGE (Xiao et al., 2023), an embedding model for retrieval, to retrieve the legal basis for each sentence in the response. By visually
presenting the legal basis to users, users can trust the advice provided by LLMs. When there is no legal basis for a sentence, it can be viewed as a warning that the sentence may be incorrect. Secondly, ELLA retrieves several legal articles based on the user’s query and presents them to users. Users can interactively select the relevant legal articles for LLMs to generate accurate and complete responses while disregarding irrelevant ones to avoid noise. Thirdly, we incorporate a legal case retrieval model in ELLA, intending to present supplementary information for users to reference. Considering the long context in legal cases, we find all the key sentences in the article through similarity matching between the query and each sentence in the legal case. We highlight all key sentences in the legal cases for users to improve their reading efficiency.
The response interpretation aids users in understanding and placing trust in the advice given by LLMs. The user study shows that our model can generate more accurate responses when users interactively select relevant legal articles. The legal case retrieval module also offers users more resourceful reference information.

heading: 2 Framework and Usage Example
text: ELLA is composed of four parts: 1) Chat Interface: visually displays the conversation between the user and the LLM. 2) Interactive Legal Article Selection: Provides retrieved legal articles for users to choose from, letting the LLM generate new responses based on the user’s selected legal articles. 3) Response Interpretation: Provides legal article and judicial interpretations to interpret each sentence of the LLM’s response. 4) Legal Case Retrieval: Displays relevant legal cases for the user to refer to.

heading: 2.1 Chat Interface
text: Our chat interface is shown in Figure 2, part 1. After clicking the input button, the chat box above will display user input and the LLM’s response. Users can have multiple rounds of chats, or click ’new conversation’ on the upper left to start a new consultation. The column on the left retains all conversations. Users can click on each chat button to view the corresponding chat content.

heading: 2.2 Interactive Legal Article Selection
text: Legal article retrieval model plays an important part in Chinese legal domain LLMs (Huang et al.,
2023; Yue et al., 2023). Lawyer LLaMA (Huang et al., 2023) mentions that when LLMs are provided with external relevant legal articles, they can generate more reliable responses. However, the current legal article retrieval models cannot ensure to retrieval all the relevant legal articles and leave out all irrelevant ones. Missed articles might reduce the completeness of the model’s response, while irrelevant articles bring noise to LLM, leading LLMs to generate irrelevant advice.
To solve this problem, ELLA allows users to interactively select legal articles. We display the top K1 = 10 relevant legal articles retrieved for the users. Users can select relevant legal articles based on their situations. The LLM will then generate responses based on the legal articles selected by the user. Note that the LLM generates its first response based on the top 3 retrieved articles by default. Subsequently, users can select legal articles for LLM to regenerate new responses multiple times.
Back to the example in Figure 1, we find that several relevant legal articles are not selected for LLM. Then we can select them, as shown in Figure 2, part 2, and click the "Regenerate" button at the bottom of the page. Then LLM generates a new response with complete information. By allowing users to participate in the legal article retrieval, it increases the consistency between the user’s situation and the referred legal articles used by the LLM, thus enabling the LLM to generate more complete and accurate responses.

heading: 2.3 Response Interpretation
text: The response interpretation module provides the legal article basis for each sentence in the LLM’s response, and helps users better understand the
terminologies in the responses. LLM is sensitive to the inputs. Users may receive different advice when ask the same questions in different ways. To facilitate users to identify which response is more reliable, or whether a response is trustworthy, the response interpretation module presents the referred legal articles for each sentence in the response. Users can verify the reliability of the response by tracing the legal article basis of each sentence.
At the same time, even though the LLM can conveniently provide legal advice to users, sometimes the responses may contain terminologies, which non-professional users may find hard to understand. Besides, some special cases lack a clear definition in the legal articles. They are both explicitly explained in China’s "judicial interpretations". To provide users with a better legal consultation experience, we use a response explanation module to provide a clear explanation of the terminology/special cases with corresponding judicial interpretation, making it easier for users to understand.
As shown in Figure 2, part 3, when the user ask "My girlfriend is married...Would living with her without being legally married be considered bigamy?", the response is "...The situation you mentioned is cohabitation rather than bigamy...cohabitation is not illegal...". To check the definition of "cohabitation", the user can hover the mouse over the sentence. Then the platform will display a hovering box, showing the corresponding judicial interpretation. We show the legal article basis for the sentence in the same way. If there is neither a legal article basis nor a judicial interpretation for the sentence, the hovering box will not display.

heading: 2.4 Legal Case Retrieval
text: Legal cases also serve as important references for users when they consult on legal issues and make judgments about their circumstances. Currently, Chinese legal domain LLMs can only make decisions for users based on internal legal knowledge and externally retrieved legal articles, unable to provide relevant legal cases for users as reference. Therefore, we introduced a legal case retrieval module in ELLA. For every query from users, we search relevant legal cases obtained from China Judgements Online 3 and display them on the platform for users, as shown in Figure 2, part 4. As the context of the legal cases may be long, we highlight the sentences in the trial proceeding records related to the user’s query. Users can directly locate these sentences to get key information. We provide multiple relevant legal cases. Users can click the button at the top of part 4 to view different legal cases.

heading: 3 System Overview
text: In this section, we detail the implementation of all back-end models of ELLA.

heading: 3.1 Legal Consultation
text: In our work, we use Lawyer LLaMA (Huang et al., 2023), a LLM adapted to the legal domain, for legal consultation. Based on Lawyer LLaMA, which focuses on answering queries about marriage, ELLA mainly provides marriage consultation services for users. Since our back-end model is pluggable, we can also replace Lawyer LLaMA with other legal domain LLMs, such as DISCLawLLM (Yue et al., 2023), ChatLaw (Cui et al., 2023) or LawGPT (Nguyen, 2023).

heading: 3.2 Legal Article Retrieval
text: We use the legal article retrieval model provided by Lawyer LLaMA. Following Lawyer LLaMA, after the user inputs a query, we retrieve the relevant legal articles, and append the top 3 legal articles to the user’s query to generate the response. Besides, we display the top K1 = 10 retrieved legal articles on the front end. If the user selects some relevant legal articles and requires a new response, in the back end, we append all selected legal articles to the input prompt, and LLM will generate a new response.
3https://wenshu.court.gov.cn

heading: 3.3 Response Interpretation
text: The response interpretation module aims to provide the legal article basis and judicial interpretations for each sentence of the response from the LLM. Here, we use BGE (Xiao et al., 2023), a state-ofthe-art embedding model for retrieval augmented generation. Since BGE has only been pre-trained on the general corpus, it lacks knowledge about the legal domain, thus being unable to distinguish between two terminologies that are semantically similar but have different definitions in the legal domain. Therefore, we need to fine-tune BGE with legal corpus to make it learn legal knowledge.
Due to the lack of training data, we construct a dataset for response interpretation. We sample 2k queries from the legal instruction tuning data published by Lawyer LLaMA. For each query q, we obtain the top 3 relevant articles [a1, a2, a3] with the legal article retrieval module, and append these three laws individually to qi. Then Lawyer LLaMA generates different responses [r1, r2, r3] based on the different legal articles. For ri = [si1, si2, ..., sin], we calculated the similarity between each sentence sij , j ∈ [1, n] and ai using BM25 (Robertson and Zaragoza, 2009). As illustrated in Figure 4, we treat the sentence with the highest BM25 score sik and gold article ai as the positive case (sik, ai), while the two most irrelevant sentences six, siy as negative cases (six, ai) and (siy, ai). We also created negative cases (sik, at), t ∈ [1, 2, 3] and t ̸= i for distinguishing relevant sentence in ri from other retrieved legal articles.
Given the similar language style and content between legal articles and judicial interpretations, and the fact that legal articles contain all the terminologies involved in judicial interpretations, we only used legal articles to construct the dataset. After fine-tuning the BGE on this dataset, we obtained a new model, which we denote as BGE1 here.
During inference, we use BGE1 to calculate the cosine similarity between the embedding of each sentence in the response and the legal articles and judicial interpretations. If the similarity exceeds a threshold Thr1, we think the corresponding legal article or judicial interpretation can explain the sentence. Thr1 is a hyper-parameter, which we set as 0.85 in our work. Then, we return the articles and judicial interpretations referenced by each sentence to the front end, to help users better understand the LLM’s responses.

heading: 3.4 Legal Case Retrieval
text: In this module, we first retrieve relevant legal cases based on the user’s input. Then we find all the key sentences in the legal case that are related to the consultation query. Finally, we re-rank the top K2 retrieved legal cases according to the number of relevant sentences in the legal case, and return the top K3 re-ranked legal cases to the front end.
Legal Case Retrieval. Similarly, due to the lack of relevant legal domain knowledge in BGE, we need to fine-tune BGE with the legal domain corpus. Here, we use the dataset LeCaRD (Ma et al., 2021), a publicly available Chinese legal case retrieval dataset. We allocated 80% of LeCaRD as the training set and 10% each as the validation and test set. We fine-tune BGE on the training set. Here we denoted the fine-tuned BGE as BGE2. When the user inputs a query, we use BGE2 to retrieve relevant legal cases.
Relevant Sentence Highlight. We use BGE2 to calculate the similarity between the user’s query and each sentence in the legal case. When the cosine similarity score is larger than Thr2, we consider this sentence to be related to the user’s query, thus this sentence can serve as a reason for this case being a relevant legal case. Thr2 is a hyperparameter, which we set to 0.65 in our work. We highlight all relevant sentences in the case for users, helping them quickly locate the parts of the case that are highly related to their query. In this way, users can quickly judge whether this legal case is relevant and helpful, and they can also quickly obtain important information that they care about.
Legal Case Re-rank. We think that the more relevant sentences in a case, the larger the possibility of the case being a relevant legal case. Therefore, we re-rank the top K2 legal cases retrieved by BGE2 according to the number of relevant sentences, and return the re-ranked top K3 legal cases to the front end. We set K2 = 50 and K3 = 15 in our work.

heading: 4 Evaluation
text: In this section, we automatically evaluate our case retrieval model. We also conduct a user study to evaluate whether ELLA helps users obtain more accurate, interpretable and informative information during the consultation.

heading: 4.1 Automatical Evaluation
text: As we mentioned in Section 3.4, we split LeCaRD into 80% for training, 10% for validation and 10% for testing. Here, we use the LeCaRD test set to evaluate our legal case retrieval model, BGE2. Following CaseEncoder (Ma et al., 2023), we use the Normalize Discounted Cumulative Gain (NDCG) metric as the evaluation metric. The experimental results are shown in Table 1.
Compared with BM25 and BGE which has not been fine-tuned, BGE2 shows a significant increase in each NDCG@K. This shows that the fine-tuned BGE can learn legal knowledge well, and better distinguish legal cases that are semantically similar but not relevant in the legal domain. Although CaseEncoder (Ma et al., 2023), SAILER (Li et al., 2023) and CaseFormer (Su et al., 2024) outperform BGE2, we use BGE2 since it can serve as an embedding model for relevant sentences similarity matching mentioned in Section 3.4. Note that our legal case retrieval model is pluggable, so we can also additionally add SOTA models mentioned above for legal case retrieval.

heading: 4.2 User Study

heading: 4.2.1 Study Design
text: We conduct a user study to validate whether ELLA can improve users’ legal consultation experience. Since LLMs deliver an impressive performance in answering simple questions, such as "Can I get married if I am younger than 20?", we randomly selected 20 consultation queries about complex marriage situations for the user study. We invited 3 non-legal professional users and asked them to obtain solutions to these queries through ELLA. Users will evaluate whether the three modules in ELLA are helpful for their legal consultation.

heading: 4.2.2 Result
text: Response Regeneration. For an average of 83% of the queries, users find that the top 3 legal articles retrieved are not entirely correct, impeding LLM from directly generating correct responses based on these articles. For an estimated 20% of the queries, LLMs can not provide correct responses due to the noise brought by irrelevant legal articles, while for 25%, LLM’s responses are incomplete, as relevant legal articles were not among the initial top three results. Another 38% of responses contained irrelevant information resulting from the inclusion of unrelated legal articles within the top three results. However, in 80% cases, users can successfully receive correct responses by selecting relevant legal articles for LLM to regenerate responses.
Response Interpretation. Users have reported that for approximately 95% of the queries, ELLA can accurately provide the legal article basis of the responses generated by the LLM. By crossreferencing the responses with the corresponding legal article, users can swiftly determine whether the responses are reliable or inaccurate. For instance, when a user asks, "I have never had children since I got married, and now I am planning to adopt a child from a relative. Can I adopt a child privately?" LLM responds "Adopters need to meet the following conditions...". ELLA justifies the response by citing Article 1098 of the Civil Code as its legal article basis. Additionally, it retrieves Article 1100 of the Civil Code, "A childless adopter may adopt two children...," which the user can select for the LLM to generate a full response. Users also noted that, in about 73% of the queries, parts of the legal articles have already been included within the responses. However, LLM may not fully rephrase the entire article. By providing the legal
articles basis, users can conveniently access to the complete information in the legal article.
In all provided judicial interpretations, roughly 30% serve the purpose of clarifying specific legal terminologies or special cases. For instance, consider a scenario where a user inquires, "My husband and I have obtained a marriage certificate but have not cohabited. We are now filing for divorce and my husband wishes to return the bride price. Is this permissible?" In response, ELLA gives additional judicial interpretation that illuminates the conditions under which the return of the bride price is allowed. However, for the remaining 70%, users claim that they are already familiar with the content in the judicial interpretations, such as, "Support payments encompass children’s living expenses, education costs, medical bills and other expenditures." Generally speaking, users assert that judicial interpretations can assist them in acquiring a better comprehension of the responses when interpretation is required, facilitate accurate judgments according to their situations, and pave the way for further consultation tailored to the specifics of their current circumstances.
Legal Case Retrieval. On average, 77% of queries proved the legal case retrieval module to be beneficial for user consultations. Users conveyed that although the retrieved legal cases might not exactly match their situations, these cases provide a reference point to gauge the possible outcomes for their unique circumstances. All users concurred that highlighting pertinent sentences significantly streamlines the process of reading cases. By emphasizing the information users are interested in, the user’s reading efficiency improves.

heading: 5 Conclusion
text: We present a novel tool, ELLA, for legal consultation. ELLA provides the legal basis and judicial interpretations that supplement the legal advice generated by LLMs, increasing users’ understanding and trust in LLM responses. It also displays retrieval results from the retrieval model and allows users to actively select relevant legal articles, thereby assisting the LLMs in generating more accurate responses. Additionally, equipped with a legal case retrieval model, users can refer to relevant legal cases for more comprehensive information. ELLA enables LLMs to provide legal advice that is easier to interpret, more precise, and more informative.

heading: Acknowledgments
text: This work is supported in part by NSFC (62161160339) and Beijing Science and Technology Program (Z231100007423011). We thank the anonymous reviewers for their valuable comments and suggestions. For any correspondence, please contact Yansong Feng.
Limitations
For simple legal queries, legal LLMs can provide correct responses in most cases. ELLA primarily assists with complex legal consultation queries. When users ask multiple questions within a single input, our legal article retrieval module may not comprehensively extract all relevant legal articles. In future work, we plan to integrate different retrieval modules to increase the diversity of retrieved legal articles.
As official judicial interpretations only contain 76 articles, ELLA can not provide interpretations for all professional terminologies. We will incorporate additional external legal knowledge, such as legal textbooks, to provide interpretations for more professional terminologies.
Due to limited computational resources, we do not use state-of-the-art case retrieval modules. We will adopt them in our future work.
Ethics Statement
The main purpose of this paper is to explore how to provide users with better legal consultation services. However, it is important to note that the outputs generated by the model may contain nonstandard, incorrect, gender-biased, or morally questionable information. Therefore, please adopt the legal advice provided by the model with caution. When legal assistance is needed, please seek help from qualified professionals.
The external knowledge used in this paper, such as legal articles and legal cases, was obtained from the official websites of the Chinese government. The data does not contain any private information and cannot be used for commercial purposes.
All participants involved in the user study are voluntary and anonymous. We did not collect any private information from the participants.

heading: A Dataset Construction
text: B Consultation Examples

heading: Future Work
text: In future work, we plan to integrate different retrieval modules to increase the diversity of retrieved legal articles. As official judicial interpretations only contain 76 articles, ELLA can not provide interpretations for all professional terminologies. We will incorporate additional external legal knowledge, such as legal textbooks, to provide interpretations for more professional terminologies. Due to limited computational resources, we do not use state-of-the-art case retrieval modules. We will adopt them in our future work