CitaLaw: Enhancing LLM with Citations in Legal Domain
Authors: Kepu Zhang, Weijie Yu, Sunhao Dai, Jun Xu
Accepted at: acl25

CitaLaw: Enhancing LLM with Citations in Legal Domain

Kepu Zhang1, Weijie Yu2*, Sunhao Dai1, Jun Xu1
1 Gaoling School of Artificial Intelligence, Renmin University of China
2 University of International Business and Economics
kepuzhang@ruc.edu.cn, yu@uibe.edu.cn

Abstract
In this paper, we propose CitaLaw, the first benchmark designed to evaluate LLMs’ ability to produce legally sound responses with appropriate citations. CitaLaw features a diverse set of legal questions for both laypersons and practitioners, paired with a comprehensive corpus of law articles and precedent cases as a reference pool. This framework enables LLM-based systems to retrieve supporting citations from the reference corpus and align these citations with the corresponding sentences in their responses. Moreover, we introduce syllogism-inspired evaluation methods to assess the legal alignment between retrieved references and LLM-generated responses, as well as their consistency with user questions. Extensive experiments on 2 open-domain and 7 legal-specific LLMs demonstrate that integrating legal references substantially enhances response quality. Furthermore, our proposed syllogism-based evaluation method exhibits strong agreement with human judgments.

1 Introduction
Generating responses supported by citations, such as relevant law articles and precedent cases, is essential for ensuring the trustworthiness of large language models (LLMs) in legal tasks. For laypersons seeking legal advice (Fei et al., 2023), LLM-generated responses grounded in citations provide verifiable information, fostering trust in the system. Conversely, for legal practitioners such as lawyers and judges, citations serve as supportive evidence that aids in analyzing complex cases, validating legal arguments, and ensuring decisions align with established legal principles (Li et al., 2024; Zhong et al., 2020; Abdallah et al., 2023).

Recently, a growing body of benchmark research (Gao et al., 2023a; Li et al., 2023) has focused on enabling LLMs to provide citations for the statements they generate. For instance, ALCE (Gao et al., 2023b) introduces a benchmark designed to evaluate the ability of LLMs to generate citation-supported outputs, aiming to improve factual accuracy. WebCiteS (Deng et al., 2024) provides a curated database of manually annotated summaries and citations to enhance performance in text summarization and citation generation.

While these studies have made notable progress in general domains, they face significant challenges when applied to the legal domain. First, laypersons and legal practitioners interact with LLMs differently and have distinct expectations for citations. Laypersons typically seek legal advice and rely on citations to verify the accuracy of LLM responses, whereas legal practitioners pose more complex queries, using LLMs for legal reasoning, with citations serving as supportive evidence. Existing studies fail to address these differences, leading to unsatisfactory performance in real-world applications. Second, existing methods often fall short in providing the diverse references required in legal contexts, such as law articles and precedent cases. Law articles establish the foundational legal framework, while precedent cases offer concrete examples and interpretive guidance. These two types of references inherently align with the distinct characteristics of civil and common law systems. Third, traditional citation evaluation measures, such as ROUGE (Lin, 2004), rely on surface-level similar-

![The framework of our CitaLaw.](page_1012_355_388_246.png)

Figure 1: The framework of our CitaLaw.
<table>
  <tr>
    <th>Major Premise</th>
    <th>Minor Premise</th>
    <th>Conclusion</th>
  </tr>
  <tr>
    <td>Question: ... The school mandatory internship. Is it legal to refuse to let the student do the internship on their own?</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Answer: The school mandatory internship, especially the school mandatory arrangement of Internship units, is not in line with the law... If such a situation occurs, the student can report it to the local education authority [A1].</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>[A1]: “Regulations on the...of Higher Learning Article 6” students enjoy the following rights...;</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Case: ... (Fact 2)...x1 suggested that we tie him up to force x2 to repay the money...(x2 serious injury)...</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Question: For Fact 2, what are the reasons for the view that x1 does not constitute transformative robbery and constitutes intentional injury crime?</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Answer: ...x1 constitutes the crime of theft and intentional injury [A1]... In this case, despite the purpose of concealing stolen goods [C1], the act of using violence against x2 [C2], who came to the building to post flyers resulting in injuries, does not constitute transformative robbery [C3]. ...</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>[C1]: xx case... concealing stolen goods...</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>[C2]: xx case... violence against...</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>[C3]: xx case... Not constituting transformative robbery...</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>[A1]: “Criminal Law Article 234” Intentional Injury Crime...</td>
    <td></td>
    <td></td>
  </tr>
</table>

(a) Layperson

(b) Practitioner

Figure 2: Examples from the two subsets of CitaLaw, with text in red, blue, and yellow representing the three dimensions of the syllogism: major premise, minor premise (circumstances, illegal acts), and conclusion (legal decisions), respectively. [A] and [C] denote citations to relevant law articles and precedent cases, respectively.

ities and are often insufficient to assess the alignment between references and LLM-generated responses. In the legal domain, effective evaluation requires a deeper understanding of logical and semantic relationships.

To overcome the above challenges, we propose CitaLaw, the first benchmark tailored to evaluate LLMs’ capabilities in generating legally grounded responses supported by accurate and context-aware citations. As shown in Figure 1, CitaLaw incorporates four distinct legal-specific features:

(1) CitaLaw has two subsets tailored for laypersons and practitioners, with examples in Figure 2. Laypersons typically ask shorter, conversational questions, while practitioners often pose specialized, detailed questions.

(2) CitaLaw includes a retrieval corpus comprising two commonly used references: law articles, which provide clear and concise guidelines for addressing user questions, and precedent cases, which offer legal reasoning and support for judicial decisions. Recognizing the distinct needs of laypersons and practitioners, we provide only law articles for laypersons to ensure clarity, while practitioners have access to both law articles and precedent cases to support more complex legal reasoning.

(3) In addition to traditional global-level metrics such as MAUVE (Pillutla et al., 2021), we propose a syllogism-based evaluation method to assess both the response correctness and the citation quality. This method provides a more granular evaluation by focusing on three key dimensions: circumstances, illegal acts, and legal decisions.

(4) We consider two types of response generation methods. The first type, Citation-Guided Generation (CGG), involves generating responses by incorporating retrieved references during generation. The second type, Answer Refinement Generation (ARG), refines the LLMs’ initial response (CloseBook) by retrieving and incorporating reference information. This category includes ARG-Q, which retrieves citations using only the user query, and ARG-QA, which retrieves citations using both the user query and the LLM’s initial response.

Extensive experiments on two open-domain and seven legal-specific LLMs reveal the following key insights: 1) Incorporating legal references into the LLM significantly improves the quality of responses; 2) Including references as part of the LLM’s input consistently outperforms answer-refinement methods; 3) Leveraging references to refine the LLM’s responses yields better alignment of responses and references. 4) For fine-tuning LLMs in legal scenarios, incorporating law articles, syllogistic reasoning, and full-scale fine-tuning achieves promising performance. 5) Open-domain LLMs surprisingly outperform legal-specific LLMs in certain scenarios; 6) Human evaluations show a strong correlation with our syllogism-based methods.

In summary, our contributions are as follows:

• To the best of our knowledge, CitaLaw is the first benchmark designed to evaluate the capability of LLMs to generate legally grounded responses with accurate and context-aware citations. CitaLaw includes questions tailored to both laypersons and practitioners, paired
with a citation corpus comprising law articles and precedent cases.

• We propose a two-level evaluation framework that combines global-level metrics with a syllogism-based reasoning approach. Additionally, we explore two mainstream methods for legal response generation: citation-guided and answer refinement.

• Through extensive experiments on two open-domain and seven legal-specific LLMs, we demonstrate the effectiveness of integrating legal references into response generation and validate our syllogism-based evaluation method. Additionally, we provide actionable insights for the practical deployment of LLMs in legal scenarios.

2 Related Work

LLM for Legal Task. A amount of work has explored applying LLMs to legal tasks (Savelka et al., 2023; Wu et al., 2023b; Yu et al., 2022a; Blair-Stanek et al., 2023). Building LLMs tailored for legal scenarios is a popular direction (Yue et al., 2023; Wu et al., 2023a; He et al., 2023). There are also some benchmarks that explore the capabilities of LLMs in legal tasks. LawBench (Fei et al., 2023) evaluates LLMs’ legal knowledge across three cognitive aspects. LAiW (Dai et al., 2023) assesses LLMs’ legal reasoning abilities based on legal practice logic. LexEval (Li et al., 2024) evaluates LLMs’ legal capabilities based on a new legal cognitive ability classification system. However, none of them have considered enhancing the trustworthiness of LLMs in legal scenarios by generating outputs with citations.

Citation in LLM. Attribution (Li et al., 2023) in LLMs refers to providing supporting evidence for the answers generated by the model, presented in the form of citations. ALCE (Gao et al., 2023b) is an automated benchmark for evaluating LLMs’ ability to generate outputs with citations, aimed at improving the factual accuracy of the generated responses. WebCiteS (Deng et al., 2024) provides a database containing 7,000 manually annotated summaries and citations to enhance LLMs’ capabilities in summarization and citation. RARR (Gao et al., 2023a) enhances LLM outputs by automatically adding citations, and modifying the responses. ExpertQA (Malaviya et al., 2024) verifies and modifies citations through expert review to ensure reliability. In contrast to the above works, Ci t aLaw focuses specifically on citation in legal scenarios.

3 Task Setup and Dataset Construction

Suppose we have a legal corpus \( D \), which consists of either a collection of precedent cases (\( D_f \)) or law articles (\( D_c \)). Given a user question \( x \) posed by either a layperson or a practitioner, the LLM-based system is tasked with retrieving supportive citations from \( D \) and generating a legally grounded response \( y \). The response \( y \) comprises a list of \( n \) sentences, i.e., \( y = [s_1, \cdots, s_n] \), where each sentence \( s_i \) refers to at most one corresponding citation. As illustrated in Figure 2, the system is further required to attach each citation to its relevant sentence, with “[A]” and “[C]” denoting references to law articles and precedent cases, respectively.

To enable the evaluation of this task, we construct the specialized dataset (Table 1 shows the statistics) as follows:

To simulate the behavior of laypersons, we include questions that are more conversational, lack detailed case descriptions, and are relatively short in length. We use the consultation section from LawBench (Fei et al., 2023), which collects user queries from the Hualv website\(^1\) and answers provided by lawyers or legal consulting firms.

To simulate the behavior of legal practitioners, we include questions that are more professional, often accompanied by detailed case descriptions, and generally longer. For this purpose, we use the open-ended question section from LexEval (Li et al., 2024), which consists of subjective questions from the National Uniform Legal Profession Qualification Examination. These questions are particularly challenging for LLMs, requiring them to understand the case fully and apply legal knowledge accurately to generate answers.

In terms of the corpus, we construct a comprehensive corpus from multiple sources, including law articles and precedent cases. Specifically, for law articles, we collect approximately 50,000 documents from LexiLaw\(^2\), covering areas such as Civil Law, Criminal Law, and judicial interpretations. For precedent cases, we include both criminal and civil cases. Criminal cases are sourced from the LeCaRD legal retrieval dataset (Ma et al., 2021b), ELAM (Yu et al., 2022b), and civil cases from the CAIL legal summary

\(^1\)www.66law.com
\(^2\)https://github.com/CSHaitao/LexiLaw
<table>
  <tr>
    <th>Dataset</th>
    <th>#Q</th>
    <th>Len<sub>Q</sub></th>
    <th>Len<sub>A</sub></th>
    <th>Q Type</th>
  </tr>
  <tr>
    <td>Layperson</td>
    <td>500</td>
    <td>57.62</td>
    <td>107.40</td>
    <td>Question</td>
  </tr>
  <tr>
    <td>Practitioner</td>
    <td>500</td>
    <td>618.96</td>
    <td>193.46</td>
    <td>Case + Question</td>
  </tr>
</table>

Table 1: Dataset statistics. #Q indicates the number of questions, Len<sub>Q</sub> and Len<sub>A</sub> denote the average lengths of questions and gold answers, and Q Type refers to the question type.

dataset, LJP-MSJudge (Ma et al., 2021a), and the pre-training data of fuzi.mingcha (Wu et al., 2023a). As a supplement to precedent cases, we also incorporate question-and-answer pairs from fine-tuning datasets of legal LLMs as part of the precedent cases. These QA pairs are collected from DISC-LawLLM (Yue et al., 2023), LawGPT_zh (Liu et al., 2023), and HanFei (He et al., 2023). In total, the constructed corpus contains approximately 500,000 documents, ensuring sufficient coverage of both law articles and precedent cases to support diverse legal tasks.

4 Method

4.1 Response Generation

We consider two types of methods in this study.

Citation-Guided Generation (CGG) produces response \( y_{cgg} \) given a user question \( x \) by referring retrieved relevant document(s) \( D_R \):

\[
y_{cgg} = f_{LLM}(x, D_R, p_1),
\]

where \( f_{LLM} \) denotes a open-domain or a legal specific LLM; \( p_1 \) is the direct generation prompt. All prompt settings are detailed in Appendix A.

Answer Refinement Generation (ARG) is a two-stage method that generates the final response \( y_{arg} \) by refining the LLM’s initial response \( y_{init} \) through the retrieval and incorporation of reference information. This process can be formulated as:

\[
y_{init} = f_{LLM}(x, p_2),
\]

where \( p_2 \) is the prompt instructing the LLM to directly generate an initial response without reference information. We refer to this step as CloseBook. The initial response \( y_{init} \) is then refined as:

\[
y_{arg} = f_{LLM}(y_{init}, D_R, p_3),
\]

where \( p_3 \) is the prompt guiding the LLM to refine the \( y_{init} \) using the retrieved documents \( D_R \).

Laypersons and practitioners interact with LLMs differently and have distinct expectations for citations. When \( x \) is submitted by a layperson, the corresponding \( D_R \) consists of relevant law articles. In contrast, when \( x \) is submitted by a practitioner, the corresponding \( D_R \) includes both relevant law articles and precedent cases. The process for retrieving \( D_R \) from \( D \) is detailed in the next subsection.

4.2 Citation Retrieval

We explore state-of-the-art open-domain dense retriever BGE (Xiao et al., 2023), along with two legal-specific dense retrievers, Criminal-BERT (Zhong et al., 2019) and Civil-BERT (Zhong et al., 2019). We also investigate two types of retrieval queries: \( x \) (the user question alone, ARG-Q) and \( [x; y_{init}] \) (the concatenation of the user query \( x \) and the initial response \( y_{init} \), where [ ; ] denotes the concatenation operation, ARG-QA). The impact of different retrieval models on performance will be analyzed in the experiments.

4.3 Citation Attachment

Building on the retrieved citations, this subsection outlines the process of attaching these law articles or precedents to specific sentences in the LLM-generated responses. This process involves answering two key questions:

What kind of sentences can be associated with citations? We utilize co-occurring words and legal entity extraction to identify sentences that explicitly reference legal concepts, actions, or terms relevant to the retrieved citations. Specifically, we construct a pool of legal terminologies using THUOCL\(^3\) and LaWGPT (Zhou et al., 2024). A sentence is considered eligible if it contains any of the terminologies from this pool. Additionally, we use SpaCy (Honnibal et al., 2020) to extract legal entities from each sentence. If a sentence includes legal entities, it is also deemed eligible for citation attachment.

How are citations attached to the identified sentences? If a sentence is deemed eligible for citation attachment, we associate it with retrieved citations as follows. For the laypersons, the retrieved law article \( c_l \in D_l \) is attached to the most relevant sentence \( s_k \in y \):

\[
C_{Lay} = \{ (s_k, c_l) \mid s_k = \arg\max_{s_i \in y} \operatorname{sim}(s_i, c_l) \},
\]

where \( (s_k, c_l) \) represents attaching the reference \( c_l \) to the sentence \( s_i \), and \( \operatorname{sim}(\cdot) \) is computed using sentence-BERT (Reimers, 2019). We set \( |C_{Lay}| = 1 \) because, typically, a layperson’s query pertains to only one specific legal article. For practitioners,
we attach the retrieved law article in the same way as for laypersons. Additionally, we associate the retrieved precedent cases \( c_c \in D_c \) with each \( s_i \in y \), which is formulated as:

\[
C_{Pra} = \{ (s_k, c_l) \mid s_k = \arg\max_{s_i \in y} \operatorname{sim}(s_i, c_l) \}
\]
\[
\cup \{ (s_i, c_c) \mid c_c = \arg\max_{c_j \in D_c} \operatorname{sim}(s_i, c_j) \},
\]

where \( |D_c| = 3 \), meaning each response \( y \) can be associated with up to three precedents\footnote{Considering the input window size of LLMs, we set up to retrieve 3 precedent cases.}.

5 Evaluation

CitaLaw provides a comprehensive evaluation framework incorporating metrics for fluency, correctness, and citation quality. This framework is divided into two levels of analysis: global level and the proposed syllogism level.

Syllogism, a foundational framework in legal reasoning, comprises three key components: the major premise, the minor premise, and the conclusion. In our legal context, these correspond to the relevant law article or precedent case (major premise), the factual circumstances and actions of a specific case (minor premise), and the resulting legal decision (conclusion). By integrating this syllogistic framework, CitaLaw goes beyond surface-level correctness to evaluate the logical coherence and alignment of LLM-generated responses with established legal principles.

5.1 Fluency (Style Consistency)

To ensure the LLM-generated responses align with the user’s requirements, the system must adapt its style based on the user’s background. For laypersons, responses should avoid excessive technical jargon to ensure accessibility and comprehension. Conversely, responses for legal practitioners should adopt a formal and professional tone to maintain credibility and utility. To achieve this aim, we concatenate the user query and the LLM-generated response and apply MAUVE (Pillutla et al., 2021) to assess their style consistency.

5.2 Correctness

At the global level, we use established metrics ROUGE (Lin, 2004) and BERTScore (Zhang et al., 2019). ROUGE measures word-level overlap between the generated and labeled responses, with scores reported for ROUGE-1, ROUGE-2, and ROUGE-L. BERTScore captures semantic similarity between the generated and labeled responses, and we report the F-score (BERT-F) for evaluation. These metrics assess the overall correctness of LLM-generated responses.

At the syllogism level, we leverage the Qwen2 (Yang et al., 2024) to extract key components, including the circumstances, illegal acts, and legal decisions. We use sentence-BERT (Reimers, 2019) to measure the alignment between the labeled responses and the generated outputs across these dimensions, resulting in Correct_c, Correct_a, and Correct_d. This syllogism-level evaluation allows us to assess the logical coherence of the responses, ensuring that they align with the underlying legal reasoning principles.

5.3 Citation Quality

As previously discussed, we assume that a question submitted by laypersons typically corresponds to a specific law article. Therefore, at the global level, we evaluate the citation quality of the retrieved law article (premise) by measuring its entailment with the associated sentence in the LLM’s response (hypothesis). Specifically, we use an NLI model to compute CitaLaw, which quantifies the degree to which the law article entails the attached sentence. This metric reflects how effectively the response aligns with the cited law article. We employ DISC-LawLLM (Yue et al., 2023) as the NLI model due to its strong agreement with human evaluations (as discussed in Sec. 6.3) and its superior performance compared to other NLI models (as detailed in Sec. 6.5).

At the syllogism level, we evaluate the quality of precedent case citations by examining three key components: circumstances, illegal acts, and legal decisions. After extracting these elements from both the retrieved cases and the associated sentence in the LLM’s response, we utilize DISC-LawLLM to assess the entailment for each component. This evaluation yields three distinct scores: Cita_c for circumstances, Cita_a for illegal acts, and Cita_d for legal decisions, providing a more detailed and nuanced assessment of citation quality within the syllogism framework.

6 Experiments

We conduct extensive experiments on our CitaLaw using the proposed two-level evaluation methods.
<table>
  <tr>
    <th rowspan="2">Category</th>
    <th rowspan="2">Model</th>
    <th colspan="4">Fluency</th>
    <th colspan="6">Correctness</th>
    <th rowspan="2">Citation</th>
    <th rowspan="2">All</th>
  </tr>
  <tr>
    <th>Mauve</th>
    <th>Rouge-1</th>
    <th>Rouge-2</th>
    <th>Rouge-L</th>
    <th>BERT-F</th>
    <th>Correct_t</th>
    <th>Correct_a</th>
    <th>Correct_q</th>
    <th>Cita_ave</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td rowspan="3">Llama3 (Llam3-8B-Instruct)</td>
    <td>CloseBook</td>
    <td>22.63</td>
    <td>16.47</td>
    <td>1.95</td>
    <td>13.34</td>
    <td>58.46</td>
    <td><b>73.05</b></td>
    <td>68.24</td>
    <td>66.87</td>
    <td>67.38</td>
    <td>43.15</td>
  </tr>
  <tr>
    <td>CGG</td>
    <td>61.01</td>
    <td><b>23.97</b></td>
    <td>6.05</td>
    <td>17.91</td>
    <td><b>65.94</b></td>
    <td>67.29</td>
    <td><b>77.31</b></td>
    <td><b>74.95</b></td>
    <td><b>86.70</b></td>
    <td><b>53.46</b></td>
  </tr>
  <tr>
    <td>ARG-Q</td>
    <td><b>61.27</b></td>
    <td>23.17</td>
    <td>5.65</td>
    <td>17.83</td>
    <td>64.23</td>
    <td>69.04</td>
    <td>75.45</td>
    <td>74.47</td>
    <td>79.10</td>
    <td>52.24</td>
  </tr>
  <tr>
    <td>ARG-QA</td>
    <td>51.83</td>
    <td>23.73</td>
    <td><b>6.96</b></td>
    <td><b>18.53</b></td>
    <td>64.84</td>
    <td>71.37</td>
    <td>74.81</td>
    <td>74.66</td>
    <td>80.80</td>
    <td>51.95</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen2 (Qwen2-7B-Instruct)</td>
    <td>CloseBook</td>
    <td>21.04</td>
    <td>15.29</td>
    <td>2.27</td>
    <td>11.31</td>
    <td>58.39</td>
    <td><b>70.89</b></td>
    <td>71.71</td>
    <td>69.93</td>
    <td>72.35</td>
    <td>43.69</td>
  </tr>
  <tr>
    <td>CGG</td>
    <td><b>75.10</b></td>
    <td><b>22.26</b></td>
    <td>4.77</td>
    <td>15.41</td>
    <td><b>65.28</b></td>
    <td>67.50</td>
    <td><b>78.62</b></td>
    <td><b>77.82</b></td>
    <td><b>77.59</b></td>
    <td><b>53.82</b></td>
  </tr>
  <tr>
    <td>ARG-Q</td>
    <td>66.55</td>
    <td>20.86</td>
    <td>4.50</td>
    <td>15.42</td>
    <td>64.59</td>
    <td>66.96</td>
    <td>77.82</td>
    <td>75.66</td>
    <td>81.48</td>
    <td>52.65</td>
  </tr>
  <tr>
    <td>ARG-QA</td>
    <td>66.80</td>
    <td>21.73</td>
    <td><b>4.78</b></td>
    <td><b>16.34</b></td>
    <td>64.85</td>
    <td>69.31</td>
    <td>76.35</td>
    <td>75.05</td>
    <td><b>82.83</b></td>
    <td>53.11</td>
  </tr>
  <tr>
    <td rowspan="7">Legal LLM (CGG)</td>
    <td>DISC-LawLLM</td>
    <td><b>72.70</b></td>
    <td>22.46</td>
    <td>4.14</td>
    <td>15.48</td>
    <td>65.06</td>
    <td>65.21</td>
    <td>78.55</td>
    <td>76.17</td>
    <td>83.46</td>
    <td>53.69</td>
  </tr>
  <tr>
    <td>fuzi.mingcha</td>
    <td>56.58</td>
    <td>24.54</td>
    <td>5.70</td>
    <td>17.48</td>
    <td><b>65.86</b></td>
    <td>63.28</td>
    <td><b>79.56</b></td>
    <td><b>77.94</b></td>
    <td>81.64</td>
    <td>52.51</td>
  </tr>
  <tr>
    <td>LexiLaw</td>
    <td>71.89</td>
    <td><b>24.96</b></td>
    <td><b>6.25</b></td>
    <td><b>18.91</b></td>
    <td>65.68</td>
    <td>68.89</td>
    <td>78.12</td>
    <td>76.72</td>
    <td>82.42</td>
    <td><b>54.87</b></td>
  </tr>
  <tr>
    <td>Tailing</td>
    <td>13.95</td>
    <td>15.93</td>
    <td>4.13</td>
    <td>12.89</td>
    <td>59.47</td>
    <td><b>72.00</b></td>
    <td>69.11</td>
    <td>68.38</td>
    <td>82.67</td>
    <td>44.28</td>
  </tr>
  <tr>
    <td>zhihai</td>
    <td>37.50</td>
    <td>20.98</td>
    <td>4.59</td>
    <td>13.69</td>
    <td>64.54</td>
    <td>67.75</td>
    <td>77.68</td>
    <td>76.99</td>
    <td>77.16</td>
    <td>48.99</td>
  </tr>
  <tr>
    <td>LawGPT_zh</td>
    <td>51.60</td>
    <td>23.33</td>
    <td>5.28</td>
    <td>16.17</td>
    <td>65.14</td>
    <td>63.72</td>
    <td>79.43</td>
    <td>77.52</td>
    <td><b>86.18</b></td>
    <td>52.04</td>
  </tr>
  <tr>
    <td>Hanfei</td>
    <td>51.12</td>
    <td>23.95</td>
    <td>5.19</td>
    <td>18.76</td>
    <td>65.12</td>
    <td>70.83</td>
    <td>75.01</td>
    <td>74.21</td>
    <td>76.97</td>
    <td>51.24</td>
  </tr>
</table>

Table 2: Performance comparisons on the Layperson dataset. The best performance is indicated in bold.

6.1 Experimental Settings

6.1.1 Evaluated Models
We selected two categories of LLMs for testing: The legal LLMs include (1) fuzi.mingcha (6B) (Wu et al., 2023a), (2) LexiLaw5 (6B), (3) Tailing6 (7B), (4) DISC-LawLLM (13B) (Yue et al., 2023), (5) zhihai (7B) (Wu et al.), (6) LawGPT_zh (6B) (Liu et al., 2023), (7) HanFei (7B) (He et al., 2023). The open-domain LLMs include Qwen2 (7B) (Yang et al., 2024) and Llama3 (8B) (AI@Meta, 2024). For these models, we tested all methods mentioned in Sec. 4, including: (1) CloseBook, (2) CGG, (3) ARG-Q and (4) ARG-QA. For the legal LLMs, we generate responses using CGG. Appendix B has the details.

6.1.2 Implementation Details
Our implementation is based on the Huggingface Transformers library (Wolf et al., 2020) with PyTorch. We use bge-base-zh-v1.5 (Xiao et al., 2023) as the retrieval model and conduct all experiments on Nvidia A6000 GPUs. Additional details are provided in Appendix C and https://github.com/ke-01/CitaLaw.

6.2 Main Results
The results on the Layperson and Practitioner datasets are presented in Table 2 and Table 3. We analyze the results from three perspectives:

6.2.1 Performance of Open-Domain LLM
Legal references improve the response quality. Compared to CloseBook, the overall performance in CGG, ARG-Q, and ARG-QA has improved. This indicates that incorporating references into the LLM helps it better understand both the question and the required direction for the answer, thereby enhancing performance in terms of style consistency, correctness, and citation quality.

CGG achieves better response quality. We observe that CGG achieves optimal performance, especially response correctness, suggesting that incorporating legal references into the LLM input is more effective than refining the LLM’s response. This is because including legal knowledge as input allows the LLM to consider relevant context when generating replies, whereas refining the response might lead to excessive alterations.

ARG improves the alignment of responses and references. We can observe that ARG outperforms CGG in citation-related metrics overall. This is because CGG merely incorporates reference information as input, which may lead the model to overlook some reference details during the generation process. In contrast, ARG modifies the answer based on the references after generation, making it easier to ensure the completeness of citations.

Chinese data fine-tuning can bring benefits. Both the Layperson and Practitioner datasets are Chinese datasets. Qwen2 (Fine-tuning on more Chinese data) achieved better performance than Llama3, demonstrating the benefits of using Chinese data for fine-tuning.

CloseBook tends to state circumstances. CloseBook performs better in terms of correctness regarding circumstances compared to the other dimensions. This suggests that when judicial knowledge references are not used, the LLM is more likely to repeat the circumstances itself, rather than providing an appropriate response to the illegal acts and the legal decision.
<table>
  <tr>
    <th rowspan="2">Category</th>
    <th rowspan="2">Metric</th>
    <th colspan="4">Fluency</th>
    <th colspan="4">Correctness</th>
    <th colspan="4">Citation</th>
    <th rowspan="2">AU</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>Mans</th>
    <th>Rouge-1</th>
    <th>Rouge-2</th>
    <th>Rouge-L</th>
    <th>BERT-F</th>
    <th>Correct.</th>
    <th>Correct.</th>
    <th>Correct.</th>
    <th>Cita.</th>
    <th>Cita.</th>
    <th>Cita.</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="3">Llama3 (Llama3-8B-Instruct)</td>
    <td>CloseBook</td>
    <td></td>
    <td>23.81</td>
    <td>23.05</td>
    <td>7.29</td>
    <td>19.23</td>
    <td>62.83</td>
    <td>76.30</td>
    <td>71.05</td>
    <td>70.32</td>
    <td>63.49</td>
    <td>66.95</td>
    <td>68.83</td>
    <td>65.46</td>
    <td>51.55</td>
  </tr>
  <tr>
    <td>CGG</td>
    <td></td>
    <td>36.37</td>
    <td><b>26.15</b></td>
    <td><b>7.84</b></td>
    <td><b>19.55</b></td>
    <td><b>65.60</b></td>
    <td>67.19</td>
    <td><b>76.36</b></td>
    <td><b>77.73</b></td>
    <td><b>73.58</b></td>
    <td>68.23</td>
    <td>67.87</td>
    <td>67.65</td>
    <td><b>54.51</b></td>
  </tr>
  <tr>
    <td>ARG-Q</td>
    <td></td>
    <td><b>42.65</b></td>
    <td>20.39</td>
    <td>5.07</td>
    <td>15.75</td>
    <td>62.82</td>
    <td>70.49</td>
    <td>73.67</td>
    <td>72.00</td>
    <td>68.61</td>
    <td><b>69.48</b></td>
    <td><b>70.51</b></td>
    <td>68.34</td>
    <td>53.31</td>
  </tr>
  <tr>
    <td></td>
    <td>ARG-QA</td>
    <td></td>
    <td>36.94</td>
    <td>18.64</td>
    <td>4.56</td>
    <td>14.63</td>
    <td>61.50</td>
    <td>71.07</td>
    <td>72.38</td>
    <td>70.32</td>
    <td>69.40</td>
    <td>68.95</td>
    <td>70.42</td>
    <td><b>69.51</b></td>
    <td>52.36</td>
  </tr>
  <tr>
    <td rowspan="4">Qwen2 (Qwen2-7B-Instruct)</td>
    <td>CloseBook</td>
    <td></td>
    <td><b>61.91</b></td>
    <td>30.44</td>
    <td>10.54</td>
    <td><b>23.53</b></td>
    <td>67.55</td>
    <td><b>74.35</b></td>
    <td>79.84</td>
    <td>78.52</td>
    <td>68.55</td>
    <td>68.03</td>
    <td>70.30</td>
    <td>69.71</td>
    <td><b>58.61</b></td>
  </tr>
  <tr>
    <td>CGG</td>
    <td></td>
    <td>39.66</td>
    <td><b>31.01</b></td>
    <td><b>10.75</b></td>
    <td>23.43</td>
    <td><b>69.06</b></td>
    <td>73.49</td>
    <td><b>80.11</b></td>
    <td><b>81.11</b></td>
    <td>70.37</td>
    <td>67.82</td>
    <td>69.53</td>
    <td>70.01</td>
    <td>57.20</td>
  </tr>
  <tr>
    <td>ARG-Q</td>
    <td></td>
    <td>41.02</td>
    <td>31.57</td>
    <td>5.18</td>
    <td>15.31</td>
    <td>63.31</td>
    <td>67.84</td>
    <td>73.38</td>
    <td>73.04</td>
    <td>69.26</td>
    <td>68.04</td>
    <td>70.04</td>
    <td>68.44</td>
    <td>56.25</td>
  </tr>
  <tr>
    <td>ARG-QA</td>
    <td></td>
    <td>21.97</td>
    <td>16.67</td>
    <td>3.06</td>
    <td>12.47</td>
    <td>60.70</td>
    <td>67.49</td>
    <td>71.16</td>
    <td>70.88</td>
    <td>71.76</td>
    <td><b>69.01</b></td>
    <td>71.04</td>
    <td>71.33</td>
    <td>50.63</td>
  </tr>
  <tr>
    <td rowspan="6">Legal LLM (CGG )</td>
    <td>DISC-LawLLM</td>
    <td></td>
    <td>38.11</td>
    <td>21.37</td>
    <td>6.75</td>
    <td>16.96</td>
    <td>60.84</td>
    <td><b>73.42</b></td>
    <td>72.14</td>
    <td>71.79</td>
    <td>63.92</td>
    <td>67.42</td>
    <td>68.22</td>
    <td>65.45</td>
    <td>52.20</td>
  </tr>
  <tr>
    <td>fuzi.mingcha</td>
    <td></td>
    <td>66.55</td>
    <td>28.95</td>
    <td>9.51</td>
    <td>22.69</td>
    <td>67.06</td>
    <td>70.73</td>
    <td>76.66</td>
    <td>77.47</td>
    <td>65.92</td>
    <td>66.94</td>
    <td><b>69.28</b></td>
    <td>68.69</td>
    <td>57.54</td>
  </tr>
  <tr>
    <td>LexiLaw</td>
    <td></td>
    <td>57.74</td>
    <td>29.01</td>
    <td>8.93</td>
    <td>23.83</td>
    <td>65.63</td>
    <td>70.36</td>
    <td>76.67</td>
    <td>75.97</td>
    <td>65.28</td>
    <td>66.93</td>
    <td>68.89</td>
    <td>68.03</td>
    <td>56.44</td>
  </tr>
  <tr>
    <td>Taling</td>
    <td></td>
    <td>50.16</td>
    <td>26.52</td>
    <td>9.16</td>
    <td>22.44</td>
    <td>65.35</td>
    <td>75.96</td>
    <td>73.83</td>
    <td>70.30</td>
    <td>64.65</td>
    <td>66.94</td>
    <td>67.56</td>
    <td>66.09</td>
    <td>54.91</td>
  </tr>
  <tr>
    <td>zhihai</td>
    <td></td>
    <td>26.29</td>
    <td>21.38</td>
    <td>6.00</td>
    <td>15.53</td>
    <td>64.47</td>
    <td>65.59</td>
    <td>76.38</td>
    <td>77.37</td>
    <td>67.93</td>
    <td>66.30</td>
    <td>63.17</td>
    <td>59.82</td>
    <td>50.85</td>
  </tr>
  <tr>
    <td>LawGPT_zh</td>
    <td></td>
    <td>47.10</td>
    <td>29.16</td>
    <td>8.92</td>
    <td>22.55</td>
    <td>67.64</td>
    <td>69.48</td>
    <td><b>79.37</b></td>
    <td><b>80.23</b></td>
    <td>66.90</td>
    <td><b>68.38</b></td>
    <td><b>67.55</b></td>
    <td><b>68.94</b></td>
    <td>56.35</td>
  </tr>
  <tr>
    <td>HanFei</td>
    <td></td>
    <td><b>75.72</b></td>
    <td><b>32.98</b></td>
    <td><b>12.46</b></td>
    <td><b>26.91</b></td>
    <td><b>68.72</b></td>
    <td>73.25</td>
    <td>78.63</td>
    <td>78.11</td>
    <td><b>67.03</b></td>
    <td>67.45</td>
    <td>68.63</td>
    <td>67.73</td>
    <td><b>59.80</b></td>
  </tr>
</table>

Table 3: Performance comparisons on the Practitioner dataset. The best performance is indicated in bold.

6.2.2 Performance of Legal LLM

Law article training achieves gains. In the Layperson dataset, LexiLaw achieves optimal performance overall. This is because the questions in the Layperson dataset often require only law articles to provide answers clearly, and LexiLaw’s training explicitly used law articles, allowing it to effectively handle such questions.

Full-parameter training offers advantages. Hanfei achieves the best results in the Practitioner dataset, as it is a fully parameter-trained legal LLM. Full-parameter fine-tuning allows it to effectively simulate a legal expert, thus performing well.

Syllogistic reasoning is useful. fuzi.mingcha performs well on syllogism evaluation metrics, particularly on the Layperson dataset. This is due to its fine-tuning of syllogism judgment data.

6.2.3 Open Domain LLM vs. Legal LLM

Impact of LLM Backbone. We can observe that some legal LLMs perform worse than open-domain LLMs. This is because Qwen2 and Llama3 are the latest open-domain LLMs, and their overall capabilities have significantly improved. In contrast, most legal LLMs are built on earlier generations of LLMs, which have weaker base models, leading to poorer overall performance.

Effectiveness of legal knowledge. Overall, the upper limit of legal LLMs is higher than that of open-domain LLMs. This is because legal LLMs, after extensive training on legal knowledge, have developed strong capabilities in solving legal issues. As a result, even though their base models are outdated, they can still perform effectively.

6.3 Human Evaluation

In this section, we compared the syllogism-level metric with human evaluation. Details of legal human annotators can be found in Appendix D. The syllogism-level evaluation of citation quality is divided into two stages: Stage 1: Extracting key components. Stage 2: Assessing the entailment using an NLI model.

Stage 1: We randomly selected 50 questions each from the Layperson and Practitioner datasets. After splitting the cases into individual clauses, annotators were provided with the full case and its clauses. They do a three-class classification of each clause. The Qwen2’s annotations were then compared with human annotations. The Cohen’s kappa coefficient (Cohen, 1960) of 0.7876 indicates substantial agreement (0.61–0.80) between the model’s and human annotators’ labels.

Stage 2: We randomly selected 50 questions from the Practitioner dataset and used Qwen2 to extract key components of pairs of responses and citations. Annotators assessed the degree to which the citations entailed the corresponding response components using a 5-point scale (1: low, 5: high), with descriptions provided in Appendix D. The entailment probabilities given by DISC-LawLLM, which range from 0 to 1, were scaled to the same 1–5 range by multiplying by 5 and rounding. We then compared the scaled model outputs with the human evaluations and calculated Cohen’s kappa coefficient. The kappa score of 0.6923 again indicates substantial agreement (0.61–0.80) between the model and human judgments.

6.4 Effects on Different Retrieval Models

We selected BGE as the retrieval model in the main experiment. In this section, we explore the impact of using different retrieval models. Specifically, we evaluate Criminal-BERT (Zhong et al., 2019) and Civil-BERT (Zhong et al., 2019), two legal domain models based on BERT, fine-tuned on large-scale criminal and civil law documents, respectively. We replaced the retrieval model and tested the CGG
Figure 3: Performance of different retrieval models. Lay is short for Layperson dataset and Pra is short for Practitioner dataset.

(a) Methods for CitaLaw metric with Layperson dataset.

(b) Metrics for CGG method with Practitioner dataset.

Figure 4: The performance of different NLI models when the LLM is Llama.

method on the Layperson dataset. The average results across all metrics are shown in Figure 3, with detailed metric results provided in Appendix E.

As shown, on the Layperson dataset, BGE significantly outperforms the other two models. This is because the dataset consists of questions from laypersons, which are more everyday in nature. In contrast, the two legal BERT models, having been trained extensively on legal cases, show a distributional mismatch with open-domain data, leading to poorer performance. On the Practitioner dataset, which features professional legal questions, BGE still achieves the best performance. This can be attributed to its extensive training on diverse data, likely including some legal data, and its use of more advanced model architectures and techniques. However, the two legal BERT models perform comparably to BGE, showcasing the benefits of their specialized training on legal data.

6.5 Effects on Different NLI Models

We opted to use legal LLMs as the NLI model in our experiments, as they support longer input lengths and incorporate substantial legal knowledge. In Section 6.3, we verified that DISC-LawLLM and human achieved good consistency. In this section, we explore the performance of several legal LLMs in the NLI task. Besides DISC-LawLLM, we evaluated LexiLaw, LawGPT_zh, and Hanfei, which demonstrated strong performance in the main experiments.

In Figures 4 (a), we examined the ability of four legal LLMs to evaluate Llama across the Close-Book, CGG, ARG-Q, and ARG-QA methods using the CitaLaw metric on the Layperson dataset. In Figures 4 (b), we investigated the performance of four legal LLMs in evaluating the CGG method applied to Llama across the metrics CitaLaw, Cita_c, Cita_a, and Cita_d on the Practitioner dataset.

We can observe that Hanfei provides lower entailment scores across both datasets. This is because it is a fully parameter-tuned legal LLM, which results in a diminished capability to handle the general task of entailment reasoning. Additionally, we found that on the Practitioner dataset, other legal LLMs achieved results closer to those of DISC-LawLLM, while on the Layperson dataset, the performance gap was significantly larger. This is because the Practitioner dataset is more judicially oriented, aligning with the knowledge seen during the fine-tuning of legal LLMs. In contrast, due to limited training on general-purpose data, other legal LLMs struggle to accurately determine entailment relationships in the Layperson dataset. Similar conclusions can be drawn when the LLM is Qwen in Appendix F.

7 Conclusion

We introduce CitaLaw, a benchmark designed to explore LLMs to generate responses with citations in legal scenarios, thus improving the trustworthiness of LLMs. CitaLaw includes two categories of questions: laypersons and practitioners. For laypersons, CitaLaw provides law articles as citations to help them understand the LLM’s response clearly. For practitioners, both law articles and precedent cases are provided as citations, better supporting their needs for complex reasoning. CitaLaw offers global-level and syllogism-level metrics and supports the integration of citations into LLM inputs to guide generation or using citations to refine LLM’s response. We conducted extensive experiments on 7 legal-domain LLMs and 2 popular open-domain LLMs, providing valuable insights for the deployment of LLMs in legal scenarios.
8 Limitations

While CitaLaw provides a robust framework for evaluating LLMs in legal scenarios, several limitations should be acknowledged to guide future extensions of this work.

First, the datasets used in CitaLaw are primarily sourced from the Chinese legal system, which may limit the benchmark’s applicability to other jurisdictions. However, by incorporating both law articles and precedent cases to align with the principles of civil and common law systems, CitaLaw demonstrates strong potential for adaptation to diverse legal contexts.

Second, the syllogism-based evaluation framework simplifies legal reasoning into three key components: the major premise (law articles or precedent cases), the minor premise (case circumstances and actions), and the conclusion (legal decision). While this structured approach is effective for systematic evaluation, real-world legal reasoning may encompass additional complexities.

9 Ethical Considerations

Data Privacy and Confidentiality. The legal datasets used in CitaLaw include law articles, precedent cases, user questions, and golden responses. These documents were sourced from publicly available databases, ensuring compliance with data privacy and confidentiality standards. We carefully reviewed the datasets to ensure that no personally identifiable information (PII) or sensitive details about individuals were inadvertently included.

Alignment with Legal Standards. Legal AI systems must align with the ethical and professional standards of the legal domain. Our work emphasizes the need for syllogism-based reasoning to ensure logical consistency and adherence to legal principles.

Transparency and Explainability. Legal reasoning must be transparent and interpretable, particularly when used in sensitive or high-stakes domains. The metrics proposed in CitaLaw, including syllogism-based evaluation, aim to improve explainability by breaking down the reasoning process into logical components.

Responsibility in System Deployment. CitaLaw is intended as a research benchmark and should not be directly deployed in high-stakes legal decision-making without human oversight. While the benchmark aims to enhance the trustworthiness of LLM-generated responses, legal professionals should always verify the citations and legal interpretations provided by such systems. Misuse of automated systems without adequate validation could lead to inaccurate legal advice or unintended consequences in legal proceedings.

10 Acknowledgements

This work was funded by the National Key R&D Program of China (2023YFA1008704), the National Natural Science Foundation of China (62472426). Supported by fund for building world-class universities (disciplines) of Renmin University of China. Work partially done at Beijing Key Laboratory of Research on Large Models and Intelligent Governance, and Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE. Supported by the Beijing Social Science Foundation Planning Project (Grant No. 24GLC041), the Fundamental Research Funds for the Central Universities in UIBE (Grant No. 24QN06, 24PYTS22).

References

Abdelrahman Abdallah, Bhawna Piryani, and Adam Jatowt. 2023. Exploring the state of the art in legal qa systems. Journal of Big Data, 10(1):127.

AI@Meta. 2024. Llama 3 model card.

Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. 2023. Can gpt-3 perform statutory reasoning? In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, pages 22–31.

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.

Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, and Hao Wang. 2023. Laiw: A chinese legal large language models benchmark (a technical report). arXiv preprint arXiv:2310.05620.

Haolin Deng, Chang Wang, Xin Li, Dezhang Yuan, Junlang Zhan, Tianhua Zhou, Jin Ma, Jun Gao, and Ruifeng Xu. 2024. Webcites: Attributed query-focused summarization on chinese web search results with citations. arXiv preprint arXiv:2403.01774.

Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen
Shen, and Jidong Ge. 2023. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289.

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, et al. 2023a. Rarr: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16477–16508.

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023b. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488.

Wanwei He, Jiabao Wen, Lei Zhang, Hao Cheng, Bowen Qin, Yunshui Li, Feng Jiang, Junying Chen, Benyou Wang, and Min Yang. 2023. Hanfei-1.0. https://github.com/siat-nlp/HanFei.

Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python.

Dongfang Li, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Ziyang Chen, Baotian Hu, Aigou Wu, and Min Zhang. 2023. A survey of large language models attribution. arXiv preprint arXiv:2311.03731.

Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yicun Liu. 2024. Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. Preprint, arXiv:2409.20288.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Hongcheng Liu, Yusheng Liao, Yutong Meng, and Yuhao Wang. 2023. Xiezhi: Chinese law large language model. https://github.com/LiuHC0428/LAW_GPT.

Luyao Ma, Yating Zhang, Tianyi Wang, Xiaozhong Liu, Wei Ye, Changlong Sun, and Shikun Zhang. 2021a. Legal judgment prediction with multi-stage case representation learning in the real court setting. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 993–1002.

Yixiao Ma, Yunqiu Shao, Yueyue Wu, Yiqun Liu, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2021b. Lecard: A legal case retrieval dataset for chinese law system. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2342–2348.

Chaitanya Malaviya, Subin Lee, Sihaoo Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024. Expertqa: Expert-curated questions and attributed answers. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3025–3045.

Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816–4828.

N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.

Jaromir Savelka, Kevin D Ashley, Morgan A Gray, Hannes Westermann, and Huihui Xu. 2023. Explaining legal concepts with augmented large language models (gpt-4). arXiv preprint arXiv:2306.09525.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierre Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45.

Shiguang Wu, Zhongkun Liu, Zhen Zhang, Zheng Chen, Wentao Deng, Wenhao Zhang, Jiyuan Yang, Zhitao Yao, Yougang Lyu, Xin Xin, Shen Gao, Pengjie Ren, Zhaochun Ren, and Zhumin Chen. 2023a. fuzi.mingcha. https://github.com/irlab-sdu/fuzi.mingcha.

Yiquan Wu, Yuhang Liu, Yifei Liu, Ang Li, Siying Zhou, and Kun Kuang. wisdominterrogatory. Available at GitHub.

Yiquan Wu, Siying Zhou, Yifei Liu, Weiming Lu, Xiaozhong Liu, Yating Zhang, Changlong Sun, Fei Wu, and Kun Kuang. 2023b. Precedent-enhanced legal judgment prediction with llm and domain-model collaboration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12060–12075.

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muenighoff, Defu Lian, and Jian-Yun Nie. 2023. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu
Cui, Zhenru Zhang, and Zhihao Fan. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Fangyi Yu, Lee Quartey, and Frank Schilder. 2022a. Legal prompting: Teaching a language model to think like a lawyer. arXiv preprint arXiv:2212.01326.

Weijie Yu, Zhongxiang Sun, Jun Xu, Zhenhua Dong, Xu Chen, Hongteng Xu, and Ji-Rong Wen. 2022b. Explainable legal case matching via inverse optimal transport-based rationale extraction. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, pages 657–668.

Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, and Zhongyu Wei. 2023. Disc-lawllm: Fine-tuning large language models for intelligent legal services. Preprint, arXiv:2309.11325.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Jec-qa: a legal-domain question answering dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9701–9708.

Haoxi Zhong, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2019. Open chinese language pretrained model zoo. Technical report.

Zhi Zhou, Jiang-Xin Shi, Peng-Xiao Song, Xiao-Wen Yang, Yi-Xuan Jin, Lan-Zhe Guo, and Yu-Feng Li. 2024. Lawgpt: A chinese legal knowledge-enhanced large language model. Preprint, arXiv:2406.04614.
A  The Used Prompts

Figure 5 illustrates the prompts used in this paper, including \( p_1, p_2, p_3 \) in Eq. 1, Eq. 2 and Eq. 3.

B  More Details of Evaluated Models and Datasets

For the Legal LLMs, we choose (1) fuzi.mingcha (6B) (Wu et al., 2023a): It leverages unsupervised judicial corpora for training and uses syllogistic reasoning judgment data for fine-tuning. (2) LexILaw\(^7\) (6B): It specifically utilizes legal articles and legal reference books for training. (3) Tailing\(^8\) (7B): It uses judicial text validation data, information extraction data, and judgment data for training. (4) DISC-LawLLM (13B) (Yue et al., 2023): In addition to fine-tuning with pairs, it also uses triplet data for fine-tuning to enhance the model’s ability to leverage external knowledge. (5) zhihai (7B) (Wu et al.): It utilizes ChatGPT to modify the existing dataset and then performs secondary pre-training. (6) LawGPT_zh (6B) (Liu et al., 2023): It primarily uses scenario-based dialogues and knowledge-based question-answering data for fine-tuning based on LoRA. (7) HanFei (7B) (He et al., 2023): It is the first fully parameter-trained legal LLM in China. Because in the main experiment, CGG has the best overall performance, for the legal LLMs, we generate responses using CGG.

Table 4 and Table 5 are the website URLs and corresponding licenses of the evaluated models and datasets.

C  More Details on Implementation

Considering the length of legal texts and the input window for the LLMs is limited, all experiments in this paper are conducted using a zero-shot setting. We use the Chinese-performing-well Qwen2-1.5B (Yang et al., 2024)\(^9\) to complete the MAUVE calculations. For RGUGE, We use version 1.0.1 of ROUGE for calculation. For BERTScore, we use bert-base-chinese (Devlin, 2018)\(^10\) to compute it. Regarding sentence-BERT, we employ paraphrase-multilingual-MiniLM-L12-v2 (Reimers, 2019)\(^11\).

\footnotetext{7https://github.com/CSHaitao/LexiLaw}
\footnotetext{8https://github.com/DUTIR-LegalIntelligence/Tailing}
\footnotetext{9https://huggingface.co/Qwen/Qwen2-1.5B}
\footnotetext{10https://huggingface.co/google-bert/bert-base-chinese}
\footnotetext{11https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2}

D  Human Evaluation

We hired four legal annotators from a Chinese university, all of whom have legal education backgrounds and are familiar with the cases in the dataset they need to annotate. We explained to the annotators that the data they annotated would be used for scientific research and paid them a reasonable remuneration based on local conditions. They are all graduate students from the judicial field, with practical experience in the legal profession. Two are male, two are female, aged between 24 and 30, and all have over five years of judicial theory study. Two annotators were responsible for the first stage of annotation, while the other two were responsible for the second stage, with all working together on the annotation process.

Table 6 shows a detailed description of each level used to evaluate the agreement of the NLI model with human evaluations.

E  Different Retrieval Models

Tables 7 and 8 present the performance of different retrieval models—Criminal-BERT, Civil-BERT, and BGE—on each metric for the CGG method across the two datasets. It can be observed that when Llama3 and Qwen2 are used as LLMs, BGE achieves the best performance as the retrieval model. Comparing the two datasets, on the Layerson dataset, where the questions are more general, Criminal-BERT and Civil-BERT, which focus on legal cases, perform relatively poorly. In contrast, on the Practitioner dataset, despite no structural or training improvements, Criminal-BERT and Civil-BERT achieve results comparable to BGE, highlighting the importance of legal knowledge in judicial QA tasks.

The differences between the two datasets also underscore the significance of selecting an appropriate retrieval model.

F  Different NLI Models

Figures 6 (a) and (b) show the entailment scores given by four legal LLMs as NLI models under different methods (CloseBook, CGG, ARG-Q, ARG-QA) and metrics(CitaLaw, CitaS, CitaB, and CitaC) when Qwen is used as the LLM. Similar conclusions to those in Section 6.5 can be drawn.
Answer the question based on the provided law article and cite it appropriately.
Only output the answer and citations, without including any additional content.
When citing the law article, use [A1] at the end of the relevant sentence.

Below is the provided law article:
Law article [A1]: {Law article 1}

Question: {Question}
Answer:

(a) Layperson

Refine the text based on the references and only output the refined text.

Text: {Answer}
Reference: {References}

Refined text (only output the Refined text, without any additional content):

(b) Practitioner

Answer the question.
Only output the answer without any additional content.

Question: {Question}
Answer:

(c) Response Refinement

(d) Without Reference

Figure 5: Prompts used in this paper. (a) The prompt \( p_1 \) is used to retrieve one law article in the Layperson dataset. (b) The prompt \( p_1 \) is used to retrieve one law article and three precedent cases in the Practitioner dataset. (c) The prompt \( p_3 \) is used to refine the LLM’s answer based on references. (d) The prompt \( p_2 \) is used for LLM responses without references.

<table>
  <tr>
    <th>Type</th>
    <th>LLM</th>
    <th>URL</th>
    <th>Licence</th>
  </tr>
  <tr>
    <td rowspan="2">Open domain</td>
    <td>Qwen2-7B-Instruct</td>
    <td>https://huggingface.co/Qwen/Qwen2-7B-Instruct</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>Llama3-8B-Instruct</td>
    <td>https://github.com/meta-llama/llama3</td>
    <td>META LLAMA 3 COMMUNITY License</td>
  </tr>
  <tr>
    <td rowspan="6">Legal Domain</td>
    <td>fuzi.mingcha</td>
    <td>https://github.com/irlab-sdu/fuzi_mingcha</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>DISC-LawLLM</td>
    <td>https://github.com/FudanDISC/DISC-LawLLM</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>LawGPT_zh</td>
    <td>https://github.com/LiuH0428/LAW-GPT</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>Hanfei</td>
    <td>https://github.com/slat-nlp/Hanfei</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>Tailing</td>
    <td>https://github.com/OUTIR-LegalIntelligence/Tailing</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>LexiLaw</td>
    <td>https://github.com/CShaito/LexiLaw</td>
    <td>MIT license</td>
  </tr>
  <tr>
    <td>zhilai</td>
    <td>https://github.com/zhilaiLLM/wisdomInterrogatory</td>
    <td>Apache-2.0 license</td>
  </tr>
</table>

Table 4: The LLM source URLs and licenses used by CitaLaw. The parts where the license is listed as empty indicate that the author has not provided a License.

![Line chart showing metrics for CGG method with Layperson dataset.](page_109_1042_627_246.png)

(a) Metrics for CGG method with Layperson dataset.

![Line chart showing methods for CitaLaw metric with Practitioner dataset.](page_109_1312_627_246.png)

(b) Methods for CitaLaw metric with Practitioner dataset.

Figure 6: The performance of different NLI models when the LLM is Qwen.
<table>
  <tr>
    <th>Type</th>
    <th>Dataset</th>
    <th>URL</th>
    <th>Licence</th>
  </tr>
  <tr>
    <td rowspan="2">Question</td>
    <td>Layperson</td>
    <td>https://github.com/open-compass/LawBench</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>Practitioner</td>
    <td>https://github.com/CSHaitao/LexEval</td>
    <td>MIT License</td>
  </tr>
  <tr>
    <td rowspan="8">Corpus</td>
    <td>LeCaRD</td>
    <td>https://github.com/myx666/LeCaRD</td>
    <td>MIT License</td>
  </tr>
  <tr>
    <td>ELAM</td>
    <td>https://github.com/Fuc-wjyu/IOT-Match</td>
    <td>MIT License</td>
  </tr>
  <tr>
    <td>CAIL2021-sfzy</td>
    <td>https://github.com/china-ai-law-challenge/CAIL2021</td>
    <td></td>
  </tr>
  <tr>
    <td>LJP-MSJudge</td>
    <td>https://github.com/mly-nlp/LJP-MSJudge</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>fuzi.mingcha</td>
    <td>https://github.com/irlab-sdu/fuzi_mingcha</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>DISC-LawLLM</td>
    <td>https://github.com/FudanDISC/DISC-LawLLM</td>
    <td>Apache-2.0 license</td>
  </tr>
  <tr>
    <td>LawGPT_zh</td>
    <td>https://github.com/LiuHC0428/LAW-GPT</td>
    <td></td>
  </tr>
  <tr>
    <td>Hanfei</td>
    <td>https://github.com/siat-nlp/HanFei</td>
    <td>Apache-2.0 license</td>
  </tr>
</table>

Table 5: The dataset source URLs and licenses used by CitaLaw. The parts where the license is listed as empty indicate that the author has not provided a License.

<table>
  <tr>
    <th>Score</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>1</td>
    <td>No Entailment: The former does not entail the latter at all, with no logical connection between the two.</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Weak Entailment: A partial entailment where the former somewhat relates to the latter, but the connection is weak and not fully conclusive.</td>
  </tr>
  <tr>
    <td>3</td>
    <td>Moderate Entailment: A moderate degree of entailment, meaning the former generally leads to the latter in most cases, but exceptions exist.</td>
  </tr>
  <tr>
    <td>4</td>
    <td>Strong Entailment: A strong logical relationship where the former can derive the latter in the vast majority of cases.</td>
  </tr>
  <tr>
    <td>5</td>
    <td>Complete Entailment: The former fully entails the latter in all cases, with an unambiguous and definitive logical connection between them.</td>
  </tr>
</table>

Table 6: Scoring Criteria for Human Evaluation of Entailment.

<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="4">Fluency</th>
    <th colspan="4">Correctness</th>
    <th rowspan="2">Citation</th>
    <th rowspan="2">All</th>
  </tr>
  <tr>
    <th>Category</th>
    <th>Retriever</th>
    <th>Mauve</th>
    <th>Rouge-1</th>
    <th>Rouge-2</th>
    <th>Rouge-L</th>
    <th>BERT-F</th>
    <th>Correct<sub>r</sub></th>
    <th>Correct<sub>c</sub></th>
    <th>Correct<sub>d</sub></th>
    <th>Cita<sub>Layp</sub></th>
    <th>Avg</th>
  </tr>
  <tr>
    <td rowspan="3">(Llama3-8B-Instruct)</td>
    <td>Criminal</td>
    <td></td>
    <td>37.44</td>
    <td>18.07</td>
    <td>2.18</td>
    <td>13.15</td>
    <td>61.71</td>
    <td>64.03</td>
    <td>63.56</td>
    <td>64.36</td>
    <td>80.34</td>
    <td>44.98</td>
  </tr>
  <tr>
    <td>Civil</td>
    <td></td>
    <td>56.16</td>
    <td>18.27</td>
    <td>2.34</td>
    <td>13.44</td>
    <td>61.90</td>
    <td>63.22</td>
    <td>63.89</td>
    <td>63.35</td>
    <td>80.97</td>
    <td>47.06</td>
  </tr>
  <tr>
    <td>BGE</td>
    <td></td>
    <td>61.01</td>
    <td>23.97</td>
    <td>6.05</td>
    <td>17.91</td>
    <td>65.94</td>
    <td>67.29</td>
    <td>77.31</td>
    <td>74.95</td>
    <td>86.70</td>
    <td><b>53.46</b></td>
  </tr>
  <tr>
    <td rowspan="3">(Qwen2-7B-Instruct)</td>
    <td>Criminal</td>
    <td></td>
    <td>55.26</td>
    <td>21.09</td>
    <td>4.53</td>
    <td>14.32</td>
    <td>64.73</td>
    <td>63.10</td>
    <td>64.89</td>
    <td>65.85</td>
    <td>61.60</td>
    <td>46.15</td>
  </tr>
  <tr>
    <td>Civil</td>
    <td></td>
    <td>52.44</td>
    <td>20.48</td>
    <td>4.16</td>
    <td>13.81</td>
    <td>64.45</td>
    <td>61.79</td>
    <td>64.94</td>
    <td>65.62</td>
    <td>59.88</td>
    <td>45.29</td>
  </tr>
  <tr>
    <td>BGE</td>
    <td></td>
    <td>75.10</td>
    <td>22.26</td>
    <td>4.77</td>
    <td>15.41</td>
    <td>65.28</td>
    <td>67.50</td>
    <td>78.62</td>
    <td>77.82</td>
    <td>77.59</td>
    <td><b>53.82</b></td>
  </tr>
</table>

Table 7: Performance comparisons on retrieval models in the Layperson dataset when the method is CGG. The best performance is indicated in bold.

<table>
  <tr>
    <th rowspan="2">Metric</th>
    <th colspan="4">Fluency</th>
    <th colspan="4">Correctness</th>
    <th rowspan="2">Citation</th>
    <th rowspan="2">All</th>
  </tr>
  <tr>
    <th>Category</th>
    <th>Retriever</th>
    <th>Mauve</th>
    <th>Rouge-1</th>
    <th>Rouge-2</th>
    <th>Rouge-L</th>
    <th>BERT-F</th>
    <th>Correct<sub>r</sub></th>
    <th>Correct<sub>c</sub></th>
    <th>Correct<sub>d</sub></th>
    <th>Cita<sub>Prac</sub></th>
    <th>Avg</th>
  </tr>
  <tr>
    <td rowspan="3">(Llama3-8B-Instruct)</td>
    <td>Criminal</td>
    <td></td>
    <td>34.25</td>
    <td>25.78</td>
    <td>7.80</td>
    <td>19.42</td>
    <td>65.02</td>
    <td>76.01</td>
    <td>76.01</td>
    <td>76.01</td>
    <td>66.01</td>
    <td>69.09</td>
    <td>54.03</td>
  </tr>
  <tr>
    <td>Civil</td>
    <td></td>
    <td>39.84</td>
    <td>26.39</td>
    <td>8.07</td>
    <td>20.02</td>
    <td>65.27</td>
    <td>65.41</td>
    <td>75.78</td>
    <td>75.73</td>
    <td>69.21</td>
    <td>75.52</td>
    <td>69.54</td>
    <td>60.16</td>
  </tr>
  <tr>
    <td>BGE</td>
    <td></td>
    <td>36.37</td>
    <td>26.15</td>
    <td>7.84</td>
    <td>19.55</td>
    <td>65.60</td>
    <td>67.19</td>
    <td>76.36</td>
    <td>77.73</td>
    <td>73.58</td>
    <td>68.23</td>
    <td>67.87</td>
    <td><b>67.65</b></td>
  </tr>
  <tr>
    <td rowspan="3">(Qwen2-7B-Instruct)</td>
    <td>Criminal</td>
    <td></td>
    <td>32.49</td>
    <td>31.79</td>
    <td>11.09</td>
    <td>23.93</td>
    <td>69.79</td>
    <td>72.00</td>
    <td>80.81</td>
    <td>81.53</td>
    <td>68.42</td>
    <td>68.42</td>
    <td>71.86</td>
    <td>71.54</td>
    <td>56.97</td>
  </tr>
  <tr>
    <td>Civil</td>
    <td></td>
    <td>33.37</td>
    <td>31.67</td>
    <td>11.06</td>
    <td>23.84</td>
    <td>69.63</td>
    <td>73.35</td>
    <td>80.57</td>
    <td>81.27</td>
    <td>69.11</td>
    <td>66.41</td>
    <td>70.09</td>
    <td>69.47</td>
    <td>56.65</td>
  </tr>
  <tr>
    <td>BGE</td>
    <td></td>
    <td>39.66</td>
    <td>31.01</td>
    <td>10.75</td>
    <td>23.43</td>
    <td>69.06</td>
    <td>73.49</td>
    <td>80.11</td>
    <td>81.11</td>
    <td>70.37</td>
    <td>67.82</td>
    <td>69.53</td>
    <td>70.01</td>
    <td><b>57.20</b></td>
  </tr>
</table>

Table 8: Performance comparisons on retrieval models in the Practitioner dataset when the method is CGG. The best performance is indicated in bold.

Reviewer paper summaries: This paper introduces CitaLaw, a benchmark for evaluating LLMs' ability to generate legal responses with appropriate citations. 

The key features of the proposed benchmark include:
- Two subsets of questions tailored for laypersons and legal practitioners
- A comprehensive corpus of law articles and precedent cases as references
- A two-level evaluation framework combining global metrics with a syllogism-based approach

The authors conduct extensive experiments using 2 open-domain and 7 legal-specific LLMs to demonstrate the effectiveness of integrating legal references into response generation. They also validate their proposed syllogism-based evaluation method and provide insights for deploying LLMs in legal scenarios.

This is an interesting applied NLP paper in the Chinese legal context. They produce a legal QA dataset, with some of the questions also requiring citation to a statute or precedent. There is an associated knowledge base of statutes ("law articles") and case precedents. 

They explore different RAG/LLM approaches to answering the questions.
Summary of strengths: 1. Generating responses with citations is important and practical in LegalAI scenarios. The proposed benchmark is in time and useful for this field.

2. The benchmark is comprehensive and well-designed. It thoughtfully incorporates different user types (laypersons and practitioners), diverse reference types (law articles and precedent cases), and a well-structured corpus.

3. The proposed syllogism-based evaluation approach is professional, which goes beyond surface-level metrics to evaluate logical coherence and alignment with legal principles.

Nice new dataset which will push forward applied NLP work in the Chinese legal domain, and I hope it inspires similar work in other languages.

Interesting findings!

* having access to the knowledge base helps a lot in answer generation. 
* general LLMs with access to the knowledge base outperform fine-tuned legal LLMs.
Summary of weaknesses: 1. The scale of the benchmark is relatively small, which only contains 1K questions in total. While I understand that collecting data and mapping the citations can be expensive, there are still prior resources that the authors can utilize. For example, the EQUALS[1] dataset contains over 6K questions from real-world users and provides the corresponding law articles, which is not mentioned in the submission.

2. The evaluation can can be further polished. (1) Though the authors included lots of results of LLaMA and other LegalLLMs, I am wondering how those closed-source LLMs perform on this task. For example, perplexity (https://www.perplexity.ai/) seems to be a good solution to this task. How's its performance look like? (2) It would be good to know human experts' performance in this task.


[1] EQUALS: A Real-world Dataset for Legal Question Answering via Reading Chinese Laws, ICAIL 2023

The project could be better situated against similar work (see in suggestions below). 

Although it is highlighted as a contribution, it was not clear to me what is added by the syllogism method.
Comments, suggestions and typos: Typo: Line 436 "Peformance" should be "Performance".

The authors should change references to "law articles" to "statutes" (or "code sections" or "legislation"). Most readers would expect "law article" to refer to an academic law journal article.  

The tables are a little hard to understand and it would be good to have notes underneath them explaining what all the rows and columns refer to. What are correct_c, correct_a, correct_d? it seems to be relative to the syllogisms but it was not explained enough.

Missing references (and discussion of how CitaLaw differs, there might also be more related things):

* Mahari et al, "LePaRD: A Large-Scale Dataset of Judicial Citations to Precedent", ACL (2024)

* Jiang et al, "Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling", ACL (2024)
Overall assessment scores: 3

4
Metareview score: 4
Metareview: The paper proposes a new Chinese legal QA dataset and experiments with providing citations to concrete statutes to increate trustworthyness of the generated answers.
Summary of reasons to publish: The reviewers agree that this paper provides enough novelty in the area (legal QA in Chinese) and the new dataset is of interest to the community.