UniLR: Unleashing the Power of LLMs on Multiple Legal Tasks with a Unified Legal Retriever
Authors: Ang Li, Yiquan Wu, Yifei Liu, Ming Cai, Lizhi Qing, Shihang Wang, Yangyang Kang, Chengyuan Liu, Fei Wu, Kun Kuang
Accepted at: acl25

UniLR: Unleashing the Power of LLMs on Multiple Legal Tasks with a Unified Legal Retriever

Ang Li1, Yiquan Wu2†, Yifei Liu3, Ming Cai1†, Lizhi Qing5, Shihang Wang5
Yangyang Kang1,4, Chengyuan Liu1, Fei Wu1, Kun Kuang1†

1College of Computer Science and Technology, Zhejiang University, 2Guanghua Law School, Zhejiang University,
3 College of Software Technology, Zhejiang University, 4Polytechnic Institute, Zhejiang University,
5Alibaba Group, Hangzhou, China
{leeyon, wuyiquan, liuyifei, cm, yangyangkang, liucy1, wufei, kunkuang}@zju.edu.cn
{yekai qlz, wangshihang.wsh}@alibaba-inc.com

Abstract

Despite the impressive capabilities of LLMs, they often generate content with factual inaccuracies in LegalAI, which may lead to serious legal consequences. Retrieval-Augmented Generation (RAG), a promising approach, can conveniently integrate specialized knowledge into LLMs. In practice, there are diverse legal knowledge retrieval demands (e.g. law articles and similar cases). However, existing retrieval methods are either designed for general domains, struggling with legal knowledge, or tailored for specific legal tasks, unable to handle diverse legal knowledge types. Therefore, we propose a novel Unified Legal Retriever (UniLR) capable of performing multiple legal retrieval tasks for LLMs. Specifically, we introduce attention supervision to guide the retriever in focusing on key elements during knowledge encoding. Next, we design a graph-based method to integrate meta information through a heterogeneous graph, further enriching the knowledge representation. These two components work together to enable UniLR to capture the essence of knowledge hidden beneath formats. Extensive experiments on multiple datasets of common legal tasks demonstrate that UniLR achieves the best retrieval performance and can significantly enhance the performance of LLM.

1 Introduction

Legal artificial intelligence (LegalAI) (Zhong et al., 2020a) focuses on applying artificial intelligence to benefit legal tasks (Zhong et al., 2018, 2020b; Wu et al., 2020, 2022). Recently, the focus has shifted towards leveraging large language models (LLMs) to enhance legal task performance (Fei et al., 2024). However, LLMs still struggle with factual inaccuracies (Mallen et al., 2023; Min et al., 2023), which can lead to serious legal consequences. A promising solution is Retrieval-Augmented Generation (RAG) (Ram et al., 2023), where a retriever fetches relevant knowledge from an external corpus and combines it with the query to generate a more accurate response.

Legal knowledge has various types such as charge definition, law articles, similar cases, and so on (Burton, 2007). As shown in Fig. 1, in LegalAI practice, addressing various legal tasks requires retrieving different types of legal knowledge. Generally, existing retrieval methods can be divided into two types: Sparse retrieval, like BM25 (Robertson and Zaragoza, 2009) and TF-IDF (Sparck Jones, 1972), can be applied to multiple tasks but struggle with processing complex legal knowledge. Dense retrieval, while effective for specific legal tasks through fine-tuning, cannot handle diverse retrieval needs (Wang et al., 2018, 2019a; Li et al., 2023a). Therefore, it is meaningful to develop a unified retriever that can address multiple legal retrieval tasks.

However, the main challenge in implementing such a retriever lies in the legal knowledge format

![A real-world legal example. For a fact description, if the query asks for applicable articles, relevant articles should be retrieved; if it asks for a possible court view, similar cases should be retrieved for reference.](page_1016_186_388_312.png)

Figure 1: A real-world legal example. For a fact description, if the query asks for applicable articles, relevant articles should be retrieved; if it asks for a possible court view, similar cases should be retrieved for reference.
problem. As shown in Fig. 1, the challenge manifests in two aspects: (1) Diverse formats across different knowledge types. For example, law articles are filled with concise legal terminology, while relevant cases are lengthy fact descriptions, which hinders the retriever’s understanding. (2) Subtle differences within the same knowledge type. In the legal domain, differing only in a few words can yield markedly different legal implications, which may trigger the retriever’s confusion. Notably, our findings show that, for the query-relevant knowledge, despite format varying, key elements\(^1\) (e.g., defendant’s identity, behavior’s target) are similar, and meta information\(^1\) (e.g., mapping knowledge, hierarchy, legal event schema) remains consistent. Irrelevant but confusing knowledge can also be distinguished using these concepts.

Based on this insight, we propose UniLR, a novel Unified Legal Retriever for handling multiple legal retrieval tasks by leveraging key elements and meta information. First, we develop a knowledge mining process. For key elements, we distill expertise from a legal model (Yao et al., 2022), using its attention distribution to identify key elements. For meta information, we construct a heterogeneous knowledge graph where legal knowledge, hierarchical structures, and legal event schemas are represented as different node types, with inter-type and cross-type relationships established based on carefully crafted edge construction rules.

To leverage the mined key elements and meta information, based on a dense embedding model, we design two innovative components: Key Element Supervisor (KES) and Graph-based Knowledge Augmenter (GKA). KES leverages recorded attention and a supervision loss to align the model’s focus on key elements. GKA combines graph attention and convolution to aggregate meta information from the graph, enhancing knowledge representation. Finally, we train UniLR using contrastive learning.

Multiple datasets for common legal tasks are experimented in this paper, including charge prediction, law article prediction, court view generation, and legal question answering. Empirical results demonstrate that UniLR achieves the best retrieval performance and significantly enhances LLM’s performance. To summarize, our contributions are:

• We investigate multiple legal retrieval tasks for RAG in the LLM era, considering the diverse formats and knowledge confusion.

• We define key elements and meta information of legal knowledge and mine them through expertise distillation and heterogeneous knowledge graph construction, respectively.

• We propose UniLR, a Unified Legal Retriever with two innovative components: KES introduces attention supervision to guide the retriever in focusing on key elements and GKA combines graph attention and convolution to aggregate meta information.

• Extensive experiments on multiple datasets of common legal tasks demonstrate that UniLR achieves the best retrieval performance and significantly enhances LLM capabilities. All data and code are publicly available \(^2\).

2 Related Work

2.1 Legal Artificial Intelligence

In recent years, researchers have focused on using NLP technology to solve specific tasks in the legal field, such as charge prediction (Zhong et al., 2018; Yang et al., 2019; Xu et al., 2020; Yue et al., 2021a; Wu et al., 2022; Chalkidis et al., 2020), article recommendation (Chen et al., 2013; Raghav et al., 2016; Louis and Spanakis, 2022), case retrieval (Raghav et al., 2016; Shao et al., 2020; Li et al., 2023a), court view generation (Wu et al., 2020; Yue et al., 2021b; Li et al., 2024b; Liu et al., 2024), and legal question answering (Zhong et al., 2020b; Kien et al., 2020; Louis et al., 2024). Recently, with the development of LLMs, researchers have transformed various legal tasks into question-and-answer pairs to fine-tune the LLMs, hoping to build unified legal LLMs to solve problems (Cui et al., 2023a; Liu et al., 2023; Huang et al., 2023). However, due to the lack of domain knowledge, both universal LLMs and legal LLMs perform poorly on some legal tasks. Previous work enhances legal judgment prediction and court view generation through various forms of knowledge injection, such as charge definitions, document templates, and reasoning rules (Li et al., 2024a; Zhou et al., 2024a; Li et al., 2025). Because LLM can read text form knowledge directly, a promising approach is enhancing LLMs through retrieval. In the legal

\footnotetext{1The detailed definitions and acquisition of “key elements” and “meta information” are provided in Sec. 3.2}
\footnotetext{2https://github.com/LIANG-star177/ULKR}
domain, researchers have explored retrieval for legal cases and articles (Li et al., 2023a; Wang et al., 2019a), yet there lacks a unified retriever for legal knowledge.

2.2 LLMs and Retrieval
LLMs can learn the knowledge in the retrieved information, which has been validated on many LLMs, such as GPT-3 (Brown et al., 2020), GPT-Neo (Black et al., 2021), and LLaMA (Touvron et al., 2023). In traditional retrieval methods, researchers generally use the BM25 algorithm or dense retrievers based on pre-trained models to retrieve from the training set (Liu et al., 2022; Rubin et al., 2022; Izacard et al., 2022). Recently, researchers have studied the harmonious integration of large models and retrieval (Li et al., 2023b; Luo et al., 2023; Lv et al., 2025). Ge et al. (2023) retrieves knowledge from diverse sources to improve query representation. Ma et al. (2023) uses reinforcement learning to train a query optimizer for semantic alignment. Wang et al. (2023) incorporates contextual recall in pre-training to familiarize models with RAG patterns. However, compared to general retrieval, legal knowledge presents diverse formats and potential confusion, requiring domain-specific solutions. This paper proposes a unified legal knowledge retriever to enhance LLMs in legal tasks.

3 Methodology
In this section, we first formalize the RAG problem for utilizing LLMs in legal tasks and then introduce our UniLR by detailing two parts: Knowledge Mining and Model Architecture. The overall approach of UniLR is illustrated in Fig. 2.

3.1 Problem Formulation
In this paper, we aim to develop a unified retriever that can perform different legal retrieval tasks. We first formulate the problem as follows:

Given a test example \((q, y)\) in task \(t\), where \(q\) is a legal query, \(y\) is the true label, and the specific knowledge corpus \(C_t = \{c_1, ..., c_{n_c}\}\) consisting of \(n_c\) knowledge entries, the probability of the LLM generating the target \(y\) based on the query \(q\) is defined as follows:

\[
p(y|q) = LLM(y|T(c_1, ..., c_k; q))
\]

Here, \(k\) is the number of retrieved knowledge entries, \(T\) is the template for packaging the retrieved knowledge and the query. We expect the retriever to seamlessly adapt to task transitions.

3.2 Knowledge Mining
3.2.1 Key elements extraction
According to previous work, legal knowledge often describes one or more events (Shen et al., 2020; Li et al., 2020). This paper’s key elements correspond to the subjects, objects, and triggers that constitute a legal event. LEVEN (Yao et al., 2022) is a specialized legal event dataset, which groups legal facts into 108 event types, annotating them based on subjects, objects, and trigger words. We observe that while the event prediction model trained on LEVEN places high attention on key elements, it is not well-suited for retrieval tasks. Therefore, we distill the expertise of DMBERT (Wang et al., 2019b), which has the best event prediction performance in the LEVEN.

Specially, given a piece of legal knowledge entry \(c\), we input it into the DMBERT to obtain the attention matrix \(A\) of the final layer. Then the attention distribution for the \(i\)-th token \(\hat{p}_i\) is calculated:

\[
\hat{p}_i = \text{softmax}(A_i) = \frac{\sum_{j=1}^{l_c} \exp(A_{ij})}{\sum_{i=1}^{l_c} \sum_{j=1}^{l_c} \exp(A_{ij})}
\]

Here, \(l_c\) is the knowledge length. Finally, we obtain the attention distribution \(\hat{P} = \{\hat{p}_1, ..., \hat{p}_{l_c}\}\) and construct a key element dictionary \(\mathcal{D} = \{(c_1 : \hat{P}_1), ..., (c_{n_c} : \hat{P}_{n_c})\}\), where knowledge entries serve as keys and attention distributions as values. In this way, key elements from all corpus are extracted in the form of attention.

3.2.2 Knowledge graph construction
A piece of legal knowledge is intricately linked to additional information, such as other knowledge, hierarchical structures, and legal events, which we refer to as its “meta information”. To represent these associations comprehensively, we construct a heterogeneous graph as illustrated in Fig. 3.

Specifically, the heterogeneous graph \(\mathcal{G} = (N, E)\) contains different types of nodes and edges. Firstly, we categorize the nodes into three types: (i) Knowledge nodes, \(N_c = \{n^c_i\}\) represent individual pieces of legal knowledge. (ii) Hierarchy nodes, \(N_h = \{n^h_i\}\) represent the hierarchical structure of knowledge within the tree-like Chinese legal system (Qin et al., 2024) (e.g., “Article 273” belongs to “criminal law/property infringement/theft”). (iii) Legal event nodes, \(N_e = \{n^e_i\}\), represent the legal
Figure 2: The overview of our UniLR. (1) is the knowledge mining process. We leverage the expertise of a legal event prediction model (Yao et al., 2022) to extract key elements into a dictionary and explore meta information to construct a heterogeneous graph. (2) is the model architecture. When encoding knowledge, KES guides the model to focus on key elements through attention supervision. GKA aggregates meta information to further enrich knowledge representation.

Figure 3: The definition of nodes and edges.

event labels predicted by DMBERT. We use the event label definitions from LEVEN. For node feature initialization, we use Lawformer (Xiao et al., 2021), a BERT variant pre-trained on extensive legal texts to obtain embedding.

The edges in the graph are defined based on specific connection rules: (i) If a knowledge node \( n_i^c \) belongs to a hierarchy, whose node is \( n_j^h \), an edge is created and belongs to \( E_{ch} \). (ii) If a knowledge node \( n_i^c \) contains the legal event, whose node is \( n_e^l \), an edge is created and belongs to \( E_{ce} \). (iii) If two knowledge nodes \( n_i^c \) and \( n_j^c \) have the same charge label, we think they are mapping, an edge is created and belongs to \( E_{cc} \).

3.3 Model Architecture

3.3.1 Key Element Supervisor
To guide the retriever to focus on key elements when encoding knowledge, we introduce an attention supervision method. For the knowledge \( c \), we input it into Lawformer. According to Eq. 2, we extract the model’s attention distribution of each token and obtain \( P = \{p_1, ..., p_{l_c}\} \). Next, we refer to the key element dictionary \( \mathcal{D} \) to obtain the desired attention distribution \( \hat{P} \) corresponding to knowledge \( c \). To align the model’s attention with the desired distribution, we introduce a loss function \( \mathcal{L}_{attn} \), which minimizes the Kullback-Leibler (KL) divergence between the two distributions:

\[
\mathcal{L}_{attn} = \sum_{i=1}^{l_c} KL(\hat{P}_i \parallel P_i)
\]

Here, \( l_c \) denotes the number of tokens in the knowledge. Then we obtain the hidden state of the final layer by applying the aligned attention distribution, denoted as \( h^c = [h_1; ...; h_{l_c}] \in \mathbb{R}^{l_c \times d} \), where \( d \) is the hidden dimension. By employing this approach, we effectively distill the expertise of DMBERT into the retriever, guiding the model to focus on key legal elements rather than the format.

3.3.2 Graph-based Knowledge Augmenter
To further enrich knowledge representation, we design a graph-based approach to aggregate meta
information. First, we extract the subgraph relevant to the knowledge from the entire graph \( \mathcal{G} \). Specifically, for a piece of knowledge \( c \), we identify the neighboring nodes to form a node set \( N' \), which consists of \( n' \) nodes. The edges connecting these nodes form an edge set \( E' \). From this, we obtain a subgraph \( \mathcal{G}' = \{N', E'\} \) relevant to the current knowledge.

Inspired by Guo et al. (2019), we employ graph attention to facilitate node interactions. Given GKA comprising \( L \) layers, the node representation at the \( l \)-th layer is denoted as \( H^{(l)} = [h_1^{(l)}; ...; h_{n'}^{(l)}] \). To ensure that a node is not influenced by nodes that are not directly connected, we prepare a mask for each node, which defines the index set \( I_i \) of connected nodes for the \( i \)-th node. We then apply the Multi-Head Attention mechanism to learn multiple sets of attention weights, with the attention matrix for the \( m \)-th head given as follows:

\[
I_i = \{j \mid i, j \in N', (n_i \leftrightarrow n_j)\}
\]
\[
\bar{A}^{(m)} = softmax(\frac{(h_i^{(l)} W_i^Q) \cdot (h_j^{(l)} W_j^K)^T}{\sqrt{d}}), \quad i \in I_i
\]

Here, \( W_i^Q \in \mathbb{R}^{d \times d} \) and \( W_i^K \in \mathbb{R}^{d \times d} \) are learnable weight matrices used for linear transformations, \( \leftrightarrow \) denotes existing edge between nodes.

To capture the high-order interactions, we design the heterogeneous graph convolution that considers different edge types. For the \( i \)-th node, we concatenate its encoding with the output from previous layers to serve as the input for the next graph layer, \( g_i^{(l)} = [h^c; h_i^{(1)}; ...; h_i^{(l)}] \). The convolutional operation is then performed as follows:

\[
h_i^{(l+1)} = \sum_{m=1}^M \sigma(\sum_{j=1}^{n'} \bar{A}_{ij}^{(k)} \cdot W_m^{(l)} \cdot g_j^{(l)} \cdot t_{ij} + b_m^{(l)})
\]

Here, \( M \) is the number of attention heads, \( W_m^{(l)} \) is learnable weights for node features, \( t_{ij} \) is a one-hot vector indicating the edge type between \( n_i \) and \( n_j \), and \( \sigma \) is the activation function. After propagating through the graph network, we aggregate the node features in the final layer:

\[
H^{(L)} = MaxPooling[h_1^{(L)}; ...; h_{n'}^{(L)}] \in \mathbb{R}^{T' \times d}
\]

To ensure stable gradient propagation, we flatten the node feature to \( H_{flat}^{(L)} \in \mathbb{R}^{T'd} \), then concatenate it with the initial feature \( h^c \) and reduce the dimension via a projection layer:

\[
\tilde{h}^c = RELU(W \cdot [H_{flat}^{(L)}; h^c] + b) \in \mathbb{R}^d
\]

Prompt for charge prediction task: Given query, please select the charge from the following charges, each with its definition. query, c1, c2, c3

![Diagram showing UniLR, Prompt, and Label/answer for charge prediction task](page_1047_186_355_120.png)

Prompt for legal question answering task: Answering the query based on the given example. For example, c1, c2, c3. Please answer: query

![Diagram showing UniLR, Prompt, and Label/answer for legal question answering task](page_1047_324_355_120.png)

Figure 4: The process of performing legal tasks using UniLR and LLM in a RAG framework.

\( W \in \mathbb{R}^{(T'd+d) \times d} \) is the weight matrix, and \( b \in \mathbb{R}^{d \times 1} \) is the bias vector.

3.4 Contrastive Learning

Inspired by Gao et al. (2021), we use contrastive learning to concurrently train our model across multiple legal retrieval tasks to achieve uniformity.

For each query \( q \), we obtain positive samples \( Pos \) by selecting knowledge with the same charge label. For example, for a Theft case, we choose the charge definitions of Theft, Article 273, and other cases involving Theft. Negative samples \( Neg \) are obtained through random sampling and hard negative sampling. When random sampling, we randomly select knowledge with different charge labels. When hard negative sampling, we select knowledge closely related within the same legal hierarchy, providing more challenging contrasts. Then, we permute positive and negative knowledge pairs and train our model by the following loss:

\[
\mathcal{L}_{con} = - \sum_{\tilde{h}_c^+ \in Pos} \log \frac{exp(sim(h_q, \tilde{h}_c^+)/\tau)}{\sum_{\tilde{h}_c^- \in Neg} exp(sim(h_q, \tilde{h}_c^-)/\tau)}
\]

\( h_q, \tilde{h}_c^+, \tilde{h}_c^- \) are representations of query, positive samples, and negative samples from UniLR, \( sim \) calculates the cosine similarity, and \( \tau \) is the temperature that adjusts the contrastive strength.

3.5 Training and Inference

In the training process, we combine data from multiple tasks for joint training. We employ contrastive learning loss to bring the relevant knowledge representation closer to the query, and attention distribution loss to emphasize the key elements. Therefore, the overall training objective of UniLR is to minimize the following combined loss:

\[
\mathcal{L}_{total} = \mathcal{L}_{con} + \lambda \mathcal{L}_{attn}
\]
<table>
  <tr>
    <th>Types</th>
    <th>Article</th>
    <th>Charge</th>
    <th>CVG</th>
    <th>QA</th>
  </tr>
  <tr>
    <td># Train</td>
    <td>70973</td>
    <td>70973</td>
    <td>97863</td>
    <td>8451</td>
  </tr>
  <tr>
    <td># Test</td>
    <td>1683</td>
    <td>1683</td>
    <td>2039</td>
    <td>387</td>
  </tr>
  <tr>
    <td># Knowledge corpus</td>
    <td>483</td>
    <td>452</td>
    <td>6124</td>
    <td>1248</td>
  </tr>
  <tr>
    <td>Avg. # query length</td>
    <td>440</td>
    <td>440</td>
    <td>558</td>
    <td>45</td>
  </tr>
  <tr>
    <td>Avg. # knowledge length</td>
    <td>113</td>
    <td>160</td>
    <td>571</td>
    <td>401</td>
  </tr>
</table>

Table 1: Dataset details and knowledge base.

In the inference process, we employ UniLR to encode query and knowledge. Then, the cosine similarity is used to compute their relevance scores. The knowledge with the highest relevance scores is selected to assist the LLM in solving legal tasks. As shown in Fig. 4, for the charge prediction task, UniLR retrieves several label definitions and LLM selects the final label from them. A similar process is used for the law article prediction task. For the legal question answering task, we retrieve similar QA pairs and employ few-shot learning to guide LLM in producing the final answer. A similar process is used for the court view generation task.

4 Experiments

4.1 Dataset details and knowledge base

We conduct experiments on multiple common legal tasks: law article prediction (Article), charge prediction (Charge), court view generation (CVG), and legal question answering (QA). The dataset details and corresponding knowledge base are shown in Tab.1. CAIL2018-Article and CAIL2018-Charge are from the Chinese AI and Law challenge legal judgment prediction dataset 3 (Xiao et al., 2018). To align with prior works (Xu et al., 2020; Yue et al., 2021a), we filter out data with multiple articles and charges. In these datasets, the query is a fact description, and the task is to predict article and charge labels, respectively. LAIC2021-CVG is the court view generation dataset from Legal AI challenge 4, where the query is a fact description and the task is to generate the court view. Lawzhidao-QA is the criminal question-answering dataset selected from Baidu Legal Question Answering competition5, where the query is a legal question and the task is to generate the answer. The details of the knowledge source can be found in Appendix A.

4.2 Baselines

We implement three parts of baselines for comprehensive comparison and set retrieved knowledge number \( k = 3 \), with experimental settings detailed in Appendix B. We also conduct further experiments in Appendix C, including performance variation with different \( k \) values and pre-trained models and runtime analysis.

Traditional task-specific methods. For prediction tasks, TopJudge (Zhong et al., 2018) captures topological dependencies among the subtasks in legal judgment prediction. LADAN (Xu et al., 2020) uses graph distillation for distinguishing charges and law articles. NeurJudge (Yue et al., 2021a) splits fact descriptions using intermediate subtask results for prediction. For generation tasks, BART (Lewis et al., 2019) is a widely used bidirectional autoregressive Transformer model. T5 (Raffel et al., 2020) is a transformer architecture model that follows a text-to-text transfer learning paradigm. C3VG (Yue et al., 2021b) follows a two-stage architecture which is extraction-generation.

LLM methods. For LLMs, we use GPT4 (Achiam et al., 2023), LLaMA-3 (Cui et al., 2023b), GLM4 (Zeng et al., 2024). For legal LLMs, we use LexiLaw6, LaWQPT (Zhou et al., 2024b).

LLM with retriever methods. To evaluate retrieval enhancements, we implement the following retrieval methods with top-performing LLMs: BM25 (Robertson and Zaragoza, 2009) is a retrieval model based on term frequency and document length. Contriever (Izacard et al., 2022) is a dense embedding model trained via contrastive learning. LED (Zhang et al., 2023) enhances dense retrieval by aligning embeddings with lexicon-aware representations through weakened knowledge distillation. SAILER (Li et al., 2023a) is a legal case retriever that incorporates structural information and legal rules.

4.3 Evaluation Metrics

4.3.1 Retrieval evaluation

We use Hit@k (Norouzi et al., 2014) as the evaluation metric. If there is relevant knowledge among the \( k \) knowledge retrieved, it is considered successful retrieval. Considering that the relevant knowledge is not unique when retrieving precedents, we
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="4">CAIL2018-Article</th>
    <th colspan="4">CAIL2018-Charge</th>
  </tr>
  <tr>
    <th>Ma-P</th>
    <th>Ma-R</th>
    <th>Ma-F</th>
    <th>Acc</th>
    <th>Hit@k</th>
    <th>Ma-P</th>
    <th>Ma-R</th>
    <th>Ma-F</th>
    <th>Acc</th>
    <th>Hit@k</th>
  </tr>
  <tr>
    <td colspan="11"><i>Traditional task-specific methods</i></td>
  </tr>
  <tr>
    <td>TopJudge</td>
    <td>74.49</td>
    <td>66.26</td>
    <td>68.68</td>
    <td>80.85</td>
    <td>-</td>
    <td>74.43</td>
    <td>68.37</td>
    <td>70.41</td>
    <td>78.42</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LADAN</td>
    <td>75.61</td>
    <td>70.29</td>
    <td>70.46</td>
    <td>80.92</td>
    <td>-</td>
    <td>75.36</td>
    <td>70.02</td>
    <td>71.19</td>
    <td>79.45</td>
    <td>-</td>
  </tr>
  <tr>
    <td>NeurJudge</td>
    <td>75.16</td>
    <td>72.01</td>
    <td>72.26</td>
    <td><b>81.94</b></td>
    <td>-</td>
    <td>75.76</td>
    <td>71.24</td>
    <td>71.59</td>
    <td>80.31</td>
    <td>-</td>
  </tr>
  <tr>
    <td colspan="11"><i>LLM methods</i></td>
  </tr>
  <tr>
    <td>GPT4</td>
    <td>14.76</td>
    <td>13.46</td>
    <td>13.47</td>
    <td>14.68</td>
    <td>-</td>
    <td>45.97</td>
    <td>35.61</td>
    <td>34.38</td>
    <td>46.94</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaMA-3</td>
    <td>15.18</td>
    <td>12.43</td>
    <td>13.19</td>
    <td>14.00</td>
    <td>-</td>
    <td>40.51</td>
    <td>31.88</td>
    <td>33.09</td>
    <td>42.00</td>
    <td>-</td>
  </tr>
  <tr>
    <td>GLM4</td>
    <td>9.42</td>
    <td>7.47</td>
    <td>6.57</td>
    <td>14.00</td>
    <td>-</td>
    <td>47.17</td>
    <td>35.27</td>
    <td>37.68</td>
    <td>41.33</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LexiLaw</td>
    <td>12.27</td>
    <td>6.54</td>
    <td>7.41</td>
    <td>9.33</td>
    <td>-</td>
    <td>36.59</td>
    <td>32.81</td>
    <td>37.73</td>
    <td>42.33</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LaWGPT</td>
    <td>14.49</td>
    <td>12.17</td>
    <td>12.64</td>
    <td>16.35</td>
    <td>-</td>
    <td>32.21</td>
    <td>28.81</td>
    <td>34.11</td>
    <td>43.37</td>
    <td>-</td>
  </tr>
  <tr>
    <td colspan="11"><i>Best LLM (GPT4) with retriever methods</i></td>
  </tr>
  <tr>
    <td>+ BM25</td>
    <td>31.90</td>
    <td>25.58</td>
    <td>26.70</td>
    <td>36.82</td>
    <td>71.13</td>
    <td>52.63</td>
    <td>42.87</td>
    <td>46.17</td>
    <td>50.75</td>
    <td>73.28</td>
  </tr>
  <tr>
    <td>+ Contriever</td>
    <td>72.35</td>
    <td>63.41</td>
    <td>67.83</td>
    <td>73.76</td>
    <td>75.06</td>
    <td>73.10</td>
    <td>65.90</td>
    <td>66.09</td>
    <td>71.37</td>
    <td>81.58</td>
  </tr>
  <tr>
    <td>+ LED</td>
    <td>70.73</td>
    <td>63.24</td>
    <td>67.97</td>
    <td>74.54</td>
    <td>78.31</td>
    <td>74.41</td>
    <td>66.79</td>
    <td>68.42</td>
    <td>69.62</td>
    <td>82.25</td>
  </tr>
  <tr>
    <td>+ SAILER</td>
    <td>73.75</td>
    <td>65.85</td>
    <td>69.76</td>
    <td>77.91</td>
    <td>80.70</td>
    <td>74.76</td>
    <td>68.95</td>
    <td>69.70</td>
    <td>76.75</td>
    <td>83.90</td>
  </tr>
  <tr>
    <td>+ UniLR</td>
    <td><b>77.24</b></td>
    <td><b>72.73</b></td>
    <td><b>72.32</b></td>
    <td><b>80.98</b></td>
    <td><b>84.88</b></td>
    <td><b>76.98</b></td>
    <td><b>73.11</b></td>
    <td><b>72.34</b></td>
    <td><b>80.36</b></td>
    <td><b>85.27</b></td>
  </tr>
</table>

Table 2: The performance of article prediction task and charge prediction task. The best is bolded, the second best is underlined, and the gray-shaded metric only measures retriever performance when \( k = 3 \).

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="5">LAIC2021-CVG</th>
    <th colspan="5">Lawzhidao-QA</th>
  </tr>
  <tr>
    <th>B-1</th>
    <th>B-2</th>
    <th>B-N</th>
    <th>R-L</th>
    <th>Hit@k</th>
    <th>R-p@k</th>
    <th>B-1</th>
    <th>B-2</th>
    <th>B-N</th>
    <th>R-L</th>
    <th>Hit@k</th>
    <th>R-p@k</th>
  </tr>
  <tr>
    <td colspan="13"><i>Traditional task-specific methods</i></td>
  </tr>
  <tr>
    <td>Bart</td>
    <td>61.01</td>
    <td>52.01</td>
    <td>48.97</td>
    <td>56.95</td>
    <td>-</td>
    <td>-</td>
    <td>37.58</td>
    <td>23.71</td>
    <td>20.74</td>
    <td>21.93</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>T5</td>
    <td>61.24</td>
    <td>51.24</td>
    <td>47.68</td>
    <td>58.09</td>
    <td>-</td>
    <td>-</td>
    <td>36.32</td>
    <td>21.45</td>
    <td>20.24</td>
    <td>18.31</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>C3VG</td>
    <td><b>63.35</b></td>
    <td>52.70</td>
    <td>49.30</td>
    <td>60.71</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td colspan="13"><i>LLM methods</i></td>
  </tr>
  <tr>
    <td>GPT4</td>
    <td>41.23</td>
    <td>26.90</td>
    <td>23.06</td>
    <td>27.46</td>
    <td>-</td>
    <td>-</td>
    <td>11.98</td>
    <td>7.63</td>
    <td>5.73</td>
    <td>14.38</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaMA-3</td>
    <td>35.98</td>
    <td>19.74</td>
    <td>18.01</td>
    <td>23.17</td>
    <td>-</td>
    <td>-</td>
    <td>20.70</td>
    <td>15.27</td>
    <td>14.10</td>
    <td>18.39</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>GLM4</td>
    <td>42.15</td>
    <td>28.01</td>
    <td>22.32</td>
    <td>32.31</td>
    <td>-</td>
    <td>-</td>
    <td>33.02</td>
    <td>20.10</td>
    <td>15.43</td>
    <td>20.73</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LexiLaw</td>
    <td>39.30</td>
    <td>23.26</td>
    <td>20.59</td>
    <td>23.91</td>
    <td>-</td>
    <td>-</td>
    <td>28.99</td>
    <td>16.56</td>
    <td>14.42</td>
    <td>19.23</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LaWGPT</td>
    <td>15.17</td>
    <td>10.03</td>
    <td>8.59</td>
    <td>12.85</td>
    <td>-</td>
    <td>-</td>
    <td>24.32</td>
    <td>16.10</td>
    <td>14.22</td>
    <td>17.78</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td colspan="13"><i>Best LLM (GLM4) with retriever methods</i></td>
  </tr>
  <tr>
    <td>+ BM25</td>
    <td>53.47</td>
    <td>46.53</td>
    <td>44.46</td>
    <td>58.30</td>
    <td>83.20</td>
    <td>65.43</td>
    <td>39.04</td>
    <td>28.24</td>
    <td>24.49</td>
    <td>30.72</td>
    <td>59.63</td>
    <td>37.47</td>
  </tr>
  <tr>
    <td>+ Contriever</td>
    <td>57.35</td>
    <td>51.07</td>
    <td>48.98</td>
    <td>63.51</td>
    <td>87.50</td>
    <td>77.32</td>
    <td>42.26</td>
    <td>30.38</td>
    <td>26.42</td>
    <td>32.95</td>
    <td>65.22</td>
    <td>55.10</td>
  </tr>
  <tr>
    <td>+ LED</td>
    <td>57.49</td>
    <td>51.78</td>
    <td>48.84</td>
    <td>63.52</td>
    <td>87.88</td>
    <td>79.46</td>
    <td>43.80</td>
    <td>32.45</td>
    <td>27.54</td>
    <td>32.87</td>
    <td>66.78</td>
    <td>55.73</td>
  </tr>
  <tr>
    <td>+ SAILER</td>
    <td>59.06</td>
    <td>53.02</td>
    <td>50.95</td>
    <td>65.05</td>
    <td>90.97</td>
    <td>87.86</td>
    <td>45.00</td>
    <td>34.93</td>
    <td>30.82</td>
    <td>36.26</td>
    <td>70.43</td>
    <td>60.73</td>
  </tr>
  <tr>
    <td>+ UniLR</td>
    <td><b>63.94</b></td>
    <td><b>58.03</b></td>
    <td><b>55.81</b></td>
    <td><b>72.68</b></td>
    <td><b>91.41</b></td>
    <td><b>90.23</b></td>
    <td><b>48.38</b></td>
    <td><b>37.13</b></td>
    <td><b>33.27</b></td>
    <td><b>40.87</b></td>
    <td><b>75.90</b></td>
    <td><b>64.10</b></td>
  </tr>
</table>

Table 3: The performance of court view generation task and legal question answering task. The best is bolded, the second best is underlined, and the gray-shaded metric measures retriever performance when \( k = 3 \).

select R-p@k (Chen et al., 2023) to do further evaluation. It is calculated as \( r/R \), where \( R \) is the number of retrieved knowledge and \( r \) is the number of relevant knowledge.

4.3.2 Task performance evaluation

For the prediction tasks, we employ macro precision (Ma-P), macro recall (Ma-R), macro f1 score (Ma-F), and accuracy (Acc). For the generation task, we use BLEU-1 (B-1), BLEU-2 (B-2), BLEU-N (B-N, the average value of BLEU-1 ~4), and ROUGE-L (R-L) (Lin, 2004). To comprehensively test the generation results, we conduct the human evaluation, with details provided in Appendix D.

4.4 Experimental Results

4.4.1 The performance of prediction tasks

From Tab.2, we can conclude that: (1) Traditional task-specific methods excel over LLMs. This is because LLMs are generative, selecting a token from a vast vocabulary in decoding. (2) Legal LLMs underperform universal LLMs like GPT4, possibly due to a decrease in in-context learning ability during fine-tuning. (3) In retrieval evaluation, our UniLR outperforms the best baseline SAILER by 5.18% in Hit@k for article prediction and 1.63% for charge prediction. This indicates the effectiveness of introducing key elements and meta information. (4) In prediction evaluation, UniLR significantly improves GPT4’s perfor-
<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="4">CAIL2018-Article</th>
    <th colspan="4">CAIL2018-Charge</th>
  </tr>
  <tr>
    <th>Ma-P</th><th>Ma-R</th><th>Ma-F</th><th>Acc</th>
    <th>Hit@k</th><th>Ma-P</th><th>Ma-R</th><th>Ma-F</th><th>Acc</th>
    <th>Hit@k</th>
  </tr>
  <tr>
    <td>UniLR</td>
    <td>77.24</td><td>72.73</td><td>72.32</td><td>80.98</td>
    <td>84.88</td><td>76.98</td><td>73.11</td><td>72.34</td><td>80.36</td>
    <td>85.27</td>
  </tr>
  <tr>
    <td>w/o KES</td>
    <td>74.15</td><td>71.31</td><td>70.76</td><td>78.87</td>
    <td>81.33</td><td>75.25</td><td>69.02</td><td>69.83</td><td>77.89</td>
    <td>81.68</td>
  </tr>
  <tr>
    <td>w/o GKA</td>
    <td>75.70</td><td>71.80</td><td>71.04</td><td>79.57</td>
    <td>81.96</td><td>74.18</td><td>71.07</td><td>68.59</td><td>78.69</td>
    <td>82.37</td>
  </tr>
  <tr>
    <td colspan="11"><i>Ablation of meta information in GKA</i></td>
  </tr>
  <tr>
    <td>w/o \( E_{ec} \)</td>
    <td>75.41</td><td>71.28</td><td>70.91</td><td>79.09</td>
    <td>82.94</td><td>75.97</td><td>70.24</td><td>70.07</td><td>78.53</td>
    <td>83.01</td>
  </tr>
  <tr>
    <td>w/o \( E_{ce} \)</td>
    <td>76.45</td><td>72.64</td><td>72.07</td><td>80.57</td>
    <td>84.14</td><td>76.42</td><td>72.26</td><td>71.83</td><td>79.59</td>
    <td>84.58</td>
  </tr>
  <tr>
    <td>w/o \( E_{ch} \)</td>
    <td>75.10</td><td>71.65</td><td>71.06</td><td>80.12</td>
    <td>84.05</td><td>76.15</td><td>69.48</td><td>70.30</td><td>77.39</td>
    <td>82.94</td>
  </tr>
</table>

Table 4: Ablation experiment of the best performing LLM with UniLR in prediction tasks.

![t-SNE plots of legal cases](page_1012_370_410_180.png)

Figure 5: The t-SNE plots of legal cases.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="2">CVG</th>
    <th colspan="2">QA</th>
  </tr>
  <tr>
    <th>Flu.</th><th>Rat.</th>
    <th>Flu.</th><th>Rat.</th>
  </tr>
  <tr>
    <td>BART</td>
    <td>4.44</td><td>3.64</td>
    <td>2.18</td><td>2.52</td>
  </tr>
  <tr>
    <td>GLM4</td>
    <td>3.37</td><td>2.71</td>
    <td>3.09</td><td>2.91</td>
  </tr>
  <tr>
    <td>GLM4+SAILER</td>
    <td>4.46</td><td>3.93</td>
    <td>3.95</td><td>3.47</td>
  </tr>
  <tr>
    <td>GLM4+UniLR</td>
    <td><b>4.62</b></td><td><b>4.04</b></td>
    <td><b>4.28</b></td><td><b>3.99</b></td>
  </tr>
</table>

Table 5: Human evaluation on legal generation tasks.

mance, surpassing all baselines, including the traditional SOTA method. This suggests that UniLR effectively bridges the gap between LLMs and legal tasks.

4.4.2 The performance of generation tasks
From Tab.3 and Tab.5, we have the following observations: (1) LLM methods show poor performance, indicating their tendency to generate imaginative outputs are unsuitable for the precision required in the legal domain. (2) Many LLM with retriever methods surpass traditional task-specific methods emphasizing the importance of knowledge. (3) GLM4 surpasses GPT4, suggesting that its training involved more Chinese legal documents. (4) UniLR achieves top retrieval performance and significantly boosts GLM4’s downstream task performance. For example, in legal QA, it outperforms the second-best baseline by 7.77% in Hit@k and 12.71% in R-L. (5) In human evaluation, UniLR excels in Fluency and Rationality, aligning with automatic evaluations. Additionally, BART performs well in CVG but lags in QA, while GLM4 shows the opposite trend due to the structured nature of court views versus the flexibility of QA tasks.

4.4.3 Ablation study
We conduct ablation experiments on the GPT4+UniLR method in two prediction tasks, as shown in Tab. 4. (1) w/o KES eliminates the key elements supervision, relying solely on Lawformer encoding. We find a significant performance degradation, demonstrating the importance of focusing on key elements. (2) w/o GKA removes the graph-based knowledge augmenter. Noticeable performance degradation indicates that aggregating meta information can effectively capture a comprehensive knowledge representation. (3) We conduct ablation experiments on GKA’s associations, including mapping knowledge, legal events, and hierarchy. Removing associations between knowledge results in the most significant performance decline.

4.5 Visual Analysis
Following Li et al. (2023a), to explore UniLR’s distinguish ability for legal knowledge representation, we select 5000 illegal facts from CAIL2018 that involve confusing charges and visualize their encoding by using different retriever methods. Specifically, the charges included Robbery, Theft, Snatch, Intentional injury, Intentional homicide, each with 1000 cases. As observed from Fig. 5, UniLR significantly increases the distance between encoded illegal facts related to different charges, demonstrating its strongest ability to distinguish between confusing knowledge. Case Study in Appendix E also confirms the effectiveness of UniLR.
5 Conclusion and Future Work

In conclusion, we address the challenges in enhancing LLMs in multiple legal tasks by introducing a Unified Legal Knowledge Retriever (UniLR). By incorporating key elements and meta information, UniLR significantly alleviates disparities in multiple retrieval tasks and reduces knowledge confusion. Extensive experiments on multiple common legal tasks demonstrate that UniLR outperforms state-of-the-art retrieval methods and significantly enhances the performance of LLMs in legal applications. In the future, to better explore Retrieval-Augmented Generation (RAG) in legal tasks, we aim to make efforts in two directions: (1) Training an LLM capable of flexibly utilizing knowledge obtained from retrievers. (2) Leveraging feedback from the LLM to train a retriever that is better suited for LLM.

6 Ethical Issue Discussion

Legal AI has benefited from the emergence of LLMs, but it’s a sensitive technology that demands ethical considerations. Our UniLR is designed to enhance LLMs in legal tasks, mitigating the risk of factual errors to some extent. However, even minor inaccuracies could have significant consequences. Our goal is to provide suggestions to judges rather than replace them. In practice, human judges should be the final safeguard to protect justice and fairness. Although our method demonstrates promising results on legal task datasets, it does not imply that it can endow LLMs with human-like empathy, experience, and intuition. It is necessary to prevent misuse. Additionally, given that the model retrieves knowledge from the external corpus, ensuring the quality and fairness of the corpus is crucial.

7 Limitations

In this section, we discuss the limitations of our works as follows:

• We do not pretrain or fine-tune LLMs for the RAG process. Training an LLM capable of flexibly utilizing knowledge obtained from retrievers, may produce better output.

• We validate the effectiveness of designing a legal retriever to meet LLM needs. Exploring the application of such retrievers in other fields like medicine and education is worth considering.

• Our research is conducted on the Chinese legal system. We are also very interested in exploring the generalization of our methods to other languages.

8 Acknowledgments

This work was supported in part by "Pioneer" and "Leading Goose" R&D Program of Zhejiang (2025C02037), National Natural Science Foundation of China (62376243, 62441605), and National Key Research and Development Program of China (2024YFE0203700). All opinions in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Rose Biderman. 2021. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

S.J. Burton. 2007. An Introduction to Law and Legal Reasoning. Academic Success Series. Walters Kluwer Law & Business.

Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. Preprint, arXiv:2010.02559.

Jiangui Chen, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yiqun Liu, Yixing Fan, and Xueqi Cheng. 2023. A unified generative retriever for knowledge-intensive language tasks via prompt learning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23. ACM.

Yen-Liang Chen, Yi-Hung Liu, and Wu-Liang Ho. 2013. A text mining approach to assist the general public in the retrieval of legal documents. Journal of the American Society for Information Science and Technology, 64(2):280–290.

Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023a. Chatlaw: Open-source legal large language model with integrated external knowledge bases. Preprint, arXiv:2306.16092.
Yiming Cui, Ziqing Yang, and Xin Yao. 2023b. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177.

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, et al. 2024. Lawbench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7933–7962.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Suyu Ge, Chenyan Xiong, Corby Rosset, Arnold Overwijk, Jiawei Han, and Paul Bennett. 2023. Augmenting zero-shot dense retrievers with plug-in mixture-of-memories. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1796–1812.

Zhijiang Guo, Yan Zhang, and Wei Lu. 2019. Attention guided graph convolutional networks for relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 241–251.

Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. Lawyer llama technical report. ArXiv, abs/2305.15062.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Preprint, arXiv:2112.09118.

Phi Manh Kien, Ha-Thanh Nguyen, Ngo Xuan Bach, Vu Tran, Minh Le Nguyen, and Tu Minh Phuong. 2020. Answering legal questions by learning neural attentive text representation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 988–998, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. Preprint, arXiv:1412.6980.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.

Ang Li, Qiangchao Chen, Yiquan Wu, Ming Cai, Xiang Zhou, Fei Wu, and Kun Kuang. 2024a. From graph to word bag: Introducing domain knowledge to confusing charge prediction. Preprint, arXiv:2403.04369.

Ang Li, Yiquan Wu, Ming Cai, Adam Jatowt, Xiang Zhou, Weiming Lu, Changlong Sun, Fei Wu, and Kun Kuang. 2025. Legal judgment prediction based on knowledge-enhanced multi-task and multi-label text classification. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6957–6970, Albuquerque, New Mexico. Association for Computational Linguistics.

Ang Li, Yiquan Wu, Yifei Liu, Fei Wu, Ming Cai, and Kun Kuang. 2024b. Enhancing court view generation with knowledge injection and guidance. Preprint, arXiv:2403.04366.

Haitao Li, Qingyao Ai, Jia Chen, Qian Dong, Yueyue Wu, Yiqun Liu, Chong Chen, and Qi Tian. 2023a. Sailer: structure-aware pre-trained language model for legal case retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1035–1044.

Qingquan Li, Qifan Zhang, Junjie Yao, and Yingjie Zhang. 2020. Event extraction for criminal legal text. In 2020 IEEE International Conference on Knowledge Graph (ICKG), pages 573–580.

Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023b. Unified demonstration retriever for in-context learning. CoRR.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Hongcheng Liu, Yusheng Liao, Yutong Meng, and Yuhao Wang. 2023. Lawgpt: Chinese legal dialogue language model. https://github.com/LiuHC0428/LAW_GPT.

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3?. DeeLIO 2022, page 100.

Yifei Liu, Yiquan Wu, Ang Li, Yating Zhang, Changlong Sun, Weiming Lu, Fei Wu, and Kun Kuang. 2024. Unleashing the power of LLMs in court view generation by stimulating internal knowledge and incorporating external knowledge. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2782–2792, Mexico City, Mexico. Association for Computational Linguistics.

Antoine Louis and Gerasimos Spanakis. 2022. A statutory article retrieval dataset in french. Preprint, arXiv:2108.11792.
Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2024. Interpretable long-form legal question answering with retrieval-augmented large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22266–22275.

Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Seyed Mehran Kazemi, Chitta Baral, Vaiva Imbrasaitė, and Vincent Y Zhao. 2023. Dr. icl: Demonstration-retrieved in-context learning. CoRR.

Zheqi Lv, Tianyu Zhan, Wenjie Wang, Xinyu Lin, Shengyu Zhang, Wenqiao Zhang, Jiwei Li, Kun Kuang, and Fei Wu. 2025. Collaboration of large language models and small recommendation models for device-cloud recommendation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD ’25, page 962–973, New York, NY, USA. Association for Computing Machinery.

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303–5315.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. Preprint, arXiv:2305.14251.

Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S. Corrado, and Jeffrey Dean. 2014. Zero-shot learning by convex combination of semantic embeddings. Preprint, arXiv:1312.5650.

Weicong Qin, Zelin Cao, Weijie Yu, Zihua Si, Sirui Chen, and Jun Xu. 2024. Explicitly integrating judgment prediction with legal document retrieval: A law-guided generative approach. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2210–2220, New York, NY, USA. Association for Computing Machinery.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.

K. Raghav, Krishna Reddy, and V. Balakista Reddy. 2016. Analyzing the extraction of relevant legal judgments using paragraph-level and citation information.

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331.

Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. Preprint, arXiv:2112.08633.

Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Ken Satoh, Min Zhang, and Shaoping Ma. 2020. Bertpli: Modeling paragraph-level interactions for legal case retrieval. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3501–3507. International Joint Conferences on Artificial Intelligence Organization. Main track.

Shirong Shen, Guilin Qi, Zhen Li, Sheng Bi, and Lusheng Wang. 2020. Hierarchical Chinese legal event extraction via pedal attention mechanism. In Proceedings of the 28th International Conference on Computational Linguistics, pages 100–113, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.

Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, Anima Anandkumar, and Bryan Catanzaro. 2023. Shall we pretrain autoregressive language models with retrieval? a comprehensive study. Preprint, arXiv:2304.06762.

Pengfei Wang, Yu Fan, Shuzi Niu, Ze Yang, Yongfeng Zhang, and Jiateng Guo. 2019a. Hierarchical matching network for crime classification. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 325–334. ACM.

Pengfei Wang, Ze Yang, Shuzi Niu, Yongfeng Zhang, Lei Zhang, and Shaozhang Niu. 2018. Modeling dynamic pairwise attention for crime classification over legal articles. In The 41st International ACM
SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 485–494. ACM.

Xiaozhi Wang, Xu Han, Zhiyuan Liu, Maosong Sun, and Peng Li. 2019b. Adversarial training for weakly supervised event detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 998–1008. Minneapolis, Minnesota. Association for Computational Linguistics.

Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. 2020. De-biased court’s view generation with causality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 763–780. Association for Computational Linguistics.

Yiquan Wu, Yifei Liu, Weiming Lu, Yating Zhang, Jun Feng, Changlong Sun, Fei Wu, and Kun Kuang. 2022. Towards interactivity and interpretability: A rationale-based legal judgment prediction framework. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4787–4799.

Chaojun Xiao, Xueyu Hu, Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021. Lawformer: A pre-trained language model for chinese legal long documents. AI Open, 2:79–84.

Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. 2018. Cail2018: A large-scale legal dataset for judgment prediction. Preprint, arXiv:1807.02478.

Nuo Xu, Pinghui Wang, Long Chen, Li Pan, Xiaoyan Wang, and Junzhou Zhao. 2020. Distinguishing confusing law articles for legal judgment prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3086–3095. Online. Association for Computational Linguistics.

Wenmian Yang, Weijia Jia, Xiaojie Zhou, and Yutao Luo. 2019. Legal judgment prediction via multi-perspective bi-feedback network. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization.

Feng Yao, Chaojun Xiao, Xiaozhi Wang, Zhiyuan Liu, Lei Hou, Cunchao Tu, Juanzi Li, Yun Liu, Weixing Shen, and Maosong Sun. 2022. LEVEN: A large-scale chinese legal event detection dataset. In Findings of the Association for Computational Linguistics, ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 183–201. Association for Computational Linguistics.

Linan Yue, Qi Liu, Binbin Jin, Han Wu, Kai Zhang, Yanqing An, Mingyue Cheng, Biao Yin, and Dayong Wu. 2021a. Neurjudge: A circumstance-aware neural framework for legal judgment prediction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 973–982.

Linan Yue, Qi Liu*, Han Wu, Yanqing An, Li Wang, and Senchao Yuan. 2021b. Circumstances enhanced criminal court view generation. The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.

Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. CoRR.

Kai Zhang, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, Binxing Jiao, and Daxin Jiang. 2023. Led: Lexicon-enlightened dense retriever for large-scale retrieval. In Proceedings of the ACM Web Conference 2023, pages 3203–3213.

Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judgment prediction via topological learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3540–3549, Brussels, Belgium. Association for Computational Linguistics.

Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020a. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218–5230, Online. Association for Computational Linguistics.

Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020b. Jecqa: a legal-domain question answering dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9701–9708.

Xiang Zhou, Yudong Wu, Ang Li, Ming Cai, Yiquan Wu, and Kun Kuang. 2024a. Unlocking authentic judicial reasoning: A template-based legal information generation framework for judicial views. Knowledge-Based Systems, 301:112232.

Zhi Zhou, Jiang-Xin Shi, Peng-Xiao Song, Xiaowen Yang, Yi-Xuan Jin, Lan-Zhe Guo, and Yu-Feng Li. 2024b. Lawgpt: A chinese legal knowledge-enhanced large language model. CoRR.

A Knowledge Source.

We collect multiple legal knowledge sources to construct the knowledge base. In our UniLR, they are further processed together into a legal essence dictionary and a heterogeneous knowledge metagraph.
(1) Article definitions: are detailed content about all criminal law articles. It assists in the article prediction task. (2) Charge definitions: are specific definitions for all criminal charges from the Criminal Law, formatted similarly to Xu et al. (2020). It assists in the charge prediction task. (3) CVG cases: are selected from the training set of LAIC2021 in a balanced way. It contains fact descriptions and corresponding court views. It assists in the court view generation task. (4) QA cases: are similarly selected from the training set of Lawzhidao-QA dataset. It assists in the legal QA task. (5) Legal system tree: is extracted from the Chinese Criminal Law, which has a tree-like structure comprising chapters and specific offenses. Each knowledge can be brought into a leaf node in the hierarchical structure, utilizing paths from the root to the leaf. (6) Legal events: are from the legal event detection dataset LEVEN (Yao et al., 2022). We focus on charge-oriented events and leverage their definitions.

B Experiment Settings

All training and inference were conducted on 2 NVIDIA Tesla A100 GPUs. We rerun the experiments five times with different random seeds and report the average. We also use the Fisher randomization test to ensure the significance of the results.

B.1 Retrieval setting

We set the maximum length for input query and each knowledge to 512. For the BM25 algorithm, we set k1 to 1.5 and b to 0.75. Other retrieval methods utilizing pre-trained models (Contriever, LED, SAILER, UniLR) leverage the Lawformer (Xiao et al., 2021) as the pre-trained embedding model, followed by fine-tuning. To ensure fairness, the dense retrieval baselines are jointly fine-tuned on multiple legal retrieval tasks, similar to UniLR. In the fine-tuning, all retrievers are trained using the Adam optimizer (Kingma and Ba, 2017) with a learning rate of 1e-6 for 50 epochs. In our UniLR, we set the GKA layers to 2, and the hyperparameter controlling loss \( \lambda \) to the best-performing value of 0.2. In the reference of retrieval, we use top-k selection to select the final knowledge and set k=3. Referring to Langchain Chatchat 7, we use FAISS to pre-vectorize the knowledge base, allowing for quick computation of relevance scores with the query during inference.

B.2 Legal tasks setting

For traditional methods, they are trained and inferred according to their original papers. For all LLMs, we set the inference temperature to 0.8 and the maximum output length to 512. Regarding LLM with retrieval methods, since they require concatenating the query and knowledge, we set the maximum input length to 2048.

C Further Experiments

C.1 The performance variation with different k values

In this paper, we set the number of retrieved knowledge number \( k = 3 \) for each query. We explore the impact of different \( k \) values on the performance of multiple legal retrieval tasks, testing experimental results for \( k \) ranging from 1 to 5. As shown Tab. 6, when \( k \leq 3 \), the performance gain from adding retrieved knowledge number is relatively significant. However, further increasing \( k \) brings minimal improvement. Moreover, inference with the LLM consumes substantial computational resources. Considering the trade-off between performance gain and resource consumption, we chose \( k = 3 \) for the main experiment.

C.2 Sensitivity analysis of the combined loss hyperparameter

To evaluate the effect of the hyperparameter \( \lambda \) in the combined loss function, we conducted experiments on four datasets by varying \( \lambda \) within the range [0.1,0.4]. The results are shown in Tab. 7. The results indicate that when \( \lambda < 0.2 \), the attention supervision signal is insufficient, resulting in lower performance. Conversely, when \( \lambda > 0.2 \), the attention supervision component dominates the optimization objective and suppresses the primary task loss, leading to degraded performance. Optimal results are achieved when \( \lambda = 0.2 \), which provides a balanced trade-off between task-specific loss and attention supervision.

C.3 The performance variation with different pre-trained model

To assess the robustness of our pipeline, we replaced Lawformer with BERT-base Chinese. We also set the retrieved knowledge number \( k = 3 \). The experimental results for law article prediction task and charge prediction task are shown in Tab. 8 and Tab. 9. The BERT pre-trained model was not specifically trained on legal texts, yet it did
<table>
  <tr>
    <th>k</th>
    <th>CAIL2018-Article</th>
    <th>CAIL2018-Charge</th>
    <th>LAIC2021-CVG</th>
    <th>Lawzhidao-QA</th>
  </tr>
  <tr>
    <td>1</td>
    <td>75.72</td>
    <td>76.37</td>
    <td>90.63</td>
    <td>65.96</td>
  </tr>
  <tr>
    <td>2</td>
    <td>80.73</td>
    <td>81.91</td>
    <td>91.21</td>
    <td>72.17</td>
  </tr>
  <tr>
    <td>3</td>
    <td>84.88</td>
    <td>85.27</td>
    <td>91.41</td>
    <td>75.90</td>
  </tr>
  <tr>
    <td>4</td>
    <td>86.05</td>
    <td>86.25</td>
    <td>91.60</td>
    <td>77.76</td>
  </tr>
  <tr>
    <td>5</td>
    <td>86.84</td>
    <td>87.22</td>
    <td>91.99</td>
    <td>78.39</td>
  </tr>
</table>

Table 6: Performance of Different k Values

<table>
  <tr>
    <th>λ</th>
    <th>CAIL2018-Article</th>
    <th>CAIL2018-Charge</th>
    <th>LAIC2021-CVG</th>
    <th>Lawzhidao-QA</th>
  </tr>
  <tr>
    <td>0.1</td>
    <td>83.78</td>
    <td>84.11</td>
    <td>90.41</td>
    <td>73.93</td>
  </tr>
  <tr>
    <td>0.2</td>
    <td><b>84.88</b></td>
    <td><b>85.27</b></td>
    <td><b>91.41</b></td>
    <td><b>75.90</b></td>
  </tr>
  <tr>
    <td>0.3</td>
    <td>84.71</td>
    <td>83.59</td>
    <td>91.54</td>
    <td>74.11</td>
  </tr>
  <tr>
    <td>0.4</td>
    <td>84.40</td>
    <td>82.49</td>
    <td>90.78</td>
    <td>73.20</td>
  </tr>
</table>

Table 7: Performance with different values of \( \lambda \) in the combined loss function.

<table>
  <tr>
    <th>Methods</th>
    <th>Ma-F</th>
    <th>Acc</th>
    <th>Hit@k</th>
  </tr>
  <tr>
    <td>UniLR (BERT)</td>
    <td>71.84</td>
    <td>80.73</td>
    <td>82.84</td>
  </tr>
  <tr>
    <td>UniLR (Lawformer)</td>
    <td>72.32</td>
    <td>80.98</td>
    <td>84.88</td>
  </tr>
</table>

Table 8: The performance of law article prediction task.

<table>
  <tr>
    <th>Methods</th>
    <th>Ma-F</th>
    <th>Acc</th>
    <th>Hit@k</th>
  </tr>
  <tr>
    <td>UniLR (BERT)</td>
    <td>72.22</td>
    <td>79.89</td>
    <td>84.96</td>
  </tr>
  <tr>
    <td>UniLR (Lawformer)</td>
    <td>72.34</td>
    <td>80.36</td>
    <td>85.27</td>
  </tr>
</table>

Table 9: The performance of charge prediction task.

not cause a significant performance drop in UniLR. These results demonstrate that UniLR maintains robust performance when replacing the pre-trained model.

C.4 Runtime analysis

We further analyze the runtime as follows: During training, we used a distributed setup on a single machine with 2 A100 GPUs. Each GPU utilized 16,964MB of memory, with a batch size of 8. The model was trained for 50 epochs, completing in approximately 5.56 hours. For inference, the model uses FAISS to pre-store knowledge as a vector database, which is 44.4MB in size. During inference, the model occupies 4,356MB of GPU memory. Retrieving results from the model is quick, with the primary time consumption being LLMs generating answers based on the retrieved knowledge. On average, generating each response takes 1.53 seconds.

D Human Evaluation Metric

We select baselines with good similarity performance to do human evaluation. We randomly select 200 samples and shuffle them to ensure fairness. We invite five annotators (10 Ph.D. students from the Law major) to evaluate every sample from two perspectives referencing true labels:

• Fluency. The annotators rate the fluency of generation texts on a scale of 1-5.

• Rationality. The annotator needs to score 1-5 on whether the answer to the question is reasonable.

When scoring on a scale of 1-5, they are required to provide integer scores, i.e., selecting from the range [1, 2, 3, 4, 5].

E Case Study

We conduct the case study to demonstrate the application details of UniLR in legal prediction tasks. In Fig. 6, based on the fact description, the defendant was discovered stealing property and then engaged in violent behavior (forcibly dragging the victim), which constituted a robbery. UniLR correctly identified this and accurately predicted Robbery, while NeurJudge predicted theft based on the act of Theft.

We also provided a case for legal QA in Fig. 7, we use gray highlights to represent confusing semantics. Red highlights represent key elements, while green highlights represent meta information. It is observed that SAILER, influenced by the format in order to and the word detain, retrieves a Detention case. In reality, due to the defendant’s intention for valuables and the violent actions, the charge has transformed into Kidnapping. UniLR focuses on key elements such as valuables and violence in the knowledge, and further enriches the knowledge information by associating the legal
<table>
  <tr>
    <th>Fact Description</th>
    <td>At around 2 a.m. on August 4, 2015, the defendant, W, drove a tricycle to a foot bath shop in Cixi City and stole a white Apple 4S phone worth RMB 750 while the victim, L, was asleep. As W was escaping, L woke up and grabbed the tricycle. Despite knowing this, W accelerated, dragging L for about 20 meters and causing L to fall and sustain minor injuries. The stolen phone was recovered and returned to L.</td>
    <th colspan="3">Ground truth</th>
    <th>NetJudge</th>
    <th>UniLR</th>
  </tr>
  <tr>
    <th rowspan="2">Judgment</th>
    <th>Law articles</th>
    <th>Article 263</th>
    <th>Article 263 (Anyone who steals a significant amount of public or private property, commits theft multiple times, breaks into a residence to steal, carries a weapon during theft, or engages in pickpocketing shall be sentenced to up to 3 years of imprisonment, detention, or control, and fined.)</th>
    <th>Article 263 (Anyone who robs public or private property by violence, threat, or other means shall be sentenced to 3 to 10 years of imprisonment and fined)</th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <th>charges</th>
    <th>Robbery</th>
    <th>Theft (Definition: Theft is the act of secretly taking a significant amount of public or private property, or repeatedly secretly taking public or private property, with the intent of illegal possession.)</th>
    <th>Robbery (Definition: Robbery is the act of unlawfully taking public or private property by force, threat, or other means with the intent of illegal possession.)</th>
    <th></th>
    <th></th>
  </tr>
</table>

Figure 6: A case study of article and charge prediction tasks.

<table>
  <tr>
    <th>Query</th>
    <td>In order to quickly obtain valuables, defendant A detained victim B for an extended period and violently demanded cash. Is the sentence in this case serious?</td>
    <th colspan="3"></th>
    <th colspan="2"></th>
  </tr>
  <tr>
    <th>Ground Truth</th>
    <td>According to Article 239 of the Criminal Law, the defendant's behavior seriously violates the individual's right to freedom and personal safety. Defendant A kidnapped the victim for the purpose of forcing him to provide property, and he shall be sentenced to fixed-term imprisonment of not less than ten years or life imprisonment.</td>
    <th>SAILER</th>
    <th>UniLR</th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <th>Retrieved Knowledge</th>
    <td>Query: In order to seek revenge, defendant A illegally detained and insulted Victim B. How will he be sentenced? Answer: According to the law, illegally detaining others shall be sentenced to fixed-term imprisonment of not more than three years, detention, public surveillance, or deprivation of political rights.</td>
    <td>Query: Defendant A kidnapped and extorted Victim B 5 million yuan. The stolen money has been spent and A can not repay civil compensation. What is the verdict? Answer: According to the law, kidnapping for ransom of 5 million yuan is suspected of kidnapping and should be sentenced to life imprisonment.</td>
    <td>Legal event Kidnapping<br>Using violence to take hostages in exchange for benefits, the target is a person.</td>
    <td>Article knowledge: Article 239, Anyone who kidnaps another person to demand ransom, extort money or property shall be sentenced to at least ten years...</td>
    <td></td>
    <td></td>
  </tr>
</table>

Figure 7: A case study of Legal QA task.

event and the article. Ultimately, UniLR successfully retrieves the appropriate QA case of Kidnapping.

Reviewer paper summaries: This paper introduces UniLR (Unified Legal Retriever), designed to enhance LLMs' performance on multiple legal tasks through improved knowledge retrieval. The key innovation is addressing two main challenges in legal knowledge retrieval: diverse formats across different knowledge types and subtle differences within the same knowledge type. UniLR incorporates two main components: Key Element Supervisor (KES) that guides focus on important elements through attention supervision, and Graph-based Knowledge Augmenter (GKA) that enriches knowledge representation using a heterogeneous graph structure. The authors evaluate UniLR on multiple legal tasks including charge prediction, law article prediction, court view generation, and legal question answering, demonstrating superior performance compared to existing methods.

- This paper introduces a unified retrieval framework that can be utilized to complete various legal AI tasks. 

- The paper integrates multiple types of legal data, using the structured information between legal data to construct a graph structure for retrieval. 

- The content retrieved through this framework can assist LLM in accomplishing a wide range of legal tasks.
Summary of strengths: The paper presents a well-structured and comprehensive solution to an important problem in legal AI. The authors clearly identify and address specific challenges in legal knowledge retrieval, and  the technical approach combining attention supervision and graph-based knowledge representation is theoretically sound and well-motivated. The experimental evaluation is thorough, covering multiple legal tasks and including both automated metrics and human evaluation. The ablation studies effectively demonstrate the contribution of each component, and the case studies provide concrete examples of how UniLR improves over baseline methods.

1. The paper propose a unified legal retriever, which is quite interesting. It integrates various kinds of legal data, providing assistance for LLMs to accomplish legal AI tasks.

2. This paper takes advantage of the hierarchical structure of legal data. For example, it extracts legal events, hierarchy in legal articles and legal cases. It models the associations between legal data based on mapping knowledge.

3. The experimental design is comprehensive. It adopts graph learning and contrastive learning methods to conduct more effective retrieval of legal data.
Summary of weaknesses: 1. There is a lack of discussion of the effects of the hyper-parameters such as the lambda of the combined loss, which is a very important hyper-parameter;
2. More details about the graph construction process could be given;
3. More quantitative analysis of the attention supervision mechanism's effectiveness could be included.

1. Although this paper proposes a unified legal retriever, the dataset it chooses is very limited, only including criminal law articles, and the dataset of legal cases is incomplete. A lot of important legal data are omitted, which would significantly impact the retrieval performance.

2. Regarding the CAIL2018-Article task, I don't understand why LLMs are used for this task. The essence of this task is to retrieve the most similar legal articles, essentially a retrieval task. In this experiment, a LLM is used for output after retrieval. I think the experiment design needs to be improved.

3. The scope of this article is very limited. The data used is limited, and the downstream legal tasks are also quite confined. Although it proposes the use of graph structures to construct hierarchical information for legal data, this method can only be used within the realm of criminal law and is completely inapplicable in areas such as civil law and administrative law.

4. I suggest referring to the following articles when revising the paper:

- Ma Y, Shao Y, Wu Y, et al. LeCaRD: a legal case retrieval dataset for Chinese law system[C]//Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 2021: 2342-2348.

- Yu W, Sun Z, Xu J, et al. Explainable legal case matching via inverse optimal transport-based rationale extraction[C]//Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 2022: 657-668.
Comments, suggestions and typos: None

1. I suggest that the author first expand the dataset used in the paper, as the current dataset is too limited. If the dataset is not expanded to include other areas of law, I recommend adding "in criminal law" to the title, abstract, and intro. At the same time, the legal case data in the criminal law dataset should be expanded.

2. I advise the author to conduct a more detailed experimental design on how to utilize the LLMs when completing downstream tasks. For instance, when the LLMs should retrieve, and how it uses the knowledge after hierarchical retrieval. This would improve the performance of the downstream tasks and make full use of the capabilities of the LLMs.
Overall assessment scores: 3

2.5