Venue: NAACL 2025

# Elevating Legal LLM Responses: Harnessing Trainable Logical Structures

and Semantic Knowledge with Legal Reasoning

 Rujing Yao1, Yang Wu2, Chenghao Wang3, Jingwei Xiong4, Fang Wang1, Xiaozhong Liu2\({}^{*}\)

###### Abstract

Large Language Models (LLMs) have achieved impressive results across numerous domains, yet they experience notable deficiencies in legal question-answering tasks. LLMs often generate generalized responses that lack the logical specificity required for expert legal advice and are prone to hallucination, providing answers that appear correct but are unreliable. Retrieval-Augmented Generation (RAG) techniques offer partial solutions to address this challenge, but existing approaches typically focus only on semantic similarity, neglecting the logical structure essential to legal reasoning. In this paper, we propose the Logical-Semantic Integration Model (LSIM), a novel supervised framework that bridges semantic and logical coherence. LSIM comprises three components: reinforcement learning predicts a structured fact-rule chain for each question, a trainable Deep Structured Semantic Model (DSSM) retrieves the most relevant candidate questions by integrating semantic and logical features, and in-context learning generates the final answer using the retrieved content. Our experiments on a real-world legal QA dataset-validated through both automated metrics and human evaluation-demonstrate that LSIM significantly enhances accuracy and reliability compared to existing methods.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 5630–5642

**Rujing Yao1, Yang Wu2, Chenghao Wang3, Jingwei Xiong4, Fang Wang1, Xiaozhong Liu2\({}^{*}\)**

1Nankai University, Tianjin, China

2Worcester Polytechnic Institute, USA

3Fayuan Technology Co., Ltd., Hangzhou, China

4University of California, Davis, USA rjyao@mail.nankai.edu.cn, ywu19@wpi.edu, chenghao.woh@gmail.com, jwxiong@ucdavis.edu, wangfangnk@anankai.edu.cn, xliu14@wpi.edu

## 1 Introduction

The rapid advancement of Large Language Models (LLMs) has provided ordinary individuals with a way to access affordable legal services, substantially expanding their opportunities to obtain legal assistance Cheong et al. (2024); Louis et al. (2024). However, given the diversity, complexity, and vague expressions of legal questions, responses generated by LLMs often lack logical specificity and may include hallucinations, exposing users to potential legal risks and financial losses Dahl et al. (2024). Retrieval-Augmented Generation (MAG) Lewis et al. (2020); Li et al. (2024) has emerged as a promising approach to mitigate hallucinations and improve response accuracy by retrieving relevant legal cases and provisions as contextual knowledge sources for LLMs. Nevertheless, existing RAG methods primarily focus on semantic similarity and frequently overlook the intricate logical structures and reasoning essential for addressing complex legal issues, limiting their effectiveness in real-world legal applications.

As illustrated in Fig. 1, this paper introduces a novel LLM framework specifically designed to address complex legal QA tasks. Our approach integrates learnable chain-of-thought (CoT) reasoning as logical structures with supervised RAG, enhanced by the Deep Structured Semantic Model (DSSM) Huang et al. (2013). By embedding logical reasoning into the retrieval process, the framework ensures that both semantic relevance and logical coherence are maintained. Additionally, context learning is leveraged to synthesize high-quality answers, utilizing the retrieved auxiliary information to deliver precise, contextually appropriate responses.

The contributions of this work are fourfold:

* We propose a novel LSIM framework, which

Figure 1: An illustration of our model.



consists of three components: learnable fact-rule chain, supervised DSSM-powered RAG, and legal in-context learning for precise legal answer generation.
* We employ reinforcement learning to estimate the logical structure of users' legal questions, which integrates both logical structures and semantic information, and navigates LLMs to generate responses like legal professional.
* We extract fact-rule information in the form of chain of thought from users' legal questions, identifying essential facts and applicable legal rules. This enables the system to understand complex legal issues with precision, facilitating the retrieval of highly relevant case law.
* We conducted extensive experiments on a real-world legal QA dataset collected specifically for this study. The results validate the effectiveness and reliability of the proposed framework.

## 2 Related work

### Retrieval-Augmented Generation (RAG)

RAG can significantly improve the model performance by leveraging additional knowledge and has been widely applied in various tasks, such as question & answering (Q&A) Lewis et al. (2020); Mao et al. (2020), machine translation Gu et al. (2018), and summarization Liu et al. (2020); Parvez et al. (2021). With the emergence of LLMs such as LLaMA and ChatGPT, the integration of RAG with LLMs has gained significant popularity and led to significant advancements in multiple tasks Liu et al. (2023); Kim et al. (2023); Sharma et al. (2024); Feng et al. (2024).

RAG is also widely applied in research within the legal domain, such as legal Q&A Cui et al. (2023); Louis et al. (2024); Wiratunga et al. (2024), legal judgment prediction Wu et al. (2023), legal text evaluation Ryu et al. (2023), and terminology drafting for legislative documents Chouhan and Gertz (2024).

However, most prior research primarily concentrates on improving the performance of retrieval models from a semantic perspective. While semantic information is undoubtedly important, the significance of logical structure is particularly prominent in dealing with legal questions. Legal reasoning often relies on a well-defined logical flow. To address this challenge, our study emphasizes the integration of both semantic information and logical structure in retrieval processing.

### Question & Answering (Q&A)

Q&A is an active research area in NLP that aims to develop systems capable of providing accurate and relevant answers to questions posed in natural language by users based on large knowledge sources Rogers et al. (2023). Current Q&A studies mainly focus on 1) knowledge retrieval which aims to develop effective and efficient methods to retrieve relevant information from large knowledge bases or corpora Karpukhin et al. (2020), 2) reading comprehension which aims to build models that can comprehend passages to identify answerelevant information Baradaran et al. (2022), 3) multi-hop reasoning, which aims to perform multi-step reasoning by combining information from multiple sources Wang et al. (2022), and 4) explainable Q&A which aims to generate human-understandable explanations or rationales to support their answers Latcinnik and Berant (2020).

### AI Applications in Law

The legal domain has seen increasing interest in applying AI and machine learning techniques to assist with various tasks in Law. One active area of research is using NLP for legal document analysis and information extraction Zhong et al. (2020). Mistica et al. (2020) created a schema based on related information that legal professionals seek within judgements and performed classification based on it. Sun et al. (2023) proposed a model-agnostic causal learning framework to for legal case matching. There is also work on using AI for legal judgment prediction, as in Liu et al. (2023b) who develop a neural framework to predict judgments from fact descriptions.

Another emerging application is using AI for legal QA, legal reasoning, and argument mining from texts. Chen et al. (2023) proposed a well-annotated real-world dataset for legal QA. Mumford et al. (2023) established a new dataset and explored neural methods to capture patterns of reasoning over legal texts. Zhang et al. (2023) investigated extracting argumentative components like claims and evidence from legal cases. Some researchers are also exploring constitutionality analysis, with Sert et al. (2022) proposing an AI system to predict decisions of the Turkish constitutional court. While promising, these AI-based legal methods still face chal 

lenges around interpretability, generalization, and capturing the nuanced reasoning required in law.

## 3 Methodology

In this section, we propose our Logical-Semantic Integration Model (LSIM), as depicted in Fig. 2.

### Learnable fact-rule chain

In the legal domain, judging a case requires comprehensive consideration of the facts of the case and relevant legal rules. Each judgment process is a reasoning process that needs to combine the facts of the case with legal rules to reach a final conclusion. The user's question and the lawyer's response can be viewed as a complete legal case. Constructing a fact-rule chain for the case helps clarify the entire logical structure and more clearly identify the core issues for the case. Therefore, in this study, we represent the logical structure as the fact-rule chain of the question and its answer. Each node in the fact-rule chain consists of either a fact or a rule. Fact nodes correspond to the specific circumstances of the case, such as illegal search, arrest, and indictment. Rule nodes correspond to the relevant legal basis applicable to legal case circumstances, such as Fourth Amendment, harmless error doctrine, and Federal Rules of Evidence. The complete fact-rule chain illustrates the comprehensive reasoning process for the legal question/answer.

#### 3.1.1 Logical structure extraction

Following Wu et al. (2024), a fact-rule graph \({\cal G}\) is constructed using the LLM. Assume our training set \(T=\{(q_{i},a_{i})\}^{N}_{i=1}\) contains \(N\) instances, where \(q_{i}\) is the \(i\)-th question, and \(a_{i}\) is the real lawyer's answer to \(q_{i}\). For each question-answer pair \((q_{i},a_{i})\), the LLM, guided by a tailored prompt, is employed to extract the most relevant fact-rule chain from graph \({\cal G}\). The prompt is provided in Appendix A.11. Then, the fact-rule chain \(C_{q_{i}}\) for question \(q_{i}\) is obtained, and \(C_{q_{i}}=\{c_{q_{i},1},c_{q_{i},2},...,c_{q_{i},t}\}\) where \(c_{q_{i},t}\) is the \(t\)-th fact-rule chain node of the question \(q_{i}\). Similarly, the fact-rule chain \(C_{a_{i}}\) for answer \(a_{i}\) is obtained, and \(C_{a_{i}}=\{c_{a_{i},1},c_{a_{i},2},...,c_{a_{i},t}\}\) where \(c_{a_{i},t}\) is the \(t\)-th fact-rule chain node of the answer \({a_{i}}^{2}\). Consequently, the fact-rule chain for all questions \(C_{Q}\) and the fact-rule chain for all answers \(C_{A}\) in the training set are obtained, where \(C_{Q}=\{C_{q_{i}}\}^{N}_{i=1}\) and \(C_{A}=\{C_{a_{i}}\}^{N}_{i=1}\)3.

Footnote 3: \(C_{Q}\) is a set composed of fact-rule chains for \(N\) legal questions, and \(C_{A}\) is a set composed of fact-rule chains for \(N\) legal answers.

#### 3.1.2 Learning to predict fact-rule chain

The prediction of fact-rule chain is inherently a cumulative and continuous process. Therefore, we consider the fact-rule chain prediction task as a sequential decision-making process, and a reinforcement learning-based approach is empolyed. Specifically, reinforcement learning is employed to predict the fact-rule chain of the lawyer's response based on the information detected from the user's question. This prediction serves to supplement the information in the user's question. Given the fact-rule chain \(C_{q_{i}}\) for the legal question \(q_{i}\), we first encode \(C_{q_{i}}\) using BERT Kenton and Toutanova (2019) to obtain its embedding representation:

\[h_{C_{q_{i}}}=Encode(C_{q_{i}}). \tag{1}\]

Then, we utilize a policy network \(\pi_{\theta}(n_{t}|s_{t})\) to predict the fact-rule chain for answer \(a_{i}\) step by step, where \(s_{t}\) represents the current state at time step \(t\), and \(n_{t}\) denotes the action (next fact-rule chain node) predicted by the policy network. The initial fact-rule chain \(C_{q_{i}}^{t=0}\) is set to \(Cq_{i}\), and the initial state \(s_{0}\) is set to \(h_{C_{q_{i}}}\). At step \(t\), the policy network selects an action \(n_{t}\) based on the current state \(s_{t}\). Then, the selected node \(n_{t}\) is appended to the current fact-rule chain:

\[C_{q_{i}}^{t+1}=[C_{q_{i}}^{t},n_{t}],~{}~{}~{}n_{t}\sim\pi_{\theta}(n_{t}|s_{ t}) \tag{2}\]

Subsequently, the state embedding is updated using the new fact-rule chain:

\[s_{t+1}=Encode(C_{q_{i}}^{t+1}). \tag{3}\]

This process is repeated until a maximum number of steps is reached or no valid next node can be selected. The policy network is implemented as a multi-layer perceptron (MLP). The REINFORCE algorithm Williams (1992) is employed to train the policy network, which is a classic policy gradient method in reinforcement learning. The REINFORCE algorithm focuses strongly on maximizing long-term rewards. This enables the model to consider more far-reaching goals and impacts, rather than merely immediate predictive performance. In complex tasks such as legal reasoning, each thought 

step is built upon the foundation of previous logical reasoning. The REINFORCE algorithm can effectively simulate this process, learning the logical relationships between nodes in the fact-rule chain, thereby ensuring that the generated fact-rule chain is logically coherent and accumulative. The training objective is to maximize the expected cumulative reward:

\[J(\theta)={\mathbb{E}}\pi_{\theta}[\sum^{T}_{t=0}r_{t}], \tag{4}\]

where \(r_{t}\) is the reward at step \(t\), and \(T\) is the maximum number of steps. The reward \(r_{t}\) is defined as follows:

\[r_{t}=\begin{cases}1,&\text{if $n_{t}\in C_{a_{i}}$}\\ 0,&\text{otherwise}\end{cases} \tag{5}\]

where \(C_{a_{i}}\) is the ground-truth fact-rule chain for answer \(a_{i}\).

During inference, the trained policy network is employed to predict the fact-rule chain for a given legal question. Assume the inference step is \(z\), the predicted fact-rule chain is \(C^{z}_{q_{i}}\) for question \(q_{i}\), and \(C^{z}_{q_{i}}\) is the predicted logical structure. Appendix B provides examples illustrating the fact-rule chains.

### Supervised DSSM-powered RAG

Deep Structured Semantic Model (DSSM) Huang et al. (2013) is utilized to retrieve the most relevant questions from the database that are relevant to the user's question \(q_{i}\) in terms of legal knowledge. These retrieved questions, along with corresponding responses from lawyers, are provided to the LLM as context, assisting it in generating more accurate responses to the current user's question.

Let \(D\) be the database of candidate questions, and \(D=\{(q^{D}_{j},a^{D}_{j})\}^{M}_{j=1}\) contains \(M\) instances, where \(q^{D}_{j}\) is the \(j\)-th candidate question in \(D\), and \(a^{D}_{j}\) is the real lawyer's answer to \(q^{D}_{j}\).

Given a legal question \(q_{i}\in T\), its logical structure \(C^{z}_{q_{i}}\) can be obtained by the method described in Section 3.1.2. Similarly, for each candidate question \(q^{D}_{j}\in D\), its logical structure \(C^{z}_{q^{D}_{j}}\) can also be obtained. Then we encode each of them independently using the same encoder:

\[h_{q_{i}}=Encode(q_{i}),\] \[h_{C_{q_{i}}}=Encode(C^{z}_{q_{i}}),\] \[h_{q^{D}_{j}}=Encode(q^{D}_{j}),\] \[h_{C_{q^{D}_{j}}}=Encode(C^{z}_{q^{D}_{j}}). \tag{6}\]

Subsequently, \(h_{C_{q_{i}}}\), which represents the logical structure features, and \(h_{q_{i}}\), which represents the semantic features, are concatenated together to form the features for the current question \(q_{i}\):

\[e_{q_{i}}=[h_{C_{q_{i}}},h_{q_{i}}]. \tag{7}\]

Similarly, the features for candidate question \(q^{D}_{j}\) can be obtained:

\[e_{q^{D}_{j}}=[h_{C_{q^{D}_{j}}},h_{q^{D}_{j}}]. \tag{8}\]

The DSSM model is composed of a multi-layer perceptron (MLP) and computes a relevance score \(p_{ij}\) between \(q_{i}\) and candidate question \(q^{D}_{j}\):

\[x_{1}=\tanh(W_{1}[e_{q_{i}},e_{q^{D}_{j}}]+b_{1})\] \[x_{2}=\tanh(W_{2}x_{1}+b_{2})\] \[x_{3}=\tanh(W_{3}x_{2}+b_{3})\] \[p_{ij}=W_{4}x_{3}+b_{4}, \tag{9}\]

where \(W_{1},W_{2},W_{3}\), and \(W_{4}\) are weights, and \(b_{1},b_{2},b_{3}\) , and \(b_{4}\) are bias.

The margin ranking loss is employed, which encourages the model to assign higher scores to more relevant cases. For each question \(q_{i}\), we select the

Figure 2: The overall framework of LSIM. The network consists of three modules: learnable fact-rule chain, supervised DSSM-powered RAG, and legal in-context learning.

 

candidate question in the database with the highest annotated relevance score as the positive example \(c_{i}^{+}\), and the candidate question with the lowest score as the negative example \(c_{i}^{-}\). The annotated relevance scores are generated by the LLM. Specifically, the LLM evaluates the relevance between the current query and each candidate question. These relevance scores are assigned on a scale from 0 to 5, where a score of 0 indicates minimal relevance and a score of 5 denotes the highest level of similarity.The prompt is provided in Appendix A.2 The loss function is defined as:

\[\mathcal{L}(q_{i},c_{i}^{+},c_{i}^{-})=\max(0,\alpha-p(q_{i},c_{i}^{+})+p(q_{ i},c_{i}^{-})), \tag{10}\]

where \(\alpha\) is a hyperparameter.

During inference, for each question \(q_{i}\), we compute the relevance scores between \(q_{i}\) and all candidate questions in the database \(D\) using the trained DSSM model. The top-K candidate questions with the highest scores are the final retrieval results.

### Legal in-context learning

After retrieving the top-K most relevant questions \(q_{j_{1}}^{D},q_{j_{2}}^{D},...,q_{j_{K}}^{D}\) from the database \(D\) for the current question \(q_{i}\), we concatenate them with their corresponding answers \(a_{j_{1}}^{D},a_{j_{2}}^{D},...,a_{j_{K}}^{D}\) to form the context for in-context learning:

\[context_{i}=[(q_{j_{1}}^{D},a_{j_{1}}^{D}),(q_{j_{2}}^{D},a_{j_{2}}^{D}),..., (q_{j_{K}}^{D},a_{j_{K}}^{D})]. \tag{11}\]

This context provides the LLM with relevant examples of how relevant legal questions have been answered by real lawyers in the past. Following Wu et al. (2023) and considering the length limit of the prompt, K is set to 3 in our experiments.

Then, the current question \(q_{i}\), the logical structure \(C_{q_{i}}\), and \(context_{i}\) are provided to the LLM to generate an answer. The prompt is provided in Appendix A.3:

\[a_{i}^{\prime}=LLM(q_{i},C_{q_{i}},context_{i}). \tag{12}\]

## 4 Experiments

### Datasets

We use real-world legal question and answer (Q&A) data collected from JUSTIA4. The dataset comprises 16,190 legal questions posed by users in the field of criminal law, with each question receiving responses from an average of 1.26 lawyers. The average length of the questions is 67 words, and the responses average 40 words. In the experiment, the total dataset comprising 16,190 samples is divided using an 8:2 ratio: 80% of the data (12,952 samples) is used as the database for retrieval, and 20% (3,283 samples) is used for training and testing. Subsequently, the 3,238 samples are further divided into training and testing sets using the same 8:2 ratio. The specific information is presented in Table 1. Appendix C presents some samples of the collected data5.

Footnote 4: [https://www.justia.com/](https://www.justia.com/)

### Baselines and evaluation metrics

**Baselines.** We implement the following baselines for comparison: **BM25**Robertson and Walker (1994), a classic bag-of-words information retrieval model, is used to retrieve the question from the database that most closely matches the user's query. **Bert**Kenton and Toutanova (2019), **Roberta**Liu et al. (2019), **Sentence T5**Ni et al. (2022), **INSTRUCTOR**Su et al. (2023), **GTR**Ni et al. (2022b), **BGE-m3**Chen et al. (2024), **text-embedding-ada-002**, **text-embedding-3-small**, and **text-embedding-3-large**, are employed to generate embeddings for the user's query and questions in the database. Similarity calculations are then used to determine the closest match. Following Louis et al. (2024); Wu et al. (2023, 2024), **LLaMA-2-13B**, **LLaMA-3-SB6**, and ** GPT-4o**, serve as the LLM baselines in our study. They can generate responses to the posed questions.

Footnote 5: The code and data are available at [https://github.com/RujingYao/LSIM](https://github.com/RujingYao/LSIM).

Footnote 6: [https://llama.meta.com/](https://llama.meta.com/)

**Evaluation metrics.** To evaluate our model, both automatic and human evaluations are used. For automatic evaluation, the commonly used text generation metrics, **ROUGE** (ROUGE-1, ROUGE-2, and ROUGE-L) Lin (2004), **METEOR**Baner

\begin{table}
\begin{tabular}{|c|c|} \hline
**Type** & **Value** \\ \hline
**Data Characteristics** & \\ \hline Total number of questions & 16,190 \\ \hline Total number of responses & 20,400 \\ \hline Mean length of questions (words) & 67 \\ \hline Mean length of responses (words) & 40 \\ \hline \multicolumn{2}{|c|}{**Data Split**} \\ \hline Database & 12,952 \\ \hline Training & 2,590 \\ \hline Testing & 648 \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of data we collected.

 

jee and Lavie, 2005), and **BERTScore**(Zhang et al., 2019) are employed. Human evaluation focuses on three aspects: 1) **Accuracy**: the aspect evaluates whether the generated answers are correct and free from factual errors. 2) **Specificity**: this aspect measures whether the responses are directly related to the specific issues raised in the question, providing clear and targeted answers rather than generalized responses. 3) **Adoptability**: this aspect assesses whether the responses generated by the model are practically useful and can be directly provided to users.

Three legal professionals were invited to evaluate the answers generated by LLaMA-3-8B, text-embedding-3-large, and our proposed Model LSIM. Each dimension is rated on a scale of 1-5, with 5 being the highest score. To ensure a fair and unbiased evaluation, each of the three professionals was presented with outputs from all tested LLMs in a randomized order, with no indication of which outputs corresponded to which models. The three professionals independently scored all the results, and the average score was calculated.

### Experiment Settings

For GPT-4o, the sampling parameters are set with a temperature of 0.8. For LLaMA-2-13B and LLaMA-3-8B, the sampling parameters are set with a temperature of 0.8 and a top-p value of 0.9. The maximum token limit per generation is set at 4096. For the LSIM method, the word embeddings are initialized using BERT. Adam is used as the optimizer. The learning rates for the policy network and the DSSM are both set to 1e-4. The number of epochs for the policy network and the DSSM are set to 30 and 50, respectively. All the results are the average values of three repeated runs.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline Method & METEOR & ROUGE-1 & ROUGE-2 & ROUGE-L & BERTScore \\ \hline LLaMA-2-13B w/o RAG & 17.09 & 13.13 & 1.91 & 12.09 & 81.24 \\ BM25 & 17.75 & 13.98 & 2.22 & 12.81 & 81.10 \\ Bert-Base & 17.96 & 13.94 & 2.18 & 12.69 & 81.33 \\ Roberta & 17.87 & 13.85 & 2.21 & 12.71 & 81.27 \\ SentenceT5-Base & 17.86 & 13.99 & 2.28 & 12.85 & 81.74 \\ INSTRUCTOR-Base & 17.92 & 14.12 & 2.26 & 12.82 & 81.81 \\ GTR-Base & 17.76 & 13.95 & 2.22 & 12.81 & 81.58 \\ BGE-m3 & 17.85 & 13.75 & 2.14 & 12.59 & 81.35 \\ text-embedding-ada-002 & 17.76 & 13.89 & 2.22 & 12.70 & 81.45 \\ text-embedding-3-small & 17.89 & 14.02 & 2.16 & 12.76 & 81.53 \\ text-embedding-3-large & 18.03 & 14.13 & 2.23 & 12.93 & 81.62 \\ \hline LSIM & **20.55** & **16.10** & **2.58** & **14.52** & **83.12** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance on legal response generation using LLaMA-2-13B (%).

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline Method & METEOR & ROUGE-1 & ROUGE-2 & ROUGE-L & BERTScore \\ \hline LLaMA-3-8B w/o RAG & 17.13 & 11.56 & 1.69 & 10.62 & 81.91 \\ BM25 & 17.84 & 13.34 & 2.09 & 12.10 & 82.47 \\ Bert-Base & 17.68 & 13.13 & 1.99 & 11.92 & 82.53 \\ Roberta & 18.03 & 13.38 & 2.14 & 12.15 & 82.56 \\ SentenceT5-Base & 18.40 & 13.58 & 2.16 & 12.40 & 82.50 \\ INSTRUCTOR-Base & 18.39 & 13.61 & 2.28 & 12.29 & 82.49 \\ GTR-Base & 18.10 & 13.54 & 2.15 & 12.32 & 82.48 \\ BGE-m3 & 18.08 & 13.46 & 2.08 & 12.12 & 82.52 \\ text-embedding-ada-002 & 17.87 & 13.24 & 2.04 & 12.01 & 82.48 \\ text-embedding-3-small & 18.32 & 13.70 & 2.22 & 12.44 & 82.49 \\ text-embedding-3-large & 18.62 & 13.82 & 2.24 & 12.53 & 82.52 \\ \hline LSIM & **21.00** & **16.30** & **2.63** & **14.74** & **83.23** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance on legal response generation using LLaMA-3-8B (%).

 



### Experiment Results

Tables 2, 3, and 4 present the experimental results obtained when LLaMA-2-13B , LLaMA-3-8B, and GPT-4o are used as base architectures, respectively. When LLaMA-2-13B is employed as the LLM baseline, our proposed LSIM algorithm also achieves the best performance across all metrics. Compared with the LLaMA-2-13B w/o RAG, our proposed LSIM model achieves improvements of 3.46% on METEOR, 2.97% on ROUGE-1, 0.67% on ROUGE-2, 2.43% on ROUGE-L, and 1.88% on BERTScore. When LLaMA-3-8B is employed as the LLM baseline, our proposed LSIM algorithm achieves the best performance across all metrics. Compared with the LLaMA-3-8B w/o RAG, our proposed LSIM model achieves improvements of 3.87% on METEOR, 4.74% on ROUGE-1, 0.94% on ROUGE-2, 4.12% on ROUGE-L, and 1.32% on BERTScore. Compared to the best performing baseline model text-embedding-3-large, LSIM model achieves improvements of 2.85%, 0.93%, and 0.64% on METEOR, ROUGE-L, and BERTScore, respectively. All results demonstrate that our LSIM algorithm achieves the best perfor

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline Method & METEOR & ROUGE-1 & ROUGE-2 & ROUGE-L & BERTScore \\ \hline GPT-4o w/o RAG & 17.92 & 12.36 & 2.20 & 11.46 & 81.60 \\ BM25 & 18.41 & 12.88 & 2.22 & 11.84 & 82.01 \\ Bert-Base & 18.53 & 12.90 & 2.24 & 11.91 & 82.11 \\ Roberta & 18.50 & 13.01 & 2.24 & 11.94 & 82.16 \\ SentenceT5-Base & 18.57 & 12.97 & 2.23 & 11.90 & 82.05 \\ INSTRUCTOR-Base & 18.55 & 12.92 & 2.30 & 11.92 & 82.04 \\ GTR-Base & 18.40 & 12.84 & 2.25 & 11.80 & 82.00 \\ BGE-m3 & 18.51 & 12.96 & 2.25 & 11.88 & 82.05 \\ text-embedding-ada-002 & 18.35 & 12.65 & 2.23 & 11.73 & 81.98 \\ text-embedding-3-small & 18.47 & 12.88 & 2.21 & 11.86 & 81.97 \\ text-embedding-3-large & 18.69 & 13.02 & 2.31 & 12.01 & 82.04 \\ \hline LSIM & **21.54** & **14.09** & **2.31** & **12.94** & **82.68** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance on legal response generation using GPT-4o (%).

Figure 3: Illustrative example 1. The sentences highlighted in red indicate remaining silent, while those highlighted in blue indicate seeking professional legal counsel.

\begin{table}
\begin{tabular}{|p{113.8pt} p{113.8pt}|p{113.8pt}|} \hline \multicolumn{4}{|c|}{**User’s Question**} \\ \hline Someone I know has been accused of touching a child. She is 18 now and is making these allegations. How can he clear himself. A friend of mine has been accused of inappropriately touching a young child when she was six, that’s what she is saying. She is 17 or 18 now. My friend is very upset and is wrongfully accused. Can he take her to court to get his name cleared? If so, what steps should be go about this? \\ \hline
**Lawyer before speaking with them** & **LLaMA-2-13B w/o RAG** & **LSIM** \\ Depending on the allegations and to whom she is making these claims, this could result in serious criminal charges against your friend. It at were me. Id keep silent about the allegations. And if he is contacted by law enforcement, definitely gets a lawyer before speaking with them & I understand that your friend is upset and wants to clear their name. However, the appropriate course of action depends on various factors, including the specific circumstances and legal jurisdiction. In other words, I don’t have and address the allegations. It’s essential to remain silent and avoid any contact with the account, as any statements or interactions may be used as evidence. The attorney can help your friend understand their rights and options, including the possibility of being a defamation claim. However, the primary focus should be on addressing the criminal allegations and ensuring your friends name is cleared. \\ \hline \hline  & & \\ \end{tabular}
\end{table}
Table 4: Performance on legal response generation using GPT-4o (%).



mance.

The results of the human evaluation are shown in Table 5. Our LSIM model achieves the best performance in terms of accuracy, specificity, and adoptability. These results highlight the effectiveness of our proposed LSIM model.

### Case Study

Figures 3, 4 and 5 present the comparison of LLaMA-2-13B directly answering the legal questions (LLaMA-2-13B w/o RAG) and utilizing our LSIM framework to respond to the legal questions.

For the given question in Figure 3, there are two main points in a real lawyer's response: keep silent and get a lawyer. However, the response generated by LLaMA-2-13B w/o RAG is comparatively generic, merely mentioning seeking legal counsel. By leveraging our LSIM model, we retrieve relevant questions that are related to the given query. Tabel 6 presents the most relevant question retrieved by LSIM. The answer to this question advises the user to "keep your mouth shut" and "hire a competent attorney".

By incorporating insights from the retrieved relevant questions, the LSIM model generates a response that covers two crucial aspects: remain silent and seek legal counsel. These key points align closely with the advice given by the real lawyer. The responses generated by our LSIM framework exhibit a higher degree of professionalism and more closely mirror the advice typically provided by a lawyer.

In Figure 4, the lawyer stated that the user could no longer drop the charges, whereas LLaMA-2-13B w/o RAG responded that the user has the right to drop the charges at any time. The response from our proposed LSIM model aligns with the lawyer's perspective, indicating that the user may not have the authority to drop the charges.

In Figure 5, the lawyer responded with, "The phone must be returned," whereas LLaMA-2-13B w/o RAG only noted that "the detective's actions may have been inappropriate or unreasonable." The response from our proposed LSIM model aligns with the key points in the lawyer's reply, explicitly stating that the user has the right to have their phone returned. These examples demonstrate the effectiveness of the proposed LSIM model.

### Ablation Study

An ablation study is also conducted on the LSIM framework when using LLaMA-3-8B as base LLM. The results are presented in Tabel 7. LSIM w/o LS refers to the LSIM model without the Logical Structure module. LSIM w/o SI refers to the LSM model without the Semantic Information module. The ablation study demonstrates that both the logical structure (LS) and semantic information (SI) modules contribute positively to the overall performance of the LSIM model. The best results are achieved by the full LSIM model, which combines the effects of both the LS and SI components.

\begin{table}
\begin{tabular}{c c c c c c} \hline Method & METEOR & ROUGE-1 & ROUGE-2 & ROUGE-L & BERTScore \\ \hline LSIM & **21.00** & **16.30** & **2.63** & **14.74** & **83.23** \\ LSIM w/o LS & 19.10 & 14.18 & 2.22 & 12.88 & 82.45 \\ LSIM w/o SI & 18.77 & 13.83 & 2.17 & 12.62 & 82.48 \\ \hline \end{tabular}
\end{table}
Table 6: The most relevant questions retrieved by LSIM.

\begin{table}
\begin{tabular}{l l l l l} \hline  & \multicolumn{4}{c}{**User’s Question**} \\ \hline Someone I know has been accused of touching a child. She is 18 now and is making these allegations. How can he clear himself. A friend of mine has been accused of inappropriately touching a young child when she was six, that’s what she is saying. She is 17 or 18 now. My friend is very upset and is wrongfully accused. Can he take her to court to get his name cleared? If so, what steps should he go about this? & \\ \hline \multicolumn{4}{c}{**The most relevant questions retrieved by LSIM**} \\ \hline A false allegation of inappropriate touching was made and a polygraph is being requested is here a way to dismiss this? My teen step-daughter has a history of bad behavior and being unruptly. During a recent counseling session, she accused me of touching her while giving her a bug last June. CPS is involved and I’m now being asked to take a polygraph. Since the incident, she has been sent to stay with her grandmother after sneaking out and breaking a neighbors window. & \\ \hline \multicolumn{4}{c}{**Lawyer’s Answer**} \\ \hline I recommend you keep your mouth shut and do not post anything else online. Hire a competent attorney today to counsel you on your possible criminal charges and how to conduct yourself during this DCS and/or LEO investigation. Again do not talk to anyone about this and have no contact with the girl. & \\ \hline \end{tabular}
\end{table}
Table 7: Ablation study when using LLaMA-3-8B as the LLM architecture (%).

 



## 5 Conclusion

This paper addresses the inherent limitations of LLMs in generating professional legal responses. We propose a novel framework, LSIM, designed to enhance the legal LLM reasoning by integrating the learnable logical structure and semantic information of legal questions. The LSIM framework is composed of three key components: (1) Reinforcement learning predicts the fact-rule chain of thought for a given legal query, guiding the reasoning process; (2) a supervised RAG retrieves the most relevant questions by integrating logical and semantic information; and (3) the fact-rule chain, relevant retrieved questions, and their corresponding answers are provided as auxiliary reference information to the LLM, enabling it to generate precise, contextually relevant responses.

We validate the effectiveness of LSIM through experiments on a real-world legal QA dataset. Results from both automated metrics and manual evaluations demonstrate the superior performance of the proposed framework in delivering accurate and expert-level legal answers.

In the future, we plan to extend the LSIM framework to other specialized domains, such as healthcare and finance. Additionally, we aim to incorporate multi-turn interactions with users, leveraging their real-time feedback to further refine the model's performance and adaptability.

Figure 4: Illustrative example 2. The sentences highlighted in blue indicate responses from different models regarding the user’s right to drop charges.

Figure 5: Illustrative example 3. The sentences highlighted in blue indicate responses from different models regarding the user’s right to property recovery.





### Limitations

The effectiveness of the RAG-based model heavily depends on the availability of databases. Consequently, our model's performance may degrade due to the lack of sufficient relevant legal cases to retrieve, which hinders the model's adaptability and utility in regions where legal cases are scarce.

As another limitation, our study is limited to single-turn interactions with LLMs. Expanding to multi-turn interactions could enhance the model's ability to develop a more nuanced and comprehensive understanding of user queries. However, achieving this will require a redesign of the proposed model to effectively manage contextual continuity and iterative feedback across multiple exchanges.

## References

* Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_, pages 65-72.
* Baradaran et al. (2022) Razieh Baradaran, Razieh Ghiasi, and Hossein Amirkhani. 2022. A survey on machine reading comprehension systems. _Natural Language Engineering_, 28(6):683-732.
* Chen et al. (2023) Andong Chen, Feng Yao, Xinyan Zhao, Yating Zhang, Changlong Sun, Yun Liu, and Weixing Shen. 2023. Equals: A real-world dataset for legal question answering via reading chinese laws. In _Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law_, pages 71-80.
* Chen et al. (2024) Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. _arXiv preprint arXiv:2402.03216_.
* Cheong et al. (2024) Inyoung Cheong, King Xia, KJ Kevin Feng, Quan Ze Chen, and Amy X Zhang. 2024. (a) i am not a lawyer, but...: Engaging legal experts towards responsible llm policies for legal advice. In _The 2024 ACM Conference on Fairness, Accountability, and Transparency_, pages 2454-2469.
* Chouhan and Gertz (2024) Ashish Chouhan and Michael Gertz. 2024. Lexdrafter: Terminology drafting for legislative documents using retrieval augmented generation. _arXiv preprint arXiv:2403.16295_.
* Cui et al. (2023) Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. _arXiv preprint arXiv:2306.16092_.
* Dahl et al. (2024) Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. _arXiv preprint arXiv:2401.01301_.
* Feng et al. (2024) Zhangying Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, and Bing Qin. 2024. Retrieval-generation synergy augmented large language models. In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 11661-11665. IEEE.
* Gu et al. (2018) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018. Search engine guided neural machine translation. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
* Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_, pages 2333-2338.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen Tau Yih. 2020. Dense passage retrieval for open-domain question answering. In _EMNLP_, pages 6769-6781.
* Ming-Wei Chang Kenton and Kristina Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of NAACL-HLT_, pages 4171-4186.
* Kim et al. (2023) Gangwoo Kim, Sungdong Kim, Byeongguk Jeon, Joonsuik Park, and Jaewoo Kang. 2023. Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 996-1009.
* Latcinnik and Berant (2020) Veronica Latcinnik and Jonathan Berant. 2020. Explaining question answering models through text generation. In _arXiv:2004.05569_.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.
* Li et al. (2024) Jiarui Li, Ye Yuan, and Zehua Zhang. 2024. Enhancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases. _arXiv preprint arXiv:2403.10446_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Liu et al. (2023a) Jiongnan Liu, Jiajie Jin, Zihan Wang, Jiehan Cheng, Zhicheng Dou, and Ji-Rong Wen. 2023a. Reta-llm: A retrieval-augmented large language model toolkit. _arXiv preprint arXiv:2306.05212_

* Liu et al. (2020) Shangqing Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. 2020. Retrieval-augmented generation for code summarization via hybrid gnn. In _International Conference on Learning Representations_.
* Liu et al. (2023b) Yifei Liu, Yiquan Wu, Yating Zhang, Changlong Sun, Weiming Lu, Fei Wu, and Kun Kuang. 2023b. Mlljp: Multi-law aware legal judgment prediction. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 1023-1034.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Louis et al. (2024) Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2024. Interpretable long-form legal question answering with retrieval-augmented large language models. In _Proceedings of the AAAI Conference on Artificial Intelligence_, pages 22266-22275.
* Mao et al. (2020) Yunning Mao, Pengcheng He, Xiaodong Liu, Yeulong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020. Generation-augmented retrieval for open-domain question answering. _arXiv preprint arXiv:2009.08553_.
* Mistica et al. (2020) Meladel Mistica, Geordie Z. Zhang, Hui Chia, Kabir Manandhar Shrestha, Rohit Kumar Gupta, Saket Khandelwal, Jeannie Paterson, Timothy Baldwin, and Daniel Beck. 2020. Information extraction from legal documents: A study in the context of common law court judgements. In _Proceedings of the 18th Annual Workshop of the Australasian Language Technology Association_, pages 98-103.
* Mumford et al. (2023) Jack Mumford, Katie Atkinson, and Trevor Bench-Capon. 2023. Combining a legal knowledge model with machine learning for reasoning with legal cases. In _Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law_, pages 167-176.
* Ni et al. (2022a) Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022a. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 1864-1874.
* Ni et al. (2022b) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al. 2022b. Large dual encoders are generalizable retrievers. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 9844-9855.
* Parvez et al. (2021) Md Rizwan Parvez, Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Retrieval augmented code generation and summarization. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 2719-2734.
* Robertson and Walker (1994) Stephen E Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In _SIGIR'94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University_, pages 232-241. Springer.
* Rogers et al. (2023) Anna Rogers, PictureMatt Gardner, and PictureIsabelle Augenstein. 2023. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. _ACM Computing Surveys_, 55(10):1-45.
* Ryu et al. (2023) Cheol Ryu, Soelhwa Lee, Subeen Pang, Chanyeol Choi, Hojun Choi, Myeonggee Min, and Jy-Yong Sohn. 2023. Retrieval-based evaluation for llms: A case study in korean legal qa. In _Proceedings of the Natural Legal Language Processing Workshop 2023_, pages 132-137.
* Sert et al. (2022) Mehmet Fatih Sert, Engin Yildirm, and Irfan Haslak. 2022. Using artificial intelligence to predict decisions of the turkish constitutional court. _Social Science Computer Review_, 40(6):1416-1435.
* Sharma et al. (2024) Sanat Sharma, David Seunghyun Yoon, Franck Dernoncourt, Dewang Sultania, Karishma Bagga, Mengjiao Zhang, Trung Bui, and Varun Kotte. 2024. Retrieval augmented generation for domain-specific question answering. _arXiv preprint arXiv:2404.14760_.
* Su et al. (2023) Hongjin Su, Weija Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finetuned text embeddings. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 1102-1121.
* Sun et al. (2023) Zhongxiang Sun, Jun Xu, Xiao Zhang, Zhenhua Dong, and Ji-Rong Wen. 2023. Law article-enhanced legal case matching: A causal learning approach. In _Proceedings of ACM SIGIR_, pages 1549-1558.
* Wang et al. (2022) Yu Wang, Vijay Srinivasan, and Hongxia Jin. 2022. A new concept of knowledge based question answering (KBQA) system for multi-hop reasoning. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4007-4017.
* Williams (1992) Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_, 8:229-256.
* Wiartunaga et al. (2024) Nirmalie Wiartunaga, Ramitha Abeyratne, Lasal Jayawardena, Kyle Martin, Stewart Massie, Ikechukwu Nikis-Orji, Ruvan Weersasinghe, Anne Liret, and Bruno Fleisch. 2024. Cbr-rag: Case-based reasoning for retrieval augmented generation in llmrs for legal question answering. _arXiv preprint arXiv:2404.04302_.
* Wu et al. (2024) Yang Wu, Chenghao Wang, Ece Gumusel, and Xiaozhong Liu. 2024. Knowledge-infused legal wisdom: Navigating llm consultation through the lens of diagnostics and positive-unlabeled reinforcement 

learning. In _Findings of the Association for Computational Linguistics: ACL 2024_.
* Wu et al. (2023) Yiquan Wu, Siying Zhou, Yifei Liu, Weiming Lu, Xiaozhong Liu, Yating Zhang, Changlong Sun, Fei Wu, and Kun Kuang. 2023. Precedent-enhanced legal judgment prediction with llm and domain-model collaboration. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12060-12075.
* Zhang et al. (2023) Gechuan Zhang, Paul Nulty, and David Lillis. 2023. Argument mining with graph representation learning. In _Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law_, page 371-380.
* Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhong et al. (2020) Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How does nlp benefit legal system: A summary of legal artificial intelligence. In _ACL_, pages 5218-5230.

## Appendix A Prompt details

### Relevant fact-rule path extraction

"Please select 1 to 4 nodes from the provided graph that are most relevant to the legal question/answer. Ensure that the selected nodes are interconnected."

### Similarity Score

"Please score the similarity of Question2 to Question1, focusing specifically on the events described in each legal question. Rate the similarity of Question2 to Question1 on a scale from 0 to 5, where 0 indicates that Question2 is completely different from Question1, and 5 indicates that Question2 is exactly the same as Question1."

### Generation of the final response

"Your task is to provide legal advice on the user's question. I will provide you with the logical structure of the user's question, along with similar questions previously asked by other users and the responses given by real lawyers. Please use this information to generate a response to the user's question."

## Appendix B Illustrative examples of the fact-rule chains

Illustrative examples of the fact-rule chains are provided in Figure 6.

## Appendix C Illustrative samples of the collected data

Illustrative samples of the collected data are provided in Table 8.

 



\begin{table}
\begin{tabular}{c c|p{142.3pt} p{56.9pt}|p{85.4pt}} \hline
**Question** & **Fact-rule chain of the question** & **Answer** & **Fact-rule chain of the answer** & **Logical structure** \\ \hline A range 1 offender gets sentenced as a range 2 on 2nd felony conviction. Is that legal? & criminal history, prior conviction, sentence sentence & It is definitely possible. The Sentencing can be appealed but it must be brought up in a Motion For New Trial first. However, the difference is 35% instead of 30% to be served prior to Parole Eligibility, in addition to a larger number of minimum years Sentence. & appeal, sentence enhancement, appeal, sentence guideline lines \\ \hline Can someone use text message as Evidence against another person court, Or Investigated Even citizens arrested. & evidence & Yes, but the text messages have to be authenticated and authenticated by calling as witness the person who sent it or received the message. & evidence, adjmissibility & evidence admissible \\ \hline I was arrested with a methapipe only. If untransphantaries or possession of Paraphernalia? & evidence, possession & If the government sends the pipe out for testing, and they are able to scrap any meth out of it, they could charge you with possession of methamphetamine. Its possible for this type of a charge to be filed several months after the police took the pipe as they wait for the lab results. It might be a good idea to consult with an attorney so you can prepare yourself for potential charges. If the pipe was clean, then you can’t be charged with possession of meth. & controlled substance, criminal law & **evidence, possession, controlled substance, criminal law** \\ \hline \end{tabular}
\end{table}
Table 8: Illustrative samples of the collected data.

Figure 6: Illustrative examples of the fact-rule chains. The red text represents the fact-rule chain for the question, and the green text represents the fact-rule chain for the answer. Together, they form the logical structure.