# Does ChatGPT Understand the Law? A Case Study on Road Homicide in Italy

Grazia Garzo, Alessandro Palumbo

<figure><img src="image_1.png" /></figure>

► To cite this version:

Grazia Garzo, Alessandro Palumbo. Does ChatGPT Understand the Law? A Case Study on Road Homicide in Italy. JURIX 2025 - 38th International Conference on Legal Knowledge and Information Systems, Dec 2025, Turin, Italy. pp.1-12. hal-05325502

HAL Id: hal-05325502

https://hal.science/hal-05325502v1

Submitted on 22 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Does ChatGPT Understand the Law?
## A Case Study on Road Homicide in Italy

Grazia GARZO a,<fnref n="1" />, Alessandro PALUMBO b

**Abstract.** This manuscript proposes a structured methodology and a replicable framework for empirically assessing the argumentative and interpretative capabilities of Large Language Models (LLMs) in the field of justice. As a demonstrative case, the framework is applied to the specific offence of vehicular homicide under Italian law, using GPT-4o as the tested model.

The evaluation is structured in two complementary phases. The first phase investigates the model's conceptual understanding in isolation: 60 legal concepts were tested through targeted prompts eliciting definitions, legal nuances, and illustrative examples. The responses were scored to assess the model's abstract comprehension of core legal notions. The second phase evaluates the model's ability to recognize, interpret, and apply these same legal concepts within real judicial reasoning. The model was provided with complete rulings from the Italian Court of Cassation and asked to summarize the decisions, identify the ratio decidendi, and reconstruct the legal reasoning underlying the Court's conclusions.

Outputs from both phases were assessed by a legal expert, who evaluated coherence, conceptual depth, and interpretative accuracy. The results highlight some correlation between the model's prior conceptual grounding and its ability to understand and replicate complex judicial reasoning. These findings underscore the importance of expert oversight in any forensic or judicial application of LLMs.

Beyond the specific case of vehicular homicide, the study proposes a generalizable framework for evaluating the legal reasoning capabilities of AI systems across different domains.

**Keywords.** Generative Artificial Intelligence, Large Language Models, Legal Understanding, Criminal Law, Road Homicide

## 1. Introduction and Motivation

In recent years, generative artificial intelligence has assumed a prominent role in debates on the future of legal scholarship, emerging as a transformative technology for the analysis, interpretation, and even application of the law [1,2]. Large Language Models (LLMs) such as GPT have distinguished themselves by their ability to generate coherent texts, answer complex questions, and simulate argumentative reasoning in natural language [3,4]. Their apparent competence, even in specialized domains like medicine, law, and public administration [5,6], raises a pressing question: do these systems genuinely understand legal concepts, or do they simply reproduce statistical correlations?

<footnote>a University of Siena, Italy</footnote>

<footnote>b CentraleSupélec, Inria, CNRS, IRISA, France</footnote>

<footnote n="1">¹Corresponding Author: Grazia Garzo, g.garzo@student.unisi.it</footnote>
This question is especially relevant in practical contexts where LLMs are deployed: drafting legal documents, providing automated advice, simplifying regulations, or analyzing case law [7]. Some jurisdictions have already introduced AI tools to support predictive jurisprudence [8], while others are evaluating their use in procedural and administrative domains [9,10]. In such scenarios, conceptual accuracy becomes crucial: if a model fails to distinguish between negligence and intent, or between causation and intervening causes, its answers may compromise fundamental principles of legality.

The Italian offence of vehicular homicide (Art. 589-bis c.p.), introduced by Law No. 41/2016, provides an emblematic case study. Unlike involuntary manslaughter (Art. 589 c.p.), it introduces harsher penalties, a detailed typification of conduct, and an aggravated structure aimed at punishing particularly dangerous driving [11,12]. The law presumes heightened culpability when the driver is impaired, narrowing judicial discretion in assessing negligence. Scholars have noted tensions with the principles of proportionality and personal responsibility [12].

Causality further complicates judicial evaluation: courts must apply the “more likely than not” standard, the lawful alternative conduct test, and analyze possible intervening causes. In multi-actor cases, liability depends on a differentiated assessment of contributory negligence and foreseeability of the outcome [11]. Another delicate issue is the distinction between conscious negligence and conditional intent, particularly in cases of egregious misconduct (street racing, driving with a revoked license, repeated violations). The Court of Cassation has repeatedly stressed that mere awareness of risk is not intent unless the agent accepts the possibility of the harmful outcome.

These features make vehicular homicide an ideal testbed for assessing whether LLMs can:

* Distinguish subjective and objective legal configurations;

* Correctly apply causal attribution standards;

* Differentiate aggravated negligence, conscious negligence, and conditional intent;

* Understand the implications of aggravating factors.

The degree of consistency in model outputs within this dense normative framework allows us to evaluate not only linguistic competence but also the capacity to generate logically grounded legal reasoning. This study explores three key dimensions: the influence of conceptual familiarity on legal reasoning; the recurrence of errors in applying criminal law concepts; and the possibility of constructing a replicable empirical framework for evaluating computational legal understanding. Unlike much of the existing literature, which focuses on predictive accuracy, this work emphasizes conceptual depth, jurisprudential grounding, and qualitative error taxonomies [6,13,14]. The contribution is twofold: it proposes a replicable methodology for assessing legal reasoning in LLMs, and it provides evidence of structural limitations that underscore the indispensable role of human experts in safeguarding legality and proportionality.

The remaining part of the manuscript is organised as follows: Section 2 reviews the main contributions in the literature concerning the legal capabilities of LLMs, with particular attention to the distinction between predictive applications and reasoning-oriented evaluations. Section 3 introduces the experimental methodology, outlining the structure and rationale of the two-phase framework and describing the procedures used to elicit and evaluate the model's legal reasoning. Section 4 presents the empirical results obtained from the application of this framework to GPT-4o, highlighting key patterns in
the model’s performance, correlations between conceptual competence and argumenta-
tive behavior, and recurring errors. Section 5 discusses the broader implications of these
findings, including methodological, ethical, and legal considerations. Finally, Section 6
concludes the paper by summarizing the main contributions.

## 2. Related Work

In recent years, the use of LLMs has attracted growing interest within both the legal and computer science research communities, generating a vast body of literature exploring the potential and limitations of artificial intelligence in the legal domain [15]. Two main research directions have emerged: on the one hand, the study of the predictive capabilities of the models (*legal prediction*); on the other, the analysis of their conceptual and argumentative competence (*legal reasoning*).

From a predictive standpoint, Zhong et al. developed an automated judgment pre-
diction system within the Chinese legal context, based on deep neural networks and se-
mantic representations of facts. The system achieved promising accuracy but struggled
to generalize to borderline cases [16]. Chalkidis et al. applied LSTM architectures and
transformer models to predict legal labels in cases before the European Court of Hu-
man Rights, reporting difficulties with complex or atypical cases [17]. Medvedeva et
al. conducted a systematic comparison of various machine learning models applied to
Dutch case law, noting a decline in performance on tasks requiring causal reasoning [18].
Aletras et al. proposed an SVM-based predictive model for ECtHR cases, demonstrat-
ing that models tend to exploit lexical correlations rather than grasp the underlying le-
gal rules [19]. The study in [20] presents a predictive approach to identifying vehicular
homicide using AI techniques to extract relevant features from crime scenes. Based on
Boolean logic, the system anticipates legal outcomes with an accuracy of 83.2%. A fur-
ther development is illustrated in [21], which introduces the first hardware circuit capable
of replicating judicial reasoning, achieving an accuracy of 86%.

In terms of legal reasoning, Bommarito and Katz tested GPT-3.5 on the U.S. bar
exam, finding high performance in civil law questions but significant instability in sub-
stantive criminal law [3]. Zheng et al. introduced CaseHOLD, a multiple-choice bench-
mark based on judicial decisions, highlighting that models tend to rely on superficial
linguistic patterns rather than normative comprehension [22]. Nanda et al. emphasized
the issue of *fairness* in legal question answering systems, showing a disconnection be-
tween surface-level accuracy and argumentative adequacy [23]. Dahl et al. documented
recurring *hallucination* phenomena in LLMs, including the generation of fictitious legal
or jurisprudential references [6].

To support the empirical evaluation of the legal capabilities of language models, nu-
merous benchmarks have been proposed. Hendrycks et al. introduced MMLU, a multi-
disciplinary test that includes legal questions [4]. Zheng et al. published LegalBench, a
suite of legal reasoning benchmarks [24]. Chalkidis et al. presented LexGLUE, a col-
lection of NLP legal datasets for various tasks [25]. Shu et al. evaluated GPT-4 on legal
reasoning tasks within the LawLLM project, highlighting both its strengths and its limi-
tations in handling complex cases [5]. Li et al. developed a knowledge-based approach,
K-LJP, to predict relevant charges and articles in Chinese legal judgment prediction [26].
Rabelo et al. curated the Japanese COLIEE benchmark for legal entailment and retrieval
tasks [27].
Despite the breadth of these contributions, many benchmarks adopt multiple-choice formats that neglect the evaluation of argumentative structure and favor answers based on lexical similarity. Guha et al. showed how simplified prompts can lead to an overestimation of models’ argumentative capacities [10,10]. Blair-Stanek et al. tested GPT-3.5 and GPT-4 on complex tasks, revealing difficulties in maintaining causal coherence and normative consistency in their explanations [28].

The literature has also addressed the ethical and regulatory implications of using AI in the legal domain. Livermore and Rockmore analyzed the risk of propagating ju-risprudential bias, emphasizing the need for human oversight in generative systems [7]. Hurshman warned against the opacity of training data and the challenges of model au-ditability [29]. Zhang et al. proposed an innovative ethical evaluation framework specif-ically designed to test LLM capabilities in legal contexts, considering not only knowl-edge but also ethical requirements [30]. Isaac et al. advocated for a legal regulation of AI systems grounded in transparency, traceability, and legal accountability [31]. Garzo et al. explored the legal and ethical implications of integrating predictive AI technologies into the Italian criminal justice system, with a particular focus on decision transparency, judicial autonomy, and responsibility [32]. Recent approaches have also explored the use of controlled languages for the formalization of legal norms. In [33] the authors pro-pose a human-in-the-loop methodology for translating EU legal provisions and case law into Attempto Controlled English, contributing to the debate on legal formalization and computable law.

In the European context, Luciano Floridi has outlined a model of *Trustworthy AI* aimed at the responsible integration of AI into legal systems, with emphasis on human oversight and explainability of decisions [34]. Giovanni Sartor analyzed the tension be-tween algorithmic efficiency and the principle of legality, raising concerns about the compatibility of automated decision-making with procedural guarantees in civil law sys-tems [9]. In the state of the art, there are areas that still leave room for methodological improvement, particularly with regard to how legal LLMs’ reasoning is evaluated. This study addresses three specific dimensions:

1. It proposes a general-purpose conceptual framework to assess whether (and in what way) LLMs understand and apply fundamental principles of criminal law;

2. It complements existing evaluation benchmarks (typically quantitative and rooted in English-language common law traditions) with an approach tailored to the normative structures of civil law systems;

3. It offers a structured classification of conceptual errors, enabling a more precise analysis of the epistemic limitations and reasoning behaviors exhibited by the model.

### 3. Methodology

In order to rigorously and replicably assess the ability of the ChatGPT-4o model to un-derstand, manipulate, and apply complex legal concepts within the framework of Ital-ian criminal law—with particular reference to the offence of vehicular homicide under Article 589-bis of the Penal Code—this study adopts a two-phase experimental protocol. The methodological goal is to move beyond traditional benchmarks based on predictive accuracy or surface-level performance, and instead to propose an empirical framework
September 2025

<figure><img src="image_3.png" /><figcaption>**Figure 1.** The workflows: Phase 1 (left); Phase 2 (right).</figcaption></figure>

capable of evaluating the depth of conceptual understanding, the consistency of legal reasoning, and the referential soundness of the model’s outputs.

The proposed approach is grounded in the distinction between two complementary dimensions of legal understanding, both examined in relation to concepts central to the offence of vehicular homicide under Italian criminal law. These dimensions are operationalized through a two-phase experimental methodology, with each phase designed to evaluate one aspect of the LLM’s legal competence:

1. **Stand-alone Conceptual Competence:** testing the model's ability to define, explain, and recognize criminal law concepts in isolation, without contextual anchoring in specific cases;

2. **Contextual Application in Jurisprudential Reasoning:** assessing the model's capacity to apply those same concepts within complex argumentative structures derived from real judicial decisions.

Figure 1 illustrates the overall structure of the two-phase experimental workflow. Each phase is articulated through a series of targeted prompts administered to the model, designed to elicit legal knowledge and reasoning. The outputs generated in response to these prompts were manually evaluated by a legal expert, using a structured scoring system. The assessment took into account the precision, clarity, and legal soundness of the responses, assigning scores that reflect the model's degree of conceptual understanding and argumentative reliability. While the present study focuses on GPT-4o and the offence of vehicular homicide, the methodology is designed to be model-agnostic and broadly applicable to other criminal offences and legal domains.

The crime of vehicular homicide was selected as the use case for applying the proposed methodology, due to its high conceptual density, legal complexity, and significance within contemporary Italian criminal jurisprudence. As an offence governed by a stratified and nuanced legal framework, it requires the interpreter to navigate intricate distinctions between negligence and intent, assess complex causal relationships, and apply aggravating factors with normative precision. These features make it an ideal benchmark for testing the model's ability to handle structured legal reasoning and to engage meaningfully with the doctrinal foundations of criminal liability.

### 3.1. Phase 1: Stand-alone Conceptual Competence

The first phase of the experimental protocol focused on evaluating the model's ability to understand legal concepts in isolation, independently of any specific factual or jurispru-
dential context. A total of 60 key concepts were identified as central to the interpretation and application of Article 589-bis of the Italian Penal Code, which governs the offence of vehicular homicide. The selected concepts cover a broad range of substantive criminal law notions, including negligence, intent, causation, aggravating circumstances, and forms of liability, and are documented and explained in detail in Table 1.

Once the glossary was constructed, each concept was submitted to the model through three distinct prompts, designed to test different facets of conceptual understanding:

* **Definition (D) prompt:** the model was asked to define the concept in technical-legal terms;

* **Example (E) prompt:** the model was asked to provide a concrete or case-based example illustrating the concept;

* **Reverse Identification (RI) prompt:** the model was given a descriptive formulation of the concept and asked to identify which legal notion it referred to.

For each prompt, the model's response was evaluated independently by a legal expert and assigned a numerical **Accuracy** score according to the following scheme: 2 for fully correct answers (i.e., the maximum score), 1 for partially correct answers (e.g., responses containing ambiguity, simplification, or minor inaccuracies), and 0 for incorrect or misleading outputs. This three-prompt evaluation was performed for each of the 60 designated legal concepts, resulting in a total of 180 individual interactions and corresponding expert assessments.

## 3.2. Phase 2: Contextual Application in Jurisprudential Reasoning

The second phase of the experimental protocol focuses on evaluating the model's ability to apply legal concepts within complex argumentative structures derived from real-world case law. Specifically, this phase assesses how well the model can interpret, reconstruct, and reason through actual judicial decisions concerning vehicular homicide.

To this end, a set of ten sentences from the Italian Court was previously selected based on their relevance, conceptual density, and doctrinal complexity. For each sentence, the model was presented with three distinct prompts, designed to probe different dimensions of its jurisprudential competence:

* **Decision Summary (DS) prompt:** the model was asked to summarize the content of the ruling, including the key facts and the outcome;

* **Factual Reasoning (FR) prompt:** the model was asked to explain the reasoning adopted by the Court in relation to the events that occurred and the causal chain leading to the decision;

* **Legal Justification (LJ) prompt:** the model was asked to identify and articulate the legal arguments and normative foundations that support the Court's decision.

Each response was manually evaluated by a legal expert using a structured scoring system, taking into account factors such as legal accuracy, argumentative coherence, clarity of exposition, and the correct use of legal concepts previously assessed in Phase 1. Five distinct scoring dimensions were applied to each prompt output:

1. **Concept Coverage Score:** quantifies the proportion of relevant legal concepts (out of the 60 defined in Phase 1) that are actually activated in the ruling under analysis. The score is expressed as a percentage;
2. **Concept Understanding Score:** evaluates the degree to which the model correctly interprets and applies the activated concepts in context, based on consistency with doctrine and case law. The score ranges from 0 (incorrect or inconsistent use) to 2 (fully accurate and contextually appropriate);

3. **Reasoning Quality Score:** assesses the overall structure, clarity, and internal coherence of the model's response, including its argumentative soundness and ability to reconstruct judicial logic. The score ranges from 0 (poor or fragmented reasoning) to 2 (complete, clear, and logically consistent);

4. **Ratio Identification Score:** specifically measures the model's capacity to correctly identify and formulate the ratio decidendi expressed by the Court in the given ruling. The score ranges from 0 (incorrect or missing) to 2 (accurate and well-articulated);

5. **Legal Error Penalty:** a corrective factor applied in cases where the output contains significant legal inaccuracies, such as the invention of non-existent case law, misattribution of legal principles, or serious conceptual confusion. The penalty ranges from 0 (no significant error) to -2 (severe legal error).

The Phase 2 covered a total of 10 sentences, each subjected to three distinct prompts, resulting in 30 model outputs. As reported in Figure 1, a total of 150 individual evaluations across the full set of judicial cases were applied. The cumulative score for each sentence ranges from a theoretical minimum of -2 to a maximum of 7. Specifically, the maximum is obtained by: (i) achieving 100% in the *Concept Coverage Score* (i.e., 1 point); (ii) receiving a perfect score of 2 in *Concept Understanding, Reasoning Quality, and Ratio Identification*; and (iii) incurring no penalty in the *Legal Error* metric (i.e., 0). This aggregation allows for a compact yet multidimensional representation of the model's jurisprudential performance on each case analysis.

## 4. Results

All experiments were conducted in Italian, including the legal sources, conceptual mappings, and prompts. The choice of language was crucial: since the task revolved around Italian legal doctrines specifically (the road homicide crime) as well as norms from the Italian *penal code* and *highway code*, using Italian ensured conceptual and terminological accuracy. The material presented in this manuscript has been subsequently translated into English for clarity and consistency in presentation. In Phase 1, the model's knowledge of the 60 legal concepts (listed in Table 1) was evaluated along three distinct prompting tasks: definition recall (D), example generation (E), and reverse identification (RI). Results show that the model demonstrates a strong grasp of legal definitions, with an average score of 1.95 out of 2 in the D prompt. Performance drops significantly in the E prompt, with an average of 1, indicating difficulty in generating legally appropriate and contextually accurate examples. However, in the reverse identification task (RI), where the model had to recognize a concept from its description, performance improved again, reaching an average score of 1.97. Table 2 reports the results of Phase 2, in which GPT-4o was asked to analyse ten publicly available rulings from the Italian Court of Cassation concerning road homicide. Each ruling is identified by its case number and year (e.g., 74/2022), and was evaluated across five dimensions.
Overall, the model demonstrated a not-so-small level of accuracy in identifying the legal concepts present in each ruling, with an average coverage score of 91.97%. The summaries generated by GPT-4o in response to the DS prompt were generally well-formed and comprehensible, with an average *Understanding* score of 2 out of 2. The model also performed well in recognising the legal reasoning embedded in these summaries, achieving an average *reasoning* score of 2. With regard to the identification of the judicial *Ratio decidendi*, the model's performance was slightly lower, with an average score of 1.8 out of 2, suggesting that it occasionally missed or only partially captured the core argumentative structure of the decisions. Penalties were applied when the output included errors or misinterpretations, leading to an average penalty score of -0.2 across all cases. For the FR prompt, the model maintained high levels of comprehension and reasoning accuracy, scoring a perfect 2 on both metrics. Additionally, it performed better than DS on the identification of the judicial ratio, with an average score of 2 and no penalties. The LJ prompt was more challenging. Although GPT-4o achieved an average score of 1.9 in understanding and maintained a strong performance in reasoning, its ability to capture the ratio of the decision dropped to 1.5. Moreover, this prompt accrued a slightly higher average penalty of -0.2.

<table><caption>Table 1.: List and scores of legal concepts evaluated in Phase 1</caption><thead><tr><th>Concept</th><th>Legal Source</th><th>D</th><th>E</th><th>RI</th></tr></thead><tbody><tr><td>Road Homicide</td><td>Art. 589-bis c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Serious or very serious road injuries</td><td>Art. 590-bis c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Drunk driving</td><td>Art. 186 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Driving under the influence of drugs</td><td>Art. 187 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Driver fleeing the scene</td><td>Art. 189 para. 1 and 6 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Failure to assist</td><td>Art. 189 para. 7 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>General negligence</td><td>Art. 43 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Specific negligence</td><td>Art. 43 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Conscious negligence</td><td>Case law</td><td>1</td><td>1</td><td>2</td></tr><tr><td>Conditional intent</td><td>Art. 43 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Causal link</td><td>Art. 40-41 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Joint participation in crime</td><td>Art. 110 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Aggravating factor: no license</td><td>Art. 186 para. 2-bis CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Aggravating factor: fleeing</td><td>Art. 589-bis para. 2 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Aggravating factor: altered state</td><td>Art. 589-bis para. 2 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Generic mitigating circumstances</td><td>Art. 62-bis c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Plea bargaining</td><td>Art. 444 c.p.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Abbreviated trial</td><td>Art. 438 c.p.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Contributory negligence</td><td>Case law</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Foreseeability of the event</td><td>Case law</td><td>2</td><td>1</td><td>2</td></tr><tr><td>BAC over 0.8 g/l</td><td>Art. 186 para. 2(b) CdS</td><td>1</td><td>1</td><td>2</td></tr><tr><td>BAC over 1.5 g/l</td><td>Art. 186 para. 2(c) CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>License revocation</td><td>Art. 222 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>License suspension</td><td>Art. 218 CdS</td><td>1</td><td>1</td><td>2</td></tr><tr><td>Driving with revoked license</td><td>Art. 116 CdS</td><td>2</td><td>0</td><td>2</td></tr><tr><td>Driving without a license</td><td>Art. 116 para. 15 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Refusal of alcohol test</td><td>Art. 186 para. 7 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Refusal of drug test</td><td>Art. 187 para. 8 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Mandatory arrest in flagrante delicto</td><td>Art. 380 c.p.p.</td><td>2</td><td>0</td><td>2</td></tr><tr><td>Detention of suspect</td><td>Art. 384 c.p.p.</td><td>2</td><td>2</td><td>2</td></tr><tr><td>Immediate trial</td><td>Art. 453 c.p.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Sentence upon request</td><td>Art. 444 c.p.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Conditional suspension of sentence</td><td>Art. 163 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Recidivism</td><td>Art. 99 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Dangerous driving</td><td>Art. 141 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Excessive speed</td><td>Art. 141 para. 3 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Failure to maintain safe distance</td><td>Art. 149 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Running a red light</td><td>Art. 146 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Failure to yield right of way</td><td>Art. 145 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Aggravating factor: high BAC</td><td>Art. 589-bis para. 2 c.p.</td><td>2</td><td>2</td><td>2</td></tr><tr><td>Use of mobile phone while driving</td><td>Art. 173 CdS</td><td>2</td><td>1</td><td>1</td></tr><tr><td>Dangerous U-turn</td><td>Art. 154 CdS</td><td>2</td><td>1</td><td>1</td></tr><tr><td>Illegal turn</td><td>Art. 154 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Disregard of road signs</td><td>Art. 146 CdS</td><td>2</td><td>2</td><td>2</td></tr><tr><td>Driving against traffic</td><td>Art. 143 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Psychophysical impairment</td><td>Art. 187 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Driving in dangerous conditions</td><td>Art. 187 CdS</td><td>2</td><td>0</td><td>2</td></tr></tbody></table>
<table><caption>Table 2. Evaluation scores for the output of prompts in Phase 2</caption><thead><tr><th>Concept</th><th>Legal Source</th><th>D</th><th>E</th><th>RI</th></tr></thead><tbody><tr><td>Alcohol test with breathalyzer</td><td>Art. 379 Reg. CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Drug test</td><td>Art. 187 CdS</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Criminal responsibility</td><td>Art. 85 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Mental capacity</td><td>Art. 85 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>State of necessity</td><td>Art. 54 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Self-defense</td><td>Art. 52 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Fortuitous event</td><td>Art. 45 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Mistake of fact</td><td>Art. 47 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Mistake of law</td><td>Art. 5 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Balancing of circumstances</td><td>Art. 69 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Accessory sanctions</td><td>Art. 19 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Fine</td><td>Art. 24 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Penalty</td><td>Art. 26 c.p.</td><td>2</td><td>1</td><td>2</td></tr><tr><td>Average</td><td></td><td>1.95</td><td>1</td><td>1.97</td></tr></tbody></table>

<table><thead><tr><th rowspan="2">Sentence</th><th rowspan="2">1. Coverage</th><th colspan="3">2. Understanding</th><th colspan="3">3. Reasoning</th><th colspan="3">4. Ratio</th><th colspan="3">5. Penalty</th><th rowspan="2">Score</th></tr><tr><th>DS</th><th>FR</th><th>LJ</th><th>DS</th><th>FR</th><th>LJ</th><th>DS</th><th>FR</th><th>LJ</th><th>DS</th><th>FR</th><th>LJ</th></tr></thead><tbody><tr><td>74/2022</td><td>100%</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>2</td><td>0</td><td>-2</td><td>0</td><td>0</td><td>13</td></tr><tr><td>20253/2023</td><td>75%</td><td>2</td><td>2</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>-1</td><td>14.75</td></tr><tr><td>5862/2021</td><td>100%</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>0</td><td>19</td></tr><tr><td>6366/2016</td><td>92.86%</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>1</td><td>0</td><td>0</td><td>-1</td><td>16.93</td></tr><tr><td>24175/2023</td><td>80%</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>18.8</td></tr><tr><td>17209/2023</td><td>90%</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>18.9</td></tr><tr><td>50110/2019</td><td>100%</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>19</td></tr><tr><td>42099/2021</td><td>100%</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>19</td></tr><tr><td>42018/2022</td><td>100%</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>19</td></tr><tr><td>36814/2023</td><td>81.81%</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>0</td><td>0</td><td>0</td><td>18.81</td></tr><tr><td>Averages</td><td>91.97%</td><td>2</td><td>2</td><td>1.9</td><td>2</td><td>2</td><td>2</td><td>1.8</td><td>2</td><td>1.5</td><td>-0.2</td><td>0</td><td>-0.2</td><td>17.72</td></tr></tbody></table>

## 5. Discussion

The evaluation of GPT-40's performance in the example generation task (E) reveals a consistent pattern of partial comprehension. In almost all cases, the model was able to provide responses that were thematically relevant, but superficial, lacking the legal precision, normative anchoring, and factual coherence expected in a proper legal example. One of the most recurrent issues was the fabrication or misuse of case references: many examples included invented or misattributed citations to judicial rulings, which undermined the legal credibility of the outputs. In some cases, GPT-40 presented what appeared to be real case scenarios but attached them to incorrect or imaginary sentence numbers. In other instances, it invented illustrative examples that, while plausible, were not explicitly framed as hypothetical, thereby generating potential confusion between real jurisprudence and constructed narratives. The examples also often lacked adequate legal framing. Concepts were introduced in vague or generic terms, without specifying the legal consequences, doctrinal interpretations, or relevant codified norms. While the model showed an ability to associate the correct semantic domain with each concept, the examples it produced fell short of what would be expected from a jurist or law student trained in producing concrete, normatively grounded illustrations.

These findings highlight an important distinction between linguistic-pattern recognition and true legal competence. The model performs well in tasks that rely on identifying or reproducing familiar formulations, such as definition recall (D) and reverse identification (RI). In both cases, its strong performance appears to stem more from its ability to match lexical and syntactic patterns than from a deep conceptual understanding: GPT-40 effectively retrieves key terms and legal phrasing, producing coherent outputs without necessarily engaging in substantive legal reasoning. The situation changes in the exam-
ple generation task (E), where the model is required to construct a concrete and legally accurate illustration of the concept.

While GPT-4o demonstrated notable competence in identifying key legal concepts and reproducing elements of judicial reasoning in Italian rulings, a deeper examination of the outputs generated in Phase 2 reveals several recurring limitations. Although the model's average scores were relatively high, the accuracy and reliability of its responses were uneven when examined more closely. One recurrent issue lies in normative misclassifications. For example, in multiple instances, the model misattributed cooperative liability to Article 110 of the Penal Code, when the correct reference should have been Article 113, which governs negligent cooperation. Similarly, it occasionally introduced non-existent legal expressions (e.g., implausible causality") or employed inappropriate legal phrasing which, although colloquially intuitive, lacks technical accuracy in the Italian legal context.

Beyond structural and legal errors, the model also exhibited a number of semantic inaccuracies that significantly affect the interpretability and epistemic reliability of its outputs. Some responses were vague, incomplete, or lacked appropriate normative grounding. GPT-4o fabricated or altered judicial identifiers, associating real case numbers with fictional fact patterns, or vice versa. These forms of referential distortion not only compromise clarity but may also mislead users by attributing false legitimacy to legal arguments. Even more critically, the model at times created hallucinated legal authority, presenting entirely invented case citations as if they were real or assigning existing ones to the wrong doctrinal contexts.

## 6. Conclusion

This study presented an empirical framework for evaluating the legal reasoning abilities of Large Language Models (LLMs), with a specific focus on GPT-4o and the offence of vehicular homicide under Italian criminal law. The findings reveal that GPT-4o exhibits strong performance in definition recall and reverse identification tasks, suggesting an ability to retrieve and recognize legal formulations with high lexical fidelity. However, it struggles in tasks that require generative precision and normative contextualization, such as the construction of legal examples. Critical errors persisted, including normative misclassifications, improper references to legal sources, vague or inaccurate expressions, and occasional hallucinations. These results reinforce a crucial distinction: LLMs do not exhibit legal understanding in the human sense but rather demonstrate a form of pattern-based linguistic competence. They are far from being autonomous legal agents, but their capabilities as decision-support tools should not be underestimated. When properly supervised, they can assist in tasks such as summarization, concept retrieval, and preliminary legal analysis. Future research should explore ways to integrate these systems within legal workflows, ensuring transparency, accountability, and human oversight. From a methodological standpoint, the framework proposed in this paper offers a replicable approach for dissecting and scoring LLM outputs across conceptual and argumentative dimensions, while highlighting recurring error types that may be domain-specific. This dual-layered evaluation, grounded in legal expertise, allows for a more granular and principled assessment than traditional benchmarks based solely on predictive accuracy.
References

[1] H. Surden. Artificial intelligence and law: An overview. Ga. St. UL Rev., 35:1305, 2018.

[2] Kevin D Ashley. Artificial intelligence and legal analytics: new tools for law practice in the digital age. Cambridge University Press, 2017.

[3] M. Bommarito II and D. M. Katz. Gpt takes the bar exam. arXiv:2212.14402, 2022.

[4] D. Hendrycks et al. Measuring massive multitask language understanding. arXiv:2009.03300, 2020.

[5] D. Shu et al. Lawllm: Law large language model for the us legal system. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4882–4889, 2024.

[6] M. Dahl et al. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1):64–93, 2024.

[7] M. A Livermore et al. Law search in the age of the algorithm. Mich. St. L. Rev., page 1183, 2020.

[8] Rachel E Stern, Benjamin L Liebman, Margaret E Roberts, and Alice Z Wang. Automating fairness? artificial intelligence in the chinese courts. Colum. J. Transnat'l L., 59:515, 2020.

[9] G. Sartor. La legalità algoritmica nella giurisprudenza amministrativa. Rivista Italiana di Informatica e Diritto, XXV(2):1–18, 2022.

[10] N. Guha et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123–44279, 2023.

[11] R. Giovagnoli. Manuale di Diritto Penale: Parte Generale. Itaedizioni, Roma, Italia, 2021.

[12] R. Giovagnoli. Manuale di Diritto Penale: Parte Speciale. Itaedizioni, Roma, Italia, 2023.

[13] J. H Choi and D. Schwarcz. Ai assistance in legal analysis: An empirical study. 2023.

[14] G. Garzo and A. Palumbo. Come l'intelligenza artificialità può supportare le decisioni giuridiche: Un caso di studio sugli omicidi stradali. Journal of Inclusive Methodology and Technology in Learning and Teaching, 5(1), 2025.

[15] M. Siino et al. Exploring llms applications in law: A literature review on current legal nlp approaches. IEEE Access, 2025.

[16] H. Zhong et al. How does nlp benefit legal system: A summary of legal artificial intelligence. arXiv:2004.12158, 2020.

[17] I. Chalkidis et al. Neural legal judgment prediction in english. arXiv:1906.02059, 2019.

[18] M. Medvedeva et al. Using machine learning to predict decisions of the european court of human rights. Artificial Intelligence and Law, 28(2):237–266, 2020.

[19] N. Aletras et al. Predicting judicial decisions of the european court of human rights: A natural language processing perspective. PeerJ computer science, 2:e93, 2016.

[20] G. Garzo et al. Opening the black box: How boolean ai can support legal analysis. In 2024 4th International Conference on Computer Communication and Artificial Intelligence (CCAI). IEEE, 2024.

[21] G. Garzo and A. Palumbo. The first hardware circuit emulating italian road homicides legal logic, daje! In IS 2025-18th IADIS International Conference Information Systems, 2025.

[22] L. Zheng et al. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the eighteenth international conference on artificial intelligence and law, pages 159–168, 2021.

[23] V. Nanda et al. Fairness through robustness: Investigating robustness disparity in deep learning. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021.

[24] L. Zheng et al. A reasoning-focused legal retrieval benchmark. In Proceedings of the 2025 Symposium on Computer Science and Law, pages 169–193, 2025.

[25] I. Chalkidis et al. Lexglue: A benchmark dataset for legal language understanding in english. arXiv:2110.00976, 2021.

[26] A. Li et al. Legal judgment prediction based on knowledge-enhanced multi-task and multi-label text classification. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 6957–6970, 2025.

[27] J. Rabelo et al. Coliee 2020: methods for legal document retrieval and entailment. In New Frontiers in Artificial Intelligence: JSAI-isAI 2020 Workshops, JURISIN, LENLS 2020 Workshops, Virtual Event, November 15–17, 2020, Revised Selected Papers 12, pages 196–210. Springer, 2021.

[28] A. Blair-Stanek et al. Blt: Can large language models handle basic legal text? arXiv:2311.09693, 2023.

[29] C. Hurshman. Do opaque algorithms have functions? Synthese, 204(3):91, 2024.

[30] R. Zhang et al. Evaluation ethics of llms in legal domain. arXiv:2403.11152, 2024.

[31] O. Isaac and O. Ifeoluwa. Artificial intelligence in law: Regulatory adaptation and ethical considerations.
<footnote>[32] Grazia G. and A. Palumbo. Legal & ethical implications of predictive digital techniques in the judicial criminal proceedings. In *13th International Symposium on Digital Forensics and Security (ISDFS)*, 2025.</footnote>

<footnote>[33] G. Garzo and A. Palumbo. Human-in-the-loop: Legal knowledge formalization in attempt to controlled english. In *13th International Symposium on Digital Forensics and Security (ISDFS)*, 2025.</footnote>

<footnote>[34] L. Floridi. Establishing the rules for building trustworthy ai. *Nature Machine Intelligence*, 1(6), 2019.</footnote>