<figure><img src="image_1.png" /></figure>

HAL
open science

# Discovering the Potential of LLMs in Annotating Legal
Texts for Argument Mining

Christina Berghegger, César Philippe, Karla Salas-Jimenez, Jean-Guy Mailly,
Leila Moudjari, Laurent Perrussel

► To cite this version:

Christina Berghegger, César Philippe, Karla Salas-Jimenez, Jean-Guy Mailly, Leila Moudjari, et al. Discovering the Potential of LLMs in Annotating Legal Texts for Argument Mining. Second International Workshop on Argumentation and Applications (Arg&App 2025), Nov 2025, Melbourne (AUS), Australia. hal-05339537

HAL Id: hal-05339537

https://hal.science/hal-05339537v1

Submitted on 30 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Discovering the Potential of LLMs in Annotating Legal
## Texts for Argument Mining

Christina Berghegger<sup>1,†</sup>, César Philippe<sup>1,†</sup>, Karla Salas-Jimenez<sup>1,2,†</sup>, Jean-Guy Mailly<sup>1,†</sup>,
Leila Moudjari<sup>3</sup> and Laurent Perrussel<sup>1</sup>

**Abstract**

Legal reasoning is a key domain for the application of computational argumentation techniques. However, the scarcity of datasets of annotated legal text tailored for argumentation-related tasks—aside from a single dataset comprising decisions from the European Court of Human Rights (ECHR)—poses a significant challenge to developing automated legal reasoning systems grounded in formal argumentation techniques. To address this challenge, we explore the use of Large Language Models (LLMs) to facilitate the annotation of legal texts, specifically focusing on identifying premises and claims within the ECHR dataset. Our preliminary results demonstrate the promising potential of LLMs to assist legal experts in annotating additional legal corpora. This advancement paves the way for more robust, argumentation-driven automated reasoning systems in the legal domain.

**Keywords**

Argument Mining, Legal Argumentation, Dataset annotation, ECHR, LLM

# 1. Introduction

Computational models of argumentation [1, 2] have shown their usefulness in modeling various kinds of real world situations, including legal reasoning. A key part of legal reasoning is based on the use of arguments, e.g., to convince a jury about the innocence or guilt of a defendant or to justify the decision of a court. The resolution of a case depends on the arguments provided by each party's theory of the case. In other words, the arguments are the cornerstone of the law and developing tools to extract these arguments would help litigants greatly. It will help improve the definition of the case, find contradictions made by the other party or provide a reasoned and motivated resolution. An illustrative and recent example of an application of computational argumentation for legal reasoning is the use of the ANGELIC method [3] to predict the decisions of the European Court of Human Rights (ECHR) [4], showing that this kind of technique based on symbolic reasoning has better accuracy than approaches purely based on Machine Learning.

The practical application of argument-based reasoning to legal scenarios relies on the extraction of arguments and their relations from a variety of legal text. Such an extraction requires the development of efficient argument mining techniques [5], but a weak point in the current state of the art in legal argument mining is the lack of annotated datasets.

The recent emergence of LLMs and their success for various tasks [6, 7] offer opportunities to improve the extraction of arguments from legal text. Various LLM tasks have been performed on the ECHR dataset, such as sentence classification [8], prompt engineering [9], or argument extraction [10]. The fact that there is no comparable dataset in the legal field creates little room to test generalizability or compare results on other baselines.

Arg&App 2025: International Workshop on Argumentation and Applications, November 2025, Melbourne, Australia

<figure><img src="image_3.png" /></figure>

<footnote n="1"><sup>1</sup>Université Toulouse Capitole, IRIT, Toulouse, France</footnote>

<footnote n="2"><sup>2</sup>PCIC, UNAM, Ciudad de México, México.</footnote>

<footnote n="3"><sup>3</sup>IRIT, Université de Toulouse, CNRS, Toulouse INP, UT3, Toulouse, France</footnote>

<footnote>†These authors were funded by the French National Research Agency (grant AIDAL, ANR-22-CPJ1-0061-01).</footnote>

<footnote>✉ christina.berghegger@ut-capitole.fr (C. Berghegger); cephilip@insa-toulouse.fr (C. Philippe); karla_dsj@ciencias.unam.mx (K. Salas-Jimenez); jean-guy.mailly@irit.fr (J. Mailly); leila.moudjari@irit.fr (L. Moudjari); laurent.perrussel@ut-capitole.fr (L. Perrussel)</footnote>

<footnote>© 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).</footnote>
The goal of this paper is to demonstrate how LLM can facilitate the annotation of text by legal experts. As in Gray et al. [11], we will use LLM for determining whether a sentence in a legal opinion describes a legal factor. Our main contributions are as follows:

* **Novel methodology for LLM as a legal annotator:** We propose a methodology that addresses a critical challenge in legal NLP. It is based on several LLM prompts that could help with daily tasks for legal practitioners.

* **Structured Evaluation Pipeline:** We design and compare multiple annotation pipelines (Direct, Direct with Complaints, and Modular), demonstrating thoughtful engineering and methodological rigor.

* **In-depth Evaluation:** We conduct both quantitative (similarity thresholds, match ratios) and qualitative (manual annotation reviews, iterative prompt refinement) evaluations to assess performance and guide improvements.

The paper is structured as follows: Section 2 explores the usage of LLMs for applications in the field of computational argumentation. In Section 3, we detail the setup of our annotation pipeline<fnref n="1" />. The experiments performed with different models and the results are explained in Section 4. Finally, we sketch out some future ideas for our project in Section 5.

## 2. Related Work

As most fields in Natural Language Processing (NLP), natural argumentation has been heavily impacted by the emergence of LLM. Researchers started evaluating the performance of LLMs in well-known argumentation tasks, such as premise and claim identification [12] or argument scheme classification [13], but also on new use cases, such as interactions between LLM agents [14]. The present work focuses on LLMs for data annotation for argument mining.

**LLMs to annotate data** One of the most relevant is the work of Trajano et al. [15], they observe that it can be challenging to find experts available to evaluate the correctness of the LLM results. Next, Lindahl and Borin [16] compare the inter-annotator agreement (IAA) for different texts, and state that a “less than substantial agreement is a quite common result in argumentation mining” due to the lack of a unified or generally accepted argumentation model and the fact that approaches to argumentation differ depending on the domain type. Pietron et al. [17] observe “a noticeable group of incorrect or problematic classifications made by human evaluators”. Savelka and Ashley [18] explore the ability of LLMs to semantically annotate legal texts in a zero-shot learning settings, which is a relevant research question in fields where either little data for finetuning is available, or finetuning is computationally not possible. They consider tasks such as contract review or case analysis, and compare the performance of different models, namely on semantic annotation tasks.

**LLMs for argument mining** There are papers that use LLMs for argument mining. For example, Trajano et al. [15] explore an approach to translating natural language arguments into computational arguments using LLMs with an argumentation scheme provided from Retrieval Augmented Generation (RAG). Zubaer et al. [19] compare the performance of GPT-like models in legal argument mining via prompting between zero-shot and with some examples, specifically in clause classification, using the legal corpus from the European Court of Human Rights (ECHR). They explore in-context learning (ICL), which involves guiding a language model using instructions and examples rather than fine-tuning. They highlight that prompts act like hyperparameters needing careful tuning and find that providing examples generally improves performance over zero-shot approaches. De Wynter and Yuan [9] evaluate the abilities of LLMs in argument mining and argument pair extraction, considering different levels of abstraction in their input and output. Gorur et al. [20] shows the interest of LLMs for identifying the

<footnote n="1">¹Source code and complete prompts available at https://github.com/KarlaDSJ/LegalArgumentsLLM</footnote>
attacks/supports relations between arguments. The approach outperforms state-of-the-art methods, but requires that the arguments have already been correctly identified. Faugier et al. [14] propose an LLM-based debate builder aiming at verifying the existing relations between arguments in a debate and assisting users to create new arguments, namely of type support and attack.

Moreover, although computational argumentation is a highly interdisciplinary field in which many tasks can be applied to different domains, each domain has particularities that must be considered. For illustration, Otiefy and Alhamzeh [12] explore argument mining tasks such as claim/premise and argument relation classification in the financial field, and Hong et al. [21] propose an agent-based framework based on LLM discussions via argumentation schemes, aiming at explainable clinical decision reasoning.

The work by Mumford et al. [22], presenting a new annotated dataset covering Article 6 of the European Convention on Human Rights (ECHR), shows the extensive effort which has to be undertaken to create new annotations. This motivated us to take a step backwards and, before proceeding to argument mining tasks, have a closer look at the potential of data annotation with the help of Large Language Models. We want to explore the possibility to annotate datasets for a variety of tasks, and in a variety of domains, minimizing time and costs. Notice that our goal is not to extract arguments with LLMs (i.e. to perform the argument mining task with LLMs), but to assist annotators in their task to create new datasets. The identification of additional arguments is an implicit aspect of this support, not the main focus. The dataset published by Poudyal et al. [23] is commonly used for legal argumentation tasks, and will serve as a baseline for our work.

## 3. Argumentation Pipeline

Let us now detail the pipeline for extracting arguments for annotations: in a first step, we detail the input, i.e. the dataset and second we describe the technical configuration.

### 3.1. Selected cases

<table><caption>**Table 1**</caption><thead><tr><th colspan="5">Short texts</th><th colspan="5">Long texts</th></tr><tr><th>text</th><th>length</th><th>words</th><th>tokens</th><th>ratio</th><th>text</th><th>length</th><th>words</th><th>tokens</th><th>ratio</th></tr></thead><tbody><tr><td>00</td><td>39</td><td>1789</td><td>2315</td><td>29%</td><td>01</td><td>127</td><td>2002</td><td>2381</td><td>19%</td></tr><tr><td>04</td><td>41</td><td>978</td><td>1281</td><td>31%</td><td>02</td><td>158</td><td>1462</td><td>2118</td><td>45%</td></tr><tr><td>05</td><td>24</td><td>993</td><td>1242</td><td>25%</td><td>17</td><td>175</td><td>2002</td><td>2739</td><td>37%</td></tr><tr><td>06</td><td>26</td><td>1130</td><td>1435</td><td>27%</td><td>19</td><td>130</td><td>2002</td><td>2505</td><td>25%</td></tr><tr><td>10</td><td>32</td><td>2002</td><td>2456</td><td>23%</td><td>27</td><td>128</td><td>2002</td><td>2544</td><td>27%</td></tr><tr><td>13</td><td>27</td><td>851</td><td>1068</td><td>25%</td><td>29</td><td>118</td><td>2002</td><td>2613</td><td>31%</td></tr><tr><td>16</td><td>32</td><td>2002</td><td>2537</td><td>27%</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>20</td><td>54</td><td>2002</td><td>2465</td><td>23%</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>21</td><td>70</td><td>2002</td><td>2513</td><td>26%</td><td></td><td></td><td></td><td></td><td></td></tr></tbody><tfoot><tr><th>mean</th><td>38.3</td><td>1527.7</td><td>1923.6</td><td>26%</td><td>139.3</td><td>1912</td><td>2483.3</td><td>31%</td><td></td></tr></tfoot></table>

The lengths of the texts are presented in terms of the number of lines, words, tokens, and ratios (ratio stands for 'word to token ratio' because in the tokenizer, a word is not necessarily one token).

As mentioned in previous sections, the only annotated dataset of legal texts (for argument mining purposes) is the one created by Poudyal et al. [23], the ECHR dataset. It is composed of 42 decisions of the European Court of Human Rights (ECHR). It is annotated by premise, conclusion, and non-argument. Recently, Habernal et al. [10] proposed an extended version of the ECHR dataset with token span labels. However, this new dataset only annotates arguments, not the parts of arguments (claims and premises).

As a baseline, we identified nine texts of the ECHR dataset with rather short text length. We chose short texts as input during the development process of the pipeline to accelerate testing and to rapidly identify possible shortcomings. In order to test the generalization of the pipelines in longer text we
identified six longer texts from the ECHR dataset. Table 1 details the characteristics of the short and long text used in this work.

## 3.2. Technical Setup

<figure><img src="image_4.png" /><figcaption>**Figure 1:** Schematic description of the three variants of our pipeline, where the text input is split in several text windows, and then either (1) the LLM is used to extract arguments directly, or (2) the complaints in the windows are identified before the argument extraction, or (3) for each complaint, there is a paragraph extraction before argument extraction.</figcaption></figure>

To capture the idea of modularity in terms of task execution, we initiate our project with three different types of pipelines, each having a different degree of modularity (see Figure 1). All pipelines have two versions, one of whose prompts are extended by examples, inspired by the literature around In-Context Learning as mentioned in Section 2.

**Pipeline 1: Direct from text** retrieve arguments directly from a legal text, without any intermediary tasks, pre- or post-processing. This is the least modular pipeline.

**Pipeline 2: Direct with complaints** First, we identify the applicant's complaints. Then, the LLM retrieves arguments from the original text that address the complaints.

**Pipeline 3: Modular** The idea of this pipeline is to replicate a behaviour which we consider to be a logical way of human reasoning: First, we search for relevant topics (complaints), then we identify their contexts (paragraphs) and finally draw conclusions (arguments).

**Technical Configuration** We used two models for comparison: the *meta-llama/Meta-Llama-3-8B-Instruct* [24], which has eight billion parameters, and the specialized law model *Equall/Saul-7B-Instruct-v1* [25] based on Mistral, which has seven billion parameters and implements a continuous pretraining using an English legal corpus of over 30 billion tokens from various jurisdictions (in the U.S., Europe and Australia). We set the temperature to 0.5 and the maximum number of output tokens to 1024.

We limited output tokens due to the 2000-word maximum output length of current LLMs, as shown by Bai et al. [26]. Since we lacked access to *Meta-Meta-Llama-3-8B-Instruct* to determine its word-token ratio, we used the tokenizer from *sentence-transformers/all-MiniLM-L6-v2* to estimate the ratio based on our input texts.

We analyzed the first 2000 words of each input text and found an average word-token ratio of 29%, equivalent to 2576 tokens (see Table 1). For now, we decided to limit the output pipeline to a far smaller number to account for the possible usage of models with smaller ratios. However, especially when working with longer texts as presented in subsection 4.2, testing larger output threshold should be explored as a way to improve pipeline performance. Given that we limit our output size, we also decided to split the input text into smaller windows. This has advantages and disadvantages. On the one hand, in case the entire text window is relevant and composed of arguments, the output size is large enough to retrieve all relevant information. On the other hand, the LLM might lose context, because it can only
refer to information from one text window at a time. We try to limit the latter by ensuring an overlap between text windows (see Figure 1).

**Annotation scheme.** For each pipeline, we provide a prompt and a guideline for annotating every argument found in it. We expect granular annotations of each argument, identifying the claims and premises. Guideline example:

n. Argument

**Claim:** {text of the claim}

**Premises:**

- {premise 1}

- {premise 2}

...

### 3.3. Pipelines Evaluation

We evaluated the LLM generated arguments in terms of their similarity with original annotations from the ECHR dataset. We calculate the similarity using the tokenizer and embeddings of the *sentence-transformers/all-MiniLM-L6-v2* model. Using vector similarity as comparison metric, we give some more freedom to the LLM, and introduce some degree of flexibility regarding the identification and interpretation of arguments. This is in line with the low annotators' agreement we could observe from many related studies, as presented in Section 2, and which suggests that argument annotations in the legal context are seldom straightforward.

We proposed two ways for evaluating the similarity of an argument: 1) Evaluating whole argument, without taking into account their internal structure, and 2) Evaluating every part of the argument, in a structured way. For example:

#### a) Unstructured argument

**Argument:** The Government submit that the Magistrates made an error of law within their jurisdiction which has been corrected on appeal. They rely on the absence of any suggestion in the High Court's judgment that the applicant's detention was unlawful.

#### b) Structured argument

**Claim:** The Government submit that the Magistrates made an error of law within their jurisdiction which has been corrected on appeal.

**Premises:** They rely on the absence of any suggestion in the High Court's judgment that the applicant's detention was unlawful.

**Unstructured argument** We use vector similarity as our comparison metric. The similarity is computed in two directions: 1) from each argument in the original dataset to the argument generated by the LLM (*base on original*), and 2) from each argument generated by the LLM to the arguments in the original dataset (*base on LLM*). An argument *a* and an argument *b* are considered a match if their vector similarity (denoted by Function *sim(a, b)*) exceeds a predefined threshold *t*. For this work, we set *t* = 0.75. Therefore, the *matching ratio* indicates the percentage of arguments that matched the process described above. In Appendix A, we present experiments for setting value *t*.

**Structured argument** We take the matched arguments for the *Unstructured argument*, computed the similarity between each claim and premises and evaluate the granular similarity by assigning different weights to premise and claim similarity: for $\alpha \in [0, 1]$, $sim\_arg(A_1, A_2) = \alpha \times simP(P_1, P_2) + (1 - \alpha) \times simC(C_1, C_2)$ with $A_i$ an argument made of premises $P_i$ and a claim $C_i$, and $simP$ (resp. $simC$) the function evaluating the similarity between premises (resp. claims) [27]. In our case, these functions are again the vector similarity, so the main point is to determine the relative importance of premises and claims (i.e. the value of $\alpha$). We ran several experiments to determine the value of $\alpha$ as 0.7. The experiments are presented in Appendix B.
# 4. Experiments and results

In this section, we first present a comparison of the three pipelines with zero-shot and one-shot prompts. Then, we conduct an in-depth study of Pipeline 1 with a one-shot and a rework prompt, using Llama [24] and Saul [25], and a state-of-the-art Argument Mining methodology (namely ArgueMapper/ArgueBuf) [28, 29] to test similarity in structured and unstructured arguments. Finally, we test the generalisation using long texts.

## 4.1. Comparison of the 3 pipelines on short texts

We ran all the pipelines on Llama [24], with and without examples, for the short texts, and did a high-level comparison with the original annotations. We took into account the mean number of arguments per text, the mean number of premises per argument, as well as the maximum number of premises belonging to one argument. As presented in Table 2, the pipeline *Direct from text with examples* is the closest to the original, and was thus used to generate outputs for all following evaluations and analysis. While the high-level evaluation gives a first indication that the pipeline with examples tends to output less arguments with more premises compared to its zero-shot setting, we want to get a more detailed understanding of the impact of providing examples. Hence, we keep the zero-shot pipeline to compare annotation results.

<table><caption>**Table 2**</caption><thead><tr><th>Pipeline</th><th>mean #Arg.</th><th>mean #Prem.</th><th>max #Prem.</th></tr></thead><tbody><tr><td><strong>Original dataset</strong></td><td><strong>13.56</strong></td><td><strong>2.49</strong></td><td><strong>5.56</strong></td></tr><tr><td>Direct from text (-)</td><td>25.22</td><td>2.01</td><td>4.56</td></tr><tr><td>Direct from text (+)</td><td><strong>16.44</strong></td><td><strong>2.21</strong></td><td><strong>4.56</strong></td></tr><tr><td>Direct with complaints (-)</td><td>25.56</td><td>1.99</td><td>4.22</td></tr><tr><td>Direct with complaints (+)</td><td>16.56</td><td>1.91</td><td>4.33</td></tr><tr><td>Modular (-)</td><td>23.22</td><td>1.82</td><td>4.44</td></tr><tr><td>Modular (+)</td><td>19.89</td><td>1.57</td><td>3.89</td></tr></tbody></table>

High-level pipeline evaluation on mean number of arguments, and mean and max number of premises per arguments, for the original dataset and the three pipelines with (+) or without (-) examples

Let us focus on Pipeline 1 *Direct from Text* and analyse the impact of providing examples to the LLM on annotation performance by evaluating the matching ratio, as well as similarity scores. We observed that the zero-shot pipeline extracted a lot more arguments than present in the original annotations. As shown in Table 3 column *zero-shot*, the matching ratio is lower on LLM arguments for the zero-shot setting.

<table><caption>**Table 3**</caption><thead><tr><th>Based on</th><th>zero-shot</th></tr></thead><tbody><tr><td>Original</td><td>70.69%</td></tr><tr><td>LLM</td><td>55.16%</td></tr></tbody></table>

Matching ratio 0.75 for Pipeline 1 *Direct from Text* with zero-shot. The column *Based on* refers to the base data set for calculating similarity.

The fact that the matching ratio based on original arguments is higher for the zero-shot setting indicates a low precision of LLM generated arguments. In other words, the LLM extracts too many arguments which increases chances of one of them being a good match for an original argument, but leaving a higher number of extracted arguments unmatched, decreasing the precision score. Besides this, the quality of matches is rather equal for the prompt with examples and the zero-shot setting, with a slightly better performance for the prompt with examples. Mean values over all texts are provided in Table 4.
<table><caption>**Table 4**</caption><thead><tr><th rowspan="2">Metric</th><th colspan="2">Based on original</th><th colspan="2">Based on LLM</th></tr><tr><th>examples</th><th>zero-shot</th><th>examples</th><th>zero-shot</th></tr></thead><tbody><tr><td>mean sim</td><td>83.22%</td><td>82.58%</td><td>83.33%</td><td>82.49%</td></tr><tr><td>max sim</td><td>85.30%</td><td>85.05%</td><td>85.46%</td><td>84.05%</td></tr></tbody></table>

The mean and maximum similarity are calculated after matching the arguments based on the original and those generated by the LLM, both with and without examples.

Recall that our aim is to create a pipeline which generates annotations on legal texts. Hence, precision is of high importance and a pipeline which just creates an excessive amount of arguments, which however are not relevant, cannot be used. Therefore, we still believe that the pipeline using examples, and extracting arguments with a higher precision, is the best one to go forward with. Hence, the following sections are based on the setting with examples.

## 4.2. Improving the Generation of Annotations

Aiming to increase quality of the generated annotations, we adopt an iterative prompt modification approach, targeting different weaknesses of a pipeline. For example, to increase the ratio of matched original arguments, it is important to understand the reasons why a specific argument of the original annotations cannot be identified by the LLM. Therefore, as a first iteration step, we did a manual evaluation of all non-matched original arguments and identified the following recurring characteristics:

1. The LLM does not identify arguments of complex structures. It is good at identifying arguments where the premises indicates in an explicit manner the support of a claim, but it cannot capture situations where this support is less evident.

2. LLM arguments are often too short, e.g. composed of only a claim and lacking premises completely.

3. The LLM missed several arguments where references were made to prior cases or legal articles.

We changed the LLM prompt accordingly and added the following instructions:

1. We provided specific structures and syntax arguments can have and explained the respective relations between premises and claims. For this iteration, we used three schemes that are widely discussed in literature, e.g. argument schemes around position to know, rules and precedent cases which we found relevant for our application [30].

2. We gave specific instructions about the requirement of premises, and indicated a preference for longer arguments.

3. We explained the role of prior cases and references to articles, and gave specific instructions to use them to form arguments.

After modifying the prompt, we ran the pipeline again on Llama [24] and re-evaluated the match ratio. Let us illustrate the impact of this change on Pipeline 1: *Direct from text*: we could observe that it increases for the values of the original arguments (cf. Table 5 columns *before* and *after*), which is an indication for a positive impact of our modifications to match original arguments.

This can be explained by the fact that our first iteration of prompt modifications was targeting unmatched original arguments, and unmatched LLM arguments were not taken into account when altering the prompt. In the following prompt improvement iterations, we should thus consider unmatched LLM arguments to increase precision.

**Long text** Similarly to the shorter texts, we ran the pipeline on Llama [24] with the initial and modified prompt for long texts. As we can see in Table 5, the mean matching ratio looks generally better for long texts, than for shorter texts. It must be noted that the aforementioned matching ratio on LLM arguments is a lot worse for the long text inputs. This means, that the LLM generated a lot more arguments than annotated in the original dataset, and while many original arguments can be matched, the majority of LLM arguments does not appear in the original annotations. This is another motivation to look at matching ratios from both perspectives.
**Limitation** Figure 2 details the results for each text. Improvement cannot always be guaranteed: when comparing values *before* with values *after*, the similarity of arguments decreased for some texts (namely texts 00, 13 and 20). It requires a more fine-grained analysis during the prompt improvement iteration. Moreover, the modifications did not lead to an equal increase in the precision score of LLM extracted arguments, where the ratio stayed rather stable as presented in Table 5 columns *before* and *after*.

<table><caption>**Table 5**</caption><thead><tr><th>Based on</th><th>zero-shot</th><th>before</th><th>after</th><th>long-texts before</th><th>long-texts after</th><th>Saul</th></tr></thead><tbody><tr><td>Original</td><td>70.69%</td><td>64.85%</td><td>76.40%</td><td>70.79%</td><td>84.81%</td><td>71.04%</td></tr><tr><td>LLM</td><td>55.16%</td><td>62.44%</td><td>61.92%</td><td>41.01%</td><td>46.47%</td><td>72.06%</td></tr></tbody></table>

Mean matching ratio 0.75 for pipeline 1. We show the value for all prompt modifications. The column *Base on* refers to the base data set for calculating similarity.

**Towards an interactive annotation** We believe that this iterative prompt improvement is a really interesting perspective to understand strengths and weaknesses of our pipeline. In a real system, these feedback loops could be part of the pipeline, allowing for output improvement through human-computer interaction. Alternatively, the interaction could also be automatized and completed by multiple LLM-agents.

**Considering the structure of the arguments** We computed the granular similarity for the structured arguments, as can be seen in Figure 2 in the graph on the right. For all the documents, the similarity decreased significantly. In a detailed analysis, we identified four main reasons for this:

* The Argumentative Discourse Units (ADUs) are the same, but they are not ordered. The claim is a premise, and one premise is the claim. The rest of the premises are also correct premises.

* The same ADUs, but with more information. The premises found by the LLM are in the original argument, but the claim did not appear.

* The ADUs are disorganized and missing information. The claim found is a premise, and one premise is the original claim, but the LLM did not find some premises.

* The ADUs are partially found. The claim was not found, and the premises coincide partially.

We list the cases in order of difficulty, with the easiest ones at the top. We prefer to have the entire argument, even if it is somewhat disorganised. This makes it easier for a lawyer to organise the argument than to search for missing information. These points could be interesting to take into account for Pipeline 2.

## 4.3. Comparison with Other Approaches

To understand the impact of the LLM model on results, we ran our pipeline with another LLM: *Equall/Saul-7B-Instruct-v1* [25].

As presented in Figure 2, the orange bar indicates the results of the reworked prompt. As can be seen, the performance of Saul is lower than that of Llama. This is not surprising because Saul is smaller than Llama. The advantage is that Saul produces fewer arguments, as can be seen in Table 5.

We also evaluated a state-of-the-art argument mining approach (namely ArgueMapper/ArgueBuf [28, 29]) to identify arguments in our dataset. The pipeline is designed to transform texts into structured arguments and comprises stages to extract linguistic features, classify premises and claims, and associate premises with their corresponding claim. It works with classic NLP (Natural Language Processing) and ML (Machine Learning) algorithms. We trained the model on all texts from the dataset except for the 9 short texts for which we generated annotations and see that the model outperforms the LLM approaches.
For this algorithm, we achieved better results in terms of both similarity and granular similarity for the documents: 06, 13 and 16. For the rest of the documents, the performance is similar to that of Llama. Regarding granular similarity, there are cases where the algorithm only finds a claim without premises.

Although the results look promising, we must again mention our goal of annotating data from scratch, with a versatile, modular, and independent pipeline. Approaches like these need already annotated data to function properly, which can be a drawback for our use case, despite the high performance. Moreover, given that train and test data belong to the same dataset, we would also have to test generalizability of the model on other datasets.

<figure><img src="image_5.png" /><figcaption>**Figure 2:** The mean and granular similarities are presented. The first nine texts are the shortest texts, compared with other approaches, and the last six are a generalization on Llama for long texts.</figcaption></figure>

# 5. Conclusion

The emergence of LLMs has introduced new possibilities to the field of computational argumentation. However, there are still challenges to overcome, such as the limited availability of data, the difficulty of finding experienced evaluators, and the need for task-specific annotations, which are often time-consuming and expensive to produce. To address these issues, we explored the potential of LLMs for creating annotations.

We propose a pipeline based on the ECHR dataset and use prompting with examples to extract arguments from the original texts. We then compare the LLM-extracted arguments to the ground truth based on vector similarity of arguments and use the matching ratio, i.e. the number of original arguments that were identified by the LLM, as performance metric. We observe that the ratio fluctuates significantly between texts, but reaches a mean score of more than 64% for the initial data input and prompt. After prompt improvement, this ratio increases by more than 11 percentage points to over 76%. This project creates a baseline which we believe can be significantly improved by extending the pipeline. For example, we can think of an automated prompt improvement process where LLM-agents challenge and evaluate each other aiming at reaching as much similarity with the ground truth as possible.

For future work, the first step we want to explore is evaluating the efficiency of our pipeline as a tool for legal experts to annotate legal texts. We want to compare the results of two groups of annotators: one that uses the pipeline and another that does not. This comparison will reveal the capabilities of LLMs for annotation tasks, and assess the time saved for the annotation by using the pipeline. In a later step we aim to evaluate the versatility of our pipeline beyond the ECHR dataset, increase its modularity, and generalise it to different tasks. This will make it useful for real-life applications.
References

[1] P. M. Dung, On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games, Artif. Intell. 77 (1995) 321–358. doi:10.1016/0004-3702(94)00041-X.

[2] P. Baroni, D. Gabbay, M. Giacomin, L. van der Torre (Eds.), Handbook of Formal Argumentation, College Publications, 2018. URL: https://www.collegepublications.co.uk/handbooks/?00003.

[3] K. Atkinson, T. J. M. Bench-Capon, ANGELIC II: an improved methodology for representing legal domain knowledge, in: Proc. of ICAIL 2023, 2023, pp. 12–21. doi:10.1145/3594536.3595137.

[4] J. Collenette, K. Atkinson, T. J. M. Bench-Capon, Explainable AI tools for legal reasoning about cases: A study on the european court of human rights, Artif. Intell. 317 (2023) 103861. doi:10.1016/J.ARTINT.2023.103861.

[5] E. Cabrio, S. Villata, Five years of argument mining: a data-driven analysis, in: Proc. of IJCAI 2018, 2018, pp. 5427–5433. doi:10.24963/IJCAI.2018/766.

[6] J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, R. McHardy, Challenges and applications of large language models, CoRR abs/2307.10169 (2023). doi:10.48550/ARXIV.2307.10169.

[7] M. A. K. Raiaan, M. S. H. Mukta, K. Fatema, N. M. Fahad, S. Sakib, M. M. J. Mim, J. Ahmad, M. E. Ali, S. Azam, A review on large language models: Architectures, applications, taxonomies, open issues and challenges, IEEE Access 12 (2024) 26839–26874. doi:10.1109/ACCESS.2024.3365742.

[8] I. Chalkidis, M. Fergadiotis, D. Tsarapatsanis, N. Aletras, I. Androutsopoulos, P. Malakasiotis, Paragraph-level rationale extraction through regularization: A case study on european court of human rights cases (2021) 226–241. doi:10.18653/V1/2021.NAACL-MAIN.22.

[9] A. De Wynter, T. Yuan, “I’d like to have an argument, please”: Argumentative reasoning in large language models, in: Proc. of COMMA, 2024, pp. 73–84.

[10] I. Habernal, D. Faber, N. Recchia, S. Bretthauer, I. Gurevych, I. Spiecker genannt Döhmann, C. Burchard, Mining legal arguments in court decisions, Artif. Intell. Law 32 (2024) 1–38. doi:10.1007/S10506-023-09361-Y.

[11] M. A. Gray, J. Savelka, W. M. Oliver, K. D. Ashley, Can GPT alleviate the burden of annotation?, in: Proc. of JURIX 2023, 2023, pp. 157–166. doi:10.3233/FAIA230961.

[12] Y. Otiefy, A. Alhamzeh, Exploring large language models in financial argument relation identification, in: Proc. of FinNLP-KDF-ECONLP 2024, 2024, pp. 119–129. URL: https://aclanthology.org/2024.finnlp-1.12/.

[13] T. Bench-Capon, K. Atkinson, Using argumentation schemes to model legal reasoning, arXiv preprint arXiv:2210.00315 (2022).

[14] E. Faugier, F. Armetta, A. Bonifati, B. Yun, Assisted debate builder with large language models, in: Proc. of ECAI 2024, 2024, pp. 4447–4450. doi:10.3233/FAIA241026.

[15] G. Trajano, D. C. Engelmann, R. H. Bordini, S. Sarkadi, J. Mumford, A. R. Panisson, Translating natural language arguments to computational arguments using LLMs, in: Proc. of COMMA 2024, 2024, pp. 289–300. doi:10.3233/FAIA240329.

[16] A. Lindahl, L. Borin, Annotation for computational argumentation analysis: Issues and perspectives, Lang. Linguistics Compass 18 (2024). doi:10.1111/LNC3.12505.

[17] M. Pietron, R. Olszowski, J. Gomulka, Efficient argument classification with compact language models and ChatGPT-4 refinements, in: Proc. of ICCCI 2024, 2024, pp. 249–262. doi:10.1007/978-3-031-70816-9_20.

[18] J. Savelka, K. D. Ashley, The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts, Frontiers Artif. Intell. 6 (2023). doi:10.3389/FRAI.2023.1279794.

[19] A. A. Zubaer, M. Granitzer, J. Mitrovic, Performance analysis of large language models in the domain of legal argument mining, Frontiers Artif. Intell. 6 (2023). doi:10.3389/FRAI.2023.1278796.

[20] D. Gorur, A. Rago, F. Toni, Can large language models perform relation-based argument mining?, in: Proc. of COLING 2025, 2025, pp. 8518–8534. URL: https://aclanthology.org/2025.coling-main.569/.

[21] S. Hong, L. Xiao, X. Zhang, J. Chen, ArgMed-agents: Explainable clinical decision reasoning
with llm dissclusion via argumentation schemes, 2024. URL: https://arxiv.org/abs/2403.06294.
arXiv:2403.06294.

[22] J. Mumford, K. Atkinson, T. J. M. Bench-Capon, Annotated insights into legal reasoning: A dataset of article 6 ECHR cases, Argument Comput. 15 (2024) 113–119. doi:10.3233/AAC-240002.

[23] P. Poudyal, J. Savelka, A. Ieven, M. F. Moens, T. Goncalves, P. Quaresma, ECHR: Legal corpus for argument mining, in: Proceedings of the 7th Workshop on Argument Mining, 2020, pp. 67–75. URL: https://aclanthology.org/2020.argmining-1.8/.

[24] AI@Meta, Llama 3 model card, 2024. URL: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.

[25] P. Colombo, T. P. Pires, M. Boudiaf, D. Culver, R. Melo, C. Corro, A. F. T. Martins, F. Esposito, V. L. Raposo, S. Morgado, M. Desa, Saullm-7b: A pioneering large language model for law, 2024. URL: https://arxiv.org/abs/2403.03883.

[26] Y. Bai, J. Zhang, X. Lv, L. Zheng, S. Zhu, L. Hou, Y. Dong, J. Tang, J. Li, Longwriter: Unleashing 10,000+ word generation from long context llms, in: Proc. of ICLR 2025, 2025. URL: https://openreview.net/forum?id=kQ5s9Yh0WI.

[27] L. Amgoud, V. David, Measuring similarity between logical arguments, in: Proc. of KR 2018, 2018, pp. 98–107. URL: https://aaai.org/ocs/index.php/KR/KR18/paper/view/17999.

[28] M. Lenz, P. Sahitaj, S. Kallenberg, C. Coors, L. Dumani, R. Schenkel, R. Bergmann, Towards an argument mining pipeline transforming texts to argument graphs, in: Proc. of COMMA 2020, 2020, pp. 263–270. doi:10.3233/FAIA200510.

[29] M. Lenz, R. Bergmann, User-centric argument mining with ArgueMapper and Arguebuf, in: Proc. of COMMA 2022, 2022, pp. 367–368. doi:10.3233/FAIA220176.

[30] T. F. Gordon, D. Walton, Legal reasoning with argumentation schemes, in: Proc. of ICAIL 2009, ACM, 2009, pp. 137–146. doi:10.1145/1568234.1568250.

## A. Defining the value of *t*.

To define the value of *t*, we did a manual evaluation of all possible combinations between original arguments and LLM generated arguments of the nine selected articles, and classified them as either *acceptable* or *not acceptable*, without knowing their vector similarity. We obtained the results presented in Table 6. A threshold below 0.6 leads to mostly wrong matches, while matches above 0.85 are almost exclusively correct. For this project, we proceeded with a *t*-value of 0.75, where the majority of matches can be considered acceptable.

Figure 3 and Figure 4 show examples of matches between LLM generated arguments and original arguments from the dataset, with a similarity of 98.4% and 75.1% respectively. The first LLM argument is an almost complete replication of the original, with only one subordinate clause (underlined) missing. The second argument match, which is just above the threshold, shows how the LLM is able to capture the relevant information (exhaustion of domestic remedies, reference to article 26 of the Convention, lodging of the application within a certain time frame) but is not able to replicate the wording and setup of the original argument.

<table><caption>**Table 6**
Argument Evaluation per Threshold</caption><thead><tr><td>t</td><td>acceptable</td><td>not acceptable</td><td>ratio</td></tr></thead><tbody><tr><td>0.60</td><td>29</td><td>97</td><td>23%</td></tr><tr><td>0.65</td><td>26</td><td>56</td><td>32%</td></tr><tr><td>0.70</td><td>26</td><td>63</td><td>29%</td></tr><tr><td>0.75</td><td>32</td><td>30</td><td>52%</td></tr><tr><td>0.80</td><td>37</td><td>22</td><td>63%</td></tr><tr><td>0.85</td><td>24</td><td>3</td><td>89%</td></tr><tr><td>0.90</td><td>12</td><td>1</td><td>92%</td></tr></tbody></table>
<figure><img src="image_6.png" /><figcaption>**Figure 3:** Original (3a) and LLM generated (3b) arguments with a similarity of 98.4%. Differences are underlined.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 4:** Original (4a) and LLM generated (4b) arguments with a similarity of 75.1%. Matches of relevant information are underlined.</figcaption></figure>

## B. Defining the value of α.

To choose the value of α, we compute the granular similarity based on the original arguments with the improved prompt on Llama for α ∈ {0.3, 0.4, 0.5, 0.6, 0.7}. Table 7 shows that the highest granular similarity is obtained for α = 0.7 in most cases, indicating that premises contribute more information to the argument.

<table><caption>**Table 7**</caption><thead><tr><th>text</th><th>0.3</th><th>0.4</th><th>0.5</th><th>0.6</th><th>0.7</th></tr></thead><tbody><tr><td>00</td><td>52.68</td><td>54.32</td><td>55.97</td><td>57.61</td><td><strong>59.26</strong></td></tr><tr><td>04</td><td><strong>63.4</strong></td><td>63.23</td><td>63.07</td><td>62.9</td><td>62.74</td></tr><tr><td>05</td><td>48.17</td><td>49.55</td><td>50.93</td><td>52.32</td><td><strong>53.7</strong></td></tr><tr><td>06</td><td>49.91</td><td>50.45</td><td>51</td><td>51.55</td><td><strong>52.09</strong></td></tr><tr><td>10</td><td>55.6</td><td>57.91</td><td>60.22</td><td>62.53</td><td><strong>64.85</strong></td></tr><tr><td>13</td><td>55.29</td><td>57.74</td><td>60.18</td><td>62.63</td><td><strong>65.07</strong></td></tr><tr><td>16</td><td>63.15</td><td>66.09</td><td>69.03</td><td>71.97</td><td><strong>74.91</strong></td></tr><tr><td>20</td><td>47.04</td><td>49.78</td><td>52.52</td><td>55.26</td><td><strong>58</strong></td></tr><tr><td>21</td><td>61.23</td><td>62.29</td><td>63.36</td><td>64.42</td><td><strong>65.48</strong></td></tr></tbody></table>

Mean granular similarity for different values of α. Boldface correspond to the higher mean similarity, for each text.