<figure><img src="image_1.png" /></figure>

HAL
open science

# Extracting Proceedings Data from Court Cases with Machine Learning

Bruno Mathis

<figure><img src="image_2.png" /></figure>

## To cite this version:

Bruno Mathis. Extracting Proceedings Data from Court Cases with Machine Learning. Stats, 2022, 5 (4), pp.1305-1320. 10.3390/stats5040079. hal-04526289

HAL Id: hal-04526289

https://hal.science/hal-04526289

Submitted on 21 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_1.png" /></figure>

HAL
open science

# Extracting Proceedings Data from Court Cases with Machine Learning

Bruno Mathis

<figure><img src="image_2.png" /></figure>

## To cite this version:

Bruno Mathis. Extracting Proceedings Data from Court Cases with Machine Learning. Stats, 2022, 5 (4), pp.1305-1320. 10.3390/stats5040079. hal-04526289

HAL Id: hal-04526289

https://hal.science/hal-04526289v1

Submitted on 21 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_4.png" /></figure>

Article

# Extracting Proceedings Data from Court Cases with Machine Learning

Bruno Mathis<fnref n="1" />,<fnref n="2" />*

**Abstract:** France is rolling out an open data program for all court cases, but with few metadata attached. Reusers will have to use named-entity recognition (NER) within the text body of the case to extract any value from it. Any court case may include up to 26 variables, or labels, that are related to the proceeding, regardless of the case substance. These labels are from different syntactic types: some of them are rare; others are ubiquitous. This experiment compares different algorithms, namely CRF, SpaCy, Flair and DeLFT, to extract proceedings data and uses the learning model assessment capabilities of Kairntech, an NLP platform. It shows that an NER model can apply to this large and diverse set of labels and extract data of high quality. We achieved an 87.5% F1 measure with Flair trained on more than 27,000 manual annotations. Quality may yet be improved by combining NER models by data type.

**Keywords:** machine learning; named-entity recognition; information extraction; judicial data; civil procedur

<figure><img src="image_5.png" /></figure>

check for
updates

**Citation:** Mathis, B. Extracting Proceedings Data from Court Cases with Machine Learning. *Stats* 2022, 5, 1305–1320. https://doi.org/10.3390/stats5040079

Academic Editor: Stéphane Mussard

Received: 5 November 2022

Accepted: 5 December 2022

Published: 13 December 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_6.png" /></figure>

**Copyright:** © 2022 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

# 1. Introduction

## 1.1. Context and Objective

France produces between three million and four million court cases a year. A small number of them are supplied with metadata. Appellate cases often include metadata at the beginning of the text but less frequently than supreme court cases do. First-instance cases mostly do not.

Fortunately, French law prescribes that the text of any case indicates:

- The jurisdiction (tribunal or court of appeal) that rendered the case;

- The name of the judges who deliberated on it;

- Its date;

- The name of the representative of the public prosecutor, if any;

- The name of the clerk;

- The first name or denomination of the parties and their domicile or registered office;

- If applicable, the names of the lawyers or of any person who represented or assisted the parties;

- In noncontentious matters, the name of the persons who must be notified.

These data make up the metadata of the case and often show up in the header of the text. The law also says that the judgment must be motivated. It sets out the case in the form of an operative part. The French legislator has decided to have all court cases made available to the public as open data. Though some provisions restrict the content of cases that is eligible to open data, the scope of the final law is unprecedented among developed countries. A first version has been rolled out in September 2021, and the last one is scheduled in December 2025. This project opens up many interesting reuse opportunities.

Predictive justice is the most hyped type of application. Indeed, using machine learning for court cases may help predict the outcome of upcoming cases and help attorneys

<footnote n="1">¹ CHROME Laboratory, Nimes University, 5 Rue du Docteur Georges Salan CS 13019, 30021 Nîmes, France; brunomathis16@gmail.com</footnote>

<footnote n="2">² European Centre of Law & Economics of ESSEC Business School, 3 Av. Bernard Hirsch, 95000 Cergy, France</footnote>
<figure><img src="image_4.png" /></figure>

Article

# Extracting Proceedings Data from Court Cases with Machine Learning

Bruno Mathis<fnref n="1" />,<fnref n="2" />*

**Abstract:** France is rolling out an open data program for all court cases, but with few metadata attached. Reusers will have to use named-entity recognition (NER) within the text body of the case to extract any value from it. Any court case may include up to 26 variables, or labels, that are related to the proceeding, regardless of the case substance. These labels are from different syntactic types: some of them are rare; others are ubiquitous. This experiment compares different algorithms, namely CRF, SpaCy, Flair and DeLFT, to extract proceedings data and uses the learning model assessment capabilities of Kairntech, an NLP platform. It shows that an NER model can apply to this large and diverse set of labels and extract data of high quality. We achieved an 87.5% F1 measure with Flair trained on more than 27,000 manual annotations. Quality may yet be improved by combining NER models by data type.

**Keywords:** machine learning; named-entity recognition; information extraction; judicial data; civil procedur

<figure><img src="image_5.png" /></figure>

check for
updates

**Citation:** Mathis, B. Extracting Proceedings Data from Court Cases with Machine Learning. *Stats* 2022, 5, 1305–1320. https://doi.org/10.3390/stats5040079

Academic Editor: Stéphane Mussard

Received: 5 November 2022

Accepted: 5 December 2022

Published: 13 December 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_6.png" /></figure>

**Copyright:** © 2022 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

# 1. Introduction

## 1.1. Context and Objective

France produces between three million and four million court cases a year. A small number of them are supplied with metadata. Appellate cases often include metadata at the beginning of the text but less frequently than supreme court cases do. First-instance cases mostly do not.

Fortunately, French law prescribes that the text of any case indicates:

- The jurisdiction (tribunal or court of appeal) that rendered the case;

- The name of the judges who deliberated on it;

- Its date;

- The name of the representative of the public prosecutor, if any;

- The name of the clerk;

- The first name or denomination of the parties and their domicile or registered office;

- If applicable, the names of the lawyers or of any person who represented or assisted the parties;

- In noncontentious matters, the name of the persons who must be notified.

These data make up the metadata of the case and often show up in the header of the text. The law also says that the judgment must be motivated. It sets out the case in the form of an operative part. The French legislator has decided to have all court cases made available to the public as open data. Though some provisions restrict the content of cases that is eligible to open data, the scope of the final law is unprecedented among developed countries. A first version has been rolled out in September 2021, and the last one is scheduled in December 2025. This project opens up many interesting reuse opportunities.

Predictive justice is the most hyped type of application. Indeed, using machine learning for court cases may help predict the outcome of upcoming cases and help attorneys

<footnote n="1">¹ CHROME Laboratory, Nimes University, 5 Rue du Docteur Georges Salan CS 13019, 30021 Nîmes, France; brunomathis16@gmail.com</footnote>

<footnote n="2">² European Centre of Law & Economics of ESSEC Business School, 3 Av. Bernard Hirsch, 95000 Cergy, France</footnote>
advise their customers accordingly. However, using machine learning for court cases may also serve general-interest applications, which have been little talked about until now. This research seeks to pave the way for the design of a database of proceedings data that could be used by social science researchers and legal technology startups ('legaltech'). One possible use case of such a database is to statistically compare different judicial pathways: cases not followed by an appeal, cases followed by an appeal, judgments followed by an appeal in Cassation or by a referral to another appellate court, etc. These different pathways could be compared in terms of case numbers and average duration, according to jurisdictions, areas of law and years of production.

The objective of this research is to show that named-entity recognition (NER) can be applied not only to the traditional named entities (people, places, organizations) but also to the whole set of data of different kinds that are necessary to describe the proceedings of the case, regardless of its substance.

## 1.2. Related Work

The first experimentation of using machine learning on case law dates back to 2016. Researchers investigated how well the machine could reach the same case as the judges of the European Court of Human Rights (ECHR) on the basis of the facts and the legal proceedings of a dispute [1]. They extracted from Hudoc, ECHR's database, all the judgments of the court, in their English version, in which the judges were asked to rule on Articles 3, 6 and 8 of the European Convention on Human Rights, and divided them to as many datasets. They masked the operative part, then manually annotated in a separate file the expected result of the test with the tag "article violation" or "no violation of the article". They used support vector machine (SVM), a linear model with features based on groups of words and topics to represent legal textual information extracted from the cases. This training led to the same outcome as judges in 79% of cases.

This landmark experience was generalized, in 2019, to all the articles of the European Convention on Human Rights, again using the SVM algorithm [2]. With approximately the same corpus, a comparison between several neural models showed they delivered higher quality than SVM [3]. Again on an ECHR corpus, but with only 30 cases, a comparison between different extraction mechanisms of event dates demonstrated that transformer models achieved the best result and that dates of procedural acts were easier to extract than other dates relative to the circumstances of the case [4]. Where a court database was previously applied a rules-based program to generate metadata, an opportunity arose to conduct training on a large dataset, because no preliminary classification was required. This is what was done for 28,000 cases of the Supreme Court of the US (SCOTUS) [5] and for 130,000 ones of the French Court of Cassation [6]. The aim of the former research was to reproduce the verdict, which it successfully did in 70% of cases, and that of the latter was also to reproduce the subject of law and to estimate the time frame during which the case had been rendered.

Google's 2018 release of its BERT model [7], based on a broad corpus, was followed by sectoral variants. The so-called LEGAL-BERT model [8], shows that pretraining from scratch on legal content is more efficient than pretraining only on the original BERT base supplemented with legal content.

Most experiments on court cases have been carried out on datasets from supreme courts. These have some points in common. The text of the judgments is rather long and well structured: an SVM could provide good results because the structure of the text in different sections (basically: circumstances, facts, procedure, law, outcome) allowed for segregating groups of words by section. The description of the facts and the procedure is exhaustive. The verdict has a binary character (violation or non-violation). The dataset is large and presumably statistically representative. However, because supreme courts rule on the correct application of the law, not on the merits of cases, such experiments can hardly be extrapolated to lower jurisdictions. These experiments use classification, not NER.
advise their customers accordingly. However, using machine learning for court cases may also serve general-interest applications, which have been little talked about until now. This research seeks to pave the way for the design of a database of proceedings data that could be used by social science researchers and legal technology startups ('legaltech'). One possible use case of such a database is to statistically compare different judicial pathways: cases not followed by an appeal, cases followed by an appeal, judgments followed by an appeal in Cassation or by a referral to another appellate court, etc. These different pathways could be compared in terms of case numbers and average duration, according to jurisdictions, areas of law and years of production.

The objective of this research is to show that named-entity recognition (NER) can be applied not only to the traditional named entities (people, places, organizations) but also to the whole set of data of different kinds that are necessary to describe the proceedings of the case, regardless of its substance.

## 1.2. Related Work

The first experimentation of using machine learning on case law dates back to 2016. Researchers investigated how well the machine could reach the same case as the judges of the European Court of Human Rights (ECHR) on the basis of the facts and the legal proceedings of a dispute [1]. They extracted from Hudoc, ECHR's database, all the judgments of the court, in their English version, in which the judges were asked to rule on Articles 3, 6 and 8 of the European Convention on Human Rights, and divided them to as many datasets. They masked the operative part, then manually annotated in a separate file the expected result of the test with the tag "article violation" or "no violation of the article". They used support vector machine (SVM), a linear model with features based on groups of words and topics to represent legal textual information extracted from the cases. This training led to the same outcome as judges in 79% of cases.

This landmark experience was generalized, in 2019, to all the articles of the European Convention on Human Rights, again using the SVM algorithm [2]. With approximately the same corpus, a comparison between several neural models showed they delivered higher quality than SVM [3]. Again on an ECHR corpus, but with only 30 cases, a comparison between different extraction mechanisms of event dates demonstrated that transformer models achieved the best result and that dates of procedural acts were easier to extract than other dates relative to the circumstances of the case [4]. Where a court database was previously applied a rules-based program to generate metadata, an opportunity arose to conduct training on a large dataset, because no preliminary classification was required. This is what was done for 28,000 cases of the Supreme Court of the US (SCOTUS) [5] and for 130,000 ones of the French Court of Cassation [6]. The aim of the former research was to reproduce the verdict, which it successfully did in 70% of cases, and that of the latter was also to reproduce the subject of law and to estimate the time frame during which the case had been rendered.

Google's 2018 release of its BERT model [7], based on a broad corpus, was followed by sectoral variants. The so-called LEGAL-BERT model [8], shows that pretraining from scratch on legal content is more efficient than pretraining only on the original BERT base supplemented with legal content.

Most experiments on court cases have been carried out on datasets from supreme courts. These have some points in common. The text of the judgments is rather long and well structured: an SVM could provide good results because the structure of the text in different sections (basically: circumstances, facts, procedure, law, outcome) allowed for segregating groups of words by section. The description of the facts and the procedure is exhaustive. The verdict has a binary character (violation or non-violation). The dataset is large and presumably statistically representative. However, because supreme courts rule on the correct application of the law, not on the merits of cases, such experiments can hardly be extrapolated to lower jurisdictions. These experiments use classification, not NER.
In the legal domain, NER can apply not only to names of people, places or organizations but also to names of laws and procedures or to concepts [9]. It can also apply to amounts: a Brazilian team extracted information from cases of the Appellate State Court of Rio de Janeiro (ASCRJ) cases confirming or modifying compensation for moral damage [10]. Amounts were annotated with one out of three mutually exclusive labels, representing either an increase, a decrease or a confirmation of the damage amount granted by the first-instance judge. A more recent NER-based study focused on extracting catchphrases from Indian case law [11].

More research studies have focused on legal and regulatory texts rather than on case law and on the English language rather than on French. One study, applied to court cases, did attempt to generalize extraction to all possible judicial concepts, including facts and pretentions of the parties [12]. This is the first working of this kind on French-language case law. The French Court of Cassation launched a project to pseudonymize court cases [13]. They annotated a dataset with four labels, namely natural person, legal person, address and date of birth, and had these annotations trained by a neuronal network on the basis of a bidirectional long-short term memory (LSTM) algorithm [14]. CamemBERT [15] is the first general-purpose French linguistic model, and JuriBERT [16] is the first one dedicated to the French legal domain.

## 2. Methodology

### 2.1. Approach

Within the NLP research flow specializing on the legal domain, this work positions itself among the downstream tasks (Table 1). Like [10,13], it uses NER to address a real-life issue, but with a greater number of labels, of different data types and frequencies: where [10] dealt with labels of roughly balanced annotation numbers, we deal with some labels expected to annotate a few word sequences and other ones expected to annotate many such sequences. Another difference is that this study has involved a significant annotation effort: 1706 cases have been subject to 27,703 manual annotations.

<table><caption>**Table 1.** Studies on court case data extraction.</caption><thead><tr><th>Year</th><th>Research</th><th>Model</th><th>Language</th><th>Pretrained Corpus</th><th>Task</th><th>Dataset Origin</th><th>Cases</th><th>Labels</th></tr></thead><tbody><tr><td>2016</td><td>Aletras</td><td>SVM</td><td>English</td><td></td><td>classification</td><td>ECHR</td><td>584</td><td>2</td></tr><tr><td>2017</td><td>Katz</td><td>Random Forest</td><td>English</td><td></td><td>classification</td><td>SCOTUS</td><td>28,000</td><td>3</td></tr><tr><td>2018</td><td>Sulea</td><td>SVM</td><td>French</td><td></td><td>classification</td><td>Cassation</td><td>126,865</td><td>3</td></tr><tr><td>2019</td><td>Devlin</td><td>BERT</td><td>English</td><td>general</td><td>classification</td><td>ECHR</td><td>1942</td><td>2</td></tr><tr><td>2019</td><td>Medvedeva</td><td>SVM</td><td>English</td><td></td><td>classification</td><td>ECHR</td><td>11,500</td><td>66</td></tr><tr><td>2019</td><td>Chalkidis</td><td>BERT</td><td>English</td><td>legal</td><td>classification</td><td>NER</td><td>3,022</td><td>6</td></tr><tr><td>2019</td><td>Fernandes</td><td>BiLSTM-CRF</td><td>Portuguese</td><td></td><td>NER/classification</td><td>ECHR</td><td>30</td><td></td></tr><tr><td>2020</td><td>Filtz</td><td></td><td>German</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>2020</td><td>Martin</td><td>CamemBERT</td><td>French</td><td>general</td><td></td><td></td><td></td><td></td></tr><tr><td>2020</td><td>Chalkidis</td><td>Legal-BERT</td><td>English</td><td>legal</td><td></td><td></td><td></td><td></td></tr><tr><td>2020</td><td>Ngompe</td><td>CRF</td><td>French</td><td></td><td>NER</td><td>ECHR</td><td>503</td><td>11</td></tr><tr><td>2021</td><td>Mathis</td><td>Flair</td><td>French</td><td></td><td>NER</td><td>3-degrees mix</td><td>1700</td><td>26</td></tr><tr><td>2022</td><td>Douka</td><td>JuriBERT</td><td>French</td><td>legal</td><td>NER</td><td>Cassation</td><td>123,361</td><td>151</td></tr></tbody></table>

Proceedings data are common to different areas of law (or types of litigation). There is basically one set of proceedings data for all civil matters and another one for criminal matters. Criminal court cases are subject to more publicity restrictions than civil ones, are available to the public in smaller numbers and will come last in the French government's open data program schedule. That is why this research is limited to civil cases.

We built up a dataset that, though not pretending to be representative of all law areas, focuses on three different cases: road accidents, commercial leases and banking or finance-related litigation. The underlying assumption is that if the quality of extraction is about the same among these three areas of law, then it is likely other ones will provide similar quality.
In the legal domain, NER can apply not only to names of people, places or organizations but also to names of laws and procedures or to concepts [9]. It can also apply to amounts: a Brazilian team extracted information from cases of the Appellate State Court of Rio de Janeiro (ASCRJ) cases confirming or modifying compensation for moral damage [10]. Amounts were annotated with one out of three mutually exclusive labels, representing either an increase, a decrease or a confirmation of the damage amount granted by the first-instance judge. A more recent NER-based study focused on extracting catchphrases from Indian case law [11].

More research studies have focused on legal and regulatory texts rather than on case law and on the English language rather than on French. One study, applied to court cases, did attempt to generalize extraction to all possible judicial concepts, including facts and pretentions of the parties [12]. This is the first working of this kind on French-language case law. The French Court of Cassation launched a project to pseudonymize court cases [13]. They annotated a dataset with four labels, namely natural person, legal person, address and date of birth, and had these annotations trained by a neuronal network on the basis of a bidirectional long-short term memory (LSTM) algorithm [14]. CamemBERT [15] is the first general-purpose French linguistic model, and JuriBERT [16] is the first one dedicated to the French legal domain.

## 2. Methodology

### 2.1. Approach

Within the NLP research flow specializing on the legal domain, this work positions itself among the downstream tasks (Table 1). Like [10,13], it uses NER to address a real-life issue, but with a greater number of labels, of different data types and frequencies: where [10] dealt with labels of roughly balanced annotation numbers, we deal with some labels expected to annotate a few word sequences and other ones expected to annotate many such sequences. Another difference is that this study has involved a significant annotation effort: 1706 cases have been subject to 27,703 manual annotations.

<table><caption>**Table 1.** Studies on court case data extraction.</caption><thead><tr><th>Year</th><th>Research</th><th>Model</th><th>Language</th><th>Pretrained Corpus</th><th>Task</th><th>Dataset Origin</th><th>Cases</th><th>Labels</th></tr></thead><tbody><tr><td>2016</td><td>Aletras</td><td>SVM</td><td>English</td><td></td><td>classification</td><td>ECHR</td><td>584</td><td>2</td></tr><tr><td>2017</td><td>Katz</td><td>Random Forest</td><td>English</td><td></td><td>classification</td><td>SCOTUS</td><td>28,000</td><td>3</td></tr><tr><td>2018</td><td>Sulea</td><td>SVM</td><td>French</td><td></td><td>classification</td><td>Cassation</td><td>126,865</td><td>3</td></tr><tr><td>2019</td><td>Devlin</td><td>BERT</td><td>English</td><td>general</td><td>classification</td><td>ECHR</td><td>1942</td><td>2</td></tr><tr><td>2019</td><td>Medvedeva</td><td>SVM</td><td>English</td><td></td><td>classification</td><td>ECHR</td><td>11,500</td><td>66</td></tr><tr><td>2019</td><td>Chalkidis</td><td>BERT</td><td>English</td><td>legal</td><td>classification</td><td>NER</td><td>3,022</td><td>6</td></tr><tr><td>2019</td><td>Fernandes</td><td>BiLSTM-CRF</td><td>Portuguese</td><td></td><td>NER/classification</td><td>ECHR</td><td>30</td><td></td></tr><tr><td>2020</td><td>Filtz</td><td></td><td>German</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>2020</td><td>Martin</td><td>CamemBERT</td><td>French</td><td>general</td><td></td><td></td><td></td><td></td></tr><tr><td>2020</td><td>Chalkidis</td><td>Legal-BERT</td><td>English</td><td>legal</td><td></td><td></td><td></td><td></td></tr><tr><td>2020</td><td>Ngompe</td><td>CRF</td><td>French</td><td></td><td>NER</td><td>ECHR</td><td>503</td><td>11</td></tr><tr><td>2021</td><td>Mathis</td><td>Flair</td><td>French</td><td></td><td>NER</td><td>3-degrees mix</td><td>1700</td><td>26</td></tr><tr><td>2022</td><td>Douka</td><td>JuriBERT</td><td>French</td><td>legal</td><td>NER</td><td>Cassation</td><td>123,361</td><td>151</td></tr></tbody></table>

Proceedings data are common to different areas of law (or types of litigation). There is basically one set of proceedings data for all civil matters and another one for criminal matters. Criminal court cases are subject to more publicity restrictions than civil ones, are available to the public in smaller numbers and will come last in the French government's open data program schedule. That is why this research is limited to civil cases.

We built up a dataset that, though not pretending to be representative of all law areas, focuses on three different cases: road accidents, commercial leases and banking or finance-related litigation. The underlying assumption is that if the quality of extraction is about the same among these three areas of law, then it is likely other ones will provide similar quality.
The research does not intend to devise a new learning algorithm that would deliver a quality higher than that produced by existing ones. Though it does seek which algorithm, among several ones, applies in what context and delivers the best result for court data, its purpose is more practical: the identification of the algorithm, or the set of algorithms, available off the shelf that provides the highest quality for court data. We will not try to optimize further and will keep the default values of hyperparameters proposed by our tool.

Court case files are dirtier than most classical NER datasets [17]. Some legal editors have conducted preliminary data cleansing on cases originated by jurisdictions. They then used their machine-learning experience on court data by pseudonymizing and structuring them, before trying to carry out some more value-added work. Such tasks are off-putting and time-consuming while bringing few scientific lessons. This research therefore seeks to deliver results in one go, directly from raw texts, despite the rather poor quality of their rendition.

## 2.2. Dataset

For any natural-language processing, a dataset should have sufficient examples to ensure the best balance in the variety of data and language [18]. In our case, the learning of proceedings data should be based on a stock of cases, not only recent cases, and therefore, the learning dataset should be representative of the historically different writing conventions and templates. Our policy was to collect well-distributed cases from the past 15 to 20 years, depending on their level of availability, which varies by jurisdiction. Our dataset is composed of cases from the three jurisdictional degrees and focuses on three law areas (Table 2).

<table><caption>Table 2. Cases breakdown by degree and law area.</caption><thead><tr><th>Cases</th><th>1st Instance</th><th>Appeal</th><th>Final Appeal</th><th>Total</th></tr></thead><tbody><tr><td>Leases</td><td>250</td><td>208</td><td>63</td><td>521</td></tr><tr><td>Accidents</td><td>80</td><td>198</td><td>46</td><td>324</td></tr><tr><td>Bank</td><td>250</td><td>231</td><td>50</td><td>531</td></tr><tr><td>Other</td><td>234</td><td>0</td><td>96</td><td>330</td></tr><tr><td>All</td><td>814</td><td>637</td><td>255</td><td>1706</td></tr></tbody></table>

One first batch of cases, all on civil responsibility matters, comes from three judicial tribunals: 46 from Chambéry, 119 from Saint-Etienne and 149 from Grenoble, for a total of 314. They had been scanned by each tribunal as PDF images, then OCR'd to PDF text by Université Savoie Mont Blanc (USMB) and then converted to TXT files. As these cases have not been pseudonymized, they will not be published in the context of this research.

A second batch was supplied by Doctrine, a French legal technology. It includes 250 cases on banking law and another 250 on commercial leases. They have been pseudonymized with Doctrine's proprietary ML-based algorithm and randomly selected from their database upon our request. In many cases, parties were not only pseudonymized but actually erased, which damages the intelligibility of the text. USMB and Doctrine are warmly thanked for these corpora.

Three datasets of appellate cases have been downloaded from Légifrance, the portal of Direction de l'Information Légale et Administrative (DILA), the French national gazette editor. These cases have been pseudonymized by DILA and been subject to no subsequent data cleaning. They are published for having been "selected according to the methods specific to each order of jurisdiction", so they are not necessarily statistically representative of the procedural paths. The datasets are fairly well geographically distributed among appellate courts. The Court of Cassation, the supreme court and Légifrance each use their own template for the same case, and each one varies over time. We selected supreme court cases with the help of a page of the Court of Cassation's website that has been clustering cases since 2014, by area of law, such as road accidents, commercial leases and banking. We
The research does not intend to devise a new learning algorithm that would deliver a quality higher than that produced by existing ones. Though it does seek which algorithm, among several ones, applies in what context and delivers the best result for court data, its purpose is more practical: the identification of the algorithm, or the set of algorithms, available off the shelf that provides the highest quality for court data. We will not try to optimize further and will keep the default values of hyperparameters proposed by our tool.

Court case files are dirtier than most classical NER datasets [17]. Some legal editors have conducted preliminary data cleansing on cases originated by jurisdictions. They then used their machine-learning experience on court data by pseudonymizing and structuring them, before trying to carry out some more value-added work. Such tasks are off-putting and time-consuming while bringing few scientific lessons. This research therefore seeks to deliver results in one go, directly from raw texts, despite the rather poor quality of their rendition.

## 2.2. Dataset

For any natural-language processing, a dataset should have sufficient examples to ensure the best balance in the variety of data and language [18]. In our case, the learning of proceedings data should be based on a stock of cases, not only recent cases, and therefore, the learning dataset should be representative of the historically different writing conventions and templates. Our policy was to collect well-distributed cases from the past 15 to 20 years, depending on their level of availability, which varies by jurisdiction. Our dataset is composed of cases from the three jurisdictional degrees and focuses on three law areas (Table 2).

<table><caption>Table 2. Cases breakdown by degree and law area.</caption><thead><tr><td>Cases</td><td>1st Instance</td><td>Appeal</td><td>Final Appeal</td><td>Total</td></tr></thead><tbody><tr><td>Leases</td><td>250</td><td>208</td><td>63</td><td>521</td></tr><tr><td>Accidents</td><td>80</td><td>198</td><td>46</td><td>324</td></tr><tr><td>Bank</td><td>250</td><td>231</td><td>50</td><td>531</td></tr><tr><td>Other</td><td>234</td><td>0</td><td>96</td><td>330</td></tr><tr><td>All</td><td>814</td><td>637</td><td>255</td><td>1706</td></tr></tbody></table>

One first batch of cases, all on civil responsibility matters, comes from three judicial tribunals: 46 from Chambéry, 119 from Saint-Etienne and 149 from Grenoble, for a total of 314. They had been scanned by each tribunal as PDF images, then OCR'd to PDF text by Université Savoie Mont Blanc (USMB) and then converted to TXT files. As these cases have not been pseudonymized, they will not be published in the context of this research.

A second batch was supplied by Doctrine, a French legal technology. It includes 250 cases on banking law and another 250 on commercial leases. They have been pseudonymized with Doctrine's proprietary ML-based algorithm and randomly selected from their database upon our request. In many cases, parties were not only pseudonymized but actually erased, which damages the intelligibility of the text. USMB and Doctrine are warmly thanked for these corpora.

Three datasets of appellate cases have been downloaded from Légifrance, the portal of Direction de l'Information Légale et Administrative (DILA), the French national gazette editor. These cases have been pseudonymized by DILA and been subject to no subsequent data cleaning. They are published for having been "selected according to the methods specific to each order of jurisdiction", so they are not necessarily statistically representative of the procedural paths. The datasets are fairly well geographically distributed among appellate courts. The Court of Cassation, the supreme court and Légifrance each use their own template for the same case, and each one varies over time. We selected supreme court cases with the help of a page of the Court of Cassation's website that has been clustering cases since 2014, by area of law, such as road accidents, commercial leases and banking. We
downloaded the body of their text from Légifrance, in HTML format, and pasted them to as many TXT files.

Because of legal publicity restrictions and policy variations in the collection and centralization of court cases by supreme courts, the uncertain comprehensiveness of the available corpora makes up the first bias of analysis. French cases raise other issues. Though syntax is generally correct, punctuation hazards, missing blanks and words pseudonymized by error are likely to hinder entity recognition. Many cases are verbose, with redundant arguments or citations, sometimes scattered in different sections. Moreover, court cases are generally not structured. Of appellate cases, 45% do not have an explicit title for all sections [19]. The last chapter, “the outcome” (le dispositif), which describes the operative part, is the only section whose starting point is systematically showed, generally by the words “par ces motifs”.

## 2.3. Labels

Selected labels describe information related to civil proceedings. About half of them are common to the three jurisdictional orders (see Table 3).

<table><caption>**Table 3.** Set of labels (alphabetic order). Checkmarks show labels that are applicable per jurisdictional degree and that have been subject to manual annotations.</caption><thead><tr><th>Label</th><th>Label in French</th><th>Type</th><th>First-Instance Tribunal</th><th>Appellate Court</th><th>Supreme Court (Cour de Cassation)</th></tr></thead><tbody><tr><td>Case-law citation</td><td>Référence jurisprudentielle</td><td>Named entity</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Chamber</td><td>Formation</td><td>Named entity</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Costs of instance</td><td>Dépens</td><td>Amount</td><td>✓</td><td></td><td></td></tr><tr><td>Date of appeal</td><td>Date d'appel</td><td>Judicial date</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Date of appellate case</td><td>Date d'arrêt</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of bankruptcy act</td><td>Date de redressement ou liquidation</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of birth</td><td>Date de naissance</td><td>Date</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Date of expertise</td><td>Date de rapport d'expertise</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of injunction</td><td>Date de référé</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of introductory act</td><td>Date d'assignation</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of judgment</td><td>Date de jugement</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of pre-trial</td><td>Date de mise en état</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Defendant</td><td>Défendeur</td><td>Named entity (party)</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>ECLI</td><td>ECLI</td><td>Decision identifier</td><td></td><td></td><td>✓</td></tr><tr><td>Id of appellate case</td><td>RG appel</td><td>Decision identifier</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Id of final appellate case</td><td>Numéro de pourvoi</td><td>Decision identifier</td><td></td><td></td><td>✓</td></tr><tr><td>Id of first-instance case</td><td>RG première instance</td><td>Decision identifier</td><td>✓</td><td>✓</td><td></td></tr></tbody></table>
downloaded the body of their text from Légifrance, in HTML format, and pasted them to as many TXT files.

Because of legal publicity restrictions and policy variations in the collection and centralization of court cases by supreme courts, the uncertain comprehensiveness of the available corpora makes up the first bias of analysis. French cases raise other issues. Though syntax is generally correct, punctuation hazards, missing blanks and words pseudonymized by error are likely to hinder entity recognition. Many cases are verbose, with redundant arguments or citations, sometimes scattered in different sections. Moreover, court cases are generally not structured. Of appellate cases, 45% do not have an explicit title for all sections [19]. The last chapter, “the outcome” (le dispositif), which describes the operative part, is the only section whose starting point is systematically showed, generally by the words “par ces motifs”.

## 2.3. Labels

Selected labels describe information related to civil proceedings. About half of them are common to the three jurisdictional orders (see Table 3).

<table><caption>**Table 3.** Set of labels (alphabetic order). Checkmarks show labels that are applicable per jurisdictional degree and that have been subject to manual annotations.</caption><thead><tr><th>Label</th><th>Label in French</th><th>Type</th><th>First-Instance Tribunal</th><th>Appellate Court</th><th>Supreme Court (Cour de Cassation)</th></tr></thead><tbody><tr><td>Case-law citation</td><td>Référence jurisprudentielle</td><td>Named entity</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Chamber</td><td>Formation</td><td>Named entity</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Costs of instance</td><td>Dépens</td><td>Amount</td><td>✓</td><td></td><td></td></tr><tr><td>Date of appeal</td><td>Date d'appel</td><td>Judicial date</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Date of appellate case</td><td>Date d'arrêt</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of bankruptcy act</td><td>Date de redressement ou liquidation</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of birth</td><td>Date de naissance</td><td>Date</td><td>✓</td><td>✓</td><td></td></tr><tr><td>Date of expertise</td><td>Date de rapport d'expertise</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of injunction</td><td>Date de référé</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of introductory act</td><td>Date d'assignation</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of judgment</td><td>Date de jugement</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Date of pre-trial</td><td>Date de mise en état</td><td>Judicial date</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Defendant</td><td>Défendeur</td><td>Named entity (party)</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>ECLI</td><td>ECLI</td><td>Decision identifier</td><td></td><td></td><td>✓</td></tr><tr><td>Id of appellate case</td><td>RG appel</td><td>Decision identifier</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Id of final appellate case</td><td>Numéro de pourvoi</td><td>Decision identifier</td><td></td><td></td><td>✓</td></tr><tr><td>Id of first-instance case</td><td>RG première instance</td><td>Decision identifier</td><td>✓</td><td>✓</td><td></td></tr></tbody></table>
<table><caption>Table 3. *Cont.*</caption><thead><tr><td>Label</td><td>Label in French</td><td>Type</td><td>First-Instance Tribunal</td><td>Appellate Court</td><td>Supreme Court (Cour de Cassation)</td></tr></thead><tbody><tr><td>Jurisdiction</td><td>Jurisdiction</td><td>Named entity</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Legal citation</td><td>Référence juridique</td><td>Named entity</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Losing party</td><td>Condamne</td><td>Named entity (party)</td><td>✓</td><td></td><td></td></tr><tr><td>Outcome</td><td>Sens de l'arrêt</td><td>Boolean</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Plaintiff</td><td>Demandeur</td><td>Named entity (party)</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Reference for further explanation</td><td>Renvoi pour plus ample exposé</td><td>Boolean</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Referring jurisdiction</td><td>Juridiction de renvoi</td><td>Named entity</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Type of case</td><td>Type de décision</td><td>Named entity</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Unrecoverable costs</td><td>Frais irrépétibles</td><td>Amount</td><td>✓</td><td>✓</td><td>✓</td></tr></tbody></table>

Plaintiff, defendant and condemned (“losing”) party are three labels that represent one party to a litigation. The plaintiff is the party who initiates a procedure, through a so-called *assignation* or *saisine*, in a first-instance judgment, or the party who, unsatisfied with the solution of the first-instance judge, lodged an appeal. The defendant is the party seeking confirmation of the first-instance judgment, in an appellate judgment, and we also use that label name in the context of a first-instance judgment to describe the party whom the introductory act is aimed at. One might wonder why we should bother to extract the parties to a trial given that names of natural persons are, or are supposed to be, “occulted” (Art. L111-13 of the code of judicial organization). One first answer is that in the absence, most of the time, of a unique identifier of the case preceding an appellate case, every indirect attribute of a case may help the matching. One name of a legal person (ex: “BNP Paribas”), one first name (ex: “Véronique”) or one title (“Madame”, “Maître”) may be enough to identify a party among many first-instance cases made a given day by a given chamber of a given jurisdiction the case at the origin of a given appellate case. One second answer is to preserve the very intelligibility of the case, as long as cases recognize the parties to a trial by their name, not by their role (essentially plaintiff versus defendant).

The label “date of birth” is proposed as a complementary identifier of a party who is a natural person. It may help disambiguate between different judgments made the same day in the same jurisdiction, when matched with appellate cases to build judicial pathways.

The other date labels define as many judicial milestones. One of these is the date of bankruptcy act. Parties that are legal persons may be subject to acts such as *redressement judiciaire* or *liquidation judiciaire*, which may happen during the litigation timespan. This helps to avoid any such date is being mistaken for a judicial milestone of the current case and helps to avoid getting many false positives, especially under the label “date of judgment”. The other dates do refer to the current case. The label “date of introductory act” (*date d'assignation*) refers to the initial seizure of a tribunal by a party. The label “date of judgment” excludes the definition of “date of injunction” (*date de référé*), a type of ordinance that may be ordered by the same first-instance jurisdiction, which we preferred to annotate with a dedicated label. It is also distinct from the “date of appellate case”, which is issued following an appeal. The “date of expertise” refers to the date when any expert designated by a judgment hands over their report to the judge. The “date of pre-trial” (*date de mise en état*) marks an update of the litigation case, such as when the case is “joined” with one other case or several other related cases or when it closes the instruction phase. We would expect that if more than one identifier of a first-instance judgment were to appear in the text of a judgment, a pre-trial date would also appear. A long procedure may involve an
<table><caption>Table 3. *Cont.*</caption><thead><tr><td>Label</td><td>Label in French</td><td>Type</td><td>First-Instance Tribunal</td><td>Appellate Court</td><td>Supreme Court (Cour de Cassation)</td></tr></thead><tbody><tr><td>Jurisdiction</td><td>Jurisdiction</td><td>Named entity</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Legal citation</td><td>Référence juridique</td><td>Named entity</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Losing party</td><td>Condamne</td><td>Named entity (party)</td><td>✓</td><td></td><td></td></tr><tr><td>Outcome</td><td>Sens de l'arrêt</td><td>Boolean</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Plaintiff</td><td>Demandeur</td><td>Named entity (party)</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Reference for further explanation</td><td>Renvoi pour plus ample exposé</td><td>Boolean</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Referring jurisdiction</td><td>Juridiction de renvoi</td><td>Named entity</td><td></td><td>✓</td><td>✓</td></tr><tr><td>Type of case</td><td>Type de décision</td><td>Named entity</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Unrecoverable costs</td><td>Frais irrépétibles</td><td>Amount</td><td>✓</td><td>✓</td><td>✓</td></tr></tbody></table>

Plaintiff, defendant and condemned (“losing”) party are three labels that represent one party to a litigation. The plaintiff is the party who initiates a procedure, through a so-called *assignation* or *saisine*, in a first-instance judgment, or the party who, unsatisfied with the solution of the first-instance judge, lodged an appeal. The defendant is the party seeking confirmation of the first-instance judgment, in an appellate judgment, and we also use that label name in the context of a first-instance judgment to describe the party whom the introductory act is aimed at. One might wonder why we should bother to extract the parties to a trial given that names of natural persons are, or are supposed to be, “occulted” (Art. L111-13 of the code of judicial organization). One first answer is that in the absence, most of the time, of a unique identifier of the case preceding an appellate case, every indirect attribute of a case may help the matching. One name of a legal person (ex: “BNP Paribas”), one first name (ex: “Véronique”) or one title (“Madame”, “Maître”) may be enough to identify a party among many first-instance cases made a given day by a given chamber of a given jurisdiction the case at the origin of a given appellate case. One second answer is to preserve the very intelligibility of the case, as long as cases recognize the parties to a trial by their name, not by their role (essentially plaintiff versus defendant).

The label “date of birth” is proposed as a complementary identifier of a party who is a natural person. It may help disambiguate between different judgments made the same day in the same jurisdiction, when matched with appellate cases to build judicial pathways.

The other date labels define as many judicial milestones. One of these is the date of bankruptcy act. Parties that are legal persons may be subject to acts such as *redressement judiciaire* or *liquidation judiciaire*, which may happen during the litigation timespan. This helps to avoid any such date is being mistaken for a judicial milestone of the current case and helps to avoid getting many false positives, especially under the label “date of judgment”. The other dates do refer to the current case. The label “date of introductory act” (*date d'assignation*) refers to the initial seizure of a tribunal by a party. The label “date of judgment” excludes the definition of “date of injunction” (*date de référé*), a type of ordinance that may be ordered by the same first-instance jurisdiction, which we preferred to annotate with a dedicated label. It is also distinct from the “date of appellate case”, which is issued following an appeal. The “date of expertise” refers to the date when any expert designated by a judgment hands over their report to the judge. The “date of pre-trial” (*date de mise en état*) marks an update of the litigation case, such as when the case is “joined” with one other case or several other related cases or when it closes the instruction phase. We would expect that if more than one identifier of a first-instance judgment were to appear in the text of a judgment, a pre-trial date would also appear. A long procedure may involve an
assignment, an injunction, the handover of a report by a judicial expert, a first-instance judgment, an appeal, an appellate case, a pre-trial, another appellate case, a final appeal and a final appellate case. Figure 1 shows an example.

<figure><img src="image_7.png" /><figcaption>**Figure 1.** Example of annotations in a court case.</figcaption></figure>

The "date of appeal" had been included before realizing that it had little relevance, insofar as French law states any appeal can be lodged only within the fortnight following the first-instance judgment.

The label "jurisdiction" is defined as the combination of either a tribunal (of first-instance) or a court (of appeal) with a city (in French, a *ressort*), when it has competence in a given territory. Court of Cassation and *Tribunal des Conflits*, which have national competence, are also defined as jurisdictions. We added the "chamber" label to cater to large jurisdictions where cases can be rendered by different courtrooms.

We decided to dedicate a label to the "referring jurisdiction" (*juridiction de renvoi*). The appellate court may hand over the case to a lower jurisdiction or to other magistrates of the same jurisdiction that has been subject to the appeal. As opposed to other annotations of jurisdictions, it may announce an upcoming case and therefore have no date and no chamber associated to it. We believe that this distinction will help reusers separate the past proceedings from the next one in their analysis (see Figure 1).

The label "legal citation" is used to annotate the articles of law invoked by the parties or by the judge, whereas "case-law citation" refers to some case not directly related to the litigation in progress. Many fewer annotations are expected, France having a civil-law tradition, not a common-law system, where previous court cases play a decisive role in every case.

We made the choice to use two separate labels for the Id of the appellate case and the Id of the first-instance case. An analysis of their proximity to the dates of appeal and trial judgment may be sufficient to assign a unified label to either type of proceeding. So that will be considered later.

Though ECLI, the European Case Law identifier, could be generated with even greater quality through some regular expression (regexp) algorithm, "ECLI" has been added as a label because it is a case identifier like three other labels (see Table 1).

For a tribunal, the outcome consists in setting a type of case or naming the losing party (or parties). For an appellate court, it is often a confirmation or a reversal, but the judge may decide on alternative or supplemental provisions. The outcome of a case is binary only at the level of the supreme court. Three labels are therefore needed to describe the operative part of the case. We introduced "type of case" and "losing party" specifically for tribunals, and "outcome", applicable to the other two degrees, is a Boolean that basically classifies the case as a confirmation or a reversal.

Our set of labels include two labels with an amount type: "costs of instance", annotated only for first-instance cases, and "non-recoverable costs". The latter represents the amount dedicated to refund legal fees, mostly paid to lawyers, different from the costs incurred by the proceedings and paid to the jurisdiction, which are fully recoverable by law. Such amounts can be found in cases of every jurisdictional order.

We also adopted "for further account", a Boolean that, if true, invites the reuser to collect and read previous related cases for a comprehensive understanding of the context.
assignment, an injunction, the handover of a report by a judicial expert, a first-instance judgment, an appeal, an appellate case, a pre-trial, another appellate case, a final appeal and a final appellate case. Figure 1 shows an example.

<figure><img src="image_7.png" /><figcaption>**Figure 1.** Example of annotations in a court case.</figcaption></figure>

The "date of appeal" had been included before realizing that it had little relevance, insofar as French law states any appeal can be lodged only within the fortnight following the first-instance judgment.

The label "jurisdiction" is defined as the combination of either a tribunal (of first-instance) or a court (of appeal) with a city (in French, a *ressort*), when it has competence in a given territory. Court of Cassation and *Tribunal des Conflits*, which have national competence, are also defined as jurisdictions. We added the "chamber" label to cater to large jurisdictions where cases can be rendered by different courtrooms.

We decided to dedicate a label to the "referring jurisdiction" (*juridiction de renvoi*). The appellate court may hand over the case to a lower jurisdiction or to other magistrates of the same jurisdiction that has been subject to the appeal. As opposed to other annotations of jurisdictions, it may announce an upcoming case and therefore have no date and no chamber associated to it. We believe that this distinction will help reusers separate the past proceedings from the next one in their analysis (see Figure 1).

The label "legal citation" is used to annotate the articles of law invoked by the parties or by the judge, whereas "case-law citation" refers to some case not directly related to the litigation in progress. Many fewer annotations are expected, France having a civil-law tradition, not a common-law system, where previous court cases play a decisive role in every case.

We made the choice to use two separate labels for the Id of the appellate case and the Id of the first-instance case. An analysis of their proximity to the dates of appeal and trial judgment may be sufficient to assign a unified label to either type of proceeding. So that will be considered later.

Though ECLI, the European Case Law identifier, could be generated with even greater quality through some regular expression (regexp) algorithm, "ECLI" has been added as a label because it is a case identifier like three other labels (see Table 1).

For a tribunal, the outcome consists in setting a type of case or naming the losing party (or parties). For an appellate court, it is often a confirmation or a reversal, but the judge may decide on alternative or supplemental provisions. The outcome of a case is binary only at the level of the supreme court. Three labels are therefore needed to describe the operative part of the case. We introduced "type of case" and "losing party" specifically for tribunals, and "outcome", applicable to the other two degrees, is a Boolean that basically classifies the case as a confirmation or a reversal.

Our set of labels include two labels with an amount type: "costs of instance", annotated only for first-instance cases, and "non-recoverable costs". The latter represents the amount dedicated to refund legal fees, mostly paid to lawyers, different from the costs incurred by the proceedings and paid to the jurisdiction, which are fully recoverable by law. Such amounts can be found in cases of every jurisdictional order.

We also adopted "for further account", a Boolean that, if true, invites the reuser to collect and read previous related cases for a comprehensive understanding of the context.
In such a case, this means the case could be unfit for some reuses, such as predictive justice. Finally, we created the label “law area”, some metadata available only for Court of Cassation cases, to help us conduct analyses on that axis.

Case categories (*nature d'affaire civile*, or NAC) have not been annotated. They are too seldom mentioned by the judge or difficult to recognize, so learning them would prove ineffective. Nor have the names of attorneys been extracted, but [20] in their own research already studied the extraction of appellant's and appellee's lawyers.

Table 4 shows how manual annotations break down per label.

<table><caption>**Table 4.** Numbers of annotations.</caption><thead><tr><th>Label</th><th>First-Instance</th><th>Appeal</th><th>Final Appeal</th><th>All Corpora</th></tr></thead><tbody><tr><td>Number of decisions</td><td>814</td><td>637</td><td>255</td><td>1706</td></tr><tr><td>Case-law citation</td><td>6</td><td>40</td><td>22</td><td>68</td></tr><tr><td>Chamber</td><td>528</td><td>589</td><td>360</td><td>1477</td></tr><tr><td>Costs of instance</td><td>133</td><td>0</td><td>0</td><td>133</td></tr><tr><td>Date of appeal</td><td>0</td><td>340</td><td>0</td><td>340</td></tr><tr><td>Date of appellate case</td><td>58</td><td>726</td><td>642</td><td>1426</td></tr><tr><td>Date of bankruptcy act</td><td>115</td><td>79</td><td>23</td><td>217</td></tr><tr><td>Date of birth</td><td>283</td><td>407</td><td>0</td><td>690</td></tr><tr><td>Date of expertise</td><td>166</td><td>136</td><td>11</td><td>313</td></tr><tr><td>Date of injunction</td><td>288</td><td>124</td><td>4</td><td>416</td></tr><tr><td>Date of introductory act</td><td>762</td><td>386</td><td>47</td><td>1195</td></tr><tr><td>Date of judgment</td><td>832</td><td>920</td><td>47</td><td>1799</td></tr><tr><td>Date of pre-trial</td><td>130</td><td>83</td><td>3</td><td>216</td></tr><tr><td>Defendant</td><td>1375</td><td>1197</td><td>382</td><td>2954</td></tr><tr><td>ECLI</td><td>0</td><td>0</td><td>177</td><td>177</td></tr><tr><td>Id of appellate case</td><td>0</td><td>578</td><td>7</td><td>585</td></tr><tr><td>Id of final appellate case</td><td>0</td><td>0</td><td>307</td><td>307</td></tr><tr><td>Id of first-instance case</td><td>862</td><td>242</td><td>0</td><td>1104</td></tr><tr><td>Jurisdiction</td><td>1039</td><td>1607</td><td>664</td><td>3310</td></tr><tr><td>Legal citation</td><td>2355</td><td>2071</td><td>958</td><td>5384</td></tr><tr><td>Losing party</td><td>833</td><td>0</td><td>0</td><td>833</td></tr><tr><td>Outcome</td><td>0</td><td>513</td><td>298</td><td>811</td></tr><tr><td>Plaintiff</td><td>1052</td><td>889</td><td>370</td><td>2311</td></tr><tr><td>Reference for further explanation</td><td>41</td><td>136</td><td>0</td><td>177</td></tr><tr><td>Referring jurisdiction</td><td>0</td><td>47</td><td>121</td><td>168</td></tr><tr><td>Type of case</td><td>114</td><td>0</td><td>0</td><td>114</td></tr><tr><td>Type of litigation</td><td>0</td><td>0</td><td>184</td><td>184</td></tr><tr><td>Unrecoverable costs</td><td>548</td><td>433</td><td>13</td><td>994</td></tr></tbody></table>
In such a case, this means the case could be unfit for some reuses, such as predictive justice. Finally, we created the label “law area”, some metadata available only for Court of Cassation cases, to help us conduct analyses on that axis.

Case categories (*nature d'affaire civile*, or NAC) have not been annotated. They are too seldom mentioned by the judge or difficult to recognize, so learning them would prove ineffective. Nor have the names of attorneys been extracted, but [20] in their own research already studied the extraction of appellant's and appellee's lawyers.

Table 4 shows how manual annotations break down per label.

<table><caption>**Table 4.** Numbers of annotations.</caption><thead><tr><th>Label</th><th>First-Instance</th><th>Appeal</th><th>Final Appeal</th><th>All Corpora</th></tr></thead><tbody><tr><td>Number of decisions</td><td>814</td><td>637</td><td>255</td><td>1706</td></tr><tr><td>Case-law citation</td><td>6</td><td>40</td><td>22</td><td>68</td></tr><tr><td>Chamber</td><td>528</td><td>589</td><td>360</td><td>1477</td></tr><tr><td>Costs of instance</td><td>133</td><td>0</td><td>0</td><td>133</td></tr><tr><td>Date of appeal</td><td>0</td><td>340</td><td>0</td><td>340</td></tr><tr><td>Date of appellate case</td><td>58</td><td>726</td><td>642</td><td>1426</td></tr><tr><td>Date of bankruptcy act</td><td>115</td><td>79</td><td>23</td><td>217</td></tr><tr><td>Date of birth</td><td>283</td><td>407</td><td>0</td><td>690</td></tr><tr><td>Date of expertise</td><td>166</td><td>136</td><td>11</td><td>313</td></tr><tr><td>Date of injunction</td><td>288</td><td>124</td><td>4</td><td>416</td></tr><tr><td>Date of introductory act</td><td>762</td><td>386</td><td>47</td><td>1195</td></tr><tr><td>Date of judgment</td><td>832</td><td>920</td><td>47</td><td>1799</td></tr><tr><td>Date of pre-trial</td><td>130</td><td>83</td><td>3</td><td>216</td></tr><tr><td>Defendant</td><td>1375</td><td>1197</td><td>382</td><td>2954</td></tr><tr><td>ECLI</td><td>0</td><td>0</td><td>177</td><td>177</td></tr><tr><td>Id of appellate case</td><td>0</td><td>578</td><td>7</td><td>585</td></tr><tr><td>Id of final appellate case</td><td>0</td><td>0</td><td>307</td><td>307</td></tr><tr><td>Id of first-instance case</td><td>862</td><td>242</td><td>0</td><td>1104</td></tr><tr><td>Jurisdiction</td><td>1039</td><td>1607</td><td>664</td><td>3310</td></tr><tr><td>Legal citation</td><td>2355</td><td>2071</td><td>958</td><td>5384</td></tr><tr><td>Losing party</td><td>833</td><td>0</td><td>0</td><td>833</td></tr><tr><td>Outcome</td><td>0</td><td>513</td><td>298</td><td>811</td></tr><tr><td>Plaintiff</td><td>1052</td><td>889</td><td>370</td><td>2311</td></tr><tr><td>Reference for further explanation</td><td>41</td><td>136</td><td>0</td><td>177</td></tr><tr><td>Referring jurisdiction</td><td>0</td><td>47</td><td>121</td><td>168</td></tr><tr><td>Type of case</td><td>114</td><td>0</td><td>0</td><td>114</td></tr><tr><td>Type of litigation</td><td>0</td><td>0</td><td>184</td><td>184</td></tr><tr><td>Unrecoverable costs</td><td>548</td><td>433</td><td>13</td><td>994</td></tr></tbody></table>
<table><caption>**Table 4.** *Cont.*</caption><thead><tr><th>Label</th><th>First-Instance</th><th>Appeal</th><th>Final Appeal</th><th>All Corpora</th></tr></thead><tbody><tr><td>Total number of annotations</td><td>11520</td><td>11,543</td><td>4640</td><td>27,703</td></tr><tr><td>Number of segments containing annotations</td><td>4331</td><td>4853</td><td>1574</td><td>10,758</td></tr><tr><td>Average number of annotations per decision</td><td>14.15</td><td>18.12</td><td>18.20</td><td>16.24</td></tr></tbody></table>

## 2.4. Tool

Kairntech is a French startup specializing in AI applied to text. The company is developing Sherpa [21], a natural-language processing (NLP) platform for software developers. It manages both classification and NER. It includes a graphical interface to annotate and a workbench to test different training algorithms. It caters to French, among several other languages. We use the *segment* (sentence) as the basic learning unit. Sherpa defines as a “dataset” the set of documents or segments that have been subject to at least one manual annotation. It is divided into a training dataset and a test dataset. The application of an experiment to a dataset produces a model, characterized by quality indicators: precision rate, recall rate and f-measure. The precision rate divides the number of correct annotations by that of all returned annotations. The recall rate is the number of correct annotations divided by the number of annotations that should have been returned. The F1 measure is the harmonic mean of precision and recall.

For NER, the workbench proposes to run CRF (conditional random fields), Spacy, Flair or DeLFT. CRF [22], is proposed in five training methods: L-BFGS, L2SGD, PA, AROW and AP. Flair [23], may be used with embeddings, which can be combined: Flair embeddings (contextualized string embeddings), bytecode and transformer embeddings.

DeLFT (deep learning framework for text) is a Keras and TensorFlow framework for text processing, focusing on NER and classification [24]. Flair and DeLFT are both based on recurrent neural networks (RNNs). The DeLFT RNN layer may be followed by a CRF layer. This allows the model to use past and future annotations to set the current annotation [25]. DeLFT can be combined with either ELMo embeddings [26], or BERT embeddings, for English texts, but CamemBERT, its French variant, is not available for DeLFT. We used DeLFT configured with its ELMo embeddings.

The basic experiment uses the CRF algorithm with a training dataset representing 80% of the randomly selected segments in the dataset. Training on the basis of the 80% allows annotations to be projected onto the remaining 20% of segments. The comparison of the manual annotations in this 20% with the projected annotations after training allows for quality indicators to be calculated. We leave this distribution constant in our experimentations.

The tool does not automate cross validation: it is up to the user to multiply runs and to average the results.

## 2.5. Conduct of Training

To save ourselves the rental costs of a GPU, we sampled one out of 10 cases from our global dataset representing all three jurisdictional degrees. The sampled dataset resulted in 171 cases, 1136 segments and 2857 manual annotations. Each experiment was run on a reshuffled dataset.

We used each algorithm with its default set of hyperparameters. CRF was applied to segments with a maximal size of 1500 characters. Flair was used with the SGD tokenizer. Its learning rate was 0.1, its anneal factor 0.5, batch size 32, patience 3, 100 epochs max, top four layers for embeddings, no storage of embeddings, hidden size 256, 1 LSTM layer, word dropout probability 0.5 and locked dropout probability 0.05. DeLFT used ELMo embeddings. The size of the character embeddings to compute was 25, dropout 0.5, recurrent dropout 0.25, max epochs 50, optimizer was adam, learning rate 0.001, clip
<table><caption>**Table 4.** *Cont.*</caption><thead><tr><th>Label</th><th>First-Instance</th><th>Appeal</th><th>Final Appeal</th><th>All Corpora</th></tr></thead><tbody><tr><td>Total number of annotations</td><td>11520</td><td>11,543</td><td>4640</td><td>27,703</td></tr><tr><td>Number of segments containing annotations</td><td>4331</td><td>4853</td><td>1574</td><td>10,758</td></tr><tr><td>Average number of annotations per decision</td><td>14.15</td><td>18.12</td><td>18.20</td><td>16.24</td></tr></tbody></table>

## 2.4. Tool

Kairntech is a French startup specializing in AI applied to text. The company is developing Sherpa [21], a natural-language processing (NLP) platform for software developers. It manages both classification and NER. It includes a graphical interface to annotate and a workbench to test different training algorithms. It caters to French, among several other languages. We use the *segment* (sentence) as the basic learning unit. Sherpa defines as a “dataset” the set of documents or segments that have been subject to at least one manual annotation. It is divided into a training dataset and a test dataset. The application of an experiment to a dataset produces a model, characterized by quality indicators: precision rate, recall rate and f-measure. The precision rate divides the number of correct annotations by that of all returned annotations. The recall rate is the number of correct annotations divided by the number of annotations that should have been returned. The F1 measure is the harmonic mean of precision and recall.

For NER, the workbench proposes to run CRF (conditional random fields), Spacy, Flair or DeLFT. CRF [22], is proposed in five training methods: L-BFGS, L2SGD, PA, AROW and AP. Flair [23], may be used with embeddings, which can be combined: Flair embeddings (contextualized string embeddings), bytecode and transformer embeddings.

DeLFT (deep learning framework for text) is a Keras and TensorFlow framework for text processing, focusing on NER and classification [24]. Flair and DeLFT are both based on recurrent neural networks (RNNs). The DeLFT RNN layer may be followed by a CRF layer. This allows the model to use past and future annotations to set the current annotation [25]. DeLFT can be combined with either ELMo embeddings [26], or BERT embeddings, for English texts, but CamemBERT, its French variant, is not available for DeLFT. We used DeLFT configured with its ELMo embeddings.

The basic experiment uses the CRF algorithm with a training dataset representing 80% of the randomly selected segments in the dataset. Training on the basis of the 80% allows annotations to be projected onto the remaining 20% of segments. The comparison of the manual annotations in this 20% with the projected annotations after training allows for quality indicators to be calculated. We leave this distribution constant in our experimentations.

The tool does not automate cross validation: it is up to the user to multiply runs and to average the results.

## 2.5. Conduct of Training

To save ourselves the rental costs of a GPU, we sampled one out of 10 cases from our global dataset representing all three jurisdictional degrees. The sampled dataset resulted in 171 cases, 1136 segments and 2857 manual annotations. Each experiment was run on a reshuffled dataset.

We used each algorithm with its default set of hyperparameters. CRF was applied to segments with a maximal size of 1500 characters. Flair was used with the SGD tokenizer. Its learning rate was 0.1, its anneal factor 0.5, batch size 32, patience 3, 100 epochs max, top four layers for embeddings, no storage of embeddings, hidden size 256, 1 LSTM layer, word dropout probability 0.5 and locked dropout probability 0.05. DeLFT used ELMo embeddings. The size of the character embeddings to compute was 25, dropout 0.5, recurrent dropout 0.25, max epochs 50, optimizer was adam, learning rate 0.001, clip
gradients 0.9, patience 5 and model type was bidirectional LSTM cum CRF. The maximum length of character sequence to compute embeddings was 30, the dimensionality of the character LSTM embeddings output space was 25, and the dimensionality of the word LSTM embeddings output space was 100. The batch size was 20, and the maximum number of checkpoints to keep was 5.

Among the different flavors of CRF, we chose CRF-pa, which seemed to deliver the best quality for a comparable model generation time. We looked at relative performances between labels.

Table 5 shows labels whose number of annotations (support) in the test dataset is below 100. The F1 scores are those of the first run and would prove volatile after other runs. A label such as “case-law citation”, with only nine annotations to reproduce, understandably provides a fragile result, dependent on the outcome of the split between train and test data.

<table><caption>**Table 5.** Quality for rare annotations (CRF-pa on all decisions).</caption><thead><tr><td>Label</td><td>Support</td><td>%</td><td>F1</td></tr></thead><tbody><tr><td>Date of appeal</td><td>62</td><td>1.18%</td><td>62.5%</td></tr><tr><td>Date of birth</td><td>75</td><td>1.42%</td><td>94.7%</td></tr><tr><td>Date of expertise</td><td>70</td><td>1.33%</td><td>75.2%</td></tr><tr><td>Date of bankruptcy</td><td>42</td><td>0.80%</td><td>19.2%</td></tr><tr><td>Date of injunction</td><td>97</td><td>1.84%</td><td>53.9%</td></tr><tr><td>Date of pre-trial</td><td>40</td><td>0.76%</td><td>29.9%</td></tr><tr><td>Costs of instance</td><td>20</td><td>0.38%</td><td>56.3%</td></tr><tr><td>ECLI</td><td>42</td><td>0.80%</td><td>97.6%</td></tr><tr><td>Unrecoverable costs</td><td>73</td><td>1.38%</td><td>56.6%</td></tr><tr><td>Case-law citation</td><td>9</td><td>0.17%</td><td>40.0%</td></tr><tr><td>Outcome</td><td>74</td><td>1.40%</td><td>81.9%</td></tr><tr><td>Referring jurisdiction</td><td>33</td><td>0.63%</td><td>81.4%</td></tr><tr><td>Reference for further explanation</td><td>30</td><td>0.57%</td><td>78.6%</td></tr><tr><td>Other labels</td><td>4605</td><td>87.35%</td><td></td></tr><tr><td>Total</td><td>5272</td><td>100.00%</td><td></td></tr></tbody></table>

To fine-tune the analyses per jurisdictional degree, law area or label while ensuring statistical representativity, we needed to work on the consolidated dataset. This in turn required us to rent GPUs (one NVIDIA T4 Tensor Core GPUs, eight vCPUs, 32 GB RAM) to get acceptable training times.

# 3. Results

## 3.1. By Type of Algorithm

Our first analysis was to rank algorithm performances irrespective of the type of jurisdiction and the law area.

This cycle of runs shows that DeLFT is the best-performing model and CRF-pa (passive/aggressive) is the best-performing algorithm for the shortest training time. SpaCy (with 30 iterations) appears to be a good compromise between quality and training time. Table 6 sorts the performance outcomes of algorithms in different setups by ascending F1 score average over five runs.
gradients 0.9, patience 5 and model type was bidirectional LSTM cum CRF. The maximum length of character sequence to compute embeddings was 30, the dimensionality of the character LSTM embeddings output space was 25, and the dimensionality of the word LSTM embeddings output space was 100. The batch size was 20, and the maximum number of checkpoints to keep was 5.

Among the different flavors of CRF, we chose CRF-pa, which seemed to deliver the best quality for a comparable model generation time. We looked at relative performances between labels.

Table 5 shows labels whose number of annotations (support) in the test dataset is below 100. The F1 scores are those of the first run and would prove volatile after other runs. A label such as “case-law citation”, with only nine annotations to reproduce, understandably provides a fragile result, dependent on the outcome of the split between train and test data.

<table><caption>**Table 5.** Quality for rare annotations (CRF-pa on all decisions).</caption><thead><tr><td>Label</td><td>Support</td><td>%</td><td>F1</td></tr></thead><tbody><tr><td>Date of appeal</td><td>62</td><td>1.18%</td><td>62.5%</td></tr><tr><td>Date of birth</td><td>75</td><td>1.42%</td><td>94.7%</td></tr><tr><td>Date of expertise</td><td>70</td><td>1.33%</td><td>75.2%</td></tr><tr><td>Date of bankruptcy</td><td>42</td><td>0.80%</td><td>19.2%</td></tr><tr><td>Date of injunction</td><td>97</td><td>1.84%</td><td>53.9%</td></tr><tr><td>Date of pre-trial</td><td>40</td><td>0.76%</td><td>29.9%</td></tr><tr><td>Costs of instance</td><td>20</td><td>0.38%</td><td>56.3%</td></tr><tr><td>ECLI</td><td>42</td><td>0.80%</td><td>97.6%</td></tr><tr><td>Unrecoverable costs</td><td>73</td><td>1.38%</td><td>56.6%</td></tr><tr><td>Case-law citation</td><td>9</td><td>0.17%</td><td>40.0%</td></tr><tr><td>Outcome</td><td>74</td><td>1.40%</td><td>81.9%</td></tr><tr><td>Referring jurisdiction</td><td>33</td><td>0.63%</td><td>81.4%</td></tr><tr><td>Reference for further explanation</td><td>30</td><td>0.57%</td><td>78.6%</td></tr><tr><td>Other labels</td><td>4605</td><td>87.35%</td><td></td></tr><tr><td>Total</td><td>5272</td><td>100.00%</td><td></td></tr></tbody></table>

To fine-tune the analyses per jurisdictional degree, law area or label while ensuring statistical representativity, we needed to work on the consolidated dataset. This in turn required us to rent GPUs (one NVIDIA T4 Tensor Core GPUs, eight vCPUs, 32 GB RAM) to get acceptable training times.

# 3. Results

## 3.1. By Type of Algorithm

Our first analysis was to rank algorithm performances irrespective of the type of jurisdiction and the law area.

This cycle of runs shows that DeLFT is the best-performing model and CRF-pa (passive/aggressive) is the best-performing algorithm for the shortest training time. SpaCy (with 30 iterations) appears to be a good compromise between quality and training time. Table 6 sorts the performance outcomes of algorithms in different setups by ascending F1 score average over five runs.
<table><caption>**Table 6.** Quality, training time and run time by type of algorithm.</caption><thead><tr><th>Algorithm</th><th>Average f1 on 5 Reshuffled Runs</th><th>Time to Train the Model on a CPU</th><th>Time to Train the Model on a GPU</th><th>Seconds to Annotate One Decision *</th></tr></thead><tbody><tr><td>CRF—passive/aggressive</td><td>64.0%</td><td>1'35"</td><td>1'35"</td><td>0.32</td></tr><tr><td>Spacy—20 iterations</td><td>65.1%</td><td>7 mn</td><td>5 mn</td><td>0.24</td></tr><tr><td>Spacy—30 iterations</td><td>67.1%</td><td>9 mn</td><td>7 mn</td><td>0.24</td></tr><tr><td>Flair without embeddings (GRU)</td><td>69.4%</td><td>2 h</td><td>37 mn</td><td>0.31</td></tr><tr><td>Spacy—60 iterations</td><td>69.9%</td><td>19 mn</td><td>14 mn</td><td>0.25</td></tr><tr><td>Flair without embeddings (LSTM)</td><td>70.4%</td><td>3 h</td><td>1 h</td><td>0.36</td></tr><tr><td>Flair with Flair embeddings (LSTM)</td><td>75.3%</td><td>11 h</td><td>1 h</td><td>0.78</td></tr><tr><td>Flair with Flair, bytecode, transformer embeddings (LSTM)</td><td>75.3%</td><td>15 h</td><td>1 h</td><td>8.96</td></tr><tr><td>DELFT w. ELMo/bid. LSTM</td><td>76.0%</td><td>6 h</td><td>44 mn</td><td>16.04</td></tr></tbody></table>

(*): simulations conducted on a CPU server with decision RG 2016 000472 of Tribunal de Commerce de Thonon-les-Bains, 24 November 2016, from the Doctrine corpus. There may have been other computing activity on the Kairntech server upon running.

That step is not extended to transformers and to ensemble learning, which are not available in Kairntech at the time of writing.

Because CRF-pa was the configuration that performed best among the CRF Suite, we decided to keep it in our further research when comparing CRF with other families of algorithms.

## 3.2. By Degree of Jurisdiction

We wondered whether the nine labels that are common to the three jurisdictional levels (Table 2) should be trained together or be subject to degree-specific training. In this experimentation, on the whole dataset (1706 cases), we ran the same four classes of algorithms on each degree-specific dataset, with the same hyperparameters, and again on a cross-degree dataset. The algorithms were:

* CRF-pa;

* SpaCy with 30 iterations;

* Flair with its embeddings;

* DeLFT.

For practical reasons, the multijurisdictional dataset was capped at a sample of 4000 segments among the 10,750 of the aggregated segments. This represents a size comparable to the first-instance dataset (4331). Table 7 shows that the ranking between algorithms remains unchanged at every jurisdictional level. Flair outperforms DeLFT by a tiny margin. It is not clear-cut whether jurisdictional-level models outperform a single cross-jurisdictional-level one. We will therefore pursue investigations at both levels.
<table><caption>**Table 6.** Quality, training time and run time by type of algorithm.</caption><thead><tr><th>Algorithm</th><th>Average f1 on 5 Reshuffled Runs</th><th>Time to Train the Model on a CPU</th><th>Time to Train the Model on a GPU</th><th>Seconds to Annotate One Decision *</th></tr></thead><tbody><tr><td>CRF—passive/aggressive</td><td>64.0%</td><td>1'35"</td><td>1'35"</td><td>0.32</td></tr><tr><td>Spacy—20 iterations</td><td>65.1%</td><td>7 mn</td><td>5 mn</td><td>0.24</td></tr><tr><td>Spacy—30 iterations</td><td>67.1%</td><td>9 mn</td><td>7 mn</td><td>0.24</td></tr><tr><td>Flair without embeddings (GRU)</td><td>69.4%</td><td>2 h</td><td>37 mn</td><td>0.31</td></tr><tr><td>Spacy—60 iterations</td><td>69.9%</td><td>19 mn</td><td>14 mn</td><td>0.25</td></tr><tr><td>Flair without embeddings (LSTM)</td><td>70.4%</td><td>3 h</td><td>1 h</td><td>0.36</td></tr><tr><td>Flair with Flair embeddings (LSTM)</td><td>75.3%</td><td>11 h</td><td>1 h</td><td>0.78</td></tr><tr><td>Flair with Flair, bytecode, transformer embeddings (LSTM)</td><td>75.3%</td><td>15 h</td><td>1 h</td><td>8.96</td></tr><tr><td>DELFT w. ELMo/bid. LSTM</td><td>76.0%</td><td>6 h</td><td>44 mn</td><td>16.04</td></tr></tbody></table>

(*): simulations conducted on a CPU server with decision RG 2016 000472 of Tribunal de Commerce de Thonon-les-Bains, 24 November 2016, from the Doctrine corpus. There may have been other computing activity on the Kairntech server upon running.

That step is not extended to transformers and to ensemble learning, which are not available in Kairntech at the time of writing.

Because CRF-pa was the configuration that performed best among the CRF Suite, we decided to keep it in our further research when comparing CRF with other families of algorithms.

## 3.2. By Degree of Jurisdiction

We wondered whether the nine labels that are common to the three jurisdictional levels (Table 2) should be trained together or be subject to degree-specific training. In this experimentation, on the whole dataset (1706 cases), we ran the same four classes of algorithms on each degree-specific dataset, with the same hyperparameters, and again on a cross-degree dataset. The algorithms were:

* CRF-pa;

* SpaCy with 30 iterations;

* Flair with its embeddings;

* DeLFT.

For practical reasons, the multijurisdictional dataset was capped at a sample of 4000 segments among the 10,750 of the aggregated segments. This represents a size comparable to the first-instance dataset (4331). Table 7 shows that the ranking between algorithms remains unchanged at every jurisdictional level. Flair outperforms DeLFT by a tiny margin. It is not clear-cut whether jurisdictional-level models outperform a single cross-jurisdictional-level one. We will therefore pursue investigations at both levels.
<table><caption>**Table 7.** Quality by jurisdictional degree.</caption><thead><tr><th>Jurisdictional Degree <sup>1</sup></th><th>Algorithm</th><th>F1-Score <sup>2</sup></th></tr></thead><tbody><tr><td rowspan="4">Tribunals</td><td>CRF-PA</td><td>78.3%</td></tr><tr><td>Spacy</td><td>83.2%</td></tr><tr><td>Flair with its embeddings</td><td>87.2%</td></tr><tr><td>DeLFT with ELMo</td><td>86.9%</td></tr><tr><td rowspan="4">Appellate Courts</td><td>CRF-PA</td><td>76.3%</td></tr><tr><td>Spacy</td><td>80.9%</td></tr><tr><td>Flair with its embeddings</td><td>84.5%</td></tr><tr><td>DeLFT with ELMo</td><td>83.1%</td></tr><tr><td rowspan="4">Court of Cassation</td><td>CRF-PA</td><td>80.2%</td></tr><tr><td>Spacy</td><td>81.7%</td></tr><tr><td>Flair with its embeddings</td><td>87.3%</td></tr><tr><td>DeLFT with ELMo</td><td>85.7%</td></tr><tr><td rowspan="4">All jurisdictions (4000 segments)</td><td>CRF-PA</td><td>75.0%</td></tr><tr><td>Spacy</td><td>80.4%</td></tr><tr><td>Flair with its embeddings</td><td>85.7%</td></tr><tr><td>DeLFT with ELMo</td><td>83.6%</td></tr></tbody></table>

### 3.3. By Law Area

Given that the purpose of this research is to assess machine-learning quality on the extraction of civil proceedings-related information, it is necessary to verify whether the law area could be a factor of that quality. Because supreme court cases refer to the legality of the procedure, not to the merits of the litigation, we excluded Court of Cassation cases from the scope of experiences conducted on a specific law area.

The F1 score rises by some 2% or 3% under any algorithm if learning is separately applied to each law area (Table 8). This contradicts our initial assumption. It seems the law area weighs more than the jurisdictional degree as a quality factor. Though it is true that in large jurisdictions with specialized chambers, judges may repeat writing patterns, in small ones, which make up the majority in our dataset, the same judge is likely to write cases in any of our three chosen law areas. So this alone cannot explain why a consolidated model produces lesser quality. Some next steps could be to extend the experimentation to other types of litigation.

<table><caption>**Table 8.** Quality by law area: lower-court cases.</caption><thead><tr><th>F1-Score, All Labels,<br>1000 Segments,<br>avg./ Runs</th><th>Leases</th><th>Accidents</th><th>Banking</th><th>Weighted<br>avg. in 3<br>Law Areas</th><th>Consolidated<br>3 Law Areas</th></tr></thead><tbody><tr><td>CRF-pa</td><td>65.5%</td><td>68.1%</td><td>62.7%</td><td>65.0%</td><td>62.8%</td></tr><tr><td>Spacy</td><td>68.2%</td><td>75.2%</td><td>65.8%</td><td>68.9%</td><td>66.7%</td></tr><tr><td>Flair with its embed.</td><td>77.9%</td><td>81.3%</td><td>74.8%</td><td>77.4%</td><td>75.2%</td></tr><tr><td>Delft</td><td>76.0%</td><td>78.7%</td><td>73.3%</td><td>75.6%</td><td>72.4%</td></tr></tbody></table>

<footnote>(1) labels: date of introductory act, date of judgment, date of appellate decision, plaintiff, defendant, jurisdiction, chamber, legal reference, non-recoverable costs. (2) average on 5 runs after re-shuffle of training dataset.</footnote>
<table><caption>**Table 7.** Quality by jurisdictional degree.</caption><thead><tr><th>Jurisdictional Degree <sup>1</sup></th><th>Algorithm</th><th>F1-Score <sup>2</sup></th></tr></thead><tbody><tr><td rowspan="4">Tribunals</td><td>CRF-PA</td><td>78.3%</td></tr><tr><td>Spacy</td><td>83.2%</td></tr><tr><td>Flair with its embeddings</td><td>87.2%</td></tr><tr><td>DeLFT with ELMo</td><td>86.9%</td></tr><tr><td rowspan="4">Appellate Courts</td><td>CRF-PA</td><td>76.3%</td></tr><tr><td>Spacy</td><td>80.9%</td></tr><tr><td>Flair with its embeddings</td><td>84.5%</td></tr><tr><td>DeLFT with ELMo</td><td>83.1%</td></tr><tr><td rowspan="4">Court of Cassation</td><td>CRF-PA</td><td>80.2%</td></tr><tr><td>Spacy</td><td>81.7%</td></tr><tr><td>Flair with its embeddings</td><td>87.3%</td></tr><tr><td>DeLFT with ELMo</td><td>85.7%</td></tr><tr><td rowspan="4">All jurisdictions (4000 segments)</td><td>CRF-PA</td><td>75.0%</td></tr><tr><td>Spacy</td><td>80.4%</td></tr><tr><td>Flair with its embeddings</td><td>85.7%</td></tr><tr><td>DeLFT with ELMo</td><td>83.6%</td></tr></tbody></table>

### 3.3. By Law Area

Given that the purpose of this research is to assess machine-learning quality on the extraction of civil proceedings-related information, it is necessary to verify whether the law area could be a factor of that quality. Because supreme court cases refer to the legality of the procedure, not to the merits of the litigation, we excluded Court of Cassation cases from the scope of experiences conducted on a specific law area.

The F1 score rises by some 2% or 3% under any algorithm if learning is separately applied to each law area (Table 8). This contradicts our initial assumption. It seems the law area weighs more than the jurisdictional degree as a quality factor. Though it is true that in large jurisdictions with specialized chambers, judges may repeat writing patterns, in small ones, which make up the majority in our dataset, the same judge is likely to write cases in any of our three chosen law areas. So this alone cannot explain why a consolidated model produces lesser quality. Some next steps could be to extend the experimentation to other types of litigation.

<table><caption>**Table 8.** Quality by law area: lower-court cases.</caption><thead><tr><th>F1-Score, All Labels,<br>1000 Segments,<br>avg./ Runs</th><th>Leases</th><th>Accidents</th><th>Banking</th><th>Weighted<br>avg. in 3<br>Law Areas</th><th>Consolidated<br>3 Law Areas</th></tr></thead><tbody><tr><td>CRF-pa</td><td>65.5%</td><td>68.1%</td><td>62.7%</td><td>65.0%</td><td>62.8%</td></tr><tr><td>Spacy</td><td>68.2%</td><td>75.2%</td><td>65.8%</td><td>68.9%</td><td>66.7%</td></tr><tr><td>Flair with its embed.</td><td>77.9%</td><td>81.3%</td><td>74.8%</td><td>77.4%</td><td>75.2%</td></tr><tr><td>Delft</td><td>76.0%</td><td>78.7%</td><td>73.3%</td><td>75.6%</td><td>72.4%</td></tr></tbody></table>

<footnote>(1) labels: date of introductory act, date of judgment, date of appellate decision, plaintiff, defendant, jurisdiction, chamber, legal reference, non-recoverable costs. (2) average on 5 runs after re-shuffle of training dataset.</footnote>
<table><caption>Table 8. *Cont.*</caption><thead><tr><th>F1-Score, All Labels,<br>1000 Segments,<br>avg./ Runs</th><th>Leases</th><th>Accidents</th><th>Banking</th><th>Weighted<br>avg. in 3<br>Law Areas</th><th>Consolidated<br>3 Law Areas</th></tr></thead><tbody><tr><td>Nb documents</td><td>458</td><td>278</td><td>481</td><td>1217</td><td>1217</td></tr><tr><td>Nb segments in<br>dataset</td><td>2995</td><td>1872</td><td>2638</td><td>7505</td><td>7505</td></tr><tr><td>Nb annotations</td><td>6187</td><td>5328</td><td>7060</td><td>18,575</td><td>18,575</td></tr></tbody></table>

### 3.4. By Type of Label

Any multilabel learning strategy must navigate between two pitfalls. Learning all labels together would produce unreliable figures for small-population ones. Learning them all separately would result in many word sequences annotated with conflicting labels. It is worthwhile to determine whether a model mix is possible, a mix where complementary models would each run on their best-performing algorithm.

We decided to divide the labels into different groups according to their numerical weight or their syntactic nature: identifiers, which are regular expressions (Group 1); dates (Group 2); parties, which are oft-repeated, named and partly pseudonymized but not truly taxonomic entities (Group 3); legal citations, which are oft-repeated, high-volume, compoundable regular expressions (Group 4); amounts (Group 5); and jurisdictions and chambers, which are taxonomic named entities (Group 6). The rest make up Group 7, the last group. Each group is therefore subject to 12 types of training.

Table 9 shows F1 scores of random runs for each group, in each type of algorithm and for each jurisdictional degree, on the full dataset. The best scores for each label are shown in red. Figures must be welcomed with prudence since these (intensive) calculations, for practical reasons, were launched only once.

<table><caption>Table 9. Quality by type of label (first run).</caption><thead><tr><th rowspan="2">Group of Learning</th><th rowspan="2">Label</th><th colspan="3">Tribunals</th><th colspan="3">Appellate Courts</th><th colspan="3">Court of Cassation</th></tr><tr><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th></tr></thead><tbody><tr><td rowspan="5">1</td><td>Appeal id</td><td></td><td>N/A</td><td></td><td></td><td>92.9%</td><td>92.6%</td><td>95.4%</td><td>94.2%</td><td>0.0%</td><td>93.6%</td><td>95.9%</td><td>94.9%</td></tr><tr><td>First instance id</td><td>95.2%</td><td>93.6%</td><td>95.9%</td><td>94.9%</td><td>71.0%</td><td>81.9%</td><td>88.4%</td><td>90.6%</td><td></td><td>95.7%</td><td>96.7%</td><td>96.0%</td></tr><tr><td>Final appeal id</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td><td>98.3%</td><td>95.7%</td><td>96.7%</td><td>96.0%</td></tr><tr><td>ECLI</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td><td>96.1%</td><td>98.6%</td><td>98.6%</td><td>98.8%</td></tr><tr><td>Date of appeal</td><td></td><td>N/A</td><td></td><td></td><td>76.5%</td><td>74.8%</td><td>93.9%</td><td>80.0%</td><td></td><td>N/A</td><td></td><td></td></tr><tr><td rowspan="6">2</td><td>Date of appellate case</td><td>33.3%</td><td>50.0%</td><td>100.0%</td><td>92.3%</td><td>81.7%</td><td>91.0%</td><td>93.0%</td><td>91.6%</td><td>93.5%</td><td>91.8%</td><td>96.7%</td><td>94.3%</td></tr><tr><td>Date of introductory act</td><td>79.0%</td><td>87.4%</td><td>75.4%</td><td>81.4%</td><td>69.1%</td><td>70.7%</td><td>83.4%</td><td>88.3%</td><td>0.0%</td><td>27.6%</td><td>15.4%</td><td>40.0%</td></tr><tr><td>Date of judgment</td><td>75.1%</td><td>78.1%</td><td>80.0%</td><td>82.4%</td><td>78.5%</td><td>83.1%</td><td>82.7%</td><td>87.7%</td><td>28.6%</td><td>26.7%</td><td>72.7%</td><td>35.3%</td></tr><tr><td>Date of pre-trial</td><td>34.3%</td><td>57.1%</td><td>57.1%</td><td>64.0%</td><td>34.8%</td><td>43.3%</td><td>90.9%</td><td>92.3%</td><td></td><td>N/A</td><td></td><td></td></tr><tr><td>Date of bankruptcy</td><td>12.1%</td><td>26.7%</td><td>83.3%</td><td>63.2%</td><td>0.0%</td><td>26.1%</td><td>36.4%</td><td>62.5%</td><td>0.0%</td><td>0.0%</td><td>83.3%</td><td>0.0%</td></tr><tr><td>Date of injunction</td><td>67.9%</td><td>74.4%</td><td>88.9%</td><td>83.9%</td><td>38.1%</td><td>50.0%</td><td>60.0%</td><td>77.3%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td></tr><tr><td rowspan="2">Date of birth</td><td></td><td>69.4%</td><td>89.2%</td><td>57.1%</td><td>81.0%</td><td>68.2%</td><td>81.4%</td><td>87.5%</td><td>89.2%</td><td>66.7%</td><td>22.2%</td><td>66.7%</td><td>0.0%</td></tr><tr><td></td><td>96.5%</td><td>96.5%</td><td>96.2%</td><td>97.0%</td><td>98.3%</td><td>95.7%</td><td>96.7%</td><td>93.9%</td><td></td><td>N/A</td><td></td><td></td></tr></tbody></table>
<table><caption>Table 8. *Cont.*</caption><thead><tr><th>F1-Score, All Labels,<br>1000 Segments,<br>avg./ Runs</th><th>Leases</th><th>Accidents</th><th>Banking</th><th>Weighted<br>avg. in 3<br>Law Areas</th><th>Consolidated<br>3 Law Areas</th></tr></thead><tbody><tr><td>Nb documents</td><td>458</td><td>278</td><td>481</td><td>1217</td><td>1217</td></tr><tr><td>Nb segments in<br>dataset</td><td>2995</td><td>1872</td><td>2638</td><td>7505</td><td>7505</td></tr><tr><td>Nb annotations</td><td>6187</td><td>5328</td><td>7060</td><td>18,575</td><td>18,575</td></tr></tbody></table>

### 3.4. By Type of Label

Any multilabel learning strategy must navigate between two pitfalls. Learning all labels together would produce unreliable figures for small-population ones. Learning them all separately would result in many word sequences annotated with conflicting labels. It is worthwhile to determine whether a model mix is possible, a mix where complementary models would each run on their best-performing algorithm.

We decided to divide the labels into different groups according to their numerical weight or their syntactic nature: identifiers, which are regular expressions (Group 1); dates (Group 2); parties, which are oft-repeated, named and partly pseudonymized but not truly taxonomic entities (Group 3); legal citations, which are oft-repeated, high-volume, compoundable regular expressions (Group 4); amounts (Group 5); and jurisdictions and chambers, which are taxonomic named entities (Group 6). The rest make up Group 7, the last group. Each group is therefore subject to 12 types of training.

Table 9 shows F1 scores of random runs for each group, in each type of algorithm and for each jurisdictional degree, on the full dataset. The best scores for each label are shown in red. Figures must be welcomed with prudence since these (intensive) calculations, for practical reasons, were launched only once.

<table><caption>Table 9. Quality by type of label (first run).</caption><thead><tr><th rowspan="2">Group of Learning</th><th rowspan="2">Label</th><th colspan="3">Tribunals</th><th colspan="3">Appellate Courts</th><th colspan="3">Court of Cassation</th></tr><tr><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th></tr></thead><tbody><tr><td rowspan="5">1</td><td>Appeal id</td><td></td><td>N/A</td><td></td><td></td><td>92.9%</td><td>92.6%</td><td>95.4%</td><td>94.2%</td><td>0.0%</td><td>93.6%</td><td>95.9%</td><td>94.9%</td></tr><tr><td>First instance id</td><td>95.2%</td><td>93.6%</td><td>95.9%</td><td>94.9%</td><td>71.0%</td><td>81.9%</td><td>88.4%</td><td>90.6%</td><td></td><td>95.7%</td><td>96.7%</td><td>96.0%</td></tr><tr><td>Final appeal id</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td><td>98.3%</td><td>95.7%</td><td>96.7%</td><td>96.0%</td></tr><tr><td>ECLI</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td><td>96.1%</td><td>98.6%</td><td>98.6%</td><td>98.8%</td></tr><tr><td>Date of appeal</td><td></td><td>N/A</td><td></td><td></td><td>76.5%</td><td>74.8%</td><td>93.9%</td><td>80.0%</td><td></td><td>N/A</td><td></td><td></td></tr><tr><td rowspan="6">2</td><td>Date of appellate case</td><td>33.3%</td><td>50.0%</td><td>100.0%</td><td>92.3%</td><td>81.7%</td><td>91.0%</td><td>93.0%</td><td>91.6%</td><td>93.5%</td><td>91.8%</td><td>96.7%</td><td>94.3%</td></tr><tr><td>Date of introductory act</td><td>79.0%</td><td>87.4%</td><td>75.4%</td><td>81.4%</td><td>69.1%</td><td>70.7%</td><td>83.4%</td><td>88.3%</td><td>0.0%</td><td>27.6%</td><td>15.4%</td><td>40.0%</td></tr><tr><td>Date of judgment</td><td>75.1%</td><td>78.1%</td><td>80.0%</td><td>82.4%</td><td>78.5%</td><td>83.1%</td><td>82.7%</td><td>87.7%</td><td>28.6%</td><td>26.7%</td><td>72.7%</td><td>35.3%</td></tr><tr><td>Date of pre-trial</td><td>34.3%</td><td>57.1%</td><td>57.1%</td><td>64.0%</td><td>34.8%</td><td>43.3%</td><td>90.9%</td><td>92.3%</td><td></td><td>N/A</td><td></td><td></td></tr><tr><td>Date of bankruptcy</td><td>12.1%</td><td>26.7%</td><td>83.3%</td><td>63.2%</td><td>0.0%</td><td>26.1%</td><td>36.4%</td><td>62.5%</td><td>0.0%</td><td>0.0%</td><td>83.3%</td><td>0,0%</td></tr><tr><td>Date of injunction</td><td>67.9%</td><td>74.4%</td><td>88.9%</td><td>83.9%</td><td>38.1%</td><td>50.0%</td><td>60.0%</td><td>77.3%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td></tr><tr><td></td><td>Date of expertise</td><td>69.4%</td><td>89.2%</td><td>57.1%</td><td>81.0%</td><td>68.2%</td><td>81.4%</td><td>87.5%</td><td>89.2%</td><td>66.7%</td><td>22.2%</td><td>66.7%</td><td>0.0%</td></tr><tr><td></td><td>Date of birth</td><td>96.5%</td><td>96.5%</td><td>96.2%</td><td>97.0%</td><td>98.3%</td><td>95.7%</td><td>96.7%</td><td>93.9%</td><td></td><td>N/A</td><td></td><td></td></tr></tbody></table>
<table><caption>Table 9. *Cont.*</caption><thead><tr><th rowspan="2">Group of Learning</th><th rowspan="2">Label</th><th colspan="4">Tribunals</th><th colspan="4">Appellate Courts</th><th colspan="4">Court of Cassation</th></tr><tr><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th></tr></thead><tbody><tr><td rowspan="3">3</td><td>Plaintiff (pseudon.)</td><td>63.2%</td><td>82.3%</td><td>77.4%</td><td>80.4%</td><td>54.8%</td><td>67.8%</td><td>72.1%</td><td>42.4%</td><td>53.7%</td><td>64.6%</td><td>88.9%</td><td>79.6%</td></tr><tr><td>Defendant (pseudon.)</td><td>73.6%</td><td>77.5%</td><td>85.5%</td><td>83.7%</td><td>52.0%</td><td>66.1%</td><td>76.6%</td><td>60.7%</td><td>50.4%</td><td>69.4%</td><td>90.5%</td><td>83.0%</td></tr><tr><td>Losing party (pseudon.)</td><td>49.6%</td><td>63.1%</td><td>75.8%</td><td>68.7%</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td></tr><tr><td>4</td><td>Legal citation</td><td>91.6%</td><td>90.4%</td><td>92.7%</td><td>91.5%</td><td>93.2%</td><td>88.4%</td><td>91.8%</td><td>88.5%</td><td>82.9%</td><td>78.6%</td><td>81.7%</td><td>81.0%</td></tr><tr><td rowspan="2">5</td><td>Non-recoverable costs</td><td>84.5%</td><td>86.7%</td><td>88.0%</td><td>92.0%</td><td>78.7%</td><td>83.6%</td><td>84.7%</td><td>35.3%</td><td>100.0%</td><td>80.0%</td><td>0.0%</td><td>50.0%</td></tr><tr><td>Costs of instance</td><td>73.2%</td><td>69.0%</td><td>0.0%</td><td>69.2%</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td></tr><tr><td rowspan="2">6</td><td>Jurisdiction</td><td>94.5%</td><td>93.5%</td><td>91.3%</td><td>92.1%</td><td>95.0%</td><td>94.2%</td><td>91.3%</td><td>93.2%</td><td>97.6%</td><td>95.0%</td><td>92.1%</td><td>91.8%</td></tr><tr><td>Chamber</td><td>94.9%</td><td>93.1%</td><td>92.1%</td><td>93.8%</td><td>87.9%</td><td>89.3%</td><td>98.2%</td><td>95.3%</td><td>95.3%</td><td>89.1%</td><td>86.4%</td><td>90.5%</td></tr><tr><td rowspan="5">7</td><td>Case-law citation</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>25.0%</td><td>54.5%</td><td>20.0%</td><td>36.4%</td><td>40.0%</td><td>50.0%</td><td>26.7%</td><td>40.0%</td></tr><tr><td>Type of litigation</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td><td>96.0%</td><td>100.0%</td><td>87.9%</td><td>89.4%</td></tr><tr><td>For further account</td><td>93.3%</td><td>88.9%</td><td>100.0%</td><td>100.0%</td><td>79.2%</td><td>75.9%</td><td>76.7%</td><td>74.4%</td><td></td><td>N/A</td><td></td><td></td></tr><tr><td>Outcome</td><td></td><td>N/A</td><td></td><td></td><td>89.0%</td><td>88.1%</td><td>91.7%</td><td>93.5%</td><td>96.5%</td><td>96.1%</td><td>95.3%</td><td>96.8%</td></tr><tr><td>Type of case</td><td>70.6%</td><td>58.5%</td><td>28.6%</td><td>84.2%</td><td></td><td>N/A</td><td></td><td></td><td>N/A</td><td></td><td></td><td></td></tr></tbody></table>

Flair still ranks highest in a majority of labels, but not in all of them. It comes on top in neither degree for legal citations. DeLFT outperforms it for the date of birth and the outcome, and CRF for the type of case. DeLFT performs equally well globally for tribunals and appellate courts. If Flair maintains a slight advantage over DeLFT, it is at the cost of significantly longer training times. CRF is the best for legal citations in appellate and first-instance cases and still provides excellent results for other labels, as [Ngompe, 2019] already showed, especially with respect to its learning time.

Results are fairly good: five from those models give a quality higher than 95%. Only one gives a quality below 75%, and it involves parties (Group 3, appellate courts).

## 4. Conclusions and Next Steps

### 4.1. Conclusions

Our best overall performance was 87.5%, with Flair. It showed that building a database of proceedings data was feasible despite the variety of label types and the numerical weights of their annotations. In principle, results will be even better once the civil-justice open data program has been completed, allowing for a fully random sampling process to create the right-fitting, statistically representative training dataset.

It was not clear whether models built on the basis of a specific jurisdictional degree provided higher performance outcomes. A model dedicated to identifiers can extract them with a quality ranging from 90% to 98% (Table 10). Judicial dates can be extracted with a quality between 60% (date of injunction for appellate cases) and 96% (date of appellate case for Cassation). Jurisdictions and chambers obtained a quality near 95%. Scores of parties ranged from 72% to 90%, depending on the jurisdictional degree, and a preliminary rules-based propagation barely enhanced them.
<table><caption>Table 9. *Cont.*</caption><thead><tr><th rowspan="2">Group of Learning</th><th rowspan="2">Label</th><th colspan="4">Tribunals</th><th colspan="4">Appellate Courts</th><th colspan="4">Court of Cassation</th></tr><tr><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th><th>CRF-PA</th><th>Spacy</th><th>Flair</th><th>Delft</th></tr></thead><tbody><tr><td rowspan="3">3</td><td>Plaintiff (pseudon.)</td><td>63.2%</td><td>82.3%</td><td>77.4%</td><td>80.4%</td><td>54.8%</td><td>67.8%</td><td>72.1%</td><td>42.4%</td><td>53.7%</td><td>64.6%</td><td>88.9%</td><td>79.6%</td></tr><tr><td>Defendant (pseudon.)</td><td>73.6%</td><td>77.5%</td><td>85.5%</td><td>83.7%</td><td>52.0%</td><td>66.1%</td><td>76.6%</td><td>60.7%</td><td>50.4%</td><td>69.4%</td><td>90.5%</td><td>83.0%</td></tr><tr><td>Losing party (pseudon.)</td><td>49.6%</td><td>63.1%</td><td>75.8%</td><td>68.7%</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td></tr><tr><td>4</td><td>Legal citation</td><td>91.6%</td><td>90.4%</td><td>92.7%</td><td>91.5%</td><td>93.2%</td><td>88.4%</td><td>91.8%</td><td>88.5%</td><td>82.9%</td><td>78.6%</td><td>81.7%</td><td>81.0%</td></tr><tr><td rowspan="2">5</td><td>Non-recoverable costs</td><td>84.5%</td><td>86.7%</td><td>88.0%</td><td>92.0%</td><td>78.7%</td><td>83.6%</td><td>84.7%</td><td>35.3%</td><td>100.0%</td><td>80.0%</td><td>0.0%</td><td>50.0%</td></tr><tr><td>Costs of instance</td><td>73.2%</td><td>69.0%</td><td>0.0%</td><td>69.2%</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td></tr><tr><td rowspan="2">6</td><td>Jurisdiction</td><td>94.5%</td><td>93.5%</td><td>91.3%</td><td>92.1%</td><td>95.0%</td><td>94.2%</td><td>91.3%</td><td>93.2%</td><td>97.6%</td><td>95.0%</td><td>92.1%</td><td>91.8%</td></tr><tr><td>Chamber</td><td>94.9%</td><td>93.1%</td><td>92.1%</td><td>93.8%</td><td>87.9%</td><td>89.3%</td><td>98.2%</td><td>95.3%</td><td>95.3%</td><td>89.1%</td><td>86.4%</td><td>90.5%</td></tr><tr><td rowspan="5">7</td><td>Case-law citation</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>0.0%</td><td>25.0%</td><td>54.5%</td><td>20.0%</td><td>36.4%</td><td>40.0%</td><td>50.0%</td><td>26.7%</td><td>40.0%</td></tr><tr><td>Type of litigation</td><td></td><td>N/A</td><td></td><td></td><td></td><td>N/A</td><td></td><td></td><td>96.0%</td><td>100.0%</td><td>87.9%</td><td>89.4%</td></tr><tr><td>For further account</td><td>93.3%</td><td>88.9%</td><td>100.0%</td><td>100.0%</td><td>79.2%</td><td>75.9%</td><td>76.7%</td><td>74.4%</td><td></td><td>N/A</td><td></td><td></td></tr><tr><td>Outcome</td><td></td><td>N/A</td><td></td><td></td><td>89.0%</td><td>88.1%</td><td>91.7%</td><td>93.5%</td><td>96.5%</td><td>96.1%</td><td>95.3%</td><td>96.8%</td></tr><tr><td>Type of case</td><td>70.6%</td><td>58.5%</td><td>28.6%</td><td>84.2%</td><td></td><td>N/A</td><td></td><td></td><td>N/A</td><td></td><td></td><td></td></tr></tbody></table>

Flair still ranks highest in a majority of labels, but not in all of them. It comes on top in neither degree for legal citations. DeLFT outperforms it for the date of birth and the outcome, and CRF for the type of case. DeLFT performs equally well globally for tribunals and appellate courts. If Flair maintains a slight advantage over DeLFT, it is at the cost of significantly longer training times. CRF is the best for legal citations in appellate and first-instance cases and still provides excellent results for other labels, as [Ngompe, 2019] already showed, especially with respect to its learning time.

Results are fairly good: five from those models give a quality higher than 95%. Only one gives a quality below 75%, and it involves parties (Group 3, appellate courts).

## 4. Conclusions and Next Steps

### 4.1. Conclusions

Our best overall performance was 87.5%, with Flair. It showed that building a database of proceedings data was feasible despite the variety of label types and the numerical weights of their annotations. In principle, results will be even better once the civil-justice open data program has been completed, allowing for a fully random sampling process to create the right-fitting, statistically representative training dataset.

It was not clear whether models built on the basis of a specific jurisdictional degree provided higher performance outcomes. A model dedicated to identifiers can extract them with a quality ranging from 90% to 98% (Table 10). Judicial dates can be extracted with a quality between 60% (date of injunction for appellate cases) and 96% (date of appellate case for Cassation). Jurisdictions and chambers obtained a quality near 95%. Scores of parties ranged from 72% to 90%, depending on the jurisdictional degree, and a preliminary rules-based propagation barely enhanced them.
<table><caption>**Table 10.** Model mix (first run).</caption><thead><tr><th colspan="2">Full Dataset</th><th colspan="2">Tribunals</th><th colspan="2">Courts of Appeal</th><th colspan="2">Supreme Court</th></tr><tr><th>Group</th><th>Best Model</th><th>F1</th><th>Best Model</th><th>F1</th><th>Best Model</th><th>F1</th><th>Best Model</th><th>F1</th></tr></thead><tbody><tr><td>1</td><td>Flair</td><td>95.9%</td><td>Flair</td><td>93.4%</td><td>Flair</td><td>96.9%</td><td>Flair</td><td>93.4%</td></tr><tr><td>2</td><td>Flair</td><td>82.1%</td><td>Flair</td><td>89.0%</td><td>Flair</td><td>89.6%</td><td>Flair</td><td>89.0%</td></tr><tr><td>3</td><td>Flair</td><td>81.5%</td><td>Flair</td><td>74.4%</td><td>Flair</td><td>89.7%</td><td>Flair</td><td>74.4%</td></tr><tr><td>4</td><td>Flair</td><td>92.7%</td><td>CRF-pa</td><td>93.2%</td><td>CRF-pa</td><td>82.9%</td><td>CRF-pa</td><td>93.2%</td></tr><tr><td>5</td><td>Delft</td><td>80.6%</td><td>Flair</td><td>84.7%</td><td>CRF-pa</td><td>100.0%</td><td>CRF-pa</td><td>84.7%</td></tr><tr><td>6</td><td>CRF-pa</td><td>94.7%</td><td>Flair</td><td>94.8%</td><td>CRF-pa</td><td>96.8%</td><td>CRF-pa</td><td>94.8%</td></tr><tr><td>7</td><td>Delft</td><td>92.1%</td><td>Delft</td><td>84.2%</td><td>Spacy</td><td>96.8%</td><td>Spacy</td><td>84.2%</td></tr></tbody></table>

We obtained poor results on the labels “type of case” and “losing party” used for first-degree cases. This may reflect confusing the “losing party” label with the labels “plaintiff” and “defendant”—like that between jurisdiction and referring jurisdiction, seen above. Additional research will have to be conducted to determine the outcome for first-instance cases, possibly using a taxonomy of case types. On the other hand, we obtained very good results from the outcome on appellate courts (94.7%) and on Cassation (96.8%), possibly because this outcome was often treated as metadata and embedded in the text header. Contrary to most labels, DeLFT seemed to be the best-performing algorithm for it.

Our best F1 score (87.5%) was achieved thanks to “brute force”: some 27,000 annotations and one GPU. This is not optimal. Machine-learning practitioners seek the right balance between quality, annotation workload and computing power requirements.

The type of label and the numerical weight of annotations in each label appear to be key quality factors—and more clearly so than the jurisdictional degree and the law area: all date annotations should be trained together, as well as all amounts and annotations, to mitigate the risk of conflicting annotations. Another common property is the volume of annotations: jurisdictions and chambers, for instance, can be trained together because their respective numbers for annotations are roughly similar. Conversely, case-law references should be trained separately from legal citations, because their occurrences are comparatively much less frequent: a much larger dataset is required to reach comparable quality.

## 4.2. Next Steps

As already implied above, another step could be to complement the dataset with cases from other areas of civil law, to assess whether this is a factor of learning quality. A second one would be to reproduce the experimentation on a dataset of French criminal justice or administrative justice cases.

A third one could be to reapply to this dataset and this set of 26 labels the experiment of [27], who compared performances of a CamemBERT model refined in legal content (Judicial CamemBERT) with an ensemble method.

A fourth one would be to test a pretraining using the JuriBERT model.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data are available on https://github.com/Bruno-Mathis/Court-decisions.

**Conflicts of Interest:** The author declares no conflict of interest.

## References

1. Aletras, N.; Tsarapatsanis, D.; Preotioc-Petro, D.; Lampos, V. Predicting judicial decisions of the European Court of Human Rights: A Natural Language Processing perspective. *Peer J. Comput. Sci.* **2016**, *2*, e93. [CrossRef]
<table><caption>**Table 10.** Model mix (first run).</caption><thead><tr><th colspan="2">Full Dataset</th><th colspan="2">Tribunals</th><th colspan="2">Courts of Appeal</th><th colspan="2">Supreme Court</th></tr><tr><th>Group</th><th>Best Model</th><th>F1</th><th>Best Model</th><th>F1</th><th>Best Model</th><th>F1</th></tr></thead><tbody><tr><td>1</td><td>Flair</td><td>95.9%</td><td>Flair</td><td>93.4%</td><td>Flair</td><td>96.9%</td></tr><tr><td>2</td><td>Flair</td><td>82.1%</td><td>Flair</td><td>89.0%</td><td>Flair</td><td>89.6%</td></tr><tr><td>3</td><td>Flair</td><td>81.5%</td><td>Flair</td><td>74.4%</td><td>Flair</td><td>89.7%</td></tr><tr><td>4</td><td>Flair</td><td>92.7%</td><td>CRF-pa</td><td>93.2%</td><td>CRF-pa</td><td>82.9%</td></tr><tr><td>5</td><td>Delft</td><td>80.6%</td><td>Flair</td><td>84.7%</td><td>CRF-pa</td><td>100.0%</td></tr><tr><td>6</td><td>CRF-pa</td><td>94.7%</td><td>Flair</td><td>94.8%</td><td>CRF-pa</td><td>96.8%</td></tr><tr><td>7</td><td>Delft</td><td>92.1%</td><td>Delft</td><td>84.2%</td><td>Spacy</td><td>96.8%</td></tr></tbody></table>

We obtained poor results on the labels “type of case” and “losing party” used for first-degree cases. This may reflect confusing the “losing party” label with the labels “plaintiff” and “defendant”—like that between jurisdiction and referring jurisdiction, seen above. Additional research will have to be conducted to determine the outcome for first-instance cases, possibly using a taxonomy of case types. On the other hand, we obtained very good results from the outcome on appellate courts (94.7%) and on Cassation (96.8%), possibly because this outcome was often treated as metadata and embedded in the text header. Contrary to most labels, DeLFT seemed to be the best-performing algorithm for it.

Our best F1 score (87.5%) was achieved thanks to “brute force”: some 27,000 annotations and one GPU. This is not optimal. Machine-learning practitioners seek the right balance between quality, annotation workload and computing power requirements.

The type of label and the numerical weight of annotations in each label appear to be key quality factors—and more clearly so than the jurisdictional degree and the law area: all date annotations should be trained together, as well as all amounts and annotations, to mitigate the risk of conflicting annotations. Another common property is the volume of annotations: jurisdictions and chambers, for instance, can be trained together because their respective numbers for annotations are roughly similar. Conversely, case-law references should be trained separately from legal citations, because their occurrences are comparatively much less frequent: a much larger dataset is required to reach comparable quality.

## 4.2. Next Steps

As already implied above, another step could be to complement the dataset with cases from other areas of civil law, to assess whether this is a factor of learning quality. A second one would be to reproduce the experimentation on a dataset of French criminal justice or administrative justice cases.

A third one could be to reapply to this dataset and this set of 26 labels the experiment of [27], who compared performances of a CamemBERT model refined in legal content (Judicial CamemBERT) with an ensemble method.

A fourth one would be to test a pretraining using the JuriBERT model.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data are available on https://github.com/Bruno-Mathis/Court-decisions.

**Conflicts of Interest:** The author declares no conflict of interest.

## References

1. Aletras, N.; Tsarapatsanis, D.; Preotioc-Petro, D.; Lampos, V. Predicting judicial decisions of the European Court of Human Rights: A Natural Language Processing perspective. *Peer J. Comput. Sci.* **2016**, *2*, e93. [CrossRef]
2. Medvedeva, M.; Vols, M.; Wieling, M. Using machine learning to predict decisions of the European Court of Human Rights. *Artif. Intell. Law* **2020**, *28*, 237–266. [CrossRef]

3. Chalkidis, I.; Fergadiotis, M.; Malakasiotis, P.; Aletras, N.; Androutsopoulos, I. Neural Legal Judgment Prediction in English. *arXiv* **2019**, arXiv:1906.02059.

4. Filtz, E.; Navas-Loro, M.; Santos, C.; Polleres, A.; Kirrane, S. Events Matter: Extraction of Events from Court Decisions. In *Legal Knowledge and Information Systems*; IOS Press: Amsterdam, The Netherlands, 2020; ISBN 978-1-64368-151-1.

5. Katz, D.M.; Bommarito, M.J.; Blackman, J. A general approach for predicting the behavior of the Supreme Court of the United States. *PLoS ONE* **2017**, *12*, e0174698. [CrossRef] [PubMed]

6. Şulea, O.-M.; Zampieri, M.; Vela, M.; van Genabith, J. Predicting the Law Area and Decisions of French Supreme Court Cases. *arXiv* **2017**, arXiv:1708.01681.

7. Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. *arXiv* **2018**, arXiv:1810.04805.

8. Chalkidis, I.; Fergadiotis, M.; Malakasiotis, P.; Aletras, N.; Androutsopoulos, I. LEGAL-BERT: The Muppets straight out of Law School. In *Findings of the Association for Computational Linguistics: EMNLP 2020*; Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 2898–2904.

9. Cardellino, C.; Teruel, M.; Alonso Alemany, L.; Villata, S. A Low-cost, High coverage Legal Named Entity Recognizer, Classifier And Linker. In Proceedings of the 16th edition of the International Conference on Artificial Intelligence and Law, London, UK, 12–16 June 2017; pp. 9–18.

10. Fernandes, W.P.D.; Silva, L.J.S.; Frajhof, I.Z.; Konder, C.N.; Nasser, R.B.; de Carvalho, G.R.; Almeida, G.F.C.F.; Barbosa, S.D.J.; Lopes, H.C.V. Appellate Court Modifications Extraction for Portuguese. *Artif. Intell. Law* **2019**, *28*, 1–34. [CrossRef]

11. Mandal, A.; Ghosh, K.; Ghosh, S.; Mandal, S. A sequence labeling model for catchphrase identification from legal case documents. *Artif. Intell. Law* **2021**, *30*, 325–358. [CrossRef]

12. Ngompe, G.T.; Harispe, S.; Zambrano, G.; Montmain, J.; Mussard, S. Detecting sections and entities in court decisions using HMM and CRF graphical models. In *Advances in Knowledge Discovery and Management*; Springer: Berlin/Heidelberg, Germany, 2019; pp. 61–86.

13. Barrière, V.; Fouret, A. May I Check Again? A simple but efficient way to generate and use contextual dictionaries for Named Entity Recognition. Application to French Legal Texts. *arXiv* **2019**, arXiv:1909.03453.

14. Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. *Neural Comput.* **1997**, *9*, 1735–1780. [CrossRef] [PubMed]

15. Martin, L.; Muller, B.; Ortiz Suárez, P.J.; Dupont, Y.; Romary, L.; de La Clergerie, É.V.; Seddah, D.; Sagot, B. CamemBERT: A Tasty French Language Model. *arXiv* **2020**, arXiv:1911.03894v3.

16. Douka, S.; Abdine, H.; Vazirgiannis, M.; Hamdani, R.E.; Restrepo, D. Juribert: A masked-language model adaptation for French legal text. *arXiv* **2021**, arXiv:2110.01485.

17. Benesty, M. Why We Switched from Spacy to Flair to Anonymize French Case Law. Available online: *towardsdatascience.com* (accessed on 4 December 2022).

18. Xiao, R. Corpus Creation. In *Handbook of Natural Language Processing*, 2nd ed.; Indurkhya, N., Damerau, F.J., Eds.; Chapman and Hall: London, UK, 2020; pp. 146–165.

19. Miribel, A.; Chavallard, P. Structuring Legal Documents with Deep Learning. 2019. Available online: https://medium.com/doctrine/structuring-legal-documents-with-deep-learning-4ad9b03fb19 (accessed on 4 December 2022).

20. Boniol, P.; Panagopoulos, G.; Xypolopoulos, C.; Rajaa El Hamdani, R.; Restrepo Amariles, D.; Vazirgiannis, M. Performance in the Courtroom: Automated Processing and Visualization of Appeal Court Decisions in France. *arXiv* **2020**, arXiv:2006.06251.

21. Geissler, S. The Kairntech Sherpa—An ML Platform and API for the Enrichment of (not only) Scientific Content. In Proceedings of the 1st International Workshop on Language Technology Platforms, Language Resources and Evaluation Conference, Marseille, France, 11–16 May 2020; pp. 54–58.

22. Lafferty, J.; McCallum, A.; Pereira, F. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, San Francisco, CA, USA, 28 June–1 July 2001; pp. 282–289.

23. Akbik, A.; Bergman, T.; Blythe, D.; Rasul, K.; Schweter, S.; Vollgraf, R. FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (demonstrations), Minneapolis, MN, USA, 2–7 July 2019; pp. 54–59.

24. Lopez, P. Deep Learning Framework for Text. Available online: https://github.com/kermitt2/delft (accessed on 4 December 2022).

25. Huang, Z.; Xu, W.; Yu, K. Bidirectional LSTM-CRF models for sequence tagging. *arXiv* **2015**, arXiv:150801991.

26. Peters, M.E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; Zettlemoyer, L. Deep contextualized word representations. *arXiv* **2018**, arXiv:1802.05365.

27. Mahmoudi, S.-A.; Condevaux, C.; Mathis, B.; Zambrano, G.; Mussard, S. NER sur décisions judiciaires françaises: CamemBERT Judiciaire ou méthode ensembliste? In Proceedings of the Extraction et Gestion des connaissances, Blois, France, 24–28 January 2022.
2. Medvedeva, M.; Vols, M.; Wieling, M. Using machine learning to predict decisions of the European Court of Human Rights. Artif. Intell. Law **2020**, *28*, 237-266. [CrossRef]

3. Chalkidis, I.; Fergadiotis, M.; Malakasiotis, P.; Aletras, N.; Androutsopoulos, I. Neural Legal Judgment Prediction in English. arXiv **2019**, arXiv:1906.02059.

4. Filtz, E.; Navas-Loro, M.; Santos, C.; Polleres, A.; Kirrane, S. Events Matter: Extraction of Events from Court Decisions. In *Legal Knowledge and Information Systems*; IOS Press: Amsterdam, The Netherlands, 2020; ISBN 978-1-64368-151-1.

5. Katz, D.M.; Bommarito, M.J.; Blackman, J. A general approach for predicting the behavior of the Supreme Court of the United States. *PLoS ONE* **2017**, *12*, e0174698. [CrossRef] [PubMed]

6. Şulea, O.-M.; Zampieri, M.; Vela, M.; van Genabith, J. Predicting the Law Area and Decisions of French Supreme Court Cases. arXiv **2017**, arXiv:1708.01681.

7. Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv **2018**, arXiv:1810.04805.

8. Chalkidis, I.; Fergadiotis, M.; Malakasiotis, P.; Aletras, N.; Androutsopoulos, I. LEGAL-BERT: The Muppets straight out of Law School. In *Findings of the Association for Computational Linguistics: EMNLP 2020*; Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 2898-2904.

9. Cardellino, C.; Teruel, M.; Alonso Alemany, L.; Villata, S. A Low-cost, High coverage Legal Named Entity Recognizer, Classifier And Linker. In Proceedings of the 16th edition of the International Conference on Artificial Intelligence and Law, London, UK, 12-16 June 2017; pp. 9-18.

10. Fernandes, W.P.D.; Silva, L.J.S.; Frajhof, I.Z.; Konder, C.N.; Nasser, R.B.; de Carvalho, G.R.; Almeida, G.F.C.F.; Barbosa, S.D.J.; Lopes, H.C.V. Appellate Court Modifications Extraction for Portuguese. Artif. Intell. Law **2019**, *28*, 1-34. [CrossRef]

11. Mandal, A.; Ghosh, K.; Ghosh, S.; Mandal, S. A sequence labeling model for catchphrase identification from legal case documents. Artif. Intell. Law **2021**, *30*, 325-358. [CrossRef]

12. Ngompe, G.T.; Harispe, S.; Zambrano, G.; Montmain, J.; Mussard, S. Detecting sections and entities in court decisions using HMM and CRF graphical models. In *Advances in Knowledge Discovery and Management*; Springer: Berlin/Heidelberg, Germany, 2019; pp. 61-86.

13. Barrière, V.; Fouret, A. May I Check Again? A simple but efficient way to generate and use contextual dictionaries for Named Entity Recognition. Application to French Legal Texts. arXiv **2019**, arXiv:1909.03453.

14. Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput. **1997**, *9*, 1735-1780. [CrossRef] [PubMed]

15. Martin, L.; Muller, B.; Ortiz Suárez, P.J.; Dupont, Y.; Romary, L.; de La Clergerie, É.V.; Seddah, D.; Sagot, B. CamemBERT: A Tasty French Language Model. arXiv **2020**, arXiv:1911.03894v3.

16. Douka, S.; Abdine, H.; Vazirgiannis, M.; Hamdani, R.E.; Restrepo, D. Juribert: A masked-language model adaptation for French legal text. arXiv **2021**, arXiv:2110.01485.

17. Benesty, M. Why We Switched from Spacy to Flair to Anonymize French Case Law. Available online: [towardsdatascience.com](https://towardsdatascience.com) (accessed on 4 December 2022).

18. Xiao, R. Corpus Creation. In *Handbook of Natural Language Processing*, 2nd ed.; Indurkhya, N., Damerau, F.J., Eds.; Chapman and Hall: London, UK, 2020; pp. 146-165.

19. Miribel, A.; Chavallard, P. Structuring Legal Documents with Deep Learning. 2019. Available online: [https://medium.com/docrime/structuring-legal-documents-with-deep-learning-4ad9b03fb19](https://medium.com/docrime/structuring-legal-documents-with-deep-learning-4ad9b03fb19) (accessed on 4 December 2022).

20. Boniol, P.; Panagopoulos, G.; Xypolopoulos, C.; Rajaa El Hamdani, R.; Restrepo Amariles, D.; Vazirgiannis, M. Performance in the Courtroom: Automated Processing and Visualization of Appeal Court Decisions in France. arXiv **2020**, arXiv:2006.06251.

21. Geissler, S. The Kairntech Sherpa—An ML Platform and API for the Enrichment of (not only) Scientific Content. In Proceedings of the 1st International Workshop on Language Technology Platforms, Language Resources and Evaluation Conference, Marseille, France, 11–16 May 2020; pp. 54–58.

22. Lafferty, J.; McCallum, A.; Pereira, F. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, San Francisco, CA, USA, 28 June–1 July 2001; pp. 282–289.

23. Akbik, A.; Bergman, T.; Blythe, D.; Rasul, K.; Schweter, S.; Vollgraf, R. FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (demonstrations), Minneapolis, MN, USA, 2–7 July 2019; pp. 54–59.

24. Lopez, P. Deep Learning Framework for Text. Available online: [https://github.com/kermitt2/delft](https://github.com/kermitt2/delft) (accessed on 4 December 2022).

25. Huang, Z.; Xu, W.; Yu, K. Bidirectional LSTM-CRF models for sequence tagging. arXiv **2015**, arXiv:150801991.

26. Peters, M.E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; Zettlemoyer, L. Deep contextualized word representations. arXiv **2018**, arXiv:1802.05365.

27. Mahmoudi, S.-A.; Condevaux, C.; Mathis, B.; Zambrano, G.; Mussard, S. NER sur décisions judiciaires françaises: CamemBERT Judiciaire ou méthode ensembliste? In Proceedings of the Extraction et Gestion des connaissances, Blois, France, 24–28 January 2022.