# How information technology contributes to justice efficiency

Bruno Mathis, Stéphane Mussard

► To cite this version:

Bruno Mathis, Stéphane Mussard. How information technology contributes to justice efficiency. European Journal of Law and Economics, 2025, 60, pp.145 - 172. 10.1007/s10657-025-09853-z. hal-05461109

HAL Id: hal-05461109

https://hal.science/hal-05461109v1

Submitted on 16 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# How information technology contributes to justice efficiency

Bruno Mathis<fnref n="1" />☐, Stéphane Mussard<fnref n="1" />,<fnref n="2" />,<fnref n="3" />☐

Received: 18 February 2025 / Accepted: 9 August 2025 / Published online: 6 September 2025
© The Author(s) 2025

## Abstract

This paper explores how information technology (IT) contributes to judicial efficiency, that is, to what extent investment in IT helps shorten court delays. It uses information collected by the European Commission for the Efficiency of Justice (CEPEJ), a body of the Council of Europe, from the ministries of justice of 16 of its member countries. A part of this survey, conducted every two years by the CEPEJ, focuses on the judicial services information system, which every country describes by filling up checkboxes. The research augments CEPEJ data by building a maturity index from that material. That index, designed as a proxy to IT asset, is then used as an input to a data envelopment analysis (DEA), alongside two types of human assets: judges and other staff. The average court delays, in criminal and civil matters, are used as outputs to the DEA. One first stage of the analysis applies the directional distance function (DDF), a second one applies the Shapley value, so as to weigh each input's contribution to the distance from the efficient frontier. Results suggest IT does indeed contribute to shorter court delays, but also that other factors are at play.

**Keywords** Efficiency · Justice · Judiciary · DEA · Information technology · Benchmark.

<figure><img src="image_2.png" /></figure>

<footnote>Bruno Mathis
brunomathis16@gmail.com</footnote>

<footnote>Stéphane Mussard
stephane.mussard@unimes.fr</footnote>

<footnote n="1">¹ Univ. Nîmes Chrome, Rue du Dr Georges Salan, 30000 Nîmes, France</footnote>

<footnote n="2">² LISER, Esch-sur-Alzette, Luxembourg</footnote>

<footnote n="3">³ Mohammed VI Polytechnic University, Ben Guerir, Morocco</footnote>
# 1 Introduction

Many scholars, like Lewin et al. (1982), Kittelsen and Førsund (1992), Peyrache and Zago (2016), Voigt (2016), and Gomes et al. (2018), have studied the efficiency factors of the judiciary, mostly at court level, of one single country, and through easily quantifiable variables like numbers of judges or office staff. Some of them, especially Elbialy et al. (2011), Yeung and Azevedo (2011), Deyneli (2012), Andrade and Joia (2012), Lorenzani and Lucidi (2014), and Månsson et al. (2022) ponder to what extent information technology (IT) has an effect on judicial efficiency. Such studies are all the more relevant that some countries see their court delay remain high or rise despite an increased investment in the IT of their judicial services.

The European Commission for the efficiency of justice (CEPEJ) has been collecting, every two years, since 2006, raw data from participating countries, so as to help them assess how efficient they are relative to each other and how they make progress over time. The CEPEJ tracks the “disposition time” (DT), that is, the court delay, or average number of days between the filing of a case and its settlement by a judge. The CEPEJ methodology is based on self-assessment and makes little room to the analysis of information systems. Rather than analyzing the role of legacy systems, it focuses on cutting-edge developments. In 2018, the European Commission (EC) joined forces with the CEPEJ to evaluate different features of each participant’s judicial information system. Their common approach does not provide a solution to the problem of quality and comparability of data provided to the CEPEJ by the ministries of justice. Velicogna and Ontanu (2021) warn that outcomes must be received with extreme caution.

The purpose of this paper is twofold. First, assessing the relative contribution of IT, as opposed to that of human assets, to the efficiency of justice. Second, providing some methodological tools to the measurement of judicial efficiency. Like Lewin et al. (1982), Kittelsen and Førsund (1992), Hagstedt and Proos (2008), Yeung and Azevedo (2011), Deyneli (2012), a data envelopment analysis (DEA) model is employed, a model where efficiency comes from maximized output production or minimum input consumption. The directional distance function (DDF) is used, on the steps of Falavigna et al. (2015); Peyrache and Zago (2016) and Månsson et al. (2022). In particular, three methodological innovations are introduced:

* Augmentation of the CEPEJ indicators with a maturity index of the information system of each country's judiciary and use it as a proxy of its IT asset value;

* Choice of the DT as the efficiency indicator instead of the number of resolved cases, used by Schneider (2005), Hagstedt and Proos (2008), Falavigna et al. (2015), Beldowski et al. (2020), Elbialy et al. (2011) and Deyneli (2012); this avoids any uncontrollable variable in the dataset;

* Application of the Shapley value to assess the contribution of each input, instead of the Tobit or Farrell decomposition, used by Deyneli (2012) and Månsson et al. (2022) respectively. In order to provide additional robustness, the Equal Surplus (ES) value is also employed to measure the contribution of each input, in particular that of

<figure><img src="image_3.png" /></figure>
the IT system. The datasets and the python codes for Shapley contributions (as well as DEA and ES) are available (see Appendix 7.1 for code usage).<fnref n="1" />

The study unfolds in three steps: first, an information system maturity index is built from the CEPEJ data. Second a DEA is applied, where human assets (judges and office staff) and IT assets are both inputs, and average DTs in civil and criminal matters, as per CEPEJ definition, are outputs. Third, results given by the DDF are then decomposed into the contributions of each input according to Shapley and ES.

The paper is organized as follows. A review of the literature is first proposed to have a look on the methods to measure efficiency, the type of IT variable and the area among other things (Sect. 2). We then describe the DEA model and how to measure the contribution of judicial IT to justice efficiency, before analyzing the difference between efficiency and effectiveness in the sense of Marciano et al. (2019) (Sect. 3). A new maturity index is proposed, decomposed into specialized sub-indices (Sect. 4). The single index and the set of sub-indices are alternatively applied to a dataset of 13 comparable countries (Sect. 5). Methodological avenues are proposed in the conclusion (Sect. 6).

## 2 Literature review

The literature review builds upon several concepts derived from the methodology discussed in Sect. 3. In particular, it focuses on justice efficiency through three major dimensions: the role of information systems and IT tools, the type of efficiency analyzed (via the DEA method), and the nature of the decision-making units under consideration. These three dimensions are summarized in Table 1.

The efficiency of information systems (IS) is a major matter of research in computer science. Delone and McLean (1992) devised a model as a framework for conceptualizing and operationalizing IS success. Technical debt, an expression initially coined by Cunningham (1992), has been further investigated. Avgeriou et al. (2016) defined it as the result of technical compromises that are expedient in the short term, but that create a technical context that increases complexity and cost in the long term. Rinta-Kahila et al. (2023) mention it is often revealed when the organization aims to replace legacy systems.

The information system is essential to the efficiency of any large organization. In this perspective, Elbialy et al. (2011) and Yeung and Azevedo (2011) study, in unrelated works, the effect of the number of computers installed in courts on the efficiency of the Egyptian and Brazilian judicial systems respectively. Lorenzani and Lucidi (2014) find that fostering investment in what they call “in-court” IT positively affects the efficiency of civil justice. Voigt (2016) cites IT as one of the many factors of judicial efficiency. On their part, Palumbo et al. (2013) find a tenuous link between the budget allocated to the computerization of justice and the length of the trial whereas Gomes et al. (2018) conclude that the direction of causality between investment in technology and court productivity is unclear. In Italy, Mazzocchi et al. (2024) investigate the link between the database managing a type of litigation (SICID for civil and

<footnote n="1">¹ See our github DEA_Shapley Repository.</footnote>
<table><caption>**Table 1** Related efficiency studies in judicial systems</caption><thead><tr><th>Authors</th><th>Area</th><th>Matter of law</th><th>Model</th><th>DMU</th><th>IT as a factor</th><th>Efficiency indicator</th><th>Factor attribution</th></tr></thead><tbody><tr><td>Lewin et al. (1982)</td><td>North Carolina</td><td>Criminal</td><td>DEA</td><td>Tribunal</td><td></td><td>Resolved &amp; pending cases (multiple)</td><td></td></tr><tr><td>Kittel-sen and Førsund (1992)</td><td>Norway</td><td>All</td><td>DEA</td><td>Tribunal</td><td></td><td></td><td></td></tr><tr><td>Schneider (2005)</td><td>Germany</td><td>Labour</td><td>DEA</td><td>Court of appeal</td><td></td><td>Resolved cases</td><td></td></tr><tr><td>Hagstedt and Proos (2008)</td><td>Sweden</td><td></td><td>DEA</td><td>Tribunal</td><td></td><td>Resolved cases</td><td></td></tr><tr><td>Elbially et al. (2011)</td><td>Egypt</td><td>All</td><td>DEA</td><td>Tribunal</td><td>Number of computers</td><td>Resolved cases</td><td>Civil vs criminal</td></tr><tr><td>Yeung and Azevedo (2011)</td><td>Brazil</td><td></td><td></td><td>Tribunal</td><td>Rate of computers</td><td></td><td></td></tr><tr><td>Deyneli (2012)</td><td>CEPEJ</td><td></td><td>DEA</td><td>Country</td><td>Computerization score</td><td>Resolved cases</td><td>Tobit</td></tr><tr><td>Andrade and Joia (2012)</td><td>Brazil</td><td></td><td></td><td>Tribunal</td><td>IT investment</td><td></td><td></td></tr><tr><td>Palumbo et al. (2013)</td><td>35 countries</td><td></td><td></td><td>Country</td><td>IT budget share</td><td>Trial length &amp; resolution rate</td><td></td></tr><tr><td>Lo-renzani and Lucidi (2014)</td><td>CEPEJ</td><td>Civil</td><td></td><td>Country</td><td>IT budget share</td><td>Disposition time &amp; backlog ratio</td><td></td></tr><tr><td>Falavi-gna et al. (2015)</td><td>Italy</td><td>Tax</td><td>DDF</td><td>Tribunal</td><td></td><td>Resolved cases</td><td></td></tr><tr><td>Peyrache and Zago (2016)</td><td>Italy</td><td></td><td>DDF</td><td>Tribunal</td><td></td><td>Resolved cases</td><td>Per court size</td></tr><tr><td>Gomes et al. (2018)</td><td>Brazil</td><td></td><td>Least squares</td><td>Tribunal</td><td>Build &amp; Run IT spending</td><td></td><td></td></tr><tr><td>Beldows-ki et al. (2020)</td><td>Poland</td><td>Commercial</td><td>SFA</td><td>Tribunal</td><td></td><td>Resolved cases</td><td></td></tr><tr><td>Falavi-gna and Ippoliti (2021)</td><td>Italy</td><td>First instance</td><td>DDF</td><td>Tribunal</td><td></td><td>Resolved cases</td><td></td></tr><tr><td>Mâns-son et al. (2022)</td><td>Sweden</td><td></td><td>DDF</td><td>Tribunal</td><td></td><td>Resolved cases</td><td>Farrell decomposition</td></tr></tbody></table>
labour cases, SIECIC for bankruptcy proceedings) and the resulting average DT and found the former is a valid determinant of the latter.

From a methodological perspective, research aiming to understand the performance of courts or the efficiency of judicial systems relies primarily on the DEA method (see Sect. 3 for details). This method, devised by Charnes et al. (1978), measures the efficiency of decision-making units (DMU) based on the interrelation between inputs and outputs. Ruggiero (1996) shows it easily applies to service production, including public services, because it is nonparametric and does not require input price data. For instance, Tapia and Salvador (2022) show that scores, such as consumer-satisfaction opinion, can also be used as inputs or outputs, just as any physical quantity. In the field of justice (court) efficiency, studies have in common to view individual courts as DMU, as per Charnes definition, or aggregate courts at the country level. Lewin et al. (1982) first use DEA in the judicial domain. They apply this technique to criminal superior courts of North Carolina and since then others have used it to other specific jurisdictions or countries, see Lopes and Silva (2024). Yeung and Azevedo (2011) find that the lack of computers is not a sole reason of inefficiency. Deyneli (2012) applies a two-stage DEA to CEPEJ data, where countries are the DMUs, number of judges and office staff are inputs and resolved cases are outputs. In the second stage, DEA scores are regressed with variables affecting the courts by the Tobit regression model. He builds computerization scores on the basis of the rates of implementation in three application areas (judge assistance, judicial administration, communication with parties) declared by countries participating to the CEPEJ benchmark of 2008 (on 2006 data). This computerization index is not included among inputs, but it is used as an explanatory variable in the Tobit regression stage. No impact of computerization is found on the number of resolved cases.

Investigating the specific contribution of IT to court performance is challenging. Many countries have left tribunals in charge of IT spending in the early 1980 or 1990s, leading to duplicate software developments, and now tend to centralize IT at national or federal level, at least for its infrastructure layer. If, for a DEA, human resources and operating budget are still valid discretionary inputs for tribunals, the central judicial administration tends to be the most relevant DMU for IT investment. See Table 1 for a more comprehensive inventory of related researches.

## 3 Methodologies

This Section briefly exposes the directional distance function and its estimation by DEA. Then, the Shapley value and the Equal Surplus are described to understand how to compute the contribution of each feature (input) to the DEA score of the DDF. A discussion is opened to understand the difference between efficiency and effectiveness in the sense of Marciano et al. (2019). Finally, a hypothesis is formulated regarding the use of DT as an output.
## 3.1 Efficiency, DEA scores and contributions

Production technology defines how inputs are transformed into outputs, with the production possibility set representing all feasible input-output combinations. The directional distance function (DDF), introduced by Chambers et al. (1996, 1998), measures inefficiency by evaluating how much inputs need to be reduced or outputs expanded for a decision-making unit (DMU) to reach the efficiency frontier. This approach has been widely applied, including to judicial efficiency (Ippoliti et al., 2015), and can be interpreted as measuring wasted resources or lost output. Some limitations arise when assuming convexity in the technology, which may not fully capture complementarities or substitutability between inputs and outputs (Briec & Mussard, 2020).

DEA provides a non-parametric method to evaluate efficiency based on observed data. Studies such as Falavigna et al. (2015) demonstrate its benefits and limitations in judicial assessments, while Peyrache and Zago (2016) explore its use in decomposing inefficiency into technical, size-related, and reallocation inefficiencies. The DDF can be estimated via DEA using linear programming as follows ($δ$ for the DDF estimation, $x$ for $n$ inputs, $y$ for $m$ outputs, $g$ for the direction, and parameters $θ$ for different returns to scale assumptions):

$$ 
\begin{aligned}
D_T(x, y; g) = & \max \delta \\
& \text{s.t. } x_i - \delta g_i \ge \sum_k \theta_k x_{ik}, \quad \forall i = 1, \dots, n \\
& y_j + \delta g_o \le \sum_k \theta_k y_{jk}, \quad \forall j = 1, \dots, m \\
& \delta, \theta_k \ge 0.
\end{aligned}
$$

The advantage of the DDF relies on the choice of the direction, input-oriented if $g = (g_i, 0)$, output-oriented if $g = (0, g_o)$, or both. The choice of orientation in the DDF affects the efficiency analysis: input-oriented models focus on resource reduction, while output-oriented models emphasize performance improvement without cutting inputs. In contexts where resources are constrained, as in the Italian judiciary, an output-oriented approach is preferable (Peyrache & Zago, 2016). In our analysis, the choice of the direction is $g = (1, 1)$ because each country, once the organization has invested in IT, it can plan to reduce some of its other inputs (judges or non-judges) to reach a better efficiency.

The linear programming can be adapted to a technology satisfying one of the following assumption about returns to scale. The constraint $θ_k \ge 0$ imposes a constant returns to scale assumption (CRS), however other assumptions may be imposed on linear programming:

* $\sum_k \theta_k = 1$ (Banker et al., 1984) allows a variable returns to scale assumption (VRS).

* $\sum_k \theta_k \le 1$ (Seiford & Thrall, 1990) allows a decreasing returns to scale assumption (DRS), which is strictly speaking a non increasing returns to scale assumption).
• $\sum_k \theta_k \ge 1$ (Grosskopf, 1986) allows an increasing returns to scale assumption (IRS), which is a non decreasing returns to scale assumption. In the sequel, the DRS assumption is employed, considering inputs as IT tools. Indeed, as shown in Chambers et al. (1996), under the DRS assumption (convexity of the technology), the directional distance function is dual to the cost function associated with inputs (IT tools). Each additional euro spent on any combination of inputs will shorten the cost (DT) by less and less, since some parts of that time are fixed and cannot sharply decrease by virtue of procedural law as described in Sect. 3.3.

Peyrache and Zago (2012, 2016) first employed the DDF to measure the efficiency of the judicial system at the aggregate level in Italy (aggregating DMUs by region) with a special focus on the optimal size of the courts. They use DEA with variable returns to scale (VRS). In this case, efficiency does not scale proportionally in the sense that small courts might not benefit from the same economies of scale as large ones (with different constraints). Falavigna et al. (2015) also employ the estimation of the DDF (output-oriented) by DEA to measure the efficiency of the Italian tax courts over the period 2009–2011. Contrary to Peyrache and Zago (2016), Falavigna et al. (2015) use a CRS assumption, so that multiplying the inputs (judges, staff, etc.) and the outputs (number of cases resolved) by $\lambda$ imply that efficiency is $\lambda$ times more important than before improvement, a linear assumption. As a key element, the authors model resolution time as an undesirable (bad) output and use a bootstrap method (Simar & Wilson, 2007) to derive consistent statistical inference on DEA scores, see also Falavigna and Ippoliti (2021) for an application on Italian courts with and without bad outputs to control for the budget. In the same manner, Belarouci et al. (2023) use a bad-output DEA model to evaluate how well a conciliator can reduce hearings (intermediate output) while increasing number of cases (final output). Unlike standard DEA, this model prioritizes final outcomes by linking hearings and resolutions (a court is considered efficient only if any increase in resolutions is accompanied by a corresponding increase in hearings).

To assess the impact of each input on efficiency, cooperative game theory concepts such as the Shapley value (Shapley, 1953) and the equal surplus value (ES value) are employed. The Shapley value computes the marginal contribution of each input by averaging its impact across all possible coalitions, as applied in various domains like finance (Mussard & Terraza, 2007) and feature attribution in machine learning (Condevaux et al., 2022). A negative Shapley value indicates that an input reduces inefficiency, making it a key driver of performance improvements. The ES value takes a different approach by summing the direct contribution of an input with the overall efficiency gain evenly distributed among all inputs. This ensures that even inputs with no individual impact can have a nonzero contribution when considered collectively. While the Shapley value focuses on marginal contributions, the ES value highlights the independent role of each input. The use of both methods enhances robustness in identifying the most influential inputs in efficiency assessments.

## 3.2 Efficiency vs. effectiveness: Marciano et al. (2019)

Recently, Marciano et al. (2019) show that a difference must be made between judicial efficiency and effectiveness. Judicial efficiency is the optimal use of resources for
a given result. Judicial effectiveness is the capacity to satisfy demand for justice and thereby solve cases promptly. Effectiveness can be achieved even by a highly inefficient system devoting excessive resources to the judiciary. A high demand for justice may be caused by lax proceedings rules, frivolous suits encouraged by an excessive number of lawyers or inconsistencies within legal norms and cultural factors. Volume-based metrics, such as new or resolved cases, reflect demand for justice, and indeed, these variables are not controlled by the judiciary. As a result, a high volume of new cases causes congestion in the judicial system, suggesting that the average DT, that is, the number of unresolved cases at year-end multiplied by 365 and divided by the number of resolved cases in one year, is an appropriate metric of effectiveness rather than efficiency. However, the disposition time is also to judgment production what the production time is to industry, a key metric of lean manufacturing.

Deyneli (2012), using resolved cases, a demand-linked metric, as an output to a DEA, does provide an efficiency measurement. He shows England, France and Denmark, among 22 countries, with 100% technical efficiency, and recognizes the impact of the great number of cases as an output. However, the DEA is static and tells nothing on the length of the industrial process that is taken to produce outputs. Indeed, contentious civil cases took in 2006 an average of 260 days, in France, and 164 days, in Denmark, to be settled. Denmark and France have the 9th and 19th best delays of such cases among 29 countries (CEPEJ, 2006, p. 154).<fnref n="2" />

The European Convention of Human Rights states, in its article 6 §1, that “everyone is entitled to a fair and public hearing within a reasonable time”. The CEPEJ has been created precisely to help judiciaries devise public policies liable to decrease excessive judicial delays. It adopted the disposition time as an appropriate measure of judicial efficiency (Cappellina, 2017), for civil, criminal and administrative cases respectively, not necessarily in accordance with the economic definition of efficiency, but rather as a measure of effectiveness (Cappellina, 2017).

## 3.3 Court delay, processing time and efficiency

Effectiveness refers to the extent to which the judicial system meets the demand for justice. Indeed, Vereeck and Mühl (2000) highlight the impact of the parties’ behaviour on court delays. The civil trial may be split in four steps: negotiation, procedural preparation, pre-trial, and the trial itself. However, the first phase occurs before court filing and is not included in the DT as per CEPEJ definition. Among the three remaining steps, only the first one is affected by the parties’ procedure-related decisions: both plaintiff and defendant act as ”micro-decision making units”. The rest, pre-trial and trial, include processing time at the court. The parties’ agency is less significant in criminal cases: the suit is born at the prosecutor’s initiative and a smaller part of the court delay may be attributed to the accused and his lawyer or to the civil parties of the criminal proceeding.

Excess demand for justice causes court congestion and translates into an excess waiting time for the parties. But court congestion is not the only factor of court delays. The court delay, which is also in effect the disposition time as per CEPEJ,

<figure><img src="image_4.png" /></figure>

<footnote n="2">² England is absent from this benchmark.</footnote>
includes two other components. One is the minimum time necessary to the trial to be
held in a serene atmosphere, which is often reflected in procedural law provisions:
no judgment could be issued instantaneously, when tempers run high. The second
is processing time, which Falavigna et al. (2015) and Falavigna and Ippoliti (2021)
analyze as a side cost and model as a bad output: pre-trial involves collection of
case-contextual information and determination of laws applicable to the case, both
of which requiring a combination of human and IT resources; trial audience length
itself depends on the case complexity, also determining working time from judicial
staff. We therefore issue the hypothesis that DTs are also suitable as a measure of
efficiency, and therefore as outputs of a DEA. A robustness test will have to confirm
that hypothesis (see Sect. 5.4).

## 4 An augmented CEPEJ dataset

A team of OECD (2013, p. 92) considers that neither the centralized or federal char-
acter of a country, nor its size in terms of population, are determining factors in
central government IT spending. It is therefore possible to use the list of sixteen
countries qualified by the CEPEJ (2010) as comparable in terms of development
level, whatever their size and mode of government, and make use of raw data pro-
vided by each country for that survey.

A first subsection is devoted to the construction of a maturity index, then a second
subsection presents the data to measure efficiency.

4.1 Building an information system maturity index

France (2012) and Austria (2012) declare almost the same amount of 48 M€ ( ques-
tion 6§2) for 2010. The French population is eight times larger than Austria’s, we
then have some reservations about the reliability of the IT spending data. Moreover
CEPEJ budget data cover only the period since 2006. We therefore seek a substitute
to IT spending.

The CEPEJ does not collect the numbers of computers, and inefficiencies persist
today even though most judiciaries have nearly completed the equipment of their
front-office users. Moreover, this variable has lost its explanatory power because
many IT issues now lie at the back end of applications. Therefore, another proxy
must be used. We did not find any such proxy in literature, so we are ready to build
an ad-hoc one.

A proxy is built on CEPEJ countries thanks to the combined survey conducted by
CEPEJ / EU every two years. We use the CEPEJ (2020) evaluation cycle covering
data from 2018. Thirteen sections relate to the evaluation of the judicial information
system, more fine-grained than the three ones available to Deyneli (2012) for 2006
data. The proxy ‘information system maturity index’ is the sum of the responses to
the thirteen questions. It reflects the value of IT assets, not the cumulative spending
which, if higher, would reveal waste.

Austria comes out first and the Netherlands come out last. The latter case is puz-
zling, because the Netherlands may have spent substantial sums in its ICT over time
van den Hoogen et al. (2003) reckon that represented 8% of the justice budget in 2001. This suggests either wasted money or ill-reported information.

The European Commission’s Directorate-General for Justice and Consumers then transformed the collected information into quantifiable data in 2022 for comparison purposes. Its methodology is replicated to the 3 non-EU countries of England/Wales, Norway and Switzerland, see European Commission (2022, Annex 5). The provided raw values are not questioned, for example we disagree with the score of 1 for France on the question of a centralized and interoperable base for civil case management, but accept the resulting total points of 4.8 computed by the EC in the 4th line of our Table 2. A ranking is obtained by adding up together points across all variables.

This maturity index can be broken down by IT feature, into several sub-indices (Type 1 to 4), as shown in Table 3.

Scores should be welcomed with as much caution as input data. It is striking, for instance, that the Netherlands get only 7 points, against Austria with 20.2, on the productivity tools maturity index, while achieving even shorter DTs (110 and 104 days for civil and penal cases respectively, against 138 and 120 for Austria, see Table 4). Should these figures be deemed accurate, they suggest Austria spends far too much in its IT.

These information system maturity indices have their own limitations. They do not provide with a comprehensive representation of the judicial information system. The underlying CEPEJ questionnaire does not include key questions like: What are the shares of IT investment spending committed by jurisdictions and by the ministry? What is the level of database and software technical integration, in particular, to what extent do application systems rely on master data such as personal identification numbers?

## 4.2 Building the dataset to compute efficiency

This does not yet say to what extent the maturity of the information system, among other factors, influences case processing times. A DDF based on CEPEJ data for 2018 is therefore carried out, in which three inputs and two outputs are employed.

In principle, incoming cases can be regarded as input and settled cases as output, but courts have no agency on the number of incoming cases, which reflects national law and cultural patterns. Moreover, definitions of cases vary across countries. However the average case processing time, which ECHR calls the “disposition time” (DT), expressed in number of days, is an adequate metric of judicial production. Our choice of inputs and outputs is therefore as follows.

The inputs are the number of judges per million inhabitants, the number of non-judge justice personnel, again per million inhabitants, and the maturity index of the information system. The two types of populations are separated, because judges are those who settle court cases and need information tools most while other personnel, which are defined as non-judges, have a more intensive use of the information system and are more impacted by productivity tools.

There are two outputs: the average DT of contentious civil matters and that of criminal matters, both in first-instance only. No DT being available for Belgium and

<figure><img src="image_5.png" /></figure>
England/Wales in civil matters and for Greece in criminal matters, these countries
drop out of our sample.

Benchmarking administrative justice is more tricky because of huge differences
in terms of the functional scope and governance. First, Denmark and Norway do not
distinguish administrative decisions at all. Second, the French administrative juris-
dictional order is separate from the judicial order and should count for a separate
DMU. Including a DT for administrative cases would cause the country panel to lose
another 2 members, for lack of data applicable to the DDF, while administrative cases
represent only 10% of the volume of the sample’s aggregate contentious activity. We
therefore do not include the administrative first-instance DT amoung outputs. We
add a development on administratives cases, though, in §5.3. Moreover, as appellate
court cases DTs are correlated to first-instance ones, with also much smaller volumes,
we do not include their DTs either.

In a DEA-type analysis, production must increase with the increase in inputs. As
disposition times are supposed to be reduced with an increase in the three types of
inputs, the theoretical output is then defined as a time gained, by subtracting the
disposition time to 1,000 days (subtracting the disposition time from 10,000 days,
an even less attainable maximum, brings similar results). Accordingly, efficiency is
measured by the relationship between the first three variables (inputs) on the last two
(outputs), see Table 4.

The DDF is computed on three different sets of inputs:

• **Input set 1:** judges and non-judges;

• **Input set 2:** judges, non-judges and our single maturity index;

• **Input set 3:** judges, non-judges and the four maturity sub-indices, shown in Table 3.

As variables are not weighted, parallel computations on the three input sets allows
to look at gaps and check the solidity of our model.

• **Output:** There is one single output set for the three input sets. It is composed of two variables:

1000 – disposition time of civil matters;

1000 – disposition time of criminal matters.

# 5 DEA and input contributions

We present the results of the DEA scores, followed by an analysis of the contributions
of inputs using Shapley and ES methods.

## 5.1 DEA scores

Input sets 1 and 2 allow to show by difference the possible contribution of IT to a
multi-output production efficiency. Sets 2 and 3 are meant to find out what level of
detail of representation of IT best suggests causality to disposition times. Table 5
shows the respective efficiency of the 3 input sets to produce the same output set
(average of civil and criminal disposition times).
<table><caption>**Table 2** Judicial System Features and Scores by Country*</caption><thead><tr><th>Features</th><th>Type (*)</th><th>Ref.</th><th>Austria</th><th>Belgium</th><th>Switzerland</th><th>Germany</th><th>Denmark</th><th>Spain</th><th>Finland</th><th>France</th><th>UK</th><th>Greece</th><th>Italy</th><th>Nether-lands</th><th>Nor-land</th><th>Po-way</th><th>Por-tugal</th><th>Sweden</th></tr></thead><tbody><tr><td>Caselaw</td><td>1</td><td>C</td><td>62-2</td><td>2</td><td>3</td><td>2</td><td>0</td><td>3</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>2</td><td>3</td><td>2</td><td>1</td><td>2</td></tr><tr><td>civil +<br>criminal<br>(1st + 2nd<br>degree)</td><td></td><td></td><td>4-1</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Information<br>through<br>intranet</td><td>1</td><td>C</td><td>62-1</td><td>1</td><td>0</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>1</td><td>1</td><td>1</td><td>0</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Writing aid,<br>dictaphone<br>& speech<br>recognition</td><td>2</td><td>E</td><td>Table 2.8<br>6.1</td><td>0.8</td><td>0</td><td>2.3</td><td>2.3</td><td>2.3</td><td>2</td><td>0.4</td><td>1</td><td>0.8</td><td>1.5</td><td>0</td><td>1</td><td>1.9</td><td>2</td><td>2</td></tr><tr><td>Case<br>management</td><td>2</td><td>E</td><td>Table 6.4<br>6.2</td><td>5.3</td><td>3</td><td>3.3</td><td>5.1</td><td>4.9</td><td>5.6</td><td>4.8</td><td>4</td><td>3.9</td><td>6.1</td><td>4.5</td><td>6.5</td><td>6.3</td><td>6.2</td><td>5.1</td></tr><tr><td>Budget &<br>judicial fees<br>management</td><td>2</td><td>E</td><td>Table 0<br>6.3</td><td>1.5</td><td>1.5</td><td>1.3</td><td>0.4</td><td>1</td><td>1</td><td>2</td><td>1.75</td><td>1</td><td>1.8</td><td>0.5</td><td>2</td><td>2</td><td>1.5</td><td>0.5</td></tr><tr><td>Filing &<br>summons</td><td>2</td><td>E</td><td>Table 7<br>6.5</td><td>2.3</td><td>2.75</td><td>7</td><td>3.6</td><td>7</td><td>7</td><td>3.3</td><td>2</td><td>4.5</td><td>5.9</td><td>0</td><td>4</td><td>3.2</td><td>7</td><td>6.3</td></tr><tr><td>Judicial aid<br>management</td><td>2</td><td>E</td><td>Table 4<br>6.7</td><td>4</td><td>3.75</td><td>4</td><td>1.5</td><td>3</td><td>4</td><td>1.5</td><td>3.75</td><td>0</td><td>4</td><td>2</td><td>2.75</td><td>0</td><td>0</td><td>0</td></tr><tr><td>Interoper-<br>ability of<br>criminal<br>records</td><td>3</td><td>C</td><td>62-6<br>6.6</td><td>3</td><td>3</td><td>0</td><td>0</td><td>2</td><td>2</td><td>3</td><td>2</td><td>0</td><td>3</td><td>1</td><td>1</td><td>3</td><td>3</td><td>1</td></tr><tr><td>Communi-<br>cation with<br>attorneys</td><td>3</td><td>E</td><td>Table 6</td><td>4.35</td><td>4.5</td><td>6</td><td>5.17</td><td>6</td><td>6</td><td>4.64</td><td>4</td><td>3.78</td><td>5.26</td><td>1.42</td><td>4.5</td><td>2.96</td><td>5.8</td><td>6</td></tr></tbody></table>

<figure><img src="image_6.png" /></figure>
<table><caption>Table 2 (continued)</caption><thead><tr><th>Features</th><th>Type (*)</th><th>Ref.</th><th>Austria</th><th>Belgium</th><th>Switzerland</th><th>Germany</th><th>Denmark</th><th>Spain</th><th>Finland</th><th>France</th><th>UK</th><th>Greece</th><th>Italy</th><th>Nether-lands</th><th>Nor-land</th><th>Po-way</th><th>Por-tu-gal</th><th>Swe-den</th></tr></thead><tbody><tr><td>Communication with other professionals</td><td>3</td><td>E</td><td>Table 3<br>6.8</td><td>2.3</td><td>2.25</td><td>3</td><td>3</td><td>2</td><td>2</td><td>0.8</td><td>0.75</td><td>0</td><td>2</td><td>0</td><td>0</td><td>1.5</td><td>2</td><td>2</td></tr><tr><td>Communication with police forces</td><td>3</td><td>C</td><td>64-7<br>1</td><td>1</td><td>0.75</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0.75</td><td>0</td><td>0.5</td><td>0</td><td>0</td><td>0.5</td><td>1</td><td>1</td></tr><tr><td>Workload measurement tool</td><td>4</td><td>E</td><td>Table 4<br>6.4</td><td>0.8</td><td>3.5</td><td>2.5</td><td>2.4</td><td>4</td><td>3.7</td><td>1.7</td><td>1.66</td><td>1.3</td><td>3.3</td><td>0</td><td>3.83</td><td>3.5</td><td>3.3</td><td>4.7</td></tr><tr><td>Project management by IT department</td><td>4</td><td>C</td><td>65-2<br>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>0</td><td>0</td><td>0</td><td>1</td><td>1</td><td>0</td><td>1</td><td>1</td><td>0</td><td>1</td></tr><tr><td>Score</td><td></td><td></td><td>41.2</td><td>29.4</td><td>26</td><td>37.4</td><td>26.5</td><td>38.2</td><td>36.3</td><td>24.1</td><td>23.7</td><td>16.3</td><td>34.4</td><td>13.4</td><td>30.6</td><td>26.9</td><td>34.8</td><td>32.6</td></tr><tr><td>2018 ranking</td><td></td><td></td><td>1</td><td>9</td><td>12</td><td>3</td><td>11</td><td>2</td><td>4</td><td>13</td><td>14</td><td>15</td><td>6</td><td>16</td><td>8</td><td>10</td><td>5</td><td>7</td></tr></tbody></table>

<footnote>* C: CEPEJ, Profils pays, Exercice d'évaluation, Partie 1, édition 2020 (données 2018), see website: E. CEPEJ (2022)</footnote>
<table><caption>**Table 3** Maturity of judicial services information systems in 16 comparable European countries - Summary</caption><thead><tr><th>Type</th><th>1</th><th>2</th><th>3</th><th>4</th><th>Total</th></tr><tr><th>Features</th><th>Information</th><th>Productivity</th><th>Com- muni- cation</th><th>IT proj- ect</th><th></th></tr><tr><th></th><th>Tools</th><th>Tools</th><th>Tools</th><th>Gov- er- nance</th><th></th></tr></thead><tbody><tr><td>Austria (AT)</td><td>3</td><td>20.2</td><td>13</td><td>5</td><td>41.2</td></tr><tr><td>Belgium (BE)</td><td>3</td><td>13.9</td><td>10.65</td><td>1.8</td><td>29.35</td></tr><tr><td>Switzer- land (CH)</td><td>3</td><td>11</td><td>7.5</td><td>4.5</td><td>26</td></tr><tr><td>Germany (DE)</td><td>3</td><td>17.9</td><td>13</td><td>3.5</td><td>37.4</td></tr><tr><td>Denmark (DK)</td><td>1</td><td>12.9</td><td>9.17</td><td>3.4</td><td>26.47</td></tr><tr><td>Spain (ES)</td><td>4</td><td>18.2</td><td>11</td><td>5</td><td>38.2</td></tr><tr><td>Finland (FI)</td><td>2</td><td>19.6</td><td>11</td><td>3.7</td><td>36.3</td></tr><tr><td>France (FR)</td><td>2</td><td>12</td><td>8.44</td><td>1.7</td><td>24.14</td></tr><tr><td>England/ Wales (GB)</td><td>1</td><td>12.5</td><td>7.5</td><td>2.66</td><td>23.66</td></tr><tr><td>Greece (GR)</td><td>0</td><td>10.2</td><td>3.78</td><td>2.3</td><td>16.28</td></tr><tr><td>Italy (IT)</td><td>1</td><td>19.3</td><td>10.76</td><td>3.3</td><td>34.36</td></tr><tr><td>Nether- lands (NL)</td><td>3</td><td>7</td><td>2.42</td><td>1</td><td>13.42</td></tr><tr><td>Norway (NO)</td><td>4</td><td>16.25</td><td>5.5</td><td>4.83</td><td>30.58</td></tr><tr><td>Poland (PL)</td><td>2</td><td>13.4</td><td>7.96</td><td>3.5</td><td>26.86</td></tr><tr><td>Portugal (PT)</td><td>2</td><td>16.7</td><td>11.8</td><td>4.3</td><td>34.8</td></tr><tr><td>Sweden (SE)</td><td>3</td><td>13.9</td><td>10</td><td>5.7</td><td>32.6</td></tr><tr><td>cap</td><td>5</td><td>23</td><td>13</td><td>6</td><td>47</td></tr></tbody></table>

The DEA makes it possible to assess the efficiency of countries relatively with each other at a given time. That the IT factor can move some countries away from the efficient frontier does not mean that IT does not contribute to the shortening of disposition times in absolute terms. Moreover, a strict return to paper in each process - theoretical hypothesis - would mechanically increase the time taken to settle cases.

A principle of law holds that a minimum delay is necessary to ensure quality of judicial decisions. The disposition time of any case is constrained by legal provisions related to civil or criminal proceedings. There is therefore a point at which any additional euro invested in the smartest judicial information system will cause no addi-

<figure><img src="image_7.png" /></figure>
<table><caption>**Table 4** Inputs and outputs of judicial efficiency</caption><thead><tr><th>Data</th><th>Input</th><th>Input</th><th>Input</th><th>Output</th><th>Output</th></tr><tr><th>Variables</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th></tr></thead><tbody><tr><td>AT</td><td>273.3</td><td>562.9</td><td>41.2</td><td>138</td><td>120</td></tr><tr><td>BE</td><td>133.2</td><td>435.1</td><td>29.35</td><td></td><td>188 (6)</td></tr><tr><td>CH</td><td>143.3</td><td>596.9</td><td>26</td><td>111</td><td>100</td></tr><tr><td>DE</td><td>244.8</td><td>651.3</td><td>37.4</td><td>220</td><td>117</td></tr><tr><td>DK</td><td>64.6</td><td>285.2</td><td>26.47</td><td>207</td><td>41</td></tr><tr><td>ES</td><td>115.3</td><td>1013.6</td><td>38.2</td><td>361</td><td>170</td></tr><tr><td>FI</td><td>195.8</td><td>385.9</td><td>36.3</td><td>273</td><td>139</td></tr><tr><td>FR</td><td>108.6</td><td>341.0</td><td>24.14</td><td>420</td><td>399 (6)</td></tr><tr><td>GB</td><td>30.9</td><td>240.1</td><td>23.66</td><td></td><td>75</td></tr><tr><td>GR</td><td>267.6</td><td>389.1</td><td>16.28</td><td>559</td><td></td></tr><tr><td>IT</td><td>116.2</td><td>371.1</td><td>34.36</td><td>527</td><td>361</td></tr><tr><td>NL</td><td>145.9</td><td>433.5</td><td>13.42</td><td>110</td><td>104</td></tr><tr><td>NO</td><td>102.8</td><td>159.7</td><td>24.08</td><td>176</td><td>70</td></tr><tr><td>PL</td><td>254.5</td><td>1058.6</td><td>26.86</td><td>273</td><td>111</td></tr><tr><td>PT</td><td>192.6</td><td>566.1</td><td>34.8</td><td>229</td><td>205</td></tr><tr><td>SE</td><td>119.0</td><td>509.1</td><td>32.6</td><td>166</td><td>151</td></tr></tbody></table>

tional shortening of the disposition time. For this reason, DEA scores are computed with decreasing returns to scale.<fnref n="3" />

Table 5 suggests that IT does contribute to efficiency to the extent every country sees its gap to the efficient frontier reduced when the IT maturity index is included among inputs. The effect is even greater when IT is represented through four proxy variables instead of one, though this does not outline, at this stage, the contribution of IT in the efficiency gain.

Switzerland, Denmark, the Netherlands and Norway have a distance to the efficient frontier very close to zero. IT does not make a visible difference to their efficiency. In the Netherlands, the apparent lack of maturity of the judicial information system does not prevent tribunals to deliver short disposition times: on the face of it, the efficiency of the Dutch judiciary derives from a small IT resource. But efficiency does not capture the eventual waste factor. For other countries, IT does reduce their distance to the efficient frontier.

<footnote n="1">¹ Judges per million inhabitants, CEPEJ, Country profiles, Evaluation cycle, Part. 1, 2020 edition (2018 data), question 046 / question 001</footnote>

<footnote n="2">² Non-judges per million inhabitants, ibid., question 052 / question 001</footnote>

<footnote n="3">³ Single judicial services information system maturity index</footnote>

<footnote n="4">⁴ Disposition time for contentious civil, commercial and administrative cases, CEPEJ, Systèmes judiciaires européens. Rapport d'évaluation de la CEPEJ (données 2018), Partie 1, 2020, p. 115</footnote>

<footnote n="5">⁵ Disposition time for criminal cases, ibid., p. 128</footnote>

<footnote n="6">⁶ 2020 data. Empty cell means data was not supplied by the country</footnote>

<footnote n="3">³ Results from other types of returns are available in appendix, in Table 13 for a single information system maturity index and Table 14 for sub-indices.</footnote>
<table><caption>**Table 5** Technical efficiency—DDF and DRS—distances</caption><thead><tr><th></th><th>Input set 1</th><th>Input set 2</th><th>Input set 3</th></tr></thead><tbody><tr><td>Judges/million inhabitants</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Non-judges/million inhabitants</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Single information system maturity index</td><td></td><td>✓</td><td></td></tr><tr><td>Information tools maturity index</td><td></td><td></td><td>✓</td></tr><tr><td>Productivity tools maturity index</td><td></td><td></td><td>✓</td></tr><tr><td>Communication tools maturity index</td><td></td><td></td><td>✓</td></tr><tr><td>IT project governance maturity index</td><td></td><td></td><td>✓</td></tr><tr><td>Austria</td><td>22.177</td><td>21.614</td><td>1.287e-11</td></tr><tr><td>Switzerland</td><td>7.410e-10</td><td>6.547e-12</td><td>6.753e-11</td></tr><tr><td>Germany</td><td>51.864</td><td>22.096</td><td>2.008</td></tr><tr><td>Denmark</td><td>2.733e-10</td><td>6.551e-12</td><td>8.627e-12</td></tr><tr><td>Spain</td><td>55.634</td><td>19.799</td><td>3.472</td></tr><tr><td>Finland</td><td>85.298</td><td>19.669</td><td>1.361</td></tr><tr><td>France</td><td>56.822</td><td>14.997</td><td>0.072</td></tr><tr><td>Italy</td><td>68.567</td><td>22.585</td><td>3.330</td></tr><tr><td>Netherlands</td><td>2.453e-09</td><td>5.957e-12</td><td>9.374e-12</td></tr><tr><td>Norway</td><td>3.040e-10</td><td>6.988e-12</td><td>9.790e-12</td></tr><tr><td>Poland</td><td>68.851</td><td>12.335</td><td>1.695</td></tr><tr><td>Portugal</td><td>80.423</td><td>22.555</td><td>1.527</td></tr><tr><td>Sweden</td><td>11.615</td><td>11.233</td><td>1.704</td></tr></tbody></table>

Caution is advised. Tables 2 and 3 are not enough to explain efficiency gaps. Outgoing rankings do not capture the time taken for the IT investment to mature and have the expected effects on efficiency, an issue already raised by Andrade and Joia (2012). Additionally, going from input set 1 to input set 2 (or 3) does not capture all possible effects of IT on efficiency. IT must be combined with all input combinations, as in Shapley and ES, to bring out the full effect of IT on efficiency.

## 5.2 Input contributions: discussion

In most countries, IT investment does offset human resources related inefficiencies (downward green bars in Fig. 1). However, in Austria, Norway and Sweden, IT investment also adds distance to the efficient frontier, probably to the extent disposition times are already satisfactory, suggesting that these countries consume IT somewhat in excess for that outcome. Sweden is unique in that its number of judges improves efficiency. Unlike Austria and Norway, its number of judges per million inhabitants is below average. Combined with a good maturity score, this suggests that the country has been able to leverage its IT investment to keep the number of judges low.

France and Italy show a similar pattern: their proportion of judges is slightly beneficial, but that of non-judges strongly detrimental, to efficiency. IT spending partly offsets an overall human inefficiency.

<figure><img src="image_8.png" /></figure>
<figure><img src="image_9.png" /><figcaption>Fig. 1 Input contribution to the DDF: Single index, DRS, Shapley</figcaption></figure>

Switzerland, Denmark and the Netherlands show no input factor at all. They are at their optimum. This is odd for Switzerland where the judicial information system is managed at canton level, and is therefore highly fragmented. The need arose to devise a countrywide system called Justitia 4.0, currently under development, suggesting the actual IT assets do not contribute that much to judicial efficiency after all.

Norway is the only country whose relative number of non-judges positively contributes to efficiency. This number is also the lowest, by far, in the panel. Though we do not know how Norway can consume so few resources (simple procedure? short legal delays?), this does explain why this variable contributes positively to Norway's efficiency, and negatively to the others.

Results of the Equal Surplus method are displayed in Fig. 2. The method finds a more positive contribution of judges to efficiency, particularly for Spain. Overall, such results are consistent with the Shapley value and suggest the methodology applied is robust.

Splitting the information system maturity index in 4 sub-indices provides some insights, see Figs. 3 and 4. Where (in Spain, France and Italy) the number of judges reduces inefficiency in the presence of one single index, it participates to it with the four alternate ones.

Austria seems to be an exception: productivity and communication tools participate to inefficiency while information tools and IT project governance participate to its efficiency. This suggests Austria spends more than is necessary on those two axes. The 4 factors behave in parallel and positively to efficiency for other countries, except Norway. This suggests the spending mix in IT is quite similar between countries. It is unclear why information tools and project governance appear to carry more weight than the other two axes, particularly productivity tools, which account for nearly half of the cap score (23 out of 47 points, as shown in the last line of Table 3).

Due to a lack of data, the same DEA cannot be applied at an earlier date, therefore no productivity indicators can be derived. However, other variables can be analyzed
<figure><img src="image_10.png" /><figcaption>Fig. 2 Input contribution to the DDF: Single index, DRS, Equal surplus</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Fig. 3 Input contribution to the DDF: Sub-indices, DRS, Shapley</figcaption></figure>

over time such as the number of contentious decisions and judicial services staff from the 'country profiles' established by the CEPEJ. On this basis, variations can be measured between 2004 and 2018. The use of computers being more intense among clerks, a distinction can be made between the number of civil and criminal decisions by judge and another by "non-judge", the numbers of clerks, among non-judges, not being further isolated in the questionnaires submitted to the participants.

The variations computed in Table 6 reflect productivity as well as the effects of law reforms or specific events in each country. For instance, Germany experienced a sharp increase in asylum-law-related decisions between the two dates.

<figure><img src="image_12.png" /></figure>
<figure><img src="image_13.png" /><figcaption>Fig. 4 Input contribution to the DDF: Sub-indices, DRS, Equal surplus</figcaption></figure>

<table><caption>**Table 6** Number of decisionsª by judge and non-judge</caption><thead><tr><th rowspan="2">Country</th><th rowspan="2">Code</th><th colspan="2">2004</th><th colspan="2">2018</th><th colspan="2">Variation</th></tr><tr><th>Judge</th><th>Non-judge</th><th>Judge</th><th>Non-judge</th><th>Judge</th><th>Non-judge</th></tr></thead><tbody><tr><td>Austria</td><td>AT</td><td>56.5</td><td>20.5</td><td>91.0</td><td>44.2</td><td>60.9%</td><td>115.9%</td></tr><tr><td>Switzerland<sup>c</sup></td><td>CH</td><td>418.6<sup>b</sup></td><td>124.6<sup>b</sup></td><td>259.8</td><td>62.4</td><td>-37.9%</td><td>-49.9%</td></tr><tr><td>Germany</td><td>DE</td><td>46.8</td><td>11.9</td><td>154.7</td><td>58.2</td><td>230.6%</td><td>387.4%</td></tr><tr><td>Denmark</td><td>DK</td><td>701.1</td><td>181.4</td><td>548.3</td><td>124.2</td><td>-21.8%</td><td>-31.6%</td></tr><tr><td>Spain</td><td>ES</td><td>121.8</td><td>17.4</td><td>374.4</td><td>42.6</td><td>207.4%</td><td>145.2%</td></tr><tr><td>Finland</td><td>FI</td><td>111.4</td><td>37.7</td><td>85.1</td><td>43.2</td><td>-23.6%</td><td>14.6%</td></tr><tr><td>France</td><td>FR</td><td>412.9</td><td>155.7</td><td>360.1</td><td>114.7</td><td>-12.8%</td><td>-26.3%</td></tr><tr><td>Italy</td><td>IT</td><td>391.9</td><td>83.0</td><td>417.1</td><td>130.6</td><td>6.4%</td><td>57.3%</td></tr><tr><td>Netherlands<sup>c</sup></td><td>NL</td><td>590.3</td><td>226.8</td><td>585.5</td><td>197.1</td><td>-0.8%</td><td>-13.1%</td></tr><tr><td>Norway</td><td>NO</td><td>60.5</td><td>31.5</td><td>72.0</td><td>46.4</td><td>19.2%</td><td>47.2%</td></tr><tr><td>Poland</td><td>PL</td><td>189.3</td><td>54.6</td><td>216.3</td><td>52.0</td><td>14.3%</td><td>-4.7%</td></tr><tr><td>Portugal</td><td>PT</td><td>356.0</td><td>83.2</td><td>217.4</td><td>74.0</td><td>-38.9%</td><td>-11.1%</td></tr><tr><td>Sweden</td><td>SE</td><td>190.9<sup>b</sup></td><td>74.6<sup>b</sup></td><td>261.0</td><td>61.0</td><td>36.7%</td><td>-18.2%</td></tr></tbody></table>

Source: CEPEJ

## 5.3 Administrative cases

DiVita et al. (2024) study, at the regional level, the performance of administrative courts between 2018 and 2021. The authors show that the COVID-19 pandemic had a dual impact on the efficiency of Regional Administrative Courts (TAR) in Italy. On one hand, the reorganization of court activities extended the duration of administrative disputes. On the other hand, the adoption of digital tools, such as online hearings

<footnote>ª Contentious civil and commercial, administrative, and criminal decisions rendered at first instance</footnote>

<footnote>ᵇ Figure from 2006</footnote>

<footnote>ᶜ Including non-contentious civil decisions</footnote>
and ICT-based judicial procedures, improved court efficiency by reducing DTs. Their findings highlight that digitalization, beyond the pandemic context, is essential for ensuring the timely resolution of administrative disputes in Italian TARs.

The weight of administrative cases is low in proportion to civil and criminal cases and its distribution is rather unequal across countries (second column of Table 7). However, the results from the DEA score (DDF) analysis show similar trends to those in Table 5. For example, the Netherlands remains ranked in the top 3 (excluding Denmark and Norway).

## 5.4 Robustness check: efficiency vs. effectiveness

As explained in Sect. 3.2, following Marciano et al. (2019), the results must show robustness with respect to efficiency rather than effectiveness. It is well established in literature that the number of cases is an output allowing for the measurement of efficiency, i.e., the measurement of judicial performance, see Marciano et al. (2019). As disposition time has been used as a proxy number of cases, it is mandatory to check whether the proxy is of good quality to draw conclusions on judicial efficiency rather than effectiveness.

A first statistical test is performed on the relationship between disposition time and the number of cases, distinguishing between civil and penal jurisdictions. We recall that our disposition time variable is equal to ‘1000 - disposition time’, i.e., considering 1000 days as the maximum number of days needed for a case to be solved.<fnref n="4" /> If a judicial system has some effectiveness, it must tend to 1000 remaining days to treat a new case. As expected, all correlation coefficients are negative. If the number of cases (civil or penal) increases, the number of days ‘1000 - disposition time’ decreases (it remains less days to treat new cases). While the Spearman rank correlation coefficient is not significant for penal cases at a 10% risk level, the Pearson correlation coefficients are significant (Table 8).

<table><caption>**Table 7** Performance according to disposition time of administrative cases (Input set 2)</caption><thead><tr><th>Country</th><th>% Admin</th><th>1000 - DT</th><th>DDF</th><th>Rank</th></tr></thead><tbody><tr><td>AT</td><td>41.37%</td><td>551</td><td>18.70929</td><td>10</td></tr><tr><td>CH</td><td>4.13%</td><td>797</td><td>0.20661</td><td>3</td></tr><tr><td>DE</td><td>30.05%</td><td>565</td><td>15.64898</td><td>8</td></tr><tr><td>DK</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td>ES</td><td>9.25%</td><td>669</td><td>2.86e-11</td><td>7</td></tr><tr><td>FI</td><td>42.85%</td><td>765</td><td>2.36e-11</td><td>6</td></tr><tr><td>FR</td><td>8.69%</td><td>715</td><td>2.21e-11</td><td>4</td></tr><tr><td>IT</td><td>2.38%</td><td>111</td><td>16.72105</td><td>9</td></tr><tr><td>NL</td><td>23.41%</td><td>800</td><td>1.95e-11</td><td>1</td></tr><tr><td>NO</td><td>NA</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td>PL</td><td>3.39%</td><td>882</td><td>2.05e-11</td><td>2</td></tr><tr><td>PT</td><td>6.71%</td><td>72</td><td>22.55493</td><td>11</td></tr><tr><td>SE</td><td>46.85%</td><td>854</td><td>2.24e-01</td><td>5</td></tr></tbody></table>

<footnote n="4">⁴Note that rescaling disposition time by 1000 has no effect on the intensity of the correlation, just the sign.</footnote>
<table><caption>**Table 8** Correlation tests: disposition time & number of cases</caption><thead><tr><th>Tests</th><th>Correlations</th><th>p-value</th></tr></thead><tbody><tr><td>Pearson: 1000 - Disposition time (civil) & number of cases</td><td>-0.7984</td><td>0.0011</td></tr><tr><td>Spearman: 1000 - Disposition time (civil) & number of cases</td><td>-0.5475</td><td>0.0528</td></tr><tr><td>Pearson: 1000 - Disposition time (penal) & number of cases</td><td>-0.5916</td><td>0.0332</td></tr><tr><td>Spearman: 1000 - Disposition time (penal) & number of cases</td><td>-0.4505</td><td>0.1223</td></tr></tbody></table>

<table><caption>**Table 9** Correlation tests: DEA scores & number of cases (civil)</caption><thead><tr><th>DEA scores</th><th>Pearson</th><th>p-value</th><th>Spearman</th><th>p-value</th></tr></thead><tbody><tr><td>Inefficiency (input set 1)</td><td>0.5604</td><td>0.0464</td><td>0.3901</td><td>0.1876</td></tr><tr><td>Inefficiency (input set 2)</td><td>0.4923</td><td>0.0875</td><td>0.4890</td><td>0.0899</td></tr><tr><td>Inefficiency (input set 3)</td><td>0.5938</td><td>0.0324</td><td>0.5824</td><td>0.0367</td></tr></tbody></table>

<table><caption>**Table 10** Correlation tests: DEA scores & number of cases (penal)</caption><thead><tr><th>DEA scores</th><th>Pearson</th><th>p-value</th><th>Spearman</th><th>p-value</th></tr></thead><tbody><tr><td>Inefficiency (input set 1)</td><td>0.4803</td><td>0.0967</td><td>0.3187</td><td>0.2886</td></tr><tr><td>Inefficiency (input set 2)</td><td>0.4593</td><td>0.1144</td><td>0.4011</td><td>0.1744</td></tr><tr><td>Inefficiency (input set 3)</td><td>0.5828</td><td>0.0366</td><td>0.5330</td><td>0.0607</td></tr></tbody></table>

To assess whether the disposition time can be a good proxy for efficiency measurement, we follow the strategy employed by Marciano et al. (2019). They measure technical efficiency as per Ippoliti et al. (2015) with respect to the number of cases. Then, they show that the hypothesis of efficiency can be rejected if disposition time (or clearance rates) are not statistically correlated to the DEA scores of technical efficiency.

In the same manner, we compute the correlations between the DEA scores derived from the disposition time in Table 5 and the number of cases. Table 9 presents the correlations between the three types of efficiency scores obtained from our different input sets (specifically, input set 3, which includes IT). The DEA scores incorporating the IT maturity index (input set 3) exhibit a significant correlation with the number of cases, as indicated by both Pearson and Spearman correlations at a 5% risk level. Since the DEA scores reflect inefficiency, as far as the number of cases increases, the inefficiency increases. As mentioned in Marciano et al. (2019), no causality can be drawn from these correlations.

The same conclusion holds true for penal cases. Table 10 shows that the DEA scores associated with our IT maturity index (input set 3) are significantly correlated with the number of penal cases, according to the Pearson correlation at the 5% risk level. However, this correlation is only significant at the 10% level for the Spearman rank coefficient.

A last test can be conducted. DEA scores are computed with the number of cases as outputs. The coefficients of correlation between these scores and our disposition time variable are significant either for Pearson or Spearman, as exhibited in Tables 11 (civil) and 12 (penal). The correlations are negative: the greater the inefficiency, the fewer the remaining days to treat new cases.
<table><caption>**Table 11** Correlation tests: DEA scores & 1000 - disposition time (civil)</caption><thead><tr><th>DEA scores</th><th>Pearson</th><th>p-value</th><th>Spearman</th><th>p-value</th></tr></thead><tbody><tr><td>Inefficiency (input set 1)</td><td>-0.6619</td><td>0.0137</td><td>-0.6933</td><td>0.0086</td></tr><tr><td>Inefficiency (input set 2)</td><td>-0.5375</td><td>0.0582</td><td>-0.6272</td><td>0.0218</td></tr><tr><td>Inefficiency (input set 3)</td><td>-0.6311</td><td>0.0207</td><td>-0.6162</td><td>0.0249</td></tr></tbody></table>

<table><caption>**Table 12** Correlation tests: DEA scores & 1000 - disposition time (penal)</caption><thead><tr><th>DEA scores</th><th>Pearson</th><th>p-value</th><th>Spearman</th><th>p-value</th></tr></thead><tbody><tr><td>Inefficiency (input set 1)</td><td>-0.5351</td><td>0.0595</td><td>-0.7143</td><td>0.0061</td></tr><tr><td>Inefficiency (input set 2)</td><td>-0.5138</td><td>0.0725</td><td>-0.7033</td><td>0.0073</td></tr><tr><td>Inefficiency (input set 3)</td><td>-0.3502</td><td>0.2408</td><td>-0.6484</td><td>0.0165</td></tr></tbody></table>

Finally, in view of the previous results, the (in)efficiency scores related to disposition time are correlated with the output variables used to measure efficiency. While this is not always verified for specific datasets (Marciano et al., 2019)—in which case the hypothesis of effectiveness should be favored—the previous correlation tests show that disposition time can serve as a proxy for an output variable to measure technical efficiency.

# 6 Conclusion

For 9 out of 13 countries, the DEA suggests that investing in IT positively contributes to reduce disposition times, hence to justice efficiency. That is not a clear-cut outcome. Disaggregating the single maturity index brings results that are even harder to interpret. It is still difficult to determine the proportion and nature of IT contribution, because non-quantifiable variables also play significant roles.

We propose three avenues for improving the understanding of the factors of efficiency of judicial IT. They all require adding new questions to the two-year survey cycle conducted by the CEPEJ.

The first avenue is to enrich and fine-tune questions relative to IT policy to understand how much the spending mix within IT per type of functionality, between innovation and legacy support, or between investment/operations, etc... and the level of integration/interoperability weigh on efficiency.

The second avenue is to build a proxy of normative density through some counting, of words, articles or pages, of legal instruments. The underlying assumption is that, the denser the procedural norm, the more complex the case management system, and/or the longer the disposition time.

A third line of research is to build an attitudinal and behavioural index of stakeholders towards judicial IT and the very principle of replacing human assets with IT assets to foster efficiency of the judiciary. It would be built in such a way that reducing its value would entail efficiency gains. It might explain why an information system would not be mature, or how a country would miss the opportunity to reduce human assets despite a mature information system.

Such indices could then be used as additional inputs to a DEA.

<figure><img src="image_14.png" /></figure>
```r
columns = [
  "judges",
  "non-judges",
  "information system maturity index (this column is not used if the
next 4 are)",
  "information tools",
  "productivity tools",
  "communication tools",
  'IT project governance'
]
contrib DF = pd. Frame(df, index = cepej.index, columns = columns)
row_sum s = contrib DF. sum (axis = 1)
contrib_pourcentage = contrib DF. div (row_sum s, axis = 0) * 100
print("Absolute Contributions: ", "\n")
display (contrib DF)
print("Contributions (%): ", "\n")
display (contrib_pourcentage)
```

The last technique is the permutation method. Each input of $X$ is randomly permuted (30 times), then the score is averaged and compared with the total value of the DDF. Then, this is not a measure of attribution (or contribution) but rather a measure that indicates whether a feature may increase or decrease the DEA score: DEA score minus the mean of DEA with permutations of the inputs. If the score is positive then a feature contributes to increase the DEA score (then to increase inefficiency), and conversely.

```r
df_permutation = model. fit("permutation", X, Y)
contrib DF. permutation = pd. Frame(df_permutation, index = cepej.index,
columns = column)
display (contrib DF. permutation)
```

## Technical efficiency by country (DDF) - constant, increasing or variable returns

See Tables 13 and 14

Except for France, the different computation modes of returns do not significantly upset country rankings. It is likely this type of algorithm is of lesser importance than the choice of inputs, to the extent some hardly quantifiable factors of efficiency are not apprehended by DEA.

<table><caption>**Table 13** Judges/non-judges/Single IT index</caption><thead><tr><th>Country</th><th>CRS</th><th>Rank</th><th>IRS</th><th>Rank</th><th>VRS</th><th>Rank</th></tr></thead><tbody><tr><td>Austria</td><td>27.606</td><td>13</td><td>1.623</td><td>9</td><td>21.614</td><td>12</td></tr><tr><td>Switzerland</td><td>9.469</td><td>4</td><td>1.163</td><td>6</td><td>3.440e-13</td><td>1</td></tr><tr><td>Germany</td><td>23.818</td><td>12</td><td>1.868</td><td>11</td><td>22.096</td><td>13</td></tr><tr><td>Denmark</td><td>5.664e-12</td><td>3</td><td>8.627e-12</td><td>1</td><td>6.817e-12</td><td>2</td></tr><tr><td>Spain</td><td>19.799</td><td>9</td><td>3.198</td><td>13</td><td>17.117</td><td>9</td></tr><tr><td>Finland</td><td>19.669</td><td>8</td><td>1.034</td><td>5</td><td>18.716</td><td>10</td></tr><tr><td>France</td><td>14.168</td><td>7</td><td>0.634</td><td>4</td><td>3.082</td><td>5</td></tr><tr><td>Italy</td><td>22.585</td><td>11</td><td>2.327</td><td>12</td><td>13.937</td><td>8</td></tr><tr><td>Netherlands</td><td>4.956e-12</td><td>1</td><td>9.539e-12</td><td>2</td><td>7.846e-12</td><td>4</td></tr><tr><td>Norway</td><td>5.811e-12</td><td>2</td><td>1.189e-11</td><td>3</td><td>7.007e-12</td><td>3</td></tr><tr><td>Poland</td><td>13.345</td><td>6</td><td>1.546</td><td>8</td><td>12.335</td><td>7</td></tr><tr><td>Portugal</td><td>22.555</td><td>10</td><td>1.500</td><td>7</td><td>21.380</td><td>11</td></tr><tr><td>Sweden</td><td>12.649</td><td>5</td><td>1.818</td><td>10</td><td>11.233</td><td>6</td></tr></tbody></table>
<table><caption>**Table 14** Judges/non-judges/IT sub-indices</caption><thead><tr><th>Country</th><th>CRS</th><th>Rank</th><th>IRS</th><th>Rank</th><th>VRS</th><th>Rank</th></tr></thead><tbody><tr><td>Austria</td><td>1.623</td><td>7</td><td>1.266</td><td>7</td><td>0.364</td><td>6</td></tr><tr><td>Switzerland</td><td>1.163</td><td>4</td><td>0.814</td><td>5</td><td>9.382e-12</td><td>3</td></tr><tr><td>Germany</td><td>2.008</td><td>10</td><td>1.868</td><td>11</td><td>1.986</td><td>11</td></tr><tr><td>Denmark</td><td>8.629e-12</td><td>1</td><td>1.274e-11</td><td>1</td><td>8.762e-12</td><td>1</td></tr><tr><td>Spain</td><td>3.472</td><td>13</td><td>3.198</td><td>13</td><td>3.409</td><td>13</td></tr><tr><td>Finland</td><td>1.361</td><td>5</td><td>1.034</td><td>6</td><td>1.034</td><td>7</td></tr><tr><td>France</td><td>2.066</td><td>11</td><td>0.634</td><td>4</td><td>9.091e-12</td><td>2</td></tr><tr><td>Italy</td><td>3.330</td><td>12</td><td>2.327</td><td>12</td><td>2.327</td><td>12</td></tr><tr><td>Netherlands</td><td>9.543e-12</td><td>2</td><td>1.143e-11</td><td>2</td><td>1.020e-11</td><td>4</td></tr><tr><td>Norway</td><td>1.049e-11</td><td>3</td><td>1.966e-11</td><td>3</td><td>1.044e-11</td><td>5</td></tr><tr><td>Poland</td><td>1.695</td><td>8</td><td>1.546</td><td>9</td><td>1.546</td><td>9</td></tr><tr><td>Portugal</td><td>1.527</td><td>6</td><td>1.500</td><td>8</td><td>1.500</td><td>8</td></tr><tr><td>Sweden</td><td>1.818</td><td>9</td><td>1.836</td><td>10</td><td>1.697</td><td>10</td></tr></tbody></table>

**Acknowledgements** The authors would like to acknowledge Alain Marciano, one of the Editors-in-Chief of EJLE, for his helpful suggestions. They are also greatly indebted to their three reviewers for their valuable suggestions and remarks. All remaining errors are the authors' own.

**Author contributions** Bruno and Stéphane contributed equally to the article

**Funding** Open access funding provided by Université de Nimes. The authors did not receive support from any organization for the submitted work.

**Data availability** No datasets were generated or analysed during the current study.

## Declarations

**Conflict of interest** The authors declare no conflict of interest.

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

## References

Andrade, A., & Joia, L. A. (2012). Information technology and the efficiency of the Brazilian Judiciary System. In *11th International Conference on Electronic Government (EGOV)* (pp. 126–136). Norway: Kristiansand.

Austria. (2012). *Autriche: Exercice d'évaluation - Edition 2012 (données 2010)*. CEPEJ. https://rm.coe.in/t/commission-europeenne-pour-l-efficacite-de-la-justice-cepej-questionna/168078ba17
Avgeriou, P., Kruchten, P.B., Ozkaya, I., & Seaman, C.B. (2016). "Managing Technical Debt in Software Engineering (Dagstuhl Seminar 16162)". In: *Dagstuhl Reports* 6, pp. 110–138. https://api.semanticscholar.org/CorpusID:3254861.

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. *Management Science*, 30, 1078–1092.

Belarouci, M., Vaillant, N., & Wolff, F.-C. (2023). An efficiency analysis of conciliation activities in French Courts. *Revue d'économie Politique*, 133(2), 281–314. https://doi.org/10.3917/redp.132.0281

Beldowski, J., Dabros, L., & Wojciechowski, W. (2020). Judges and court performance: A case study of district commercial courts in Poland. *European Journal of Law and Economics*, 50, 171–201.

Brieč, W., & Mussard, S. (2020). Improvement of technical efficiency of firm groups. *European Journal of Operational Research*, 283, 991–1001. https://doi.org/10.1016/j.ejor.2019.11.048

Cappellina, B. (2017). Évaluer l'administration de la justice dans les pays Européens. Une co-construction entre Union Européenne et Conseil de l'Europe. *Revue FranÇaise d'Administration Publique*, 1(2), 59–72.

CEPEJ. (2006). *European judicial systems. Edition 2006 (2004 data)*. Council of Europe. https://rm.coe.int/cepej-2006-fr/168093fd8d.

CEPEJ. (2020). *European judicial systems. Edition 2020 (2018 data)*. Council of Europe.

CEPEJ. (2022). *Study on the Functioning of Judicial Systems in the EU Member States: Facts and Figures from the CEPEJ Questionnaires 2012 to 2020*. CEPEJ(2021)18rev1 - Part 1: Data Tables per Indicator for all EU Member States. https://commission.europa.eu/system/files/2022-05/part_1_-_eu_scoreBoard_-_indicators_-_deliverable_0.pdf.

CEPEJ. (2010). Rapport 2010 (données 2008). *Étude de 16 pays comparables, par Jean-Paul Jean*. Council of Europe: Président du Groupe de travail É valuation.

Chambers, R. G., Chung, Y., & Färe, R. (1996). Benefit and Distance Functions. *Journal of Economic Theory*, 70, 407–419.

Chambers, R. G., Chung, Y., & Färe, R. (1998). Profit, directional distance functions, and Nerlovian efficiency. *Journal of Optimization Theory and Applications*, 98, 351–364.

Charnes, A., William, C., & Edward, R. (1978). Measuring the efficiency of decision making units. *European Journal of Operational Research*, 2(6), 429–444.

Condevaux, C., Harispe, S., & Mussard, S. (Sept. 2022). "Fair and Efficient Alternatives to Shapley-based Attribution Methods". In *ECMLPKDD 2022 - The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases*. Grenoble, France. https://imt-mine.s-ales.hal.science/hal-03781033.

Cunningham, W. (1992). "The WyCash portfolio management system". In *Addendum to the Proceedings on Object-Oriented Programming Systems, Languages, and Applications*, (pp. 29–30).

Delone, W., & McLean, E. (1992). Information systems success: The quest for the dependent variable. *Information Systems Research*, 3, 60–95.

Deyneli, F. (2012). Analysis of relationship between efficiency of justice services and salaries of judges with two-stage DEA method. *European Journal of Law and Economics*, 34(3), 477–493. https://doi.org/10.1007/s10657-011-9258-3

DiVita, G., Ferrara, P. L., & Patti, A. (2024). An empirical study of administrative justice at times of COVID-19 in Italy. *Regional Studies, Regional Science*, 11(1), 558–582.

Elbially, N., & García-Rubio, M. A. (2011). *Assessing Judicial Efficiency of Egyptian First Instance Courts: A DEA Analysis*. MAGKS Papers on Economics 201119. Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung). https://EconPapers.repec.org/RePEc:mar:magkse:201119.

European Commission. (2022). *Study on the functioning of judicial systems in the EU Member States: facts and figures from the CEPEJ questionnaires 2012 to 2020. Part 1*. European Commission.

Falavigna, G., & Ippoliti, R. (2021). *Institutional Efficiency and Budget Constraints: A Directional Distance Function Approach to Lead a Key Policy Reform*. Working Paper 06/2021. CNR-IRCrES. http://www.ircres.cnr.it/images/wp/WP_06_2021.pdf.

Falavigna, G., Ippoliti, R., Manello, A., & Ramello, G. B. (2015). Judicial productivity, delay and efficiency: A Directional Distance Function (DDF) approach. *European Journal of Operational Research*, 240(2), 592–601. https://doi.org/10.1016/j.ejor.2014.07.014

France. (2012). *France : Exercice d'évaluation - Edition 2012 (données 2010)*. CEPEJ. https://rm.coe.int/commission-europeenne-pour-l-efficacite-de-la-justice-cepej-questionna/168078bab9
Gomes, A. O., Alves, S. T., & Silva, J. T. (2018). Effects of investment in information and communication technologies on productivity of courts in Brazil. *Government Information Quarterly*, 35(3), 480–490.
https://doi.org/10.1016/j.giq.2018.06.002

Grosskopf, S. (1986). The role of the reference technology in measuring productive efficiency. *The Economic Journal*, 96, 499–513.

Hagstedt, K., & Proos, J. (2008). Has the recent restructuring of the swedish district courts improved efficiency? A DEA Analysis. Bachelor's Thesis. Bachelor's Thesis. Uppsala, Sweden: Uppsala University, Department of Economics.

Ippoliti, R., Melcarne, A., & Ramello, G. B. (2015). Judicial efficiency and entrepreneurs’ expectations on the reliability of European Legal Systems. *European Journal of Law and Economics*, 40, 75–94.

Kittelsen, S. A. C., & Førsund, F. R. (1992). Efficiency analysis of Norwegian district courts. *The Journal of Productivity Analysis*, 3, 277–306.

Lewin, A. Y., Morey, R. C., & Cook, T. J. (1982). Evaluating the administrative efficiency of courts. *Omega*, 10(4), 401–411. https://doi.org/10.1016/0305-0483(82)90019-6

Lopes, N. Q. M., & Silva, M. C. A. (2024). Scale and scope economies in first-instance courts: Portuguese specialized vs non-specialized courts. *International Review of Law and Economics*, 79, 106–216.
https://doi.org/10.1016/j.irle.2024.106216

Lorenzani, D., & Lucidi, F (2014). *The economic impact of civil justice reforms*. Tech. rep. Directorate General Economic and Financial Affairs (DG ECFIN), European ... https://ec.europa.eu/economy_fi nance/publications/economic_paper/2014/pdf/ecp530_en.pdf.

Månsson, J., Andersson, C., & Bonander, F. (2022). What lessons can be learned from cost efficiency? The case of Swedish district courts. *European Journal of Law and Economics*, 54, 431–451.

Marciano, A., Melcarne, A., & Ramello, G. (2019). The economic importance of judicial institutions, their performance and the proper way to measure them. *Journal of Institutional Economics*, 15, 81–98.

Mazzocchi, P., Quintano, C., & Rocca, A. (2024). Efficiency analysis using SBM and PLS-SEM: Insights from the Italian judicial system. *Quality & Quantity*. https://doi.org/10.1007/s11135-024-01907-9

Mussard, S., & Terrazza, V. (2007). New trading risk indexes: Application of the shapley value in finance. *Economics Bulletin*, 3(25), 1–7.

OECD. (2013). *Government at a Glance 2013*, p. 196. https://doi.org/10.1787/gov_glance-2013-en.

Palumbo, G., Giupponi, G., Nunziata, L., & Sanguinetti, J.S.M. (2013). *The Economics of Civil Justice: New Cross-country Data and Empirics*. OECD Economics Department Working Papers 1060. OECD Publishing.

Peyrache, A., & Zago, A. (2012). *Large courts, small justice! The inefficiency and the optimal structure of the Italian Justice sector*. Working Paper WP06/2012. Centre for Efficiency and Productivity Analysis (CEPA).

Peyrache, A., & Zago, A. (2016). Large courts, small justice!: The inefficiency and the optimal structure of the Italian justice sector. *Omega*, 64, 42–56. https://doi.org/10.1016/j.omega.2015.11.002

Rinta-Kahila, T., Penttinen, E., & Lyytinen, K. (2023). Getting trapped in technical debt: Sociotechnical analysis of a legacy system’s replacement. *MIS Quarterly*, 47, 1–32. https://doi.org/10.25300/MISQ/2022/16711

Ruggiero, J. (1996). On the measurement of technical efficiency in the public sector. *European Journal of Operational Research*, 30, 553–565.

Schneider, M. R. (2005). Judicial career incentives and court performance: An empirical study of the German Labour Courts of Appeal. *European Journal of Law and Economics*, 20(2), 127–144. https://doi.org/10.1007/s10657-005-1733-2

Seiford, L. M., & Thrall, R. M. (1990). Recent developments in DEA: The mathematical programming approach to frontier analysis. *Journal of Econometrics*, 46, 7–38.

Shapley, L. (1953). A value for n-person games. *Contributions to the Theory of Games*, 2(28), 307–317.

Simar, L., & Wilson, P. W. (2007). Estimation and inference in two-stage, semiparametric models of production processes. *Journal of Econometrics*, 136(1), 321–364. https://doi.org/10.1016/j.jeconom.2007.05.009

Tapia, J., & Salvador, B. (2022). Data envelopment analysis efficiency in the public sector using provider and customer opinion: An application to the Spanish health system. *Health Care Management Science*, 25, 335–346.

van den Hoogen, R., Langbroek, P., & Reiling, D. (2003). Judicial electronic data interchange in the Netherlands. In M. Fabri & F. Contini (Eds.), *Judicial electronic data interchange in Europe: Applications, policies and trends* (pp. 3011–320). Bologne.
Velicogna, M., & Ontanu, A. (2021). The challenge of comparing EU Member States judicial data. *Onati Socio-legal Series, 11*, 446–480.

Vereeck, L., & Mühl, M. (2000). An economic theory of court delay. *European Journal of Law and Economics, 10*(3), 243–268.

Voigt, S. (2016). Determinants of judicial efficiency: A survey. *European Journal of Law and Economics, 42*, 183–208.

Yeung, L. L., & Azevedo, P. F. (2011). Measuring efficiency of Brazilian Courts with data envelopment analysis (DEA). *IMA Journal of Management Mathematics, 22*(4), 343–356. https://doi.org/10.1093/imaman/dpr002

**Publisher's Note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.