SlowGuess/NAACL_2021_OCR huggingface dataset:
Josef Valvoda $^{\delta}$  Tiago Pimentel $^{\delta}$  Niklas Stoehr $^{\S}$  Ryan Cotterell $^{\delta,\S}$  Simone Teufel $^{\delta}$
$^{\mathrm{d}}$ University of Cambridge,  ${}^{\mathrm{f}}$ ETH Zürich jv406@cam.ac.uk, sht25@cam.ac.uk
In common law, the outcome of a new case is determined mostly by precedent cases, rather than by existing statutes. However, how exactly does the precedent influence the outcome of a new case? Answering this question is crucial for guaranteeing fair and consistent judicial decision-making. We are the first to approach this question computationally by comparing two longstanding jurisprudential views; Halsbury's, who believes that the arguments of the precedent are the main determinant of the outcome, and Goodhart's, who believes that what matters most is the precedent's facts. We base our study on the corpus of legal cases from the European Court of Human Rights (ECtHR), which allows us to access not only the case itself, but also cases cited in the judges' arguments (i.e. the precedent cases). Taking an information-theoretic view, and modeling the question as a case outcome classification task, we find that the precedent's arguments share 0.38 nats of information with the case's outcome, whereas precedent's facts only share 0.18 nats of information (i.e.,  $58\%$  less); suggesting Halsbury's view may be more accurate in this specific court. We found however in a qualitative analysis that there are specific statuses where Goodhart's view dominates, and present some evidence these are the ones where the legal concept at hand is less straightforward.
Legal systems around the world can be divided into two major categories (Joutsen, 2019): civil law systems, which rely predominantly on the rules written down in statutes, and common law systems, which rely predominantly on past judicial decisions, known as the precedent. Within common law systems, jurisprudential scholars have pondered over the nature of precedent in law for at least a century (Halsbury, 1907). Is it the judges' argumentation in the precedent, or is it the
Figure 1: The text of ECtHR cases can be divided into facts, arguments and outcome. Arguments cite relevant cases, also known as the precedent.
claimants' specific individual circumstances that are the deciding factor in what becomes the law? Here, we present a new information-theoretical methodology that helps answer this question.
In common law countries, statutes establish the general idea of the law, but the actual scope of the law is determined by the courts during a trial. To keep case outcomes consistent and predictable in subsequent cases, judges are forced to apply the reasoning developed in prior cases with similar facts (precedent), to the facts of the new case under the doctrine of stare decisis (Duxbury, 2008; Lamond, 2016; Black, 2019). This is done by identifying the ratio decidendi (the reasons for the decision) as opposed to the obiter dicta (that which is said in passing). The distinction between ratio and obiter is an important one, since ratio is binding, whereas obiter is not. This means that courts will only strive to remain consistent in upholding ratio, but can freely depart from the obiter.
But what does the ratio consist of? There is no accepted overarching theory of precedent (Duxbury, 2008), but there are two tests of ratio. On the one hand, Lord Halsbury (1907) claims
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2275-2288 June 6-11, 2021. ©2021 Association for Computational Linguistics
that what is binding is the judge's reasoning and arguments. For instance, by using a high degree of abstraction, judges can analogise physical and psychological pain. A different view has been put forward by Goodhart (1930), who argues it is the analogy of the facts of the precedent and the case at hand, without the need for reasoning (e.g. comparing the pain caused by a knife to that caused by another instrument, requiring a far lower degree of abstraction). These give rise to the two well-known legal tests for ratio: Halsbury's test and Goodhart's test.
In this paper, we are the first to approach this problem from a data-driven perspective, using the European Court of Human Rights  $(\mathbf{E}\mathbf{C}\mathbf{t}\mathbf{H}\mathbf{R})^{1}$  case law; see Figure 1. We build a citation network over this corpus in order to have access to many precedents' full text. Training our model on either the facts or the arguments of the precedent, we can put Halsbury's and Goodhart's views to the test. We cast this problem as an information-theoretic study by measuring the mutual information (Shannon and Weaver, 1962) between the case outcome and either the precedent facts or arguments. We find that precedent arguments and case outcome share information to the degree of 0.38 nats, whereas facts and case outcome only share information to the degree of 0.18 nats (i.e.,  $58\%$  less). We therefore observe that—at least for ECtHR—Halsbury's view of the precedent is more accurate than that of Goodhart.
2 Legal Background
Despite the importance of the precedent in civil law, its operationalization remains shrouded in philosophical debate centred around how the precedent actually forms the binding law. Jurisprudentially, we can think of this as searching for the ratio decidendi in the judgement, i.e. separating the ratio decidendi from the obiter dicta, or binding law from merely circumstantial statements. It is the nature of ratio that distinguishes Halsbury's view from Goodhart's.
2.1 Halsbury: Arguments as ratio
The case argument contains the judge's explanation of why the case is decided the way it is. It incorporates knowledge of the precedent, facts of the case
and any new reasoning the judge might develop for the case itself. We consider the intuitive position that a legal test is formulated by the argument that the judge put forward when deciding the case.
A legal test is by its nature part of the ratio and, thus, would be binding on all subsequent cases. This is the position endorsed by Lord Halsbury (1907). Under this conception of the ratio, it is the arguments that matter, becoming the law; the facts of the case are of secondary importance. If a judge acts as Halsbury suggests they should extract the logic of the implicit legal test of the precedent, and attempt to largely ignore the specific facts of the case. Halsbury's view remains the conventional view of the precedent to this day (Lamond, 2005).
2.2 Goodhart: Facts as ratio
In contrast, Goodhart (1930) observes that many cases do not contain extensive reasoning, or any reasoning at all; judges seem to decide the outcome without these. Therefore, he claims that the facts of the case together with its outcome must form the ratio; otherwise, a hypothetical new case with the same facts as any given precedent could lead to a different outcome. Duxbury (2008) observes that judges, when in disagreement with the precedent, concentrate on the facts of a previous case more than one would expect if Halsbury's hypothesis were fully correct. Halsbury would predict that they should talk about the facts of previous cases as little as possible, and seek the most direct route to ratio in the form of argument, but they evidently do not. A potential explanation is that, when disagreement arises, it is easier for judges to claim that the facts are substantially different, than to challenge the logic of the precedent, i.e. to overrule that case. Overruling a previous judgement is a rare and significant legal event (Dunn, 2003; Spriggs and Hansford, 2001) because it threatens the stability of the legal system. By concentrating on facts rather than running the risk of overruling, the judge can avoid this problem, including the threat of overruling her own previous judgement.
In support of this view, inspection of the argumentative part of the judgement reveals judges do not usually formulate legal tests of the kind Halsbury implies (Lamond, 2005). Neither do judges usually search the precedent for such legal tests (Alexander and Sherwin, 2008). Goodhart's position suggests that the precedent operates less as an enactment of rules, but more as reasoning by
<sup>1</sup>European Court of Human Rights (ECtHR) is the court that adjudicates on cases dealing with the European Convention of Human Rights (ECHR).
Figure 2: Our formulation of Halsbury's and Goodhart's tests as a classification task. Current case facts are truncated to 512 tokens. Outcome of the precedent is concatenated with either the precedent facts or arguments, and both are jointly truncated at 512 tokens. Finally, these are concatenated together and embedded in 768 dimensions before being fed into the LONG-FORMER.
analogy; hence it is the good alignment between the facts of the two cases that leads to consistent outcomes.
3 An Information-theoretic Approach
Notation. We denote the set of cases as  $\mathcal{C}$ , writing each of its element as  $c$ . The set of cases that form the precedent for case  $c$  are denoted  $\mathcal{P}_c \subset \mathcal{C}$ . We will consider three main random variables in this work. First, we consider  $O$ , a random variable that ranges over a binary outcome space  $\mathcal{O} = \{0,1\}^K$ , where  $K$  is the number of Articles. An instance  $o \in \mathcal{O}$  tells us which Articles have been violated. Since  $o$  is a vector of binary outcomes for all Articles, we can index it as  $o_k$  to get the outcome of a specific  $k^{\text{th}}$  Article and we analogously index the random variable  $O_k$ . We will denote  $o_c$  the outcome of a specific case  $c$ . Next, we consider  $F$ , a random variable that ranges over the space of facts. We denote the space of all facts as  $\mathcal{F} = \Sigma^*$ , where  $\Sigma$  is a set of sub-word units and  $\Sigma^*$  is its Kleene closure. We denote an instance of  $F$  as  $f$ . We will further denote the facts of a specific case  $c$  as  $f_c$ . Finally, we consider  $A$ , a random variable that ranges over the space of Arguments. Analogously to facts, the space of all Arguments is  $\mathcal{A} = \Sigma^*$ . An element of  $\mathcal{A}$  is denoted
as  $a$ , which we again term  $a_{c}$  when referring to a specific case.
Operationalising Halsbury and Goodhart. In this work, we intend to measure the use of Halsbury's and Goodhart's views in practice, which we operationalise information-theoretically following the methodology proposed by Pimentel et al. (2019). To test the hypothesis, we construct two collections of random variables, which we denote  $H$  and  $G$ . We define an instance  $h_c$  of random variable  $H$  as the union of arguments and outcomes for all precedent cases of  $c$ , i.e.  $\bigcup_{c' \in \mathcal{P}_c} \{a_{c'}, o_{c'}\}$ . We will denote the instance  $h$  when referring to it in the abstract (without referring to a particular case). We analogously define instances of random variable  $G$  as  $g_c = \bigcup_{c' \in \mathcal{P}_c} \{f_{c'}, o_{c'}\}$ . While the set-theoretic notation may seem tedious, it encompasses the essence of the distinction between Halsbury's and Goodhart's view: Each view hypothesises a different group of random variables should contain more information about the outcome  $O$  of a given case. In terms of mutual information, we are interested in comparing the following:
$$
\mathrm {M I} (O; H \mid F), \quad \mathrm {M I} (O; G \mid F) \tag {1}
$$
If  $\operatorname{MI}(O;H\mid F) > \operatorname{MI}(O;G\mid F)$ , then Halsbury's view should be more widely used in practice. Conversely if the opposite is true, i.e.  $\operatorname{MI}(O;G\mid F) > \operatorname{MI}(O;H\mid F)$ , then Goodhart's view should be the one more widely used.
The MI is calculated by subtracting the outcome entropy conditioned on the case facts and either  $H$  or  $G$  from the outcome entropy conditioned on the facts alone. Therefore, to compute the MI we need to compute the Halsbury's and Goodhart's conditional entropies first:
$$
\begin{array}{l} \mathrm {H} (O \mid H, F) \tag {2} \\ = - \sum_ {o, h, f} p (o, h, f) \log p (o \mid h, f) \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {H} (O \mid G, F) \tag {3} \\ = - \sum_ {o, g, f} p (o, g, f) \log p (o \mid g, f) \\ \end{array}
$$
as well as the entropy conditioned on the facts of the current case alone:
$$
\mathrm {H} (O \mid F) = - \sum_ {o, f} p (o, f) \log p (o \mid f) \tag {4}
$$
2We note here that we overload the subscript notation in this paper. We will use subscript  $c$  to denote a specific case and subscript  $k$  to denote a specific article.
The conditional entropies above reflect the uncertainty (measured in nats) $^3$  of an event, given the knowledge of another random variable. For instance, if  $G$  completely determines  $O$ , then  $\mathrm{H}(O \mid G)$  is 0; there is no uncertainty left. Conversely, if the variables are independent, then  $\mathrm{H}(O) = \mathrm{H}(O \mid G)$ , where  $\mathrm{H}(O)$  denotes the unconditional entropy of the outcomes  $O$ . We now note a common decomposition of mutual information that will help with the approximation:
$$
\mathrm {M I} (O; H \mid F) = \mathrm {H} (O \mid F) - \mathrm {H} (O \mid H, F) \tag {5}
$$
$$
\operatorname {M I} (O; G \mid F) = \operatorname {H} (O \mid F) - \operatorname {H} (O \mid G, F) \tag {6}
$$
In this work, we consider the conditional probabilities  $p(o \mid \bullet)$  as the independent product of each Article's probability, i.e.  $\prod_{k=1}^{K} p(o_k \mid \bullet)$ . Information-theoretically, then, they are related through the following equation:
$$
\mathrm {H} (O \mid \bullet) = \sum_ {k = 1} ^ {K} \mathrm {H} \left(O _ {k} \mid \bullet\right) \tag {7}
$$
Following Williams et al. (2020), we further calculate the uncertainty coefficient (Theil, 1970) of each of these mutual informations. These coefficients are easier to interpret, representing the percentage of uncertainty reduced by the knowledge of a random variable:
$$
\mathrm {U} (O \mid H; F) = \frac {\operatorname {M I} (O ; H \mid F)}{\operatorname {H} (O \mid F)} \tag {8}
$$
$$
\mathrm {U} (O \mid G; F) = \frac {\mathrm {M I} (O ; G \mid F)}{\mathrm {H} (O \mid F)} \tag {9}
$$
We choose to work with the ECtHR corpus for three reasons. First, it can be treated as operating under precedential law, in the vein of common law countries. This is not a given, as the ECtHR is an international court of highest appeal without a formal doctrine of stare decisis (Jacob, 2014), but there is nevertheless strong evidence that it is precedential. This evidence comes from the court's own guidelines (ECtHR, 2014), but can also be found in the writings of a former judge of the ECtHR (Zupancic, 2016) and of legal scholars (Lupu and Voeten, 2010). Second, there is existing research on the neural modeling of ECtHR case law we can build upon (Aletras et al., 2016; Chalkidis et al., 2019, 2020). Third, the documents of the ECtHR
case law, unlike those of most other courts, textually separate the facts from the arguments, which is crucial for our experiments.
Case facts are descriptions of what had happened to the claimant before they went to the court; they include domestic proceedings of their case before it was appealed to the ECtHR as a form of a last resort. They do not contain any reference to European Convention of Human Rights (ECHR) Articles or ECtHR case law. Arguments on the other hand contain judges' discussion of ECHR articles and ECtHR case law in relation to the facts. The ECtHR corpus has been scraped from the HUDOC<sup>4</sup> database and contains 11,000 cases reported in English (Chalkidis et al., 2019).<sup>5</sup> Judges decide for each Article of ECHR whether it has been violated with respect to the claimant's circumstances. In the ECtHR corpus, each case therefore comes with a pre-extracted decision in form of a set of violated ECHR Article numbers. We refer to this set as the outcome of a case. Out of 30 Articles, 18 are from the Convention itself (Articles 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 18, 34, 38, 41, 46), while the rest (1.1, 1.2, 1.3, 4.2, 4.4, 6.1, 6.3, 7.1, 7.2, 7.3, 7.4, 12.1) comes from the Protocols to the Convention.
For our experiment, we need a sub-corpus where each case has at least one outgoing citation where the full text is contained in our corpus. In practice, there will be other outgoing citations we cannot resolve, for instance because the document is not in English or HUDOC happens not to contain them. We also need our citations to be de-duplicated. We create such a sub-corpus, which contains 9,585 documents (i.e., citing documents), with 94,167 in-corpus links (tokens) to 7,113 cases (types) and 65,495 out-of-corpus links to 22,328 types (cited documents). We start from the original ECtHR split of 9,000 training, 1,000 validation and 1,000 test cases, and after citation filtering arrive at 7,627 training, 976 validation and 982 test cases. For every citation, we extract the text under headings with regular expressions such as "THE FACTS" and "THE LAW", labelling it as facts and arguments, respectively.
4.1 Approximations
The mutual information values that we intend to analyse need to be approximated. We follow Pi-
<sup>4</sup>HUDOC: https://hudoc.echr.coe.int/eng.
${}^{5}$  ECtHR cases are reported either in English, French or both. Additionally, some cases are also reported in the language of the state they take place in.
3Nats are computed with ln, while bits use  $\log_2$
mentel et al.'s (2019; 2021) methodology for this, approximating them as the difference between two cross-entropies:
$$
\operatorname {M I} (O; H \mid F) \approx \mathrm {H} _ {\theta} (O \mid F) - \mathrm {H} _ {\theta} (O \mid H, F)
$$
$$
\operatorname {M I} (O; G \mid F) \approx \mathrm {H} _ {\theta} (O \mid F) - \mathrm {H} _ {\theta} (O \mid G, F)
$$
Indeed, although several estimates for the mutual information exist, McAllester and Stratos (2020) argues that estimating it as this difference is the most statistically justified way. These conditional entropies are themselves approximated through their sample estimate. For instance, we compute:
$$
\mathrm {H} _ {\theta} (O \mid G, F) \approx - \frac {1}{| \mathcal {C} |} \sum_ {c \in \mathcal {C}} \log p _ {\theta} \left(o _ {c} \mid g _ {c}, f _ {c}\right) \tag {10}
$$
which is exact as  $|\mathcal{C}| \to \infty$ . We note that the cross-entropy is an upper bound on the entropy, which uses a model  $p_{\theta}(o \mid \bullet)$  for its estimate. The better this model, the tighter our estimates will be. The only thing left to do now, is to obtain these probability estimates. We thus model Halsbury's view as a classification task (see Figure 2) estimating the probability:
$$
p _ {\theta} (o \mid h, f) = \prod_ {k = 1} ^ {K} p _ {\theta} \left(o _ {k} \mid h, f\right) \tag {11}
$$
We analogously model Goodhart's view as:
$$
p _ {\theta} (o \mid g, f) = \prod_ {k = 1} ^ {K} p _ {\theta} \left(o _ {k} \mid g, f\right) \tag {12}
$$
Finally, we model the  $p_{\theta}$  of the model conditioned only on the facts of the case at hand as:
$$
p _ {\theta} (o \mid f) = \prod_ {k = 1} ^ {K} p _ {\theta} \left(o _ {k} \mid f\right) \tag {13}
$$
These models can be approximated using deep neural networks as introduced in the next section. We train deep neural networks on our training sets, using a cross-entropy loss function and a sub-gradient descent method. Given the trained models, we can then answer if it is Halsbury's view or Goodhart's that is more widely used by the ECtHR judiciary.
All experiments are conducted using a LONGFORMER classifier (Beltagy et al., 2020).<sup>6</sup> The
LONGFORMER is built on the same TRANSFORMER (Vaswani et al., 2017) architecture as BERT (Devlin et al., 2019), but allows for up to 4,096 tokens, using an attention mechanism which scales linearly, instead of quadratically. We choose this architecture in particular as it achieves state-of-the-art performance in tasks similar to ours, e.g. on the IMDB sentiment classification (Maas et al., 2011) and Hyperpartisan news detection (Kiesel et al., 2019).
To find the probability of violation of the  $K$  Articles we compute:
$$
\mathbf {h} = \text {L O N G F O R M E R} (g, f) \tag {14}
$$
$$
p _ {\theta} (o \mid g, f) = \sigma (W ^ {(1)} \operatorname {R e L U} (W ^ {(2)} \mathbf {h}))
$$
where  $\mathbf{h} \in \mathbb{R}^{d_1}$  is a high dimensional representation,  $W^{(1)} \in \mathbb{R}^{K \times d_2}$  and  $W^{(2)} \in \mathbb{R}^{d_2 \times d_1}$  are learnable parameters in linear projections, and  $\sigma$  is the sigmoid function. Eq. (14) will thus output a  $K$ -dimensional vector with the probabilities for all articles, by indexing this vector we retrieve the probabilities of the individual articles applying. Due to resource limitations we set the models' hidden size to 50 and batch size to 16, and also truncate individual cases to 512 tokens. For the models  $p_{\theta}(o_k \mid g, f)$  and  $p_{\theta}(o_k \mid h, f)$ , which are trained on the combination of  $f$  and either  $h$  or  $g$ , we concatenate cases to the maximum length of 1,024 tokens (as exemplified in Figure 2). While we do not fully utilise the 4,096 word limit of the LONGFORMER, we are able to process twice as many tokens as standard BERT without pooling; memory limitations prevent us from using the full 4,096 tokens, though.
Our LONGFORMER models are implemented using the Pytorch (Paszke et al., 2019) and Hugginface (Wolf et al., 2020) Python libraries. We train all our models on 4 Nvidia P100 16GiB GPU's for a maximum of 6 hours using LONGFORMER-base model. Our results are reported in terms of the models cross entropy.
Model Input	Hθ	MI	U
Facts	2.99	-	-
Goodhart	2.81	0.18	6%
Halsbury	2.68	0.31	10%
Table 1: The cross entropy  ${\mathrm{H}}_{\theta }$  ,mutual information MI and uncertainty coefficient U results.
Our code is available here: https://github.com/valvoda/Precedent.
Figure 3: Uncertainty coefficient for the Articles of the ECHR Convention.
Our experimental results are contained in Table 1. We first note that both our mutual information estimates are statistically larger than zero, i.e. Goodhart's and Halsbury's cross-entropies are statistically smaller than that of the Facts. $^{7}$  The question we asked ourselves at the outset, though, concerns whether the data supports Halsbury's or Goodhart's view. We find that our estimate of  $\mathrm{MI}(O;H\mid F)$  is significantly larger at 0.31 nats than our estimate of  $\mathrm{MI}(O;G\mid F)$  at 0.18 nats. These results suggest that the information contributed by the precedent arguments give us nearly  $58\%$  more information about the outcome of the case than the information contained in the facts of the precedent. In terms of the uncertainty coefficient, the outcome entropy is reduced by  $6\%$  for facts and by  $10\%$  for arguments. We therefore observe that Halsbury's view is more widely used in the domain of ECtHR than Goodhart's.
6 Discussion & Analysis
A more nuanced story can be told if we inspect the individual Articles even though the small number of cases per Article does not allow for conclusive significance tests. The core rights of the Convention are contained in (Articles 2-18).<sup>8</sup> Figure 3 shows that for some of the core Articles, we see the opposite effect from what we observed
for the entirety of Articles, namely that facts outperform arguments, in particular for Articles 2, 4, 9, 11, 13, 18.
We hypothesise that the reason for this is either that the judges have not yet developed a functional legal method for these Articles, that the relevant precedent has been placed late in the list of precedents (and thus was truncated away by our methodology), or that the complexity of the arguments requires a reasoning ability our models are simply not capable of. We consider each hypothesis separately below.
6.1 Conceptual Uncertainty