SlowGuess/NAACL_2021_OCR huggingface dataset:
For some Articles, it is more difficult to develop a legal method than for others because the logic of the argument is elusive for some reason. This holds, for instance, for Articles encoding a vague concept such as "right to life", cf. the discussion below. If a case deals with such an Article, the argument of a potential precedent will be less useful to determine the outcome. We hypothesise that in such a case the judges will be more willing to depart from the logic of past cases, which they might perceive as unsatisfactory in search of a better legal reasoning. However, judges strive to maintain consistency between decisions as their authority is based on this consistency. Under these conditions, a judge might take the approach of trying to find precedent cases that match the current case in terms of facts even if not in terms of logic. Case law dealing with such Articles would therefore be more likely to follow Goodhart's view.
To support or disprove this hypothesis would require an in-depth legal analysis far beyond the scope of this paper; one would need to robustly argue why judges find it relatively more difficult to develop legal reasoning for certain articles. However, looking at the Articles where our data indicate that Goodhart's view is the one more widely used, it seems to us that they indeed concern legal concepts that are more slippery than others, which we categorised as follows.
6.1.1 Corporal Articles
We can contrast Articles 2 and 4, where judges follow Goodhart's view, to Article 3, for which judges follow Halsbury's view instead, see Table 2. All three Articles are concerned with the fundamental respect of human life, and we therefore consider them together as the corporal Articles.
Article 2: Right to Life prohibits the intentional
<sup>7</sup>We measure significance using the two tailed paired permutation tests with  $p < 0.05$  after Benjamini and Hochberg's (1995) correction.
<sup>8</sup>The Convention Section 1 is the first section of the ECHR and elevates some of the Universal Declaration of Human Rights principles into actionable rights of European citizens (Schindler, 1962).
Art	Hθ(Ok|F)	Goodhart	Halsbury
MI	U	MI	U
2	0.065	0.014	21.97%	0.010	15.27%
3	0.272	0.028	10.15%	0.047	17.23%
4	0.028	0.020	71.26%	0.011	39.27%
5	0.275	0.019	7.05%	0.021	7.53%
6	0.493	0.042	8.50%	0.089	17.95%
7	0.024	-0.003	-12.01%	-0.000	-1.52%
8	0.298	0.063	21.15%	0.084	28.33%
9	0.022	0.005	23.14%	-0.003	-15.74%
10	0.173	0.003	1.92%	0.034	19.90%
11	0.074	0.018	24.29%	-0.004	-5.66%
12	0.006	-0.001	-11.09%	0.003	46.60%
13	0.235	-0.000	-0.10%	-0.006	-2.38%
14	0.071	-0.005	-7.30%	-0.005	-7.28%
18	0.031	-0.003	-10.00%	-0.007	-24.01%
Table 2: The cross-entropy  $\mathrm{H}_{\theta}$ , mutual information MI and uncertainty coefficient U results of each of the core ECHR Articles. We note that these values are empirical estimates, so negative MI results are caused by an approximation error in our models.
deprivation of life, save for circumstances where it is a penalty for a crime, in defence, during an arrest, or riot suppression. In the context of the criminal code of Europe, this is a very restricted prohibition. Every country already encodes these rules. On the other hand, it raises the difficult issues of beginning and end of life. Is Article 2 for or against abortion (Cosentino, 2015)? What is its stance on euthanasia (Hendriks, 2019)? Developing a legal test for Article 2 seems very hard indeed.
Similarly, Article 4: Prohibition of slavery and forced labour, excludes work forced in detention, compulsory military service, any service during emergency or "normal" civic obligations. Due to the large number of exceptions to the general rule it seems very hard to establish what exactly this Article does prohibit.
Let us compare these to Article 3: Prohibition of torture, where Halsbury's view prevails. This Article simply states that no one shall be subjected to torture or inhuman or degrading treatment or punishment. No exceptions are given. It seems much easier to develop a legal test for Article 3 than for Articles 2 and 4. The judges are free to establish what constitutes torture; whereas when it comes to Articles 2 and 4, they are facing many restrictions—both legal and political.
6.1.2 Faith and Family Articles
Above, we compare Articles concerned with corporal matters. In a similar way we can also group
Articles 8, 9, 10, 11 and 12 as the Articles broadly concerning belief, family and religion.
The two outliers here are Articles 9 and 11. Article 9 provides the freedom of thought, conscience and religion, Article 11 provides the freedom of assembly and association. For both Articles, Goodhart's test outperforms Halsbury's.
Just like above, the nature of Articles 9 and 11 seems more complicated compared to Article 8, which is similar, but narrower in scope: Right to respect for private and family life, Article 10: Freedom of expression and Article 12: Right to marry.
We would argue that since Articles 8 and 12 provide a right as opposed to a freedom, they define more narrowly the obligation on the part of the State. Compared to the freedom of thought and association (Articles 9 and 11), the right to marry and the right to privacy (Article 8 and 12) seem to be more concrete and testable obligations.
We can further view Article 10: freedom of expression, as dealing with an action brought about by the exercise of Article 9: freedom of thought. While similar in concept, regulating speech seems far easier in practice than regulating thought.
Finally, an inspection of the ECHR guidelines to Article 11 reveals that judges seem to be often torn between Articles 10 and 11. This is because much of the cases dealing with Article 11 concern themselves with disentangling what constitutes an expression during an assembly and conversely which assembly is a form of an expression. Many cases deal with the question of religious gathering as an assembly. This is obviously not an easy position for a judge to divine a legal test for, and perhaps a good reason for turning to the facts of the precedent cases for consistency instead.
6.2 Late Precedent
There is a group of Articles in the last quarter of Figure 3 (13, 14, 18) for which neither Goodhart's nor Halsbury's view seem to hold. We speculate that the reason for this is that these Articles never appear alone, and instead always appear in conjunction with another Article, and also that they appear late in the list of precedents, so get truncated with our methodology.
Articles 13: Right to an effective remedy, 14: Prohibition of discrimination and 18: Limitation on use of restrictions on rights, are designed to
$^{9}$ Article 11 guidance: https://www.echr.coe.int/Documents/Guide_Art_11_ENG.pdf.
ensure that states provide remedy for their wrongdoing, equal access to the rights, and do not use the restrictions in Articles for Human Rights abuse.
To claim any one of these Articles, the claimant will also have to claim a violation of one of the primary Articles as their core grievance for which they seek the remedy or equal treatment, for instance Article 3: Prohibition of torture. This means that any case dealing with Articles 13, 14 and 18 is likely to focus on the violation of that primary right.
While there might be a precedent present for the secondary Articles, the probability is high that our models will not have the chance to train on them because they appear late and because our method truncates text due to computational complexity reasons. This could explain why for these Articles, all our models trained on the precedent cases underperform when compared to the models trained on the facts of the case alone.
6.3 Model Limitations
Another possible explanation for the different behaviour between Articles could lie within the limitations of the neural architecture. There could be a model bias for facts in precedent since they are more similar to the facts at hand as opposed to the arguments. If this is the case our results understate the value of arguments. While this is a concern, the overall results of our paper would not change even if we could remove this bias since we find arguments more important than facts despite this potential handicap.
On a more nuanced level, Articles 2 and 4 above might require a higher level of reasoning than their Article 3 counterpart. So while the judges might have developed a satisfying legal test for them, our models simply aren't able to learn it. For example for Article 7: No punishment without law, our precedent models fail to learn any additional information from the precedent facts or arguments.
This might simply be the result of an insufficient representation of Article 7 in training cases, or of its appearance truncated out of the input. However it also raises the question of what a TRANSFORMER model can learn.
The nascent field of BERTology has explored exactly this question (Rogers et al., 2020; Pimentel et al., 2020). In particular the work of Niven and Kao (2019), examining BERT performance on the English Argument Reasoning Comprehension Task (Habernal et al., 2018), suggest that instead of
BERT being able to reason, it is merely very good at utilising the artefacts in the data when compared to previous approaches. As Bender and Koller (2020) contend a system can't ever learn meaning from form alone. According to their view, description of the case facts alone will never fully capture the reality of the world the claimant inhabits.
On the other hand, there is some evidence towards transformers being able to reason over simple sentences Clark et al. (2020). While this is encouraging, legal documents are far more complicated than the simple sentences considered in the study above. Either way, the models' ability to reason in the way a human lawyer would is certainly limited and could explain the diminished performance for the more complicated Articles.
In this section, we contextualise our work with relation to the related research on legal AI. Computational approaches to solving legal problems go back at least as far as the late 1950's (Kort, 1957; Nagel, 1963). Early research has focused on crafting rule-based systems for case outcome prediction, achieving human-like performance by the early 2000's (Ashley, 2017). These systems however proved too brittle to keep up with the ever-changing legal landscape and never transitioned from research into industry.
More recently, a new wave of deep learning methods has reinvigorated the research interest in legal AI. The majority of this new work has been conducted on statutory legal systems which do not rely on the doctrine of precedent to nearly the same extent as their common law counterparts. For instance, in Chinese law the use of neural models for case outcome classification has already been investigated extensively (Hu et al., 2018; Zhong et al., 2018; Xu et al., 2020). In the precedential legal domain, smaller corpora of annotated cases have been investigated over the years (Grover et al., 2003; Valvoda et al., 2018). However, large-scale corpora necessary for deep learning architectures have become available only recently. The Caselaw Access Project<sup>10</sup> introduced a large dataset of American case law in 2018. Aletras et al. (2016) have introduced the ECtHR corpus, and Chalkidis et al. (2019) have run deep neural networks on it in order to predict outcome. Similarly, the Canadian Supreme Court Case corpus has been used in infor
10Caselaw Access Project:, https://case.law
mation retrieval for the first time by Rabelo et al. (2020). This improved access to a high quality common law datasets has opened up a potential for new work in the field of legal AI.
Particularly similar to our work is the study done by Sim et al. (2016), who have considered the influence of petitioners and responders (amicus) briefs on the US Supreme Court decision and opinions.
In this paper, we have shifted the focus of legal AI research from practical tasks such as precedent retrieval or outcome prediction, to a theoretical question: which aspect of the precedent is most important in forming the law? To this end, we trained a similar neural modeling approach as Chalkidis et al. (2019) to predict the outcome of a case on the ECtHR dataset, and inspected the difference in the mutual information between our operationalisations of Halsbury's and Goodhart's view. We have used a method inspired by Pimentel et al. (2019) to approximate the MI. We observe that out of the two archetypal views on precedent, that of Halsbury and Goodhart, the former has a better empirical support in the domain of ECtHR case law.
This study has demonstrated a novel method of approaching jurisprudential questions using the information-theoretic toolkit. We hope that future work can leverage our methodology towards answering other questions of legal philosophy. However, our results are not only of an interest in the context of legal theory, but they can also inform a development of better legal models in practice. Since most precedential reasoning is conducted using the arguments in the precedent, outcome prediction models should take advantage of the case arguments, instead of relying solely on the facts.
While our work is not concerned with a legal application, it is important to note that the results presented here are qualified by the limitations of contemporary NLP models' ability to process language. It should therefore serve as no indication that judges could (or should) be replaced by models or techniques discussed in this paper.
We are grateful to Prof. Ken Satoh for all the fruitful discussions leading towards this paper. We further thank the National Institute of Informatics
(NII) Japan and Huawei research UK for their financial support enabling this research.
Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preotjuc-Pietro, and Vasileios Lampos. 2016. Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective. PeerJ Computer Science, 2:e93.
Larry Alexander and Emily Sherwin. 2008. *The Mystification of Common-Law Reasoning*, Cambridge Introductions to Philosophy and Law, page 64-103. Cambridge University Press.
Kevin D. Ashley. 2017. Artificial Intelligence and Legal Analytics. Cambridge University Press, Cambridge.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, Online. Association for Computational Linguistics.
Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289-300.
Henry Black. 2019. Black's Law Dictionary, 11th edition. Thomson Reuters.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317-4323, Florence, Italy. Association for Computational Linguistics.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898-2904, Online. Association for Computational Linguistics.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as Soft Reasoners over Language. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3882-3890. International Joint Conferences on Artificial Intelligence Organization. Main track.
Chiara Cosentino. 2015. Safe and legal abortion: An emerging human right? the long-lasting dispute with state sovereignty in ECHR jurisprudence. Human Rights Law Review, 15(3):569-589.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Pintip Hompluem Dunn. 2003. How judges overrule: Speech act theory and the doctrine of stare decisis. The Yale Law Journal, 113(2):493-531.
Neil Duxbury. 2008. Distinguishing, overruling and the problem of self-reference, page 111-149. Cambridge University Press.
ECtHR. 2014. European court of human rights: Questions & answers for lawyers.
Arthur L. Goodhart. 1930. Determining the ratio decidendi of a case. The Yale Law Journal, 40(2):161-183.
Claire Grover, Ben Hachey, Ian Hughson, and Chris Korycinski. 2003. Automatic summarisation of legal documents. In Proceedings of the 9th International Conference on Artificial Intelligence and Law, ICAIL '03, page 243-251, New York, NY, USA. Association for Computing Machinery.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1930-1940, New Orleans, Louisiana. Association for Computational Linguistics.
Lord Halsbury. 1907. Halsbury's Laws of England, 1 edition. LexisNexis Butterworths.
Arend Cornelis Hendriks. 2019. End-of-life decisions. Recent jurisprudence of the European Court of Human Rights. *ERA Forum*, 19(4):561-570.
Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-shot charge prediction with discriminative legal attributes. In Proceedings of the 27th International Conference on Computational Linguistics, pages 487-498, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Marc Jacob. 2014. *Precedents and Case-Based Reasoning in the European Court of Justice: Unfinished Business*. Cambridge University Press.
Matti Joutsen. 2019. International and Transnational Crime and Justice, 2 edition. Cambridge University Press.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval 2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829-839, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Fred Kort. 1957. Predicting supreme court decisions mathematically: A quantitative analysis of the "right to counsel" cases. American Political Science Review, 51(1):1-12.
Grant Lamond. 2005. Do precedents create rules? *Legal Theory*, 11(1):1-26.
Grant Lamond. 2016. Precedent and Analogy in Legal Reasoning. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, spring 2016 edition. Metaphysics Research Lab, Stanford University.
Yonatan Lupu and Erik Voeten. 2010. The role of precedent at the european court of human rights: A network analysis of case citations.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
David McAllester and Karl Stratos. 2020. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, pages 875-884.
Stuart S. Nagel. 1963. Applying correlation analysis to case prediction. *Texas Law Review*, 42:1006.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8026-8037.
Tiago Pimentel, Arya D. McCarthy, Damian Blasi, Brian Roark, and Ryan Cotterell. 2019. Meaning to form: Measuring systematicity as information. In
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics.
Tiago Pimentel, Brian Roark, Søren Wichmann, Ryan Cotterell, and Damian Blasi. 2021. Finding concept-specific biases in form-meaning associations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Virtual. Association for Computational Linguistics.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609-4622, Online. Association for Computational Linguistics.
Juliano Rabelo, Mi-Young Kim, Randy Goebel, Masaharu Yoshioka, Yoshinobu Kano, and Ken Satoh. 2020. A summary of the COLIEE 2019 competition. In New Frontiers in Artificial Intelligence, pages 34-49, Cham. Springer International Publishing.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Dietrich Schindler. 1962. The European convention on human rights in practice. Washington University Law Review, page 152.
C. E. Shannon and W. Weaver. 1962. The Mathematical Theory of Communication. University of Illinois Press.
Yanchuan Sim, Bryan Routledge, and Noah A. Smith. 2016. Friends with motives: Using text to infer influence on SCOTUS. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1724-1733, Austin, Texas. Association for Computational Linguistics.
James F. Spriggs and Thomas G. Hansford. 2001. Explaining the overruling of u.s. supreme court precedent. The Journal of Politics, 63(4):1091-1111.
Henri Theil. 1970. On the estimation of relationships involving qualitative variables. American Journal of Sociology, 76(1):103-154.
Josef Valvoda, Oliver Ray, and Ken Satoh. 2018. Using agreement statements to identify majority opinion in UKHL case law. In Legal Knowledge and Information Systems - JURIX 2018: The Thirty-first Annual Conference, Groningen, The Netherlands, 12-14 December 2018, volume 313 of Frontiers in Artificial Intelligence and Applications, pages 141-150. IOS Press.