SlowGuess/NAACL_2021_OCR huggingface dataset:
Ilias Chalkidis†‡ Nikolaos Aletras
Manos Fergadiotis†‡ Ion Androutsopoulos‡†
Dimitrios Tsarapatsanis* Prodromos Malakasiotis†‡
† EY AI Centre of Excellence in Document Intelligence, NCSR “Demokritos”
$^{\ddagger}$  Department of Informatics, Athens University of Economics and Business
Computer Science Department, University of Sheffield
\* Law School, University of York
Interpretability or explainability is an emerging research field in NLP. From a user-centric point of view, the goal is to build models that provide proper justification for their decisions, similar to those of humans, by requiring the models to satisfy additional constraints. To this end, we introduce a new application on legal text where, contrary to mainstream literature targeting word-level rationales, we conceive rationales as selected paragraphs in multi-paragraph structured court cases. We also release a new dataset comprising European Court of Human Rights cases, including annotations for paragraph-level rationales. We use this dataset to study the effect of already proposed rationale constraints, i.e., sparsity, continuity, and comprehensiveness, formulated as regularizers. Our findings indicate that some of these constraints are not beneficial in paragraph-level rationale extraction, while others need re-formulation to better handle the multi-label nature of the task we consider. We also introduce a new constraint, singularity, which further improves the quality of rationales, even compared with noisy rationale supervision. Experimental results indicate that the newly introduced task is very challenging and there is a large scope for further research.
Model interpretability (or explainability) is an emerging field of research in NLP (Lipton, 2018; Jacovi and Goldberg, 2020). From a model-centric point of view, the main focus is to demystify a model's inner workings, for example targeting self-attention mechanisms (Jain and Wallace, 2019; Wegreffe and Pinter, 2019), and more recently Transformer-based language models (Clark et al., 2019; Kovaleva et al., 2019; Rogers et al., 2020). From a user-centric point of view, the main focus is to build models that learn to provide proper
justification for their decisions, similar to those of humans, (Zaidan et al., 2007; Lei et al., 2016; Chang et al., 2019; Yu et al., 2019) by requiring the models to satisfy additional constraints.
Here we follow a user-centric approach to rationale extraction, where the model learns to select a subset of the input that justifies its decision. To this end, we introduce a new application on legal text where, contrary to mainstream literature targeting word-level rationales, we conceive rationales as automatically selected paragraphs in multiparagraph structured court cases. While previous related work targets mostly binary text classification tasks (DeYoung et al., 2020), our task is a highly skewed multi-label text classification task. Given a set of paragraphs that refer to the facts of each case (henceforth facts) in judgments of the European Court of Human Rights (EChR), the model aims to predict the allegedly violated articles of the European Convention of Human Rights (ECHR). We adopt a rationalization by construction methodology (Lei et al., 2016; Chang et al., 2019; Yu et al., 2019), where the model is regularized to satisfy additional constraints that reward the model, if its decisions are based on concise rationales it selects, as opposed to inferring explanations from the model's decisions in a post-hoc manner (Ribeiro et al., 2016; Alvarez-Melis and Jaakkola, 2017; Murdoch et al., 2018).
Legal judgment prediction has been studied in the past for cases ruled by the European Court of Human Rights (Aletras et al., 2016; Medvedeva et al., 2018; Chalkidis et al., 2019) and for Chinese criminal court cases (Luo et al., 2017; Hu et al., 2018; Zhong et al., 2018), but there is no precedent of work investigating the justification of the models' decisions. Similarly to other domains (e.g., financial, biomedical), explainability is a key feature in the legal domain, which may potentially improve the trustworthiness of systems that abide by the principle of the right to explanation (Goodman and
Correspondence to: ihalk.aueb.gr
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 226-241 June 6-11, 2021. ©2021 Association for Computational Linguistics
1. The applicant was born in 1963 and lives in Honolulu, Hawaii.
2. The applicant was arrested by four police officers on suspicion for [...] According to the applicant, the applicant was beaten by the police officers.
3. The applicant was admitted to the hospital, where he was diagnosed with bruised ribs, a broken tooth, and concussion.
4. On May 2023, criminal charges were instituted against the applicant by the prosecutor's office.
5. On June 2023, the courts of  $X$  ruled the imprisonment of the applicant for [...].
6. Following his conviction, the applicant was jailed in the correctional centre of  $X$  for 2 years.
ECtHR Court
Application & Trial
Alleged Violations
- Article 3 - Prohibition of torture
- Article 6 - Right to a fair trial
The Court's decision
- Violation of Article 3.
Non violation of Article 6.
The parties' submissions
The applicant said that [...]
The defendant did not dispute [...]
The Court's assessment
- Applicant's beating (see p. 2) [...]
- Furthermore [...] (see p. 3) [...]
Figure 1: A depiction of the ECtHR process: The applicant(s) request a hearing from ECtHR regarding specific accusations (alleged violations of ECHR articles) against the defendant state(s), based on facts. The Court (judges) assesses the facts and the rest of the parties' submissions, and rules on the violation or not of the allegedly violated ECHR articles. Here, prominent facts referred in the court's assessment are highlighted.
Flaxman, 2017). We investigate the explainability of the decisions of state-of-the-art models, comparing the paragraphs they select to those of legal professionals, both litigants and lawyers, in alleged violation prediction. In the latter task, introduced in this paper, the goal is to predict the accusations (allegations) made by the applicants. The accusations can be usually predicted given only the facts of each case. By contrast, in the previously studied legal judgment prediction task, the goal is to predict the court's decision; this is much more difficult and vastly relies on case law (precedent cases).
Although the new task (alleged violation prediction) is simpler than legal judgment prediction, models that address it (and their rationales) can still be useful in the judicial process (Fig. 1). For example, they can help applicants (plaintiffs) identify alleged violations that are supported by the facts of a case. They can help judges identify more quickly facts that support the alleged violations, contributing towards more informed judicial decision making (Zhong et al., 2020). They can also help legal experts identify previous cases related to particular allegations, helping analyze case law (Katz, 2012). Our contributions are the following:
- We introduce rationale extraction for alleged violation prediction in ECtHR cases, a more tractable task compared to legal judgment prediction. This is a multi-label classification task that requires paragraph-level rationales, unlike previous work on word-level rationales for binary classification.
- We study the effect of previously proposed rationale constraints, i.e., sparsity, continuity (Lei et al., 2016), and comprehensiveness (Yu et al., 2019), formulated as regularizers. We show
that continuity is not beneficial and requisite in paragraph-level rationale-extraction, while comprehensiveness needs to be re-formulated for the multi-label nature of the task we consider. We also introduce a new constraint, singularity, which further improves the rationales, even compared with silver (noisy) rationale supervision.
- We release a new dataset for alleged article violation prediction, comprising 11k ECtHR cases in English, with silver rationales obtained from references in court decisions, and gold rationales provided by ECHR-experienced lawyers.
To the best of our knowledge, this is also the first work on rationale extraction that fine-tunes end-to-end pre-trained Transformer-based models.2
Legal judgment prediction: Initial work on legal judgment prediction in English used linear models with features based on bags of words and topics, applying them to ECtHR cases (Aletras et al., 2016; Medvedeva et al., 2018). More recently, we experimented with neural methods (Chalkidis et al., 2019), showing that hierarchical RNNs (Yang et al., 2016), and a hierarchical variation of BERT (Devlin et al., 2019) that encodes paragraphs, outperform linear classifiers with bag-of-word representations.
In all previous work, legal judgment prediction is tackled in an over-simplified experimental setup
<sup>1</sup>Our dataset is publicly available at https://huggingface.co/datasets/ecthr_cases, see usage example in Appendix E.
2Others fine-tuned such models only partially (Jain et al., 2020), i.e., top two layers, or not at all (DeYoung et al., 2020).
where only textual information from the cases themselves is considered, ignoring many other important factors that judges consider, more importantly general legal argument and past case law. Also, Aletras et al. (2016), Medvedeva et al. (2018), Chalkidis et al. (2019) treat ECtHR judgment prediction as a binary classification task per case (any article violation or not), while the ECtHR actually considers and rules on the violation of individual articles of the European Convention of Human Rights (ECHR).
In previous work (Chalkidis et al., 2019), we also attempted to predict which particular articles were violated, assuming, however, that the Court considers all the ECHR articles in each case, which is not true. In reality, the Court considers only alleged violations of particular articles, argued by applicants. Establishing which articles are allegedly violated is an important preliminary task when preparing an ECTHR application. Instead of oversimplifying the overall judgment prediction task, we focus on the preliminary task and use it as a test-bed for generating paragraph-level rationales in a multi-label text classification task for the first time.
Legal judgment prediction has also been studied in Chinese criminal cases (Luo et al., 2017; Hu et al., 2018; Zhong et al., 2018). Similarly to the literature on legal judgment prediction for ECtHR cases, the aforementioned approaches ignore the crucial aspect of justifying the models' predictions.
Given the gravity that legal outcomes have for individuals, explainability is essential to increase the trust of both legal professionals and laypersons on system decisions and promote the use of supportive tools (Barfield, 2020). To the best of our knowledge, our work is the first step towards this direction for the legal domain, but is also applicable in other domains (e.g., biomedical), where justifications of automated decisions are essential.
Rationale extraction by construction: Contrary to earlier work that required supervision in the form of human-annotated rationales (Zaidan et al., 2007; Zhang et al., 2016), Lei et al. (2016) introduced a self-supervised methodology to extract rationales (that supported aspect-based sentiment analysis predictions), i.e., gold rationale annotations were used only for evaluation. Furthermore, models were designed to produce rationales by construction, contrary to work studying saliency maps (generated by a model without explainability constraints) using gradients or perturbations at inference time (Ribeiro et al., 2016; Alvarez-Melis and Jaakkola,
2017; Murdoch et al., 2018). Lei et al. (2016) aimed to produce short coherent rationales that could replace the original full texts, maintaining the model's predictive performance. The rationales were extracted by generating binary masks indicating which words should be selected; and two additional loss regularizers were introduced, which penalize long rationales and sparse masks (that would select non-consecutive words).
Yu et al. (2019) proposed another constraint to ensure that the rationales would contain all the relevant information. They formulated this constraint through a minimax game, where two players, one using the predicted binary mask and another using the complement of this mask, aim to correctly classify the text. If the first player fails to outperform the second, the model is penalized. Chang et al. (2019) use a Generative Adversarial Network (GAN) (Goodfellow et al., 2014), where a generator producing factual rationales competes with a generator producing counterfactual rationales to trick a discriminator. The GAN was not designed to perform classification. Given a text and a label it produces a rationale supporting (or not) the label.
Jain et al. (2020) decoupled the model's predictor from the rationale extractor to produce inherently faithful explanations, ensuring that the predictor considers only the rationales and not other parts of the text. Faithfulness refers to how accurately an explanation reflects the true reasoning of a model (Lipton, 2018; Jacovi and Goldberg, 2020).
All the aforementioned work conceives rationales as selections of words, targeting binary classification tasks even when this is inappropriate. For instance, DeYoung et al. (2020) and Jain et al. (2020) over-simplified the task of the multipassage reading comprehension (MultiRC) dataset (Khashabi et al., 2018) turning it into a binary classification task with word-level rationales, while sentence-level rationales seem more suitable.
Responsible AI: Our work complies with the ECtHR data policy. By no means do we aim to build a 'robot' lawyer or judge, and we acknowledge the possible harmful impact (Angwin et al., 2016; Dressel and Farid, 2018) of irresponsible deployment. Instead, we aim to support fair and explainable AI-assisted judicial decision making and empirical legal studies. We consider our work as part of ongoing critical research on responsible AI (Elish et al., 2021) that aims to provide explainable and fair systems to support human experts.
Cases	Sparsity	#Allegations
Train	9K	24%	1.8
Development	1K	30%	1.7
Test	1K	31%	1.7
Table 1: Statistics of the new ECtHR dataset. 'Sparsity' is the average percentage of paragraphs included in the silver rationales. '#Allegations' is the average number of allegedly violated articles.
3 The New ECtHR Dataset
The court (ECTHR) hears allegations regarding breaches in human rights provisions of the European Convention of Human Rights (ECHR) by European states (Fig. 1). The court rules on a subset of all ECHR articles, which are predefined (alleged) by the applicants (plaintiffs). Our dataset comprises  $11\mathrm{k}$  ECTHR cases and can be viewed as an enriched version of the ECTHR dataset of Chalkidis et al. (2019), which did not provide ground truth for alleged article violations (articles discussed) and rationales. The new dataset includes the following:
Facts: Each judgment includes a list of paragraphs that represent the facts of the case, i.e., they describe the main events that are relevant to the case, in numbered paragraphs. We hereafter call these paragraphs facts for simplicity. Note that the facts are presented in chronological order. Not all facts have the same impact or hold crucial information with respect to alleged article violations and the court's assessment; i.e., facts may refer to information that is trivial or otherwise irrelevant to the legally crucial allegations against defendant states.
Allegedly violated articles: Judges rule on specific accusations (allegations) made by the applicants (Harris, 2018). In ECtHR cases, the judges discuss and rule on the violation, or not, of specific articles of the Convention. The articles to be discussed (and ruled on) are put forward (as alleged article violations) by the applicants and are included in the dataset as ground truth; we identify 40 violable articles in total.4 In our experiments, however, the models are not aware of the allegations. They predict the Convention articles that will be discussed (the allegations) based on the case's facts, and they also produce rationales for their predictions. Models of this kind could be used by potential applicants to help them formulate future allegations (articles they could claim to have been
violated), as already noted, but here we mainly use the task as a test-bed for rationale extraction.
Violated articles: The court decides which allegedly violated articles have indeed been violated. These decisions are also included in our dataset and could be used for full legal judgment prediction experiments (Chalkidis et al., 2019). However, they are not used in the experiments of this work.
Silver allegation rationales: Each decision of the ECtHR includes references to facts of the case (e.g., "See paragraphs 2 and 4.") and case law (e.g., "See Draci vs. Russia (2010)."). We identified references to each case's facts and retrieved the corresponding paragraphs using regular expressions. These are included in the dataset as silver allegation rationales, on the grounds that the judges refer to these paragraphs when ruling on the allegations.
Gold allegation rationales: A legal expert with experience in ECtHR cases annotated a subset of 50 test cases to identify the relevant facts (paragraphs) of the case that support the allegations (alleged article violations). In other words, each identified fact justifies (hints) one or more alleged violations.[5]
Task definition: In this work, we investigate alleged violation prediction, a multi-label text classification task where, given the facts of a ECtHR case, a model predicts which of the 40 violable ECHR articles were allegedly violated according to the applicant(s). The model also needs to identify the facts that most prominently support its decision.
We first describe a baseline model that we use as our starting point. It adopts the framework proposed by Lei et al. (2016), which generates rationales by construction: a text encoder sub-network reads the text; a rationale extraction sub-network produces a binary mask indicating the most important words of the text; and a prediction sub-network classifies a hard-masked version of the text. We then discuss additional constraints that have been proposed to improve word-level rationales, which can be added to the baseline as regularizers. We argue that one of them is not beneficial for paragraph-level rationales. We also consider variants of previous constraints that better suit multi-label classification tasks and introduce a new one.
For details on the annotation process and examples of annotated EChR cases, see Appendices C, F.
3The Convention is available at https://www.echr. coe.int/Documents/Convention_ENG.pdf. 4The rest of the articles are procedural, i.e., the number of judges, criteria for office, election of judges, etc.
4.1 Baseline Model
Our baseline is a hierarchical variation of BERT (Devlin et al., 2019) with hard attention, dubbed HIERBERT-HA. Each case (document)  $D$  is viewed as a list of facts (paragraphs)  $D = [P_{1},\ldots ,P_{N}]$ . Each paragraph is a list of tokens  $P_{i} = [w_{1},\dots ,w_{L_{i}}]$ . We first pass each paragraph independently through a shared BERT encoder (Fig. 2) to extract context-unaware paragraph representations  $P_{i}^{\mathrm{CLS}}$ , using the [CLS] embedding of BERT. Then, a shallow encoder with two Transformer layers (Vaswani et al., 2017) produces contextualized paragraph embeddings, which are in turn projected to two separate spaces by two different fully-connected layers,  $K$  and  $Q$ , with SELU activations (Klambauer et al., 2017).  $K$  produces the paragraph encoding  $P_{i}^{K}$ , to be used for classification; and  $Q$  produces the paragraph encoding  $P_{i}^{Q}$ , to be used for rationale extraction. The rationale extraction sub-network passes each  $P_{i}^{Q}$  encoding independently through a fully-connected layer with a sigmoid activation to produce soft attention scores  $a_{i}\in [0,1]$ . The attention scores are then binarized using a 0.5 threshold, leading to hard attention scores  $z_{i}$  ( $z_{i} = 1$  iff  $a_{i} > 0.5$ ). The hard-masked document representation  $D_{M}$  is obtained by hard-masking paragraphs and max-pooling:
$$
D _ {M} = \mathrm {m a x p o o l} \big ([ z _ {1} \cdot P _ {1} ^ {K}, \ldots , z _ {N} \cdot P _ {N} ^ {K} ] \big)
$$
$D_{M}$  is then fed to a dense layer with sigmoid activations, which produces a probability estimate per label,  $\widehat{Y} = [\hat{y}_1,\dots ,\hat{y}_{\hat{A} |}]$ , in our case per article of the Convention, where  $|A|$  is the size of the label set. For comparison, we also experiment with a model that masks no facts, dubbed HERBERT-ALL.
The thresholding that produces the hard (binary) masks  $z_{i}$  is not differentiable. To address this problem, Lei et al. (2016) used reinforcement learning (Williams, 1992), while Bastings et al. (2019) proposed a differentiable mechanism relying on the reparameterization trick (Louizos and Welling, 2017). We follow a simpler trick, originally proposed by Chang et al. (2019), where during backpropagation the thresholding is detached from the computation graph, allowing the gradients to bypass the thresholding and reach directly the soft attentions  $a_{i}$ .
Figure 2: Illustration of HIERBERT-HA. The shaded parts operate only when  $Lg$  or  $Lr$  are used.
4.2 Rationale Constraints as Regularizers
Sparsity: Modifying the word-level sparsity constraint of Lei et al. (2016) for our paragraph-level rationales, we also hypothesize that good rationales include a small number of facts (paragraphs) that sufficiently justify the allegations; the other facts are trivial or secondary. For instance, an introductory fact like "The applicant was born in 1984 and lives in Switzerland." does not support any allegation, while a fact like "The applicant contended that he had been beaten by police officers immediately after his arrest and later during police questioning." suggests a violation of Article 3 "Prohibition of Torture". Hence, we use a sparsity loss to control the number of selected facts:
$$
L _ {s} = \left| T - \frac {1}{N} \sum_ {i = 1} ^ {N} z _ {i} \right| \tag {1}
$$
where  $T$  is a predefined threshold specifying the desired percentage of selected facts per case. We can estimate  $T$  from silver rationales (Table 1).
Continuity: In their work on word-level rationales, Lei et al. (2016) also required the selected words to be contiguous, to obtain more coherent rationales. In other words, the transitions between selected  $(z_{i} = 1)$  and not selected  $(z_{i} = 0)$  words in the hard mask should be minimized. This is achieved by adding the following continuity loss:
$$
L _ {c} = \frac {1}{N - 1} \sum_ {i = 2} ^ {N} \left| z _ {i} - z _ {i - 1} \right| \tag {2}
$$
<sup>6</sup>In previous work, we proposed a hierarchical variation of BERT with self-attention (Chalkidis et al., 2019). In parallel work, Yang et al. (2020) proposed a similar Transformer-based Hierarchical Encoder (SMITH) for long document matching.
In paragraph-level rationale extraction, where entire paragraphs are masked, the continuity loss forces the model to select contiguous paragraphs. In EChHR cases, however, the facts are self-contained and internally coherent paragraphs (or single sentences). Hence, we hypothesize that the continuity loss is not beneficial in our case. Nonetheless, we empirically investigate its effect.