SlowGuess/NAACL_2021_OCR huggingface dataset:
Comprehensiveness: We also adapt the comprehensiveness loss of Yu et al. (2019), which was introduced to force the hard mask  $Z = [z_{1},\ldots ,z_{N}]$  to (ideally) keep all the words (in our case, paragraphs about facts) of the document  $D$  that support the correct decision  $Y$ . In our task,  $Y = [y_{1},\dots ,y_{|A|}]$  is a binary vector indicating the Convention articles the court discussed (gold allegations) in the case of  $D$ . Intuitively, the complement  $Z^{c}$  of  $Z$ , i.e., the hard mask that selects the words (in our case, facts) that  $Z$  does not select, should not select sufficient information to predict  $Y$ . Given  $D$ , let  $D_M, D_M^c$  be the representations of  $D$  obtained with  $Z, Z^{c}$ , respectively; let  $\widehat{Y}, \widehat{Y}^{c}$  be the corresponding probability estimates; let  $L_p, L_p^c$  be the classification loss, typically total binary cross-entropy, measuring how far  $\widehat{Y}, \widehat{Y}^{c}$  are from  $Y$ . In its original form, the comprehensiveness loss requires  $L_p^c$  to exceed  $L_p$  by a margin  $h$ .
$$
L _ {g} = \max  \left(L _ {p} - L _ {p} ^ {c} + h, 0\right) \tag {3}
$$
While this formulation may be adequate in binary classification tasks, in multi-label classification it is very hard to pre-select a reasonable margin, given that cross-entropy is unbounded, that the distribution of true labels (articles discussed) is highly skewed, and that some labels are easier to predict than others. To make the selection of  $h$  more intuitive, we propose a reformulation of  $L_{g}$  that operates on class probabilities rather than classification losses. The right-hand side of Eq. 3 becomes:
$$
\frac {1}{| A |} \sum_ {i = 1} ^ {| A |} y _ {i} \left(\hat {y} _ {i} ^ {c} - \hat {y} _ {i} + h\right) + (1 - y _ {i}) \left(\hat {y} _ {i} - \hat {y} _ {i} ^ {c} + h\right) \tag {4}
$$
The margin  $h$  is now easier to grasp and tune. It encourages the same gap between the probabilities predicted with  $Z$  and  $Z^c$  across all labels (articles).
We also experiment with a third variant of comprehensiveness, which does not compare the probabilities we obtain with  $Z$  and  $Z^c$ , comparing instead the two latent document representations:
$$
L _ {g} = \left| \cos \left(D _ {M}, D _ {M} ^ {c}\right) \right| \tag {5}
$$
where  $\cos$  denotes cosine similarity. This variant forces  $D_{M}$  and  $D_{M}^{c}$  to be as dissimilar as possible, without requiring a preset margin.
Singularity: A limitation of the comprehensiveness loss (any variant) is that it only requires the mask  $Z$  to be better than its complement  $Z^c$ . This does not guarantee that  $Z$  is better than every other mask. Consider a case where the gold rationale identifies three articles and  $Z$  selects only two of them. The model may produce better predictions with  $Z$  than with  $Z^c$ , and  $D_M$  may be very different than  $D_M^c$  in Eq. 5, but  $Z$  is still not the best mask. To address this limitation, we introduce the singularity loss  $L_r$ , which requires  $Z$  to be better than a mask  $Z^r$ , randomly generated per training instance and epoch, that selects as many facts as the sparsity threshold  $T$  allows:
$$
L _ {r} = \gamma \cdot L _ {g} \left(Z, Z ^ {r}\right) \tag {6}
$$
$$
\gamma = 1 - \cos (Z ^ {r}, Z)
$$
Here  $L_{g}(Z,Z^{r})$  is any variant of  $L_{g}$ , but now using  $Z^r$  instead of  $Z^c$ ; and  $\gamma$  regulates the effect of  $L_{g}(Z,Z^{r})$  by considering the cosine distance between  $Z^r$  and  $Z$ . The more  $Z$  and  $Z^r$  overlap, the less we care if  $Z$  performs better than  $Z^r$ .
The total loss of our model is computed as follows. Again  $L_{p}$  is the classification loss;  $L_{p}^{c}, L_{p}^{r}$  are the classification losses when using  $Z^{c}, Z^{r}$ , respectively; and all  $\lambda$ s are tunable hyper-parameters.
$$
\begin{array}{l} L = L _ {p} + \lambda_ {s} \cdot L _ {s} + \lambda_ {c} \cdot L _ {c} \\ + \lambda_ {g} \left(L _ {g} + L _ {p} ^ {c}\right) + \lambda_ {r} \left(L _ {r} + L _ {p} ^ {r}\right) \tag {7} \\ \end{array}
$$
We include  $L_p^c$  in Eq. 7, because otherwise the network would have no incentive to make  $D_M^c$  and  $\widehat{Y}^c$  competitive in prediction; and similarly for  $L_p^r$ .
Rationales supervision: For completeness we also experimented with a variant that utilizes silver rationales for noisy rationale supervision (Zaidan et al., 2007). In this case the total loss becomes:
$$
L = L _ {p} + \lambda_ {n s} \cdot \operatorname {M A E} (Z, Z ^ {s}) \tag {8}
$$
where MAE is the mean absolute error between the predicted mask,  $Z$ , and the silver mask,  $Z_{s}$ , and  $\lambda_{ns}$  weighs the effect of MAE in the total loss.
For all methods, we conducted grid-search to tune the hyper-parameters  $\lambda_{*}$ . We used the Adam optimizer (Kingma and Ba, 2015) across all experi
ments with a fixed learning rate of 2e-5.7 All methods rely on LEGAL-BERT-SMALL (Chalkidis et al., 2020), a variant of BERT (Devlin et al., 2019), with 6 layers, 512 hidden units and 8 attention heads, pre-trained on legal corpora. Based on this model, we were able to use up to 50 paragraphs of 256 words each in a single 32GB GPU. In preliminary experiments, we found that the proposed model relying on a shared paragraph encoder, i.e., one that passes the same context-aware paragraph representations  $P_{i}^{[CLS]}$  to both the  $Q$  and  $K$  sub-networks, as in Fig. 2, has comparable performance and better rationale quality, compared to a model with two independent paragraph encoders, as the one used in the literature (Lei et al., 2016; Yu et al., 2019; Jain et al., 2020). For all experiments, we report the average and standard deviation across five runs.
We evaluate: (a) classification performance, (b) faithfulness (Section 2), and (c) rationale quality, while respecting a given sparsity threshold  $(T)$ .
Classification performance: Given the label skewness, we evaluate classification performance using micro-F1, i.e., for each Convention article, we compute its F1, and micro-average over articles.
Faithfulness: Recall that faithfulness refers to how accurately an explanation reflects the true reasoning of a model. To measure faithfulness, we report sufficiency and comprehensiveness (DeYoung et al., 2020). Sufficiency measures the difference between the predicted probabilities for the gold (positive) labels when the model is fed with the whole text  $(\widehat{Y}_{+}^{f})$  and when the model is fed only with the predicted rationales  $(\widehat{Y}_{+})$ . Comprehensiveness (not to be confused with the homonymous loss of Eq. 3-5) measures the difference between the predicted probabilities for the gold (positive) labels obtained when the model is fed with the full text  $(\widehat{Y}_{+}^{f})$  and when it is fed with the complement of the predicted rationales  $(\widehat{Y}_{+}^{c})$ . We also compare classification performance (again using micro-F1) in both cases, i.e., when considering masked inputs (using  $Z$ ) and complementary inputs (using  $Z^c$ ).
Rationale quality: Faithful explanations (of system reasoning) are not always appropriate for users (Jacovi and Goldberg, 2020), thus we also evaluate rationale quality from a user perspective. The latter
can be performed in two ways. Objective evaluation compares predicted rationales with gold annotations, typically via Recall, Precision, F1 (comparing system-selected to human-selected facts in our case). In subjective evaluation, human annotators review the extracted rationales. We opt for an objective evaluation, mainly due to lack of resources. As rationale sparsity (number of selected paragraphs) differs across methods, which affects Recall, Precision, F1, we evaluate rationale quality with mean R-Precision (mRP) (Manning et al., 2009). That is, for each case, the model ranks the paragraphs it selects by decreasing confidence, and we compute Precision@k, where k is the number of paragraphs in the gold rationale; we then average over test cases. For completeness, we also report F1 (comparing predicted and gold rationale paragraphs), although it is less fair, because of the different sparsity of different methods, as noted.
ECHR article	Training cases	Classification F1↑
2 - Right to life	623	78.3 ± 2.3
3 - Prohibition of torture	1740	85.9 ± 0.9
5 - Right to liberty and security	1623	81.1 ± 1.5
6 - Right to a fair trial	5437	80.1 ± 1.0
8 - Right to respect for private life	1056	72.5 ± 1.8
10 - Freedom of expression	441	77.4 ± 1.6
11 - Freedom of assembly	162	72.1 ± 3.3
13 - Right to an effective remedy	1665	29.2 ± 3.3
14 - Prohibition of discrimination	444	44.8 ± 7.4
34 - Individual applications	547	10.0 ± 5.0
46 - Binding force of judgments	187	2.6 ± 3.2
P1-1 - Protection of property	1558	77.9 ± 1.3
Rest of the articles	< 100	< 50.0
Overall performance (micro-F1)	72.7 ± 1.2
Table 2: Classification performance of HIERBERT-ALL (no mask) across ECHR articles on development data, with respect to the number of training cases (instances).
6.1 Initial Classification Performance
Table 2 reports the classification performance of HIERBERT-ALL (no masking, no rationales), across ECHR articles. F1 is  $72.5\%$  or greater for most of the articles with 1,000 or more training instances. The scores are higher for articles 2, 3, 5, 6, because (according to the legal expert who provided the gold allegation rationales), (i) there is a sufficient number of cases regarding these articles, and (ii) the interpretation and application of these articles is more fact-dependent than those of other articles, such as articles 10 or 11 (Harris, 2018). On the other hand, although there is a fair amount of training instances for articles 13, 14, 34, and 46, these articles are triggered in a variety of ways, many of which turn on legal procedural technicalities.
In preliminary experiments, we tuned the baseline model on development data as a stand-alone classifier and found that the optimal learning rate was 2e-5, searching in the set {2e-5, 3e-5, 4e-5, 5e-5}. The optimal drop-out rate was 0.
See Appendix B for additional details and results.
Method	sparsity (aim: 30%)	Entire Input micro-F1 ↑	Masked Input (Z) micro-F1 ↑	Suff. ↓	Compl. Input (Zc) micro-F1 ↓	Comp. ↑
RANDOM CLASSIFIER	-	30.8 ± 0.3	-
HIERBERT-ALL (no masking)	-	73.7 ± 0.6	-
HIERBERT-HA + Ls (Eq. 1) (Lei et al., 2016)	31.7 ± 1.1	73.1 ± 0.6	69.5 ± 2.4	0.063	58.8 ± 1.5	0.181
HIERBERT-HA + Ls + Lg (Eq. 3) (Yu et al., 2019)	31.4 ± 1.9	72.8 ± 0.6	68.1 ± 4.4	0.069	59.0 ± 1.5	0.171
HIERBERT-HA + Ls + Lg (Eq. 5) (ours)	31.4 ± 1.3	72.6 ± 1.5	69.8 ± 0.8	0.043	59.6 ± 2.7	0.156
HIERBERT-HA + Ls + Lr (Eq. 4, 6) (ours)	31.5 ± 0.8	72.8 ± 0.5	70.5 ± 0.8	0.040	55.9 ± 2.8	0.204
HIERBERT-HA + rationale supervizion (Eq. 8)	33.1 ± 6.0	73.1 ± 0.5	69.2 ± 1.1	0.053	56.7 ± 6.6	0.191
6.2 Tuning the  $\lambda$  Hyper-parameters
Instead of tuning simultaneously all the  $\lambda_{*}$  hyperparameters of Eq. 7, we adopt a greedy, but more intuitive strategy: we tune one  $\lambda$  at a time, fix its value, and proceed to the next;  $\lambda$ s that have not been tuned are set to zero, i.e., the corresponding regularizer is not used yet. We begin by tuning  $\lambda_{s}$ , aiming to achieve a desirable level of sparsity without harming classification performance. We set the sparsity threshold of  $L_{s}$  (Eq. 1) to  $T = 0.3$  (select approx.  $30\%$  of the facts), which is the average sparsity of the silver rationales (Table 1). We found  $\lambda_{s} = 0.1$  achieves the best overall results on development data, thus we use this value for the rest of the experiments. To check our hypothesis that continuity  $(L_{c})$  is not beneficial in our task, we tuned  $\lambda_{c}$  on development data, confirming that the best overall results are obtained for  $\lambda_{c} = 0$ . Thus we omit  $L_{c}$  in the rest of the experiments.
6.3 Comprehensiveness/Singularity Variants
Next, we tuned and compared the variants of the comprehensiveness loss  $L_{g}$  (Table 4). Targeting the label probabilities (Eq. 4) instead of the losses (Eq. 3) leads to lower rationale quality. Targeting the document representations (Eq. 5) has the best rationale quality results, retaining (as with all versions of  $L_{g}$ ) the original classification performance (micro-F1) of Table 2. Hence, we keep the  $L_{g}$  variant of Eq. 5 in the remaining experiments of this section, with the corresponding  $\lambda_{g}$  value (1e-3).
Table 3: Classification performance (classification micro-F1) and faithfulness results on test data. Faithfulness is measured by considering Sufficiency (Suff.) and Comprehensiveness (Comp.), i.e., how close the label probabilities of the model are when using the rationales (masked input) or the complements of the rationales (complementary input), respectively, as opposed to using the entire input. Lower Suff.  $(\downarrow)$  and higher Comp.  $(\uparrow)$  are better. We also report micro-F1 for the masked and complementary input; higher and lower F1, respectively, are better.
Lgvariant	classification micro-F1 ↑	sparsity (aim: 30%)	rationale quality
F1 ↑	mRP ↑
Eq. 3	73.0 ± 0.5	31.4 ± 1.9	35.4 ± 5.8	38.4 ± 5.9
Eq. 4	73.1 ± 0.7	31.9 ± 1.4	30.3 ± 3.0	32.6 ± 2.6
Eq. 5	72.8 ± 0.8	31.8 ± 1.3	38.3 ± 2.3	41.2 ± 2.1
Table 4: Development results for variants of  $L_{g}$  (comprehensiveness) and varying  $\lambda_{g}$  values (omitted).
Lvariant	classification micro-F1 ↑	sparsity (aim: 30%)	rationale quality
F1 ↑	mRP ↑
Eq. 3, 6	73.4 ± 0.8	32.8 ± 2.8	36.9 ± 3.6	39.0 ± 3.9
Eq. 4, 6	72.5 ± 0.7	32.0 ± 1.0	39.7 ± 3.1	42.6 ± 3.8
Eq. 5, 6	72.8 ± 0.3	31.5 ± 0.9	33.0 ± 2.7	35.5 ± 2.6
Table 5: Development results for variants of  $L_{c}$  (singularity) and varying  $\lambda_{r}$  values (omitted).
Concerning the singularity loss  $L_{r}$  (Table 5), targeting the label probabilities (Eq. 4, 6) provides the best rationale quality, comparing to all the methods considered. Interestingly Eq. 5, which performed best in  $L_{g}$  (Table 4), does not perform well in  $L_{r}$  which uses  $L_{g}$  (Eq. 6). We suspect that in  $L_{r}$  where we use a random mask  $Z^{r}$  that may overlap with  $Z$ , requiring the two document representations  $D_{M}, D_{M}^{r}$  to be dissimilar (when using Eq. 5, 6) may be a harsh regularizer with negative effects.
6.4 Task Performance and Faithfulness
Table 3 presents results on test data. The models that use the hard attention mechanism and are regularized to extract rationales under certain constraints (HIERBERT-HA +  $L_{*}$ ) have comparable classification performance to HIERBERT-ALL. Furthermore, although paragraph embeddings are contextualized and probably have some information leak for all methods, our proposed extensions in rationale constraints better approximate faithfulness, while also respecting sparsity. Our proposed extensions lead to low sufficiency (lower is better, ↓), i.e., there is only a slight deterioration in label probabilities when we use the predicted rationale instead of the whole input. They also lead to high comprehensiveness (higher is better, ↑); we see a 20% deterioration in label probabilities when using the complement of the rationale instead of the whole input. Interestingly, our variant with the singularity loss (Eq. 4, 6) is more faithful than the model that uses supervision on silver rationales (Eq. 8).
<sup>9</sup>Consult Appendix D for more detailed results.
Method	Silver rationales (31%)	Gold rationales (36%)
mRP ↑	F1 ↑	mRP ↑	F1 ↑
RANDOM RATIONALE	30.2 ± 1.1	27.8 ± 1.1	35.1 ± 1.7	30.2 ± 2.2
HIERBERT-HA
+ Ls (Eq. 1)	43.1 ± 6.5	37.3 ± 5.4	51.9 ± 5.7	45.7 ± 5.4
+ Ls + Lg (Eq. 3)	41.0 ± 5.1	37.5 ± 6.7	48.9 ± 6.5	44.5 ± 6.8
+ Ls + Lg (Eq. 5)	43.0 ± 1.5	38.5 ± 1.9	50.9 ± 3.2	45.8 ± 3.3
+ Ls + Lr (Eq. 4, 6)	45.1 ± 2.1	40.9 ± 2.5	53.6 ± 2.3	48.3 ± 1.2
+ supervision (Eq. 8)	43.1 ± 5.0	39.1 ± 7.1	51.4 ± 6.7	46.8 ± 0.5
Table 6: Rationale quality results on the 50 test cases that have both silver and gold allegation rationales. Average silver/gold rationale sparsity  $(\%)$  in brackets.
6.5 Rationale Quality
We now consider rationale quality, focusing on HIERBERT-HA variants without rationale supervision. Similarly to our findings on development data (Tables 4, 5), we observe (Table 6) that using (a) our version of comprehensiveness loss (Eq. 5) or (b) our singularity loss (Eq. 4, 6) achieves better results compared to former methods, and (b) has the best results. The singularity loss is better in both settings (silver or gold test rationales), even compared to a model that uses supervision on silver rationales. The random masking of the singularity loss, which guides the model to learn to extract masks that perform better than any other mask, proved to be particularly beneficial in rationale quality. Similar observations are derived given the results on the full test set considering silver rationales. $^{10}$  In general, however, we observe that the rationales extracted by all models are far from human rationals, as indicated by the poor results (mRP, F1) on both silver and gold rationales. Hence, there is ample scope for further research.
6.6 Qualitative Analysis
Quality of silver rationales: Comparing silver rationales with gold ones, annotated by the legal expert, we find that silver rationales are not complete, i.e., they are usually fewer than the gold ones. They also include additional facts that have not been annotated by the expert. According to the expert, these facts do not support allegations, but are included for technical reasons (e.g., "The national court did not accept the applicant's allegations"). Nonetheless, ranking methods by their rationale quality measured on silver rationales produces the same ranking as when measuring on gold rationales in the common subset of cases (Table 6). Hence, it may be possible to use silver rationales, which are available for the full dataset, to rank systems participating in ECtHR rationale generation challenges.
Model bias: Low mRP with respect to gold rationales means that the models rely partially on non causal reasoning, i.e., they select secondary facts that do not justify allegations according to the legal expert. In other words, the models are sensitive to specific language, e.g., they misuse (are easily fooled by) references to health issues and medical examinations as support for Article 3 alleged violations, or references to appeals in higher courts as support for Article 5, even when there is no concrete evidence.[11] Manually inspecting the predicted rationales, we did not identify bias on demographics. Although such spurious features may be buried in the contextualized paragraph encodings  $(P_{i}^{\mathrm{CLS}})$ . In general, de-biasing models could benefit rationale extraction and we aim to investigate this direction in future work (Huang et al., 2020).
Plausibility: Plausibility refers to how convincing the interpretation is to humans (Jacovi and Goldberg, 2020). While the legal expert annotated all relevant facts with respect to allegations, according to his manual review, allegations can also be justified by sub-selections (parts) of rationales. Thus, although a method may fail to extract all the available rationales, the provided (incomplete) set of rationales may still be a convincing explanation. To properly estimate plausibility across methods, one has to perform a subjective human evaluation which we did not conduct due to lack of resources.
We introduced a new application of rationale extraction in a new legal text classification task concerning alleged violations on ECtHR cases. We also released a dataset for this task to foster further research. Moreover, we compared various rationale constraints in the form of regularizers and introduced a new one (singularity) improving faithfulness and rationale quality in a paragraph-level setup comparing both with silver and gold rationales.
In the future, we plan to investigate more constraints that may better fit paragraph-level rationale extraction and explore techniques to de-bias models and improve rationale quality. Paragraph-level rationale extraction can be also conceived as self-supervised extractive summarization to denoise long documents, a direction we plan to explore in the challenging task of case law retrieval (Locke and Zuccon, 2018).
<sup>10</sup>See Appendix D for rationale quality evaluation on the full test set.
11See Appendix F for examples of ECtHR cases.
We would like to thank the anonymous reviewers (esp. reviewer #2) for their constructive detailed comments. Nikolaos Aletras is supported by EP-SRC grant EP/V055712/1, part of the European Commission CHIST-ERA programme, call 2019 XAI: Explainable Machine Learning-based Artificial Intelligence.
Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preotjiuc-Pietro, and Vasileios Lampos. 2016. Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective. PeerJ Computer Science, 2:e93.
David Alvarez-Melis and Tommi Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 412-421, Copenhagen, Denmark. Association for Computational Linguistics.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias: There's software used across the country to predict future criminals. and it's biased against blacks. ProPublica.
Woodrow Barfield. 2020. *The Cambridge Handbook of the Law of Algorithms*. Cambridge Law Handbooks. Cambridge University Press.
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963-2977, Florence, Italy. Association for Computational Linguistics.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in english. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 78-87.
Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. Finding of EMNLP.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. 2019. A game theoretic approach to class-wise selective rationalization. In Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: