SlowGuess/NAACL_2021_OCR huggingface dataset:
Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT).
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2020. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443-4458, Online. Association for Computational Linguistics.
Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(10).
Madeleine Clare Elish, William Isaac, and Richard Zemel, editors. 2021. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT 2021). Association for Computing Machinery, Online.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, page 2672-2680, Montreal, Canada.
Bryce Goodman and Seth Flaxman. 2017. European union regulations on algorithmic decision-making and a "right to explanation". AI Magazine, 38(3):50-57.
David John Harris. 2018. Harris, O'Boyle, and Warbrick: Law of the European Convention on Human Rights, 4rd edition. Oxford University Press.
Zikun Hu, Xiang Li, Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018. Few-shot charge prediction with discriminative legal attributes. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 487-498.
Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020. Reducing sentiment bias in language models via counterfactual evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 65-83, Online. Association for Computational Linguistics.
Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4198-4205, Online. Association for Computational Linguistics.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT).
Sarthak Jain, Sarah Wiegrefe, Yuval Pinter, and Byron C. Wallace. 2020. Learning to faithfully rationalize by construction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4459-4473, Online. Association for Computational Linguistics.
Daniel Martin Katz. 2012. Quantitative legal prediction-or-how I learned to stop worrying and start preparing for the data-driven future of the legal services industry. Emory Law Journal, 62:909.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: a challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL).
Diederik P. Kingma and Jim Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. In 31th Int. Conf. on Neural Information Processing Systems.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China. Association for Computational Linguistics.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117, Austin, Texas.
Zachary C. Lipton. 2018. The mythos of model interpretability. Commun. ACM, 61(10):36-43.
Daniel Locke and Guido Zuccon. 2018. A test collection for evaluating legal case law search. In *The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval*, SIGIR '18, page 1261-1264, New York, NY, USA. Association for Computing Machinery.
Christos Louizos and Max Welling. 2017. Multiplicative normalizing flows for variational bayesian neural networks. In Proceedings of the International Conference on Machine Learning (ICML 2017), Sydney, Australia.
Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017. Learning to predict charges for criminal cases with legal basis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (Conference on Empirical Methods in Natural Language Processing (EMNLP)), pages 2727-2736.
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2009. Introduction to Information Retrieval. Cambridge University Press.
Masha Medvedeva, Michel Vols, and Martijn Wieling. 2018. Judicial decisions of the European Court of Human Rights: Looking into the crystal ball. In Proceedings of the Conference on Empirical Legal Studies.
W. James Murdoch, Peter J. Liu, and Bin Yu. 2018. Beyond word importance: Contextual decomposition to extract interactions from lstms. In Proceedings of International Conference on Learning Representations (ICLR).
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should I trust you?": Explaining the predictions of any classifier. CoRR, abs/1602.04938.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. arXiv preprint arXiv:2002.12327.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS).
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics.
Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256.
Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. 2020. Beyond 512 tokens: Siamese multi-depth transformer-based hierarchical encoder for long-form document matching. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, page 1725-1734, New York, NY, USA. Association for Computing Machinery.
Zichao Yang et al. 2016. Hierarchical attention networks for document classification. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 1480-1489.
Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using "annotator rationales" to improve machine learning for text categorization. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 260-267, Rochester, New York. Association for Computational Linguistics.
Ye Zhang, Iain Marshall, and Byron C. Wallace. 2016. Rationale-augmented convolutional neural networks for text classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 795-804, Austin, Texas.
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218-5230, Online. Association for Computational Linguistics.
Haoxi Zhong, Guo Zhipeng, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judgment prediction via topological learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3540-3549.
A Why is the new dataset any different?
The new dataset is noteworthy for three reasons:
Legal rationale dataset: This is the first rationale extraction dataset for the legal domain, where justifying decisions is essential and often requires complicated reasoning. Thus the dataset is a challenging test-bed that will boost rationale extraction research. Predicting and justifying alleged article violations is also helpful in practice and would assist legal judgment prediction (Aletras et al., 2016).
Paragraph-level rationales: Each case description is a carefully planned document of multiple paragraphs enumerating facts chronologically. Each paragraph concisely provides information at a granularity considered appropriate for legal reasoning. Accordingly, rationales must be extracted at this granularity, either selecting an entire factual paragraph or not, as opposed to most rationale extraction datasets where particular words or phrases can be selected (e.g., from product reviews).
B Baseline Model
In preliminary experiments, we found that the proposed model (41.9M parameters) (Table 7, third row) relying on a shared paragraph encoder (HIERBERT) to produce both context-aware representations and rationales, has comparable classification performance and better rationale quality compared to: (i) a model with two independent paragraph encoders (82.8M parameters) (Table 7, first row), similar to the one used in the literature (Lei et al., 2016; Yu et al., 2019; Jain et al., 2020); (ii) a model that omits the  $Q$  and  $K$  projection layers (41.4M parameters) (Table 7, second row). Recall that Lei et al. (2016), Yu et al. (2019), Jain et al. (2020) extract rationales at the word-level, and their encoders, either BILSTMs (Lei et al., 2016; Yu et al., 2019) or BERT (Jain et al., 2020), operate on that level of granularity.
Method	classification micro-F1	Silver rationales
mRP	F1
2x HIERBERT	73.4 ± 0.6	35.1 ± 7.9	29.3 ± 8.7
1x HIERBERT excl. (Q,K)	73.5 ± 0.7	29.2 ± 7.9	26.4 ± 7.9
1x HIERBERT + (Q,K)	73.2 ± 0.5	35.9 ± 4.7	39.0 ± 4.9
Table 7: Results on classification performance and rationale quality on development data.
C Annotation of Gold Rationales
The full dataset has the following characteristics:
- There are 1,000 cases in the test set. These are the most recent and have been ruled from October 5, 2017 until July 7, 2019.
- The average number of facts (paragraphs) per case is 25.2 ranging from 5 to 259.
- Almost half of the cases concern applications against 6 European states (defendants): Russia (229), Turkey (122), Ukraine (80), Romania (47), Moldova (44), Lithuania (43). Number of test cases in brackets.
- The allegations in the vast majority of cases (approx.  $88\%$ ) concern nine articles: '6' (394), '3' (233), '5' (197), '8' (188), 'P1-1' (155), '13' (123), '35' (107), '10' (106), '2' (76). Number of test cases in brackets.
Based on the above statistics and the opinion of the legal expert, for the gold rationales we considered a subset of 50 cases with the following characteristics:
Each case should consist of  $25 \pm 10$  facts.
- The cases should be as representative as possible with respect to the defendants (European states).
- The cases should have allegations in a subset of the following articles  $\{2, 3, 5, 6\}$ , whose interpretation is more fact-dependent based on the literature (Harris, 2018) and our presented empirical results (Table 2).
The annotation guidelines briefly were:
- The annotator (legal expert) inspects (reads) all the factual paragraphs of the case and selects one or more articles in the predefined set  $\{2, 3, 5, 6\}$ , that should have been argued by the applicants according to the text.
- The annotator selects the factual paragraphs that "clearly" indicate allegations for the selected article(s), annotated in the first step.
The legal expert performance compared to the gold allegedly violated articles, is  $92.3\%$  micro-F1 (Table 8). In few cases, the legal expert selected more articles (hypothesized allegations for articles 3 and 5) compared to the gold ones. As he suggested, it is a common trend for the applicants, based on the legal opinion of their attorneys, to raise allegations only for a few articles that they believe can be justified and proved to be violated, i.e., if a citizen has no concrete evidence (documents) for his torture, his lawyer may suggest him to not raise this issue in his application. The legal expert also missed a few allegations for articles 2 and 6. The best of our models, (HIERBERT-SA +  $L_{s} + L_{r}$ ) achieves  $87.6\%$  micro-F1 in the same subset.
ECHR article	Expert F1	Model F1
2 - Right to life	88.9	82.4
3 - Prohibition of torture	95.5	92.7
5 - Right to liberty and security	85.7	88.0
6 - Right to a fair trial	95.0	84.2
micro-F1	92.3	87.6
macro-F1	91.3	86.8
For completeness, we report results on development data for sparsity loss  $(L_{s})$  in Table 9 and continuity loss  $(L_{c})$  in Table 10 for different values of  $\lambda_{*}$  hyper-parameters.
Table 8: Classification performance of the legal expert and our best method on the 50 annotated test cases, considering only the facts of each case.
λs	classification micro-F1 ↑	sparsity (aim: 30%)	rationale quality
F1 ↑	mRP ↑
0	73.3 ± 0.9	90.3 ± 19.3	32.4 ± 4.0	35.1 ± 5.5
0.01	72.9 ± 0.4	30.3 ± 2.7	36.6 ± 10.8	36.8 ± 8.8
0.1	73.2 ± 0.5	31.7 ± 1.1	39.1 ± 4.7	39.0 ± 4.9
0.5	71.3 ± 0.9	37.6 ± 7.5	29.1 ± 8.2	30.0 ± 8.4
1.0	71.1 ± 1.7	35.7 ± 5.6	36.2 ± 8.1	38.7 ± 7.6
Table 9: Development results varying  ${\lambda }_{s}$  (sparsity).
λc	classification micro-F1 ↑	sparsity (aim: 30%)	rationale quality
F1 ↑	mRP ↑
0	73.2 ± 0.5	31.7 ± 1.1	35.9 ± 4.7	39.0 ± 4.9
0.01	73.2 ± 0.8	31.3 ± 2.5	30.9 ± 6.9	34.1 ± 6.3
0.1	72.8 ± 0.5	49.4 ± 20.6	26.1 ± 8.2	23.9 ± 1.6
Table 10: Development results varying  ${\lambda }_{c}$  (continuity).
In Section 5.6, we reported rationale quality on a subset of test data that includes silver and gold allegation rationales. For completeness, in Table 11 we report results on the full set of test data for silver rationales. We observe that all findings and particularly the ranking of the methods with respect to the subset of silver and gold rationales hold. Furthermore, we observe that the rationale quality performance on the full test set is slightly inferior in most cases (2-4%), which is expected as the sample annotated by the expert included only cases with allegations for articles that are more explainable.
Method	Silver rationales (31%)
mRP ↑	F1 ↑
RANDOM	30.7 ± 0.7	26.2 ± 0.5
HIERBERT-HA + Ls (Eq. 1)	39.0 ± 3.9	35.1 ± 3.7
HIERBERT-HA + Ls + Lg (Eq. 3)	39.1 ± 5.6	34.7 ± 5.7
HIERBERT-HA + Ls + Lg (Eq. 5)	42.7 ± 1.8	38.2 ± 1.5
HIERBERT-HA + Ls + Lr (Eq. 6)	43.3 ± 2.3	39.0 ± 2.1
Table 11: Results on rationale quality on the full set of test data for silver rationales.
E Using ECtHR dataset via
The dataset is available at https://archive.org/details/ECTHR-NAACL2021; but you can easily load and use it in python with two lines of code:
from datasets import load_dataset dataset = load_dataset("ecthr_cases")
F Examples of extracted Rationales from EChR cases with comments
In Fig. 3-7, we present examples of ECtHR cases. The highlighting (green background colour) indicates gold rationales. The dots (green dot on the left) indicate rationales extracted by our best model, HIERBERT-HA +  $L_{s} + L_{g}$  (Eq. 4, 6). In the caption of each figure, we include short comments explaining false positives (paragraphs the model wrongly selected) and false negatives (paragraphs the model wrongly missed).
Alleged Violations: 5 - Right to liberty and security Predicted Alleged Violations: 5 - Right to liberty and security
Figure 3: (KNEZEVIC v. CROATIA, No. 55133/13)} The model extracted most of the relevant facts indicating a possible violation of Article 5. Note that  $67\%$  (10 of 15) of the facts were considered relevant by the legal expert. Our model has a disadvantage in this case because, being trained to operate at a predefined sparsity level  $(30\%)$ , it extracted only 5 of the 15 facts  $(33\%)$ .
001-178361 - CASE OF K.I. v. RUSSIA
Alleged Violations: 3 - Prohibition of torture, 5 - Right to liberty and security Predicted Alleged Violations: 3 - Prohibition of torture
Model	Facts
7. The applicant was born in 1980. He arrived in Russia in 2003. He travelled to Tajikistan on a number of occasions to visit his parents for short periods of time.
8. On 3 May 2011 the applicant was charged in absentia in Tajikistan with participating in an extremist religious movement, the Islamic Movement of Uzbekistan, and an international search and arrest warrant was issued in his name. On 6 May 2011 the Tajik authorities ordered his pre-trial detention.
9. On 3 November 2013 the applicant was arrested in Moscow and detained. On 4 November 2013 the Meshchanskiy District Court of Moscow ("the District Court") ordered his detention pending extradition.
10. On 4 December 2013 the Tajik prosecution authorities requested the applicant's extradition on the basis of the above charges. The request included assurances regarding his proper treatment, which were formulated in standard terms.
11. On 12 December 2013 the District Court extended the applicant's detention until 3 May 2014.
12. An appeal by the applicant of 16 December 2013 was dismissed by the Moscow City Court ("the City Court") on 3 February 2014.
13. On 29 April 2014 the District Court again extended the applicant's detention until 3 August 2014.
14. An appeal by the applicant of 5 May 2014 was dismissed by the City Court on 23 July 2014.
15. On 9 October 2014 the applicant's extradition was refused by the Deputy Prosecutor General of the Russian Federation, owing to the absence of culpable actions under Russian criminal law.
16. On 13 October 2014 the applicant was released from detention.
17. On 13 October 2014, immediately after his release, the applicant was rearrested for violating migration regulations.
18. On 14 October 2014 the District Court found the applicant guilty of violating migration regulations, fined him and ordered his administrative removal. Allegations by the applicant regarding a real risk of ill-treatment were dismissed, and he was detained pending expulsion. The District Court assessing the risks stated that "[t]he claims of the representative ... are of a speculative nature and not confirmed by the case materials"
19. The above judgment was upheld on appeal by the City Court on 24 October 2014. Claims by the applicant under Article 3 of the Convention were dismissed with reference to the District Court's assessment of the case, which took into consideration "...the nature of the administrative offence, the character of the accused [who was criminally convicted in Russia]... the length of his stay in Russia and other circumstances of the case".
20. According to the latest submissions of his representative in 2015, the applicant was still in detention.
21. On 18 December 2013 the applicant lodged a request for refugee status, referring to persecution in Tajikistan and a real risk of ill-treatment.
22. On 15 September 2014 his request was refused by a final administrative decision of the migration authorities. The applicant challenged that decision in the courts, referring, inter alia, to the risk of ill-treatment.
23. On 12 November 2015 his appeals were dismissed by a final decision of the City Court.
Figure 4: (K.I. v. RUSSIA, No. 58182/14) Paragraphs 9, 11, 13 and 20 clearly indicate plausible violation of the right to liberty (Article 5), as they refer to continuous extension of applicant detention, but our model was unable to extract them, thus it was unable to predict this allegation. The model targeted only paragraphs that indicate ill-treatment, which is connected to plausible violation of Article 3 (Prohibition of Torture).
001-177696 - CASE OF KNEZEVIC v. CROATIA
Figure 5: (KAIMOVA AND OTHERS v. RUSSIA, No. 58182/14) Paragraphs 16 and 19 clearly indicate that the applicant's health (life) was at risk and authorities did not pay attention, but these paragraphs were not selected by the model. Instead paragraph 10 states that the applicant initially informed the authorities for his medical history and they provided medication. This is an indication of model sensitivity to language describing health issues (tuberculosis) in general and not specific well-defined allegations for ill-treatment on the merits.
001-180500 - CASE OF BRAJOVIĆ AND OTHERS V. MONTENEGRO
Figure 6: (BRAJOVIC AND OTHERS v. MONTENEGRO, No. 52529/12) A causal inference would connect paragraph 8 (initial trial) with paragraphs 20-22 (next trials) to infer mistrial, because there was is verdict after a reasonable period of time. Instead the model seems to be sensitive to references for the involvement of higher courts as justification of mistrial (paragraphs 10, 13, 18, and 21). This suggests that the model probably follows poor (greedy) reasoning, i.e., if the applicant appealed to higher courts, then the case is mistreated.
001-178748 - CASE OF KAIMOVA AND OTHERS v. RUSSIA
001-181279 - CASE OF RAJAK v. MONTENEGRO
Alleged Violations: 6 - Right to a fair trial Predicted Alleged Violations: 6 - Right to a fair trial, P1-1 - Protection of property
Figure 7: (RAJAK v. MONTENEGRO, No. 71998/11) Similarly to the case presented in Fig. 6, the main argument in this case is mistrial because there was a verdict after a reasonable period of time (paragraphs 5 and 18-19). The model selected paragraph 11, which does not indicate plausible violations. Given the model's prediction for allegations with respect to Article 1 of the 1st Protocol on the protection of property, we believe that paragraph 11 was selected by the model as justification on that matter.