VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights
Authors: Shanshan Xu
Leon Staufer
Santosh T.Y.S.S
Oana Ichim
Corina Heri
Matthias Grabmair
Venue: EMNLP 2023 Main (2023)
Keywords: nlp for social good
in legal domain
vulnerability classification
rationale dataset
robustness

Abstract: Recognizing vulnerability is crucial for understanding and implementing targeted support to empower individuals in need. This is especially important at the European Court of Human Rights (ECtHR), where the court adapts Convention standards to meet actual individual needs and thus to ensure effective human rights protection. However, the concept of vulnerability remains elusive at the ECtHR and no prior NLP research has dealt with it. To enable future research in this area, we present VECHR, a novel expert-annotated multi-label dataset comprising of vulnerability type classification and explanation rationale. We benchmark the performance of state-of-the-art models on VECHR from both prediction and explainability perspective. Our results demonstrate the challenging nature of task with lower prediction performance and limited agreement between models and experts. Further, we analyze the robustness of these models in dealing with out-of-domain (OOD) data and observe overall limited performance. Our dataset poses unique challenges offering a significant room for improvement regarding performance, explainability and robustness.

VECHR:
A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights
Shanshan Xu Leon Staufer Santosh T.Y.S.S',
Oana Ichim², Corina Heri', Matthias Grabmair'
Technical University of Munich, Germany, 'LMU Munich, Germany
3 Graduate Institute of International and Development Studies, Switzerland
*Faculty of Law, University of Zürich, Switzerland (shanshan. xu, leon. staufer, santosh. tokala, matthias grabmair de corina heri@rwi. uzh. ch
oana. ichinngradduateinstiitiee
Abstract
Diyowkeer
Urpopede Voww
Recognizing vulnerability is crucial for under- standing and implementing targeted support empower individuals in need. This is espe- cially important at the European Court of Hu- man Rights (ECHHR), where court adapts convention standards meet actual individual needs and thus to ensure effective human rights protection. However, the concept of vulnerabil- ity remains elusive at the ECCHH and no prior NLP research has dealt with To enable fu- work in this area, present VECHR, novel expert-annotateed multi-label dataset com- prised of vulnerability type classification and explanation rationale. We benchmark the per- formance of state-of-the-arl models on VECHR from both the prediction and explainability per- spective. Our results demonstrate the challeng- ing nature of the task with lower prediction performance and limited agreement between models and experts. We analyze the robustness these models in dealing with out-of-domain (OOD) data and observe limited overall perfor- mance. Our dataset poses unique challenges offering a significant room for improvement regarding performance, explainability, and ro- bustness.
20002
2002
2006
2000
2012
2015
2038
(a) Evolving distribution of types of vulnerability.
7000
2005
20200
(b) Increase in application of the vulnerability type "migration" between 2010 and 2018.
Figure Distribution changes of vulnerability types.
However, the concept of vulnerability remains clusive within the ECtHR. While legal scholars have explored vulnerability as component of legal reasoning (Peroni and Timmer, 2013), empirical work in this area remains scarce and predominantly relies on laborious manual processes. To address this challenge, NLP can offer valuable tools to as- sist experts in efficiently classifying and analyzing textual data. Besides high classification perfor- mance, the true utility of NLP in the legal field is its ability identify relevant aspects related to vulnerability in court cases. These aspects extracted, grouped into patterns, and used to in- form both litigation strategy and legal policy. Even so, a significant obstacle to progress this area is the lack of appropriate datasets. To bridge these research gaps, we present the dataset !ECHR! which comprises cases dealing with allegation of Article 3 "Prohibition of torture" and is obtained from legal expert's 's empirical study?. Our proposed
1 Introduction
Vulnerability encompasses a state of susceptibility harm, exploitation, particularly among indi- viduals or groups who face a higher likelihood of experiencing adverse outcomes due various fac- tors such as age, health, disability, or marginalized social position (Mackenzie et al., 2013; Fineman, 2016). While is impossible to eliminate vulnera- bility, society has the capacity to mitigate its impact. The European Court of Human Rights (ECtHR) in- terprets the European Convention of Human Rights (ECHR) to address the specific contextual needs of individuals and provide effective protection. This is achieved through various means, such as display- ing ficxibility in admissibility issues, and shifting the burden of proof (Heri, 2021).
be
'VECHR stands Vulnerabilityy Classification in European Court Human Rights Our dataset and code is available at https: thub. emn1p23
"Heri 2021. Heri is the fifth author of this work.

---

Vulnerable Type
Description
Including that of minors, the elderly, and those with physical, psychosocial and cognitive disabilities (i.c. mental illness and intellectual disability) Including that of detainees, military conscripts, and persons in state institutions Due to victimisation, including by domestic and sexual abuse, other violations, or because of feeling of vulnerability
Dependency
State Control
Victimisation
Migration
In the migration context, applies to detention and expulsion of seylum--eekers Due to discrimination and marginalisation, which covers ethnic, political and religious minorities, LGBTQI people, and those living with HIV/AIDS Due to pregnancy or situations of precarious reproductive health
Discrimination
Reproductive Health
Unpopular Views
Due to the espousal of unpopular views
Intersection
Intersecting vulnerabilitics
Table I: Description of each vulnerability type. For more details, see App A.
task is to identify which type of vulnerability (if any) is involved in a given ECtHR case.
Vulnerability Typology in ECtHR
The incscapable and universal nature of vulnera- bility, as posited by Fineman (2016), underscores its significance in legal reasoning. For instance, the European Union has acknowledged the concept by establishing definition for vulnerable individ- uals (Dir, 2013). However, it remains undefined within the context of ECtHR. To facilitate an exam- ination of vulnerability and its application within the ECtHR, it crucial to establish a typology recognized by the Court. Several scholars have endeavored to effectively categorize vulnerability for this purpose (Timmer, 2016; Limanté, 2022). One notable study is conducted by Heri (2021), which provides systematic and comprehensive examination of the concept of vulnerability under ECHR Article 3. Heri proposes a complete typol- ogy encompassing eight types: dependency, state control, victimization, migration, discrimination, reproductive health, unpopular views and intersec- tions thereof. Tab 1 gives a description for each type.
As model explainability is crucial for establish- ing trust, we extend the dataset with VECHRexplain. a token-level explanation dataset annotated by do- main experts
subset of VECHR. Its fine- grained token-level design mitigates performance overestimation of explainability when evaluated at the coarse paragraph level, as shown in previ- ous works (Chalkidis et al., 2021; Santosh et al., 2022; Xu et al., 2023). Further, the understanding and application of vulnerability in court proceed- ings change over time, reflecting socictal shifts and expanding to encompass a wider range of types (Fig la). The volume of cases also fluctuates sig- nificantly in response social and political events (Fig lb). To evaluate the model's robustness against distribution shifts, we further collect and annotate an additional out-of-domain (OOD) test set from cases involving non-Article 3 allegations, called VECHRcchllengge
We present comprehensive benchmark results using thtt--of-the-art (SOTA) models, revealing limited performance in vulnerability type classifica- tion in VECHR. We assess the models' aligmment with expert explanations in VECHRexplain and ob- serve limited agreement. Experiment results on VECHRchaleenge indicate that, although incorporat- ing description of the vulnerability type helps to improve the models' robustness, the performance remains low overall due to the challenges posed by the distribution shift. Our experiments under- score the difficulty of vulnerability classification in ECtHR, and highlight a need for further investi- gation on improve model accuracy, explainability, and robustness.
3 Data Collection and Annotations
3.1 Data Source and Collection Process
VECHR consists of 788 cases under Article 3. which were collected based on Heri's study of the Court's case law references of vulnerability. See App B for details on Heri's case sampling method- ology and our post-processing procedures. We di- vided the datasct chronologically into three sub- sets: training (-05/2015, 590 cases), validation (05/2015-09/2016, 90 cases) and test (09/2016- 02/2019, 108 cases).
VECHResplain We selected 40 cases (20 each) from the val and test splits for the explanation

---

dataset. Within each split, our sampling procedure involved two steps. First, we ensured coverage of all seven types by sampling one case for each type. Subsequently, we randomly selected an additional 13 cases to supplement the initial selection.
Split
#C'
T/C
P/C
#C-v
L/C
1/Cv
Train
590
5140
83
325
0.70
1.55
Validation
90
5077
27
1.41
2.02
Test
108
3992
34
1.13
1.65
Challenge
282
4176
51
144
0.61
1.24
Total
1070
4765
530
0.78
1.54
VECHRchallengg' To test the model's ability to generalize across distribution shifts, we extend VECHR by collecting and annotating additional cases not related to Article 3. Following Heri's method, we used the regular expression "vulne*" to retrieve all English relevant documents from the ECtHR's public database HUDOC* and ex- clude cases related to Article 3. We restricted the collection to the time span from 09/2016 (corre- sponding to start time of the test set) to 07/2022, In cases where multiple documents existed for a given case, we selected only the most recent doc- ument, resulting in a dataset consisting of 282 judgments. VhueRee can be regarded as an out-of-domain topical (OOD) scenario. The in- domain train/val/test of VECHR are all from the same text topic cluster of Article 3. The OOD VeCHRcealeeege consists of non-Article 3 cases from different topic clusters (e.g. Article 10: free- dom of expression), which involves different legal concepts and language usage." 4
Table Dataset statisties for each split, with number of cases (C), number of non-vulnerable cases (C-vv), mean tokens (2) per case, mean paragraphs per case, mean labels (L.) per case, and mean labels per case when only considering positive vulnerability cases (Cv).
VECHR challenge, we ask two expert annotators label the case following Heri's 's methodology". Each annotator has annotated 141 cases.
Anter-Annotator Agreement To ensure consis- tency with Heri's methodology, we conducted a two-round pilot study before proceeding with the annotation of the challenge set (details in App G). In each round, two annotators independently la- belled 20 randomly selected cases under Article 3, and we compared their annotations with Heri's 's labels. The anter-annotator agreement was calcu- lated using Fleiss Kappa, and we observed an in- crease from 0.39 in the first round to 0.64 in the sec- ond round, indicating substantial agreement across seven labels and three annotators.
3.2 Vulnerability Type Annotation
3.3 Explanation Annotation Process
We follow the typology and methodology presented by Heri 2021. She considered cases as "vulnerable- related",, only when "vulnerability had effectively been employed by the Court in its reasoning". These cases are further coded according to the trait situation (vulnerable type) giving rise to the vul- nerability. In situations where the Court considered that multiple traits contributed the vulnerability, she coded the case once for each relevant category. The resulting dataset comprises 7 labels Cases in which vulnerability was used only in its common definition, c.g. "financially vulnerability". were re- garded as 'non-vulnerable" and were labelled none of the 7 types. See App C for more details of the definition of "valneraabee-rlateed"
The explanation annotation process was done using the GLOSS annotation tool (Savelka and Ashley, 2018), see App H for details. Based on the case facts, the annotators was instructed to identify rel- evant text segments that indicate the involvement of a specific vulnerability type in the Court's rea- soning. The annotators was permitted to highlight the same text span as an explanation for multiple vulnerable types.
4 Dataset Analysis
Tab presents the key statistics of our dataset. VECHR comprises a total of 1,070 documents, with an average of 4,765 tokens per case (a = 4167). 788 and 282 cases fall under the Arti- cle 3 and non-Article 3 partitions, respectively. Among all, 530 documents are considered as "non- vulnerable". meaning they are not labelled as any of the seven vulnerable types. In the vulnerable-
For cases under Article 3, we adopted the labelling provided by Heri's protocol. For
/tttps://hudoc..r echr. coe. int
*For example, the Court recognizes vulnerability an elderly woman and provides her with protection under Article (prohibiting torture) rather than Article (freedom expression)
"See App "For the reason of why Heri contined her study to Article 3 and why the typology also applies to cases under other articles, please refer to App F.
annotators' background and expertise.
"See App D for our justification for excluding the type "intersectionalitty.

---

binary label
multi-label vector
Classification Layer
Post- rest-interaction Encoding Layer
Classification Layer
Contextual Encoding Layer
Transformer
d dg
d
Interaction intracction
Layer
h, h2
Multi-head dot product attention
1, f2 you. m
9 9 ......y
9x
Encoding Layer
Transformer
Independent Encoding Layer
Transformer
Transformer
Independent
LegalBERT
LegalBERT
LegalBERT
X, 1 X21
X
cour.ry Xm
Ogi
Concept Description
Case Facts
Case Facts
(a) Hierarchical Classification Variant
(b) Concept-aware Classification Variant
Figure Visualization of Hicrarchical and Concept-aware Hicrarchical Model architectures.
related cases, the average number of labels assigned per document is 1.54.
Model
Classification
Explanation
mac-F1
mic-Fl
Kappa
random
19.02
25.07
-0.11 + 0.02
We observe strong label distribution imbal- ance within the dataset. The label "state control" dominates, accounting for 33% of the cases, while the least common label, "reproductive health", is present in only 3% of the cases. For more detailed statistics of our dataset, including details regarding the label imbalances in Tab 6, please refer to App I.
BERT
24.31
41.78
0.02 0.06
27.31
45.16
0.04 +: 0.08
LegalBERT
27.34
42.47
0.04 + 0.07
Longformer
31.49
46.21
0.11 0.11
Hierachical
31.46
45.32
0.10 + 0.08
Table 3: Classification and explanation results. We report Fis for classification performance and Kappa score with standard error for explanation agreement.
5 Experiments
5.1 Vulnerability Type Classification
We independently encode each packet of the input text using the pretrained model and obtain repre- sentations (h(CLS)) for each packet. Then we apply a non pretrained transformer encoder to make the packet representations context-aware. Finally, we apply max-pooling on the context-aware packet representations to obtain the final representation of the case facts, which is then passed through a classification layer. Fig 2a illustrates the detailed architecture of the hierarchical model.
Task:
Our objective is to predict the set of specific vulnerability type(s) considered by the Court based on the factual text of a case.
Models: We finetune pre-trained models BERT (Devlin et al., 2019), CaselawBERT (Zheng al.. 2021), LegalBERT (Chalkidis al., 2020): on our datasct with a multi-label classification head, trun- cating the input to the maximum 512 tokens.
We finetune the Longformer model (Beltagy et al., 2020) our dataset that allows for processing up to 4,096 tokens, using sparse- attention mechanism which scales linearly, instead of quadratically.
For details on all models' configuration and train- ing, please refer to App J.
Evaluation Metrics: we report micro-F1 (mic-FI) and macro-FI (macFl) scores for 7+1 labels, where 7 labels correspond to 7 vulnerability types under consideration and an additional augmented label during evaluation to indicate non-vulnerable. Results: Tab reports the results of classification performance. We observe that -sgeific pre- training improved the performance over general
We further employ a hierarchical variant of pre- trained LegalBERT to deal with the long input limitation. We use a greedy input packing strat- egy where we merge multiple paragraphs* into one packet until it reaches the maximum of 512 tokens.
DDetails and statistics on paragraphs are reported in App I.

---

Model
2023a. We employ greedy packing strategy as described earlier and use a hierarchical model obtain the context-aware packet representations for each packet in the facts and concept description sep- arately. Subsequently, we apply scaled-dot- prodpr cross attention between the packet vectors of the facts (as Query) and concepts (as Keys and Values), generating the concept-aware representation of the facts section packets. A transformer layer is used to capture the contextual information of the updated packet vectors. Then we obtain the concept-aware representation of the case facts via max pooling and pass it through a classification layer to obtain the binary label. Fig 2b illustrates the detailed ar- chitecture of the concept-awar model. For more details, see App K.
mac-fl
mic-fl
random
12.75
14.61
BERT
20.51
43.48
CastlawBERT
24.55
57.51
LegalBERT
22.60
50.77
Longformer
25.24
55.71
Hierarchical
26.43
58,46
Concept-aware Hicrarchical
33.11
49.62
Table 4: Results on the challenge dataset.
pre-training. However, BERT models still face the input limitation constraint. Both Longformer and Hierarchical models improved compared to truncated variants and are comparable to each other. Overall, we see low overall performance across models, highlighting the challenging task.
The concept-aware model exhibits increased bustness distributional shift and shows an im- provement on the challenge set, owed to the in- corporation of the vulnerability type descriptions. Overall, our results show promise for the feasibility of the task yet indicate room for improvement.
5.2 Vulnerability Type Explanation
We use Integrated Gradient (IG) (Sundararajan et al., 2017) to obtain token-level importance from the model with respect to each vulnerable type un- der consideration. We max pool over sub-words to convert token-level IG scores into word-level scores, followed by threshold-based binarization. Tab 3 reports explainability performance expressed the average of Cohen's x between the models' focus and the experts' annotations for the test in- stances. We observe that the low explainability scores among different models reflect their trend in classification scores and also echo the challenging nature of the task.
Conclusion
We present VECHR, an ECtHR dataset consisting of 1,070 cases for vulnerability type classification and 40 cases for token-level explanation. We also release set of baseline results, revealing the chal- lenges of achieving accuracy, explainability, and robustness vulnerability classification. We hope that VECHR and the associated tasks will provide a challenging and useful resource for Legal NL.P re- searchers to advance research on the amalysis of vul- nerability within ECtHR jurisprudence, ultimately contributing to effective human rights protection.
5.3 Robustness to Distributional Shifts
We assess the robustness of models to distributional shift using the VECHRRhallenge and present the per- formance in Tab 4. Notably, we observe drop in macro-Fl score VECHR compared to the test set. We attribute this to the models relying on suboptimal information about vulnerability types, which is primarily derived from the factual content rather than a true understanding of the underlying concept. To address this limitation, we propose a Concept-aware Hierarchical model that considers both the case facts and the description of vulnera- bility type to determine if the facts align with the specified vulnerability type", inspired by Tyss et al.
Limitations
In our task, the length and complexity of the legal text require annotators with a deep understanding of ECtHR jurisprudence to identify vulnerability types. As result, acquiring a large amount of annotation through crowdsourcing is not feasible, leading to limited-sized datasets. Additionally, the high workload restricts us to collecting only one an- notation per case. There is growing body of work in mainstream NLP that highlights the presence of irreconeilable Human Label Variation(Plank, 2022: Basile et al., 2021) in subjective tasks, such as nat- ural language inference (Pavlick and Kwiatkowski, 2019) and toxic language detection (Sap et al., 2022). Future work should address this limitation
"We cast the multi-label task into a binary classification setup by pairing the text with each vulnerability type. These binary labels are transformed into multi-label vector for performance evaluation, to produce a fair comparison to multi- lahel models on the same metric.

---

and strive to incorporate multiple annotations to capture a more and potentially multi-faccted of the concept of vulnerability.
The assessment of the ethical implications of the dataset is based on the Data Statements by Ben- der and Friedman (2018). Through this, we aim to establish transparency and a more profound un- derstanding of limitations and biases. The curation is limited to the Article 3 documents in English,
This limitation is particularly pronounced be- cause of the self-referential wording of the ECtHR (Fikfak, 2021). As the court uses similar phrases in cases against the same respondent state or alleging the same violation, the model may learn that these are particularly relevant, even though this does not represent the legal reality. In this regard, it is ques- tionable whether cases of the ECtHR can be con- sidered "natural language". Moreover, the wording of case documents is likely to be influenced by the decision or judgement of the Court. This is because the documents composed by court staff after the verdict. Awareness of the case's conclusion could potentially impact the way its facts are presented, leading to the removal of irrelevant information or the highlighting of facts that were discovered dur- ing an investigation and are pertinent to the result (Medvedeva et al.). Instead, one could base the analysis on the so-called "communicated cases", which are often published years before the case is judged. However, these come with their own limi- tations and only represent the facts as characterized by the applicant applicant and not the respondent state. There are also significantly fewer communi- cated cases than decisions and judgements.
The speaker and annotator demographic are legally trained scholars, proficient in the English language. "Speaker" here refers to the authors of the case doc- uments, which are staff of the Court, rather than applicants. We do not believe that the labelling of vulnerable applicants is harmful because it done from a legally theoretical perspective, intending to support applicants. The underlying data is based exclusively on the publicly available datasets of ECtHR documents available on HUDOC!O, The documents not anonymized and contain the real names of the individuals involved. We do not consider the dataset be harmful, given that the judgments are already publicly available.
We are conscious that, by adapting pre-trate rrentrined encoders, our models inherit any biases they con- tain. The results we observed do not substantially relate to such encoded bias. Nonetheless, attention should be paid to how models on vulnerability are employed practically.
In light of the aforementioned limitations and the high stakes in human rights court, we have evaluated the potential for misuse of the vulnerabil- ity classification models. Medvedeva et al. (2020) mention the possibility of reverse engincering the model better prepare applications or defences. This approach is, however, only applicable in a fully automated system using a model with high accuracy towards an anticipated decision outcome. As this is not the case for the models presented, we assume the risk of circumventing legal reasoning to be low. On the contrary, we believe employing high recall vulnerability model could aid applicants and strengthen their legal reasoning. setting focused on vulnerability research, we do not think the model can be used in a detrimental way. Our research group is strongly committed to research on legal NLP models as a means to derive insight from legal data for purposes of increasing transparency, accountability, and explainability of data-driven systems in the legal domain.
One of the main challenges when working with corpora in the legal domain is their extensive length. To overcome this issue, we employ hierarchical models, which have a limitation that tokens across long distances cannot directly interact with each other. The exploration of this limitation in hicrarchical models is still relatively unexplored, although there are some preliminary studies avail- able (e.g., see Chalkidis et al. 2022). Additionally, we choose to freeze the weights in the LegalBERT sentence encoder. This is intended to conserve computational resources and reduce the model's vulnerability to superficial cues.
scholarly
Ethics Statement
Ethical considerations are of particular importance because the dataset deals with vulnerability and thus with people in need of special protection. In general, particular attention needs to be paid to ethics in the legal context to ensure the values of equal treatment, justification and explanation of outcomes and freedom from bias are upheld (Sur- den, 2019).
There was no significant environmental impact, as we performed no pre-training on large datasets. Computational resources were used for fine-tuning
hhttpps//hudoc.ee echr, wcw..coe.iia..... int

---

and training the models, as well as assessing the dataset. Based on partial logging of computational hours and idle time, we estimate an upper bound
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Veronika Fikfak. 2021. What Future for Human Rights? Decision-making by algorithm.
for the carbon footprint of 30 kg CO2 equivalents. This is an insignificant environmental impact.
Martha Fineman. 2016. The Vulnerable Subject: An- choring Equality in the Human Condition, Yale Jour- nal of Law & Feminism.
References
2013. Directive 2013/33/EU of the European Parlia- ment and of the Council of 26 June 2013 laying down standards for the reception of applicants for international protection (recast).
Corina Heri 2021, Responsive Human Rights: Vulner-
ability, Ill-treatment and the ECtHR. Bloomsbury Academic.
Agnè Limanté. 2022. Vulnerable Groups in the Case Law of the European Court of Human Rights. In Agnè Limanté and Dovilè Firrite-Andrkkiené. edi- tors, Legal Protection of Vulnerable Groups in Lithua- nia, Latvia, Estonia and Poland, volume 8, pages 29-54. Springer International Publishing, Cham.
Valerio Basile, Michael Fell, Tommaso Fornaciari. Dirk Hovy. Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15-21, Online. Association for Com- putational Linguistics.
Catriona Mackenzie, Wendy Rogers, and Susan Dodds. 2013. Introduction: What Is Vulnerability, and Why Does It Matter for Moral Theory? In Vulnerabil- ity: New Essays in Ethics and Feminist Philosophy. page Oxford University Press.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint 2000000000005500..
Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science Transactions of the Association for Computational Linguistics, 6:587-604.
Masha Medvedeva, Michel Vols, and Martijn Wieling. Judicial Decisions of the European Court of Human Rights:
Masha Medvedeva, Martijn Wieling, and Michel Vols. 2020. The Danger of Leveers-Engineeriig of Auto- mated Judicial Dacision-Making Systems.
Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodro- mos Malakasiotis, and Desmond Elliout. 2022. An exploration of hierarchical attention transformers for efficient long document classification. arXiv preprint arXiv:2210.05529.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Process- ing Systems 32, pages Inc.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages
Tlas Chalkidis, Manos Fergadiotis, Dimitrios Tsarapat- sanis, Nikolaos Aletras, lon Androutsopoulos and Prodromos Malakasiotis. 2021. Paragraph- ratio- nale extraction through regularization: A case study on european court of human rights cases. In Proceed- ings of the NAACL-HLT 2021. pages 226-241.
Curran Associates.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguis- tics, 7:677-694.
Francisco Charte, Antonio Rivera, María José del Jesus, and Francisco Herrera. 2013. A First Approach to Deal with Imbalance in Multi-label Datasets. In Hy- brid Artificial Intelligent Systems, Lecture Notes in Computer Science, pages 1500-60, Berlin, Heidel- berg. Springer.
Lourdes Peroni and Alexandra Timmer. 2013. Vulner- able groups: The promise of an emerging concept in European Human Rights Convention law. Inter- national Journal of Constitutional Law, 11(4):1056 1085.
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Process- ing, pages 10611- Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

Tyss.s Santosh, Shanshan Xu, Oana Iehim, and Matthias Grabmair. 2022. Deconfounding legal judgment pre- diction for European court of human rights cases to- wards better alignment with experts. In Proceedings of E'ENNP 2022.
Shanshan Xu, Santosh T.Y.S.S, Oana Ichim, Isabella Risini, and Matthias Grabmair. 2023. From disso- nance to insights: Dissecting disagreements in ratio- nale dataset construction for case outcome classific-- tion. arXiv preprint arXiv2310. 1878.
Maarten Sap, Swabha Swayamdipta. Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 5884-5906, Seattle, United States, Association for Computational Linguistics.
Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E. Ho. 2021. When does pre- training help? assessing self supervised learning for law and the casehold dataset of 53,000+ legal hold- ings, In Proceedings of the eighteenth international conference on artificial intelligence and law, pages 159-168.
Jaromir Savelka and Kevin D Ashley. 2018. Segmenting court decisions into functional and issue specific parts. JURIX, pages 111-120.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan, 2017. Axiomatic attribution for deep networks. In ICML, pages 3319-3328. PMLR.
Harry Surden. 2019. The Ethics of Artificial Intelli- gence in Law: Basic Questions.
Alexandra Timmer. 2016. A Quiet Revolution: Vulner- ability the European Court of Human Rights. In Martha Albertson Fineman and Anna Grear, editors, Vulnerability: Reflections New Ethical Founda- tion for Law and Politics, zeroth edition. Routledge.
Santosh Tyss, Oana Ichim, and Matthias Grabmair. 2023a. Zero-shot transfer of article-aware legal out- come classification for european court human rights cases. In Findings of the Association for Com- putational Linguistics: EACL 2023. pages 593-605.
Santosh Tyss, Marcel Perez San Blas, Phillip Kem- per. and Matthias Grabmair. 2023b. Leveraging task dependency and contrastive learning for case outcome classification on european court of human rights cases. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 100--1103.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, rie Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of of the natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Pier-

---

A Vulnerability Typology and Deseriptions
and/or willingness of complaining of ill-treatment and, on the other hand, their susceptibility be ex- posed to traumatic experiences/treamment Elderly: In many ways, vulnerability due to advanced age is a continuation of the vulnerability of children. All humans experience dependency at the begin- ning of life, and many experience it near the end. Intellectual and Psychosocial Disabilities: Intellec- tual disability may render those affected dependent on others be it the state or others who provide them with support. Having regard their special needs in exercising legal capacity and going about their lives, the Court considered that such situa- tions were likely to attract abuse. Persons living with intellectual disabilities experience difficulties in responding to, or even protesting against, vio- lations of their rights. In addition, persons with severe cognitive disabilities may experience a legal power imbalance because they do not enjoy legal capacity.
Here is the typology of vulnerability in ECIHR (Heri, 2021):
Dependency: dependency-based vulnerabil- ity, which concerns minors, the elderly, and those with physical, psychosocial and cogni- tive disabilities (i.e., mental illness and intel- lectual disability).
State Control: vulnerability due to state con- trol, including vulnerabilities of detainees, military conscripts, and persons in state in- stitutions.
vulnerability due to victimisa- tion, including by domestic and sexual abuse, other violations, or because of a feeling of vulnerabilityy.
Victimisation:
vulnerability in the migration con- text, applies to detention and expulsion of asylum-seekenn
Migration:
State Control Vulnerability due state control includes detainees, military conseripts, and persons in state institutions. This type of vulnerability in- cludes persons in detention, but also those who are institutionalised or otherwise under the sole author- ity of the state. When people are deprived of their liberty, they are vulnerable because they depend on the authorities both to guarantee their safety and to provide them with access to essential resources like food, hygienic conditions, and health care. addi- tion, the state often controls the flow of information and access to proof. Hence, the Court automati- cally applies the presumption of state responsibility when harm comes to those deprived of their liberty.
Discrimination: vulnerability due due to discrimination and marginalisation. which covers ethnic, political and religious minori- ties, LGBTQI people, and those living with HIV/AIDS.
Reproductive Health: vulnerability due to pregnancy or situations of precarious repro- ductive health.
Unpopular Views: vulnerability due to the espousal of unpopular views.
Victimisation Vulnerability due to victimisation refers to situations in which harm is inflicted by someone else. This type of vulnerability applies to situations of domestic and sexual abuse, and other type of abuse. The Court has also found that Convention violation may, and of itself, render someone vulnerable. Crime victims who are particularly vulnerable, through the circumstances of the crime, and can benefit from special measures best suited to their situation. Sexual and domestic violence are expressions of power and control over the victim, and inflict particularly intense forms of trauma from psychological standpoint. Failing recognise the suffering of the victims of sexual and domestic violence or engaging in stigmatising response - such as, for example, the perpetuation of so-called 'rape myths' represents a secondary
Intersection: intersecting vulnerabilities.
Following is detailed description of each type: Dependency Dependency-basee based vulnerability de- rives from the inner characteristics of the applicant and thus concerns minors, elderly people, as well as physical, psychosocial and cognitive disabilities (i.e., mental illness and intellectual disability). The Court has built special requirements around these categories to be fulfilled by States as part of their obligations under the Convention. Minors: The Court often refers children a paradigmatic example of vulnerable people and made use of the concept of vulnerability to require States to display particular diligence cases imposing child protec- tion given, on the one hand, their reduced ability

---

victimisation 'revictimisation' of the victims by the legal system
cant's choice to display such views and vulnera- bility comes from particular measures that state undertake when regulating or disregarding such choices.
Migration Vulnerability in the context of migra- tion applies to detention and expulsion of asylum- seekers. Applicants as asylum-seeker are consid- ered particularly vulnerable based on the sole expe- rience of migration and the trauma he or she was likely to have endured previously'. The feeling of arbitrariness and the feeling of inferiority and anxiety often associated with migration, as well as the profound effect conditions of detention in spe- cial centres, indubitably affect a person's 's dignity. The status of the applicants as asylum-secker is considered to require special protection because of their underprivileged (and vulnerable) status.
B More details on Data Source and Collection Process
Heri 2021 reported the details of the case sampling process. The following serves as a summary of her case sampling methodology: She used the regular expression "vulne*" to retrieve all relevant English documents related to Article 3 from HUDOC, the public database of the ECtHR, excluding communi- cated cases and legal summaries, for the time span between the inception of the Court and 28 February 2019, This yielded 1, 1,147 results.
Discrimination Vulnerability due to discrimina- tion and marginalisation covers ethnic, political and religious minorities, LGBTQI people, and those living with HIV/AIDS. The Court recognises that the general situation of these groups the usual conditions of their interaction with members of the majority or with the authorities - is particu- larly difficult and at odds with discriminatory atti- tudes. Similarly to dependency-basse based vulnerability, this type of vulnerability imposes special duties on states and depends not solely on the inner character- istics of applicants but also on their choices which, in most cases, states have to balance against other choices and interests.
Heri recorded her labeling in an Excel sheet, in- cluding the application number for each case. The application number serves as unique identifier for individual applications submitted to the ECtHR. To create VECHR, we fetch all relevant case doc- uments from HUDOC, including their metadata. During the post-processing, when one case has multiple documents, we keep the latest document and discard the rest, which yields 788 documents.
C Definition of "Vulnerable-related"
Heri (2021) specified that only cases where "vul- nerability had effectively been employed by the Court in its reasoning" are regarded as "vulnerable- related". Cases in which vulnerability was used only in its common definition, or used only in the context of other ECHR rights, or by third parties, were excluded. To ensure clarity and alignment with Heri's perspective, we communicated with her during the annotation process to clarify the def- inition of "vulnerable-relateed" As result, we determined that vulnerability labeled (primarily) in situations where:
Reproductive Health Vulnerability due preg- nancy or situations of precarious reproductive health concerns situations in which women may find themselves in particular vulnerable situations, even if the Court does not consider women vul- nerable as such. This may be due to experi- ence of victimisation, for example in the form of gender-based violence, due to various contexts that particularly affect women. Sometimes, de- pending on the circumstances, pregnancy may be reason enough for vulnerability while other times vulnerability is linked to the needs the unborn children.
Vulnerabilitty is part of the Court's main legal reasoning.
The alleged victims (or their children) exhibit vulnerability.
Unpopular Views Vulnerability due to the es- pousal of unpopular views includes: demonstrators, dissidents, and journalists exposed to ill-treatment by state actors. Where extradition request shows that an applicant stands accused of religiously and politically motivated crimes, the Court considers this sufficient demonstrate that the applicant is member of a vulnerable group. Similarly to the case of victimisation, it is also the appli-
The ECtHR, rather than domestic courts or other parties, consider the alleged victims vul- nerable.
D Omitting "Intersectionality" Label
The vulnerability type "intersectionality" was omit-
ted because of its unclear usage in cases. Even

---

more than the other typologies, it is a highly nu- anced concept without a clear legal definition. Fur- thermore, Heri (2021, 117) says that the ECtHR does not engage with the concept of intersectional- ity in this form. Given that there are only 11 cases exclusively annotated as "interscctional" (out of a total of 59), the effect of disregarding it in this work is negligible. Omission does not suggest that intersectionalityy fails play role in the reasoning of the ECIHR or that we deem it irrelevant. Instead, we suggest exploring the concept of vulnerability as a combination of the other seven vulnerabilitics.
and clarify their understanding of the vulnerability concept. In the second round, the annotators in- dependently labeled another 20 randomly selected cases. We calculated the inter-annotator agreement using Fleiss Kappa to measure the consistency be- tween Heri's labels and the annotations provided by our two annotatons. The Fleiss Kappa agreement increased from 0.39 in the first round to 0.64 in the second round, which we consider to be substantial agreement multi-label setting involving seven vulnerable types and three annotators.
H GLOSS Annotation Tool
Annotator Background & Expertise
The task of explanation annotation was done using the GLOSS annotation tool (Savelka and Ashley, 2018). Fig demonstrates the GLOSS annotation interface.
Two annotators performed the classification task. Annotator (the fourth author) is Post-doctoral Researcher European Research Centre, who has worked the European Court of Human Rights as case lawyer. Annotator 2 (the second author) is a law student, with also a philosophy and com- puter science background. Prior to annotation, both annotators had read Heri's s book, and Annotator 2 had received training session on the ECHR from Annotator 1.
Case fact
# cases
40
Avg. vulnerable type per case
1.3
Avg. length per case
2964 1191 words
Rationals from annotator
Avg. length case-allegation pair
630 + 551 words
The explanation dataset was annotated by Anno- tator
Table 5: Statistics for the explanation dataset.
F Justification of Article Applicability
Count
IRLbl
Dependency
254
23.74
1.41
Heri's (2021) typology is limited to Article 3 of the ECHR, which pertains to the Prohibition of Torture. Under Article 3, the concept of vulnerability was first coined by the ECtHR, given that it deals with inhuman, degrading treatment and torture, which represent prototypical contexts vulnerability. As such, an initial exploration under Article is rea- sonable.
State Control
358
33.46
1.00
Victimisation
70
6.54
5.11
Migration
71
6.64
5.04
Discrimination
34
3.18
10.53
Reproductive Health
11
1.03
32.55
Unpopular Views
33
3.08
10.85
Non- Von-Vulneeable
530
49.53
0.68
Applying Heri's procedure to Aot-Article 3 cases is nonetheless justified according to our legal ex- pert because Heri's underlying typology is based Timmer (2016) and relates to all articles. Fur- thermore, vulnerability is now concept that is not limited to a single article, and which the ECIHR applies across articles.
Table 6: Count, percentage of total documents (%).
imbalance ratio per label (IRLbl) (Charte et al., 2013) for each type vulnerability.
Dataset Statistics
The dataset comprises 1070 cases. On average, each case involves 0.78 vulnerable types and I. 1.54 vulnerable types for non-negative cases. Fig 3 presents the distribution of the number of anno- tated labels per document. We report the imbal- ance characteristics for each label in Tab Fig 4 shows the difference in frequency of vulnerability annotations between Article and non Article 3. Fig illustrates the difference frequency of each
Pilot study for Annotating Non-Article 3 Cases
In the first round, both annotators independently labeled 20 randomly selected cases under Article 3. After completing the labeling process, they com- pared their annotations with Heri's labels and en- gaged in a discussion to address any discrepancies

---

vulnerability label between all four datasets. Tab shows the statistics for the explanation dataset.
The hicrarchical nature the dataset is based on the naturally occurring paragraphs in the judgment texts. On average, each case consists of 71.54 para- graphs (a 67.54). The distribution of the num- ber of paragraphs by vulnerability type is shown in Fig 6a. The mean token count is 4,765; its distribu- tion by vulnerability type depicted in Fig 6b. The distribution of the mean token count per paragraph by vulnerability type shown in Fig 6c.
100
Count
300
200
4
Number of annotated labels per document
J Implementation Details
Figure 3: Distribution of number of annotated labels per document.
We use BERT bace-ss (Devlin et al., 2019), CaSelawBERT "casshold/legallertt (Zheng et al., 2021), LegalBERT ppppaucblcggal-bet-bbaee uncased" (Chalkidis et al., 2020), and Longformer Salllcni//ongfooreee-b--------
50
Art.
Non Art. 3
40
(Beltagy al., 2020). We finetune pre-trained models from the Transformers Hub (Wolf al., 2020) on our dataset with a multi-label classification head, truncating to maximum lengths of 512 and 4096 tokens for BERT and Longformer models, respectively.
30
10
Hyperparameter & Overfitting Measures: For the BERT-based models, we perform grid search for hyperparameters across the search space of batch size [4, 8, 16] and learning_ rate [5e-6, le- 5, 5e-5, le-4]. We train models with the Adam optimizer up epochs. We determine the best hyperparameters on the dev set and use early stop- ping based on the dev set macro-F1 score. For the Hierarchical models, we employ a maximum sen- tence length of 128 and document length (number of sentences) of 80. The dropout rate in all layers is 0.1. We perform grid search for the hyperpa- rameters across the search space of batch size [2, 4J and learning_ [le-6, 5e-6, le-5]. We train models with the Adam optimizer up 20 epochs. We determine the best hyperparameters on the dev set and use early stopping based on the dev set macro-Fi score. We use PyTorch (Paszke et al..
Depondency
Control
Viotimisation
Migration
Reprodactive Health
Uppopular Views
Non Vuluerablee
Disorimination
Figure 4: Difference in frequency of vulnerability anno- tations between Article and non-Article
60
Train
Validation
50
Test
Challenge
30
20
Dependency
State Control
Vietimisation
Migration
Discrimination
Reproductive Health
Unpopular Views
Non Valnersaile
K Model Architecture
Hierarchical Model: Greedy packing turns the case facts input into pack- ets
(2,, Jm). where packet 2i (011, Fy2..... ... Fnn) consists of tokens. We pass each packet o; independently into the pre- trained LegalBERT model (Chalkidis et al., 2020) to extract the hel representation for each packet.
Figure 5: Difference in frequency of vulnerability type between "train", "validation". "test", and "challenge" datasets.

---

250
Paragraph Count
200
150
100
Dopendoncy
State Control
Viotimisation
Migration
Reproductive Heallh
Non vulnorable
Discrimiiation
Uppopuler
(a) Box plot the number paragraphs per type.
10000
Token Count
8000
6000
2000
Degendency
State Control
Virtimisation
Xigration
Reproductive Health
Unpopuler Viows
Non Valnerable
Discrimanntion
(b) Box plot of token count per type.
Token Couat Paragraph
140
120
100
80
60
40
Dependency
State Control
Viclimisation
Migrotion
Discrimination
Reproductive Health
Unpopuler Views
Non vulnerable
(c) Box plat of the mean number of tokens per paragraph per type.
Figure 6: Box plots paragraph and token information for each vulnerability type.

---

All packet representations h {h1, h2. Am along with learnable position embeddings, are passed through transformer encoder to make them aware of the surrounding context. These context-aware packets representations are then max pooled to obtain the final representation of the case facts, which then pass through classification layer. Given the multi-label nature of the task, we employ binary cross-entropy loss over each vulnerability type label. Fig 2a illustrates the detailed architecture of our hierarchical model, which inspired by (Santosh et al., 2022; Tyss et al., 2023b).
which are once again passed through non- pretrained transformer module enhance them with the surrounding contextual information, and max- max-poolee operation to obtain the final concept- aware case fact representation. A classification layer then predicts the binary label indicating whether these facts give rise to the given vulnera- bility type. Fig 2b displays the architecture of the concept-aware hierarchical model in detail.
Concept-aware Hierarchical model: We cast the multi-label task into binary classification setup where pair case facts with each vulnerability type to predict whether this vulnerability type was raised by court in the case from which the facts section stems. Note that we transform these binary labels into a multi-labe vector for evaluation, to make a fair comparison with multi-label models. This is based on the article-aware outcome predic- tion setting in Tyss et al. 2023a.
The concept-aware model also takes the facts input, which after greedy packing form m packets x = {æ1, #2.......................... } and vulnerability concept description text c {ci, (2,,0...... ) with k packets. Packet x; {211, T22, ................................ Fin has n tokens and packet tokens.
{ca, C2,,.... Cip) has
Similar the hierarchical model, trained LegalBERT model (Chalkidis et al., 2020) to encode each packet in the case facts and con- cept description independently, and extract the hls representation for each packet. These are passed through non preetrarddd transformer model to obtain contexi-aware representations f [f1, .2, fm) and {91, .2. 9k }o for case facts and concept descriptions, respectively.
a pre-
The obtained packet representations of facts and concept description interact with each other us- ing a multi-head scaled dot product cross attention (Vaswani et al., 2017) similar to the decoder in the transformer layer by treating case facts packets (f)
the queries (Q) and the keys (K) and values (V)
come from the concept description packets (g).
QK Vah
Attention(Q, K.V) softmax
V (1)
Thus we obtain a concept-aware representations of the fact description packets d {d1,dz, d2,... dm

---

Types
Annotations
6.e applicant was born 1965 and lives Remanzaccio.
7. The applicant married A.T., Moldovan national, and two children were born of the marriage: daughter 1992 and in 1998.
dependency
8he applicant alleged that after their marriage her husband had started beating her. However. in 2011 the applicant followed her husband to Italy order to provide children with apportunity
discrimination
future.
migration
1.ehe first assault committed by A.T. against the applicant and her daughter 9. The applicant submitted that her husband, who was alcoholic, had already been physically abusing her long time when, June ahe requested the Intervention the police after she and her daughter had been assaulted A.T.
reproductive health
state control
10.When the police arrived, A.T. had street in a state of intoxication, with scratches on the left side of his face. The police drew report of the incident. The report stated that the applicant had left leg and
family
He found the
unpopular views
victimiiation
beaten and bitten bruises. The report also stated that the applicant's daughter had herself been hit after intervening to protect her mother and presented neck injury caused by fingernai injuries
face
number of
arms. The applicant her daughter were
Figure 7: Screenshot of the GLOSS annotation interface.


Reviews:
[{"review_id": "N3KSgD4g5A", "reviewer": "Reviewer_S2UF", "summary": "", "questions": "1.  In Section 4 \"Dataset Analysis\", you mention that among 1070 documents, 519 documents are considered as “non-vulnerable”. However, in Table 3, total 551 case documents are labelled as “non-vulnerable”. Similarly, in Section 4, the average number of labels assigned per document is mentioned to be 1.51, whereas in Table 3, it is 1.59. Could you please explain this discrepancy?\n2. What do you mean by \"positive vulnerability cases\" in Table 3? Since the stats are different for columns L/C and L/CV, are we still comparing the same no. of legal case documents? In other words, are positive vulnerability cases less than total cases/documents considered? If yes, how many documents are finally included in the dataset?\n3. In Section 4, you mention that 519 documents are considered as “non-vulnerable” among 1070 documents. Doesn't this make the dataset skewed towards the \"non-vulnerable\" class? Also, among the \"vulnerable\" documents, the majority belong to the \"Dependency\" and \"State Control\" classes. Have you taken any steps to address this skewness while training models. Since, different vulnerability types have different definitions, can you expect models trained on VECHR to generalize well given the imbalance in label distribution?\n4. In VECHR, the train set contains case documents prior to 05/2015, whereas val and test sets contain documents from non-overlapping periods different from that of train set. From Fig. 1 on page 1, we see that the distribution of vulnerability types evolves over time from 2015 to 2021. That makes the inherent label distribution different in the train set when compared to the val or test sets. Do you think this could be the reason for poor performance of models?\n5. The micro-f1 scores of the trained models on the VECHR (challenge) dataset are almost always better (except the random model) than the corresponding scores on the VECHR dataset. Could you please explain this observation? \n6. Is the label imbalance distribution reported in Table 4 for the overall VECHR dataset? Also, I cannot correlate Table 4 with Fig. 3. For example, \"state control\" has 33.5% representation as per Table 4, whereas, as per Fig. 3, it is less than 30% for Article-3 cases and only around 15% for Non-Article-3 cases. How can the value reach upto 33%? Could you please report the label distribution of the test sets separately for VECHR and VECHR (challenge)? \n7. Model Description given in appendix J is not very clear. Could you please include a model diagram to explain the steps?\n8. In Section 3.2, could you please clarify whether you recruited two or three annotators? Compare lines 180 and 194. Also, what is the inter-annotator agreement on the VECHR test set (not the challenge set which is reported)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n\n(1a) Discrepancy in the Dataset Analysis (question 1, 2, 6; rejection reason 1): Thanks for pointing it out. We will correct it to maintain consistency at both places. The final dataset consists of 1070 documents. We will also provide more details, for example the label distribution of the separately for VECHR test and VECHR challenge etc. We will also adjust the table 3 to make it less confusing. \n\n(1b) Limited Dataset Size and Quality Concern (rejection reason 2): We agree that a dataset with a large size and balanced label distribution would have been better for training purposes. However, it's important to consider that in specialized domains such as the law— for instance, our task involving the classification of specialized legal concepts within certain areas of law—the presence of limited data presents a unique challenge. This challenge would further captivate interesting directions involving few-shot, zero-shot, learning with limited data.\n\nSpecifically, to maintain the quality of our curated dataset, we engaged with experts from the beginning in several rounds of discussions, conducting a pilot study with them and finally an adjudication process to arrive at the annotation guidelines, which is reflected in the higher IAA score. \n\n(1c) Dataset Skewness and Generalizability (question 3):  We agree VECHR is very label imbalanced; therefore we propose a Concept-aware Hierarchical model to deal with the skewness, in the hope that the model will consider both the case facts and the description of vulnerability type to determine vulnerability type, rather than just learning the statistical distribution of the vulnerability type. Regarding generalizability,  the task presented here is representative of a task where expert legal researchers focus on a particular group of cases centering on a legal term that occurs sparsely. In such situations, discriminative models trained on the whole dataset may not be sensitive enough to be helpful.\n\n\n(1d)  LLM (rejection reason 3): We have experimented with ChatGPT on our curated dataset with different prompting strategies and a temperature of 0. We have found that, as expected, the prompt phrasing has significant influence on the performance, to the point producing counterintuitive metrics. The best model we obtained reaches a macro-F1 of ~.26 and a micro-F1 of ~.31, which is still substantially below our best models. Notably, this is with ChatGPT having possibly seen the cases during its pre-training phase. \n\n(1e) Diagram for Concept-aware Hierarchical model (question 7; rejection reason 4):  Thank you for the suggestion. We will add a diagram of the model structure and make the description more clear. \n\n(1f) Better Micro-f1 scores on the challenge set (question 3, 5):  This outcome can be attributed to the label distribution within the challenge set, which exhibits a more pronounced skew towards the majority class compared to the test set. The observed discrepancy—improved micro F1 but decreased macro F1 on the challenge set—suggests potential difficulties in the models' adaptability to distributional shift. These findings underscore the need to fortify the models' robustness. \n\n(1g) Number of annotators (question 8): For the 788 cases under article 3 (VECHR train/val/test set), the initial labelling was obtained from Heri’s study on Court’s case law references of vulnerability (Heri 2021). We then have our two annotators who read this vulnerability typology from Heri and participated in a pilot study of 40 randomly chosen article 3 cases to arrive at the consensus (line 190). The reported IAA score was calculated for these 40 cases using both annotators and Heri’s labels. \n\nWhile for the 282 cases in the Challenge set, we asked two expert annotators; each of them annotated 141 cases. \n\n\nCorina Heri. 2021. Responsive Human Rights: Vulnerability, Ill-treatment and the ECtHR. Bloomsbury 548 Academic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. First of all, the paper is very well motivated, presented and written. It was worth a read.\n2. It is true that datasets are scarce in the legal domain. Given the challenges and intricacies of the domain, NLP models are still far away from producing trustworthy explainable results which could be directly consumed without human intervention. In this regard, the proposed dataset is a valuable contribution. It is additionally novel given the task of vulnerability type analysis from European legal cases.\n3. The paper proposes multiple test sets to evaluate various facets of compared models. In this regard, the thought process of the authors and their efforts towards getting multiple test sets annotated is noteworthy. It adds value to the paper.\n4. Data collection and annotation steps are thoroughly explained with appropriate details. Although I found certain discrepancies. Please refer to my questions below.\n5. Despite being a short paper, the authors have made sure to include necessary details in the appendix as well.\n6. While more recent large language models could have been tried out, given that this is primarily a dataset and benchmarking paper, I am satisfied with the experiments.\n7. All compared models have fairly low scores, which gives ample scope to future researchers to come up with better NLP models over time.\n8. Finally, limitations of the work are clearly mentioned.", "reasons_to_reject": "Although I am inclined towards accepting the paper, I have a few concerns:\n1. Some of the dataset statistics reported in Section 4 and appendix I do not seem to correlate.\n2. The dataset size is small with close to 50% of the documents not labelled with any vulnerability label. Additionally, only 2 labels dominate the overall label distribution. This raises questions on the dataset quality as well the scope for future models to perform well on this dataset.\n3. At least one generative model could have been tried out leveraging the descriptions of vulnerability types to improve the classification performance. \n4. Description of the proposed \"Concept-aware Hierarchical\" model is not clear.", "questions_to_authors": "1.  In Section 4 \"Dataset Analysis\", you mention that among 1070 documents, 519 documents are considered as “non-vulnerable”. However, in Table 3, total 551 case documents are labelled as “non-vulnerable”. Similarly, in Section 4, the average number of labels assigned per document is mentioned to be 1.51, whereas in Table 3, it is 1.59. Could you please explain this discrepancy?\n2. What do you mean by \"positive vulnerability cases\" in Table 3? Since the stats are different for columns L/C and L/CV, are we still comparing the same no. of legal case documents? In other words, are positive vulnerability cases less than total cases/documents considered? If yes, how many documents are finally included in the dataset?\n3. In Section 4, you mention that 519 documents are considered as “non-vulnerable” among 1070 documents. Doesn't this make the dataset skewed towards the \"non-vulnerable\" class? Also, among the \"vulnerable\" documents, the majority belong to the \"Dependency\" and \"State Control\" classes. Have you taken any steps to address this skewness while training models. Since, different vulnerability types have different definitions, can you expect models trained on VECHR to generalize well given the imbalance in label distribution?\n4. In VECHR, the train set contains case documents prior to 05/2015, whereas val and test sets contain documents from non-overlapping periods different from that of train set. From Fig. 1 on page 1, we see that the distribution of vulnerability types evolves over time from 2015 to 2021. That makes the inherent label distribution different in the train set when compared to the val or test sets. Do you think this could be the reason for poor performance of models?\n5. The micro-f1 scores of the trained models on the VECHR (challenge) dataset are almost always better (except the random model) than the corresponding scores on the VECHR dataset. Could you please explain this observation? \n6. Is the label imbalance distribution reported in Table 4 for the overall VECHR dataset? Also, I cannot correlate Table 4 with Fig. 3. For example, \"state control\" has 33.5% representation as per Table 4, whereas, as per Fig. 3, it is less than 30% for Article-3 cases and only around 15% for Non-Article-3 cases. How can the value reach upto 33%? Could you please report the label distribution of the test sets separately for VECHR and VECHR (challenge)? \n7. Model Description given in appendix J is not very clear. Could you please include a model diagram to explain the steps?\n8. In Section 3.2, could you please clarify whether you recruited two or three annotators? Compare lines 180 and 194. Also, what is the inter-annotator agreement on the VECHR test set (not the challenge set which is reported)?", "paper_topic_and_main_contributions": "This paper proposes a new dataset with legal cases dealing with Article-3 allegations in the European Court of Human Rights expert-annotated with possibly multiple vulnerability types. Three different test sets are proposed, the first one is meant to evaluate models on the multi-label vulnerability classification task, the second one requires models to additionally extract textual spans as explanations for the identified vulnerability types, and the third one contains legal cases involving non-Article 3 allegations, thereby evaluating the robustness of models against distributional shifts/out of domain data. Several models are benchmarked against all three test sets for the respective tasks.\nOverall low results highlight the challenging nature of the task as well as scope for further analysis and improvement.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Fig 1 label is absent."}, "extra_text": {}}, {"review_id": "aTp91IkKP8", "reviewer": "Reviewer_tTjb", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments.\n\n(2a) Small Dataset Size: We agree that a dataset with a large size and balanced label distribution would  have been better for training purpsoses. However, it's important to consider that in specialized domains such as the law— for instance our task involving the classification of specialized legal concepts within certain areas of law—the presence of limited data presents a unique challenge. This challenge would further captivate interesting directions involving few-shot, zero-shot, learning with limited data.\n\n(2b)  LLM: We have experimented with ChatGPT on our curated dataset with different prompting strategies and a temperature of 0. We have found that, as expected, the prompt phrasing has significant influence on the performance, to the point producing counterintuitive metrics. The best model we obtained reaches a macro-F1 of ~.26 and a micro-F1 of ~.31, which is still substantially below our best models. Notably, this is with ChatGPT having possibly seen the cases during its pre-training phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides the dataset VECHR, which can bridge the NLP tools with experts in efficiently classifying and analyzing vulnerability, where vulnerability is important in ECtHR. ", "reasons_to_reject": "1. This dataset is small, especially the annotation scale of explanation rationale. Due to the small size of the evaluation data, the evaluation results may be unstable.\n2. This paper lacks the performance of LLM (such as LLaMA 13B, chatGPT) on this dataset.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper provides a dataset for classification of vulnerability type. This dataset consists of 1070 cases for prediction and 40 cases for explanation. This is a novel task, and this dataset can facilitate this new task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VQX5bql8EG", "reviewer": "Reviewer_DQPj", "summary": "", "questions": "1. **Comparison with State-of-the-Art Methods**: How does the Concept-aware Hierarchical model compare with other state-of-the-art methods that utilize external knowledge? Could you provide a detailed analysis of why the mic-f1 score of the Concept-aware Hierarchical model is lower than that of the Hierarchical model as shown in Table 2?\n2. **Handling Long Text Inputs**: Considering that the mean number of tokens in VECHR is 4596, while the maximum length of Longformer is only 4096, how did the authors handle instances that exceed this token limit? Have the authors considered applying retrieval or summarization methods to effectively reduce the input length and potentially improve the performance of Bert and Longformer?\n3. **Hyperparameter Selection**: How were the hyperparameters, such as the learning rate and number of training epochs, selected for the experiments?\n4. **Overfitting Concerns**: Considering that there are only 590 vulnerability cases in the training set, what measures were taken to prevent overfitting? Could you share the performance on the validation and test sets during the training process, along with any techniques used to mitigate overfitting?\n5. **Inclusion of Large Language Models**: From my perspective, this dataset seems very suitable for testing on large language models such as GPT-4 and Claude-2, which can analyze each sentence and generate reasons belonging to specific types. Have the authors considered experimenting with these models?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments.\n\n(3a) Limited Generalizability: The concept of vulnerable/protected groups is a common topic in the civil/human rights context internationally and domestically (see, e.g. for example in the inter-american human rights setting:  https://www.ilo.org/wcmsp5/groups/public/---dgreports/---inst/documents/genericdocument/wcms_818106.pdf). Additionally, specialized legal concepts (and sub-concepts) of importance to certain areas of law are a common occurrence in legal systems. As such, the task presented here is representative of a task where expert legal researchers focus on a particular group of cases centering on a legal term that occurs sparsely. In such situations, discriminative models trained on the whole dataset may not be sensitive enough to be helpful.\n\n(3b)Distribution of content: We ask for understanding that given limited space of 4 pages, we use the appendices for the details of data collection process and dataset statistics. If our paper is accepted, we will utilise the 5th page to add more details to the main article. \n\n(3c) Concept-aware models: We limit our external knowledge to the definitions of vulnerable types using the concept-aware model. We leave using external sources of knowledge like convention document, legal commentaries, precedent cases etc for future work.\n\n(3d) Long inputs: We truncate any tokens exceedings the input length of the baseline longformer model. Our hierarchical model does not have this limitation. Other possible approaches (summarization, etc.) can be explored in future work.\n\n(3e) Hyperparameter & Overfitting measures: For the BERT-based models, we perform a grid search for the hyperparametes across the search space of batch size [4,8,16], learning_rate [5e-6, 1e−5, 5e−5, 1e-4]. We train models with the Adam optimizer up to 8 epochs. We determine the best hyperparameters on the dev set and use early stopping based on the dev set macro-F1 score. \nFor the Hierarchical models, we employ a maximum sentence length of 128 and document length (number of sentences) of 80. The dropout rate in all layers is 0.1. We perform a grid search for the hyperparametes across the search space of batch size [2, 4], learning_rate [1e-6, 5e-6, 1e−5]. We train models with the Adam optimizer up to 20 epochs. We determine the best hyperparameters on the dev set and use early stopping based on the dev set macro-F1 score. \n\n(3f) Inclusion of LLM: We have experimented with ChatGPT and the dataset using different prompting strategies using a temperature of 0. We have found that, as expected, the prompt phrasing has significant influence on the performance, to the point producing counterintuitive metrics. The best model we obtained reaches a macro-F1 of ~.26 and a micro-F1 of ~.31, which is still substantially below our best models. Notably, this is with ChatGPT having possibly seen the cases during pre-training. The latter would be true in principle of GPT-4 and Claude-2, with the latter only having been available after the submission deadline. Given the variation produced by prompting and data contamination, we consider an exploration of the use of generative models for small-scale specialized legal concept identification a different research question that would warrant a separate paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **Novel Dataset Creation**: The introduction of the VECHR dataset is the main contribution. It fills a research gap by providing an expert-annotated multi-label dataset for vulnerability type classification, a previously unexplored area in NLP.\n2. **Explainability Focus**: By including explanation rationale in the dataset, the paper aligns with the growing interest in explainable AI. This aspect allows for more interpretable models, crucial for applications in legal and social contexts.\n3. **Interdisciplinary Relevance**: The paper's focus on human rights and legal applications showcases the breadth of NLP applicability, bridging the gap between technology and social sciences.\n", "reasons_to_reject": "1. **Limited Generalizability**: The focus on a very specific legal context (ECtHR) may limit the generalizability of the findings and the applicability of the dataset to other legal systems or vulnerability assessments. A broader analysis or comparison with other contexts might have enhanced the paper's value.\n2. **Distribution of content:** For a work on a dataset, specific definitions of data types, the data collection process, and detailed distribution of the dataset are crucial. However, these contents are placed in the appendix, and there is a lack of necessary analysis in the main body.", "questions_to_authors": "1. **Comparison with State-of-the-Art Methods**: How does the Concept-aware Hierarchical model compare with other state-of-the-art methods that utilize external knowledge? Could you provide a detailed analysis of why the mic-f1 score of the Concept-aware Hierarchical model is lower than that of the Hierarchical model as shown in Table 2?\n2. **Handling Long Text Inputs**: Considering that the mean number of tokens in VECHR is 4596, while the maximum length of Longformer is only 4096, how did the authors handle instances that exceed this token limit? Have the authors considered applying retrieval or summarization methods to effectively reduce the input length and potentially improve the performance of Bert and Longformer?\n3. **Hyperparameter Selection**: How were the hyperparameters, such as the learning rate and number of training epochs, selected for the experiments?\n4. **Overfitting Concerns**: Considering that there are only 590 vulnerability cases in the training set, what measures were taken to prevent overfitting? Could you share the performance on the validation and test sets during the training process, along with any techniques used to mitigate overfitting?\n5. **Inclusion of Large Language Models**: From my perspective, this dataset seems very suitable for testing on large language models such as GPT-4 and Claude-2, which can analyze each sentence and generate reasons belonging to specific types. Have the authors considered experimenting with these models?\n", "paper_topic_and_main_contributions": "The paper addresses the challenge of recognizing and classifying vulnerability types within the context of the European Court of Human Rights (ECtHR). The concept of vulnerability is vital in understanding individual needs and ensuring effective human rights protection. The author highlights that this concept remains elusive in the ECtHR, and there has been no prior NLP research addressing it.\n\nMain Contributions:\n\n1. **VECHR Dataset**: The authors present a novel expert-annotated multi-label dataset called VECHR, which includes vulnerability type classification and explanation rationale.\n2. **Benchmarking Models**: The paper also benchmarks the performance of state-of-the-art models on the VECHR dataset from both prediction and explainability perspectives.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "Parenthese of Eq.1"}, "extra_text": {}}, {"review_id": "iqRUNYla4e", "reviewer": "Reviewer_6aqx", "summary": "", "questions": "Q: Is the wrong base value used for calculating statistics reported on line #226? The sum is more than 100%. Please check and correct. \n\nQ: Section 5.2 Table 1 suggestion: For explanation Kappa, as mean is reported, it will be useful to also report standard deviation. \n\nQ: Why Concept-aware Hierarchical model is not used in experiments conducted in section 5.1?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed feedback.\n\n(4a) We will and should definitely add implementation details to the paper and also cite frameworks such as huggingface and pytorch. \n\n(4b) Line 226: Our task being Multilabel classification (i.e a datapoint can have multiple labels) and hence the sum can be over 100%\n\n(4c) Kappa-scores: We will add standard deviations.\n\n(4c) Concept-aware model has performance of 31.25 (mic-f1) and 43.11 (mac) on the test set, we would add that to Table 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "R1: It constructs a dataset for an important task. \nR2: The dataset is constructed very well, with details provided on annotation agreement, annotators’ backgrounds, etc.\nR3: The paper is very well written. I learned some things about how to write a good appendix for a data resource paper. Even though many pages of the appendix are used, it is written very cleverly. Readers who are not reviewers or want to reproduce don’t need to read the appendix. This is particularly hard to do for a short paper. Kudos for that. \nR4: Good well-thought effort is put into writing the limitations and ethics statement section. \n", "reasons_to_reject": "It would be sad to reject such a good paper for just this reason but I will state it so they can improve it. Hyperparameter information for fine-tuning like batch size, and learning rate is missing. Also, what version of BERT is used is missing. It is better to specify a base or large, cased or uncased, and other details for each model. ", "questions_to_authors": "Q: Is the wrong base value used for calculating statistics reported on line #226? The sum is more than 100%. Please check and correct. \n\nQ: Section 5.2 Table 1 suggestion: For explanation Kappa, as mean is reported, it will be useful to also report standard deviation. \n\nQ: Why Concept-aware Hierarchical model is not used in experiments conducted in section 5.1?\n", "paper_topic_and_main_contributions": "The paper presents a novel expert annotated dataset for the important task of classifying vulnerability in the legal documents of the European Court of Human Rights. \n\nIt also benchmarks various language models by fine-tuning them on the proposed task. \n\nIt goes beyond creating a main dataset and creates two additional datasets to understand the explanation rationale and test out-of-distribution performance. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Should cite papers related to packages used (pytorch, hugging face, etc.). We generally tend to forget their contribution and cite only models. "}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Article 3 is defined first time on line #157 but was mentioned earlier on line #061. It will be useful to define it earlier. \n\nI think there is a minor typo in footnote #3. Article numbers should be the opposite. \n\nLines #288-289: Word significant can be removed as a drop on micro is not much. \n\nIf the paper is accepted add details from Appendix J to the main paper utilizing the 5th page available. \n\nLine #322: It should be 1,070 not 1070. A comma is missing for consistency. \n\nLine #452: I agree with the statement, but it is not a good idea to make a judgment on what can have a significant or insignificant impact. As it doesn’t add any value, better to avoid writing it. More of a suggestion, please feel free to ignore it. \n\nIn Table 5: If I must guess the number followed by the +- is the standard deviation but it is better to explicitly state that. \n\nI will suggest taking advantage of the additional 5th page (if accepted) to provide more discussion on results reported in section 5. \n"}}]

Author rebuttal:

Meta-review: This paper introduces a dataset for research in legal NLP on assessing the "vulnerability type" (a technical legal term) of cases before the European Court of Human Rights.

Pros:
- Important task which proves to be challenging for today's NLP systems in preliminary experiments (sufficient for the scope of a short paper).
- Well-presented and well-written. It's obvious that a lot of careful thought has gone into this work, both in setting it up as well as in running the project, and then finally in the write-up. The paper was easy to follow and used a solid strategy for delegating details to the appendix, which can be consulted where details are sought, but that's certainly not necessary to understand the paper.
- Extensive data annotation efforts to ensure broad coverage of the data set, which are also thoroughly described.
- Focus on both NLP accuracy as well as explainability, which is critical in this domain.
- Compliments to the authors on well-considered Ethics and Limitations sections, critical in applications like this.

Cons:
- Limited generalizability as this is a fairly niche task (no less important, but still).
- Some implementation details (like hyperparameters) were missing in the review copy, but this will be addressed for the camera-ready, as per the review threads.
Decision: Accept-Main