From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification
Authors: Shanshan Xu
Santosh T.Y.S.S
Oana Ichim
Isabella Risini
Barbara Plank
Matthias Grabmair
Venue: EMNLP 2023 Main (2023)
Keywords: legal judgement prediction
case outcome classification
disagreement
explainability
rationale dataset

Abstract: In legal NLP, Case Outcome Classification (COC) must not only be accurate but also trustworthy and explainable. Existing work in explainable COC has been limited to annotations by a single expert. However, it is well-known that lawyers may disagree in their assessment of case facts. We hence collect a novel dataset RaVE: Rationale Variation in ECHR, which is obtained from two experts in the domain of international human rights law, for whom we observe weak agreement. We study their disagreements and build a two-level task-independent taxonomy, supplemented with COC-specific subcategories. To our knowledge, this is the first work in the legal NLP that focuses on human label variation. We quantitatively assess different taxonomy categories and find that disagreements mainly stem from underspecification of the legal context, which poses challenges given the typically limited granularity and noise in COC metadata.  We further assess the explainablility of state-of-the-art COC models on RaVE and observe limited agreement between models and experts. Overall, our case study reveals hitherto underappreciated complexities in creating benchmark datasets in legal NLP that revolve around identifying aspects of a case's facts supposedly relevant for its outcome.

From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification
Shanshan Xu Santosh T.Y.S.S** .ana Ichim², Isabella Risini,,
Barbara Plank4. Matthias Grabmair'
TTcchicaa University of Munich, Germany
"Graduate Institute of International and Development Studies, Switzerland
FFacllty of Law, Ruhr University Bochum, Germany
'IT University of Copenhagen, Denmark
MM Munich & Munich Center for Machine Learning (MCML), Germany
Abstract
Disagreement in Experts' Annotation
Expert-moder Misallgniiine
In legal NLP, Case Outcome Classification (COC) must not only be accurate but also trustworthy and explainable. Existing work in explainable COC has been limited to notations by a single expert. However, it is well-known that lawyers may disagree in their assessment of case facts. We hence collect a novel dataset RAVE: Rationale Variation ECHR!, which obtained from perts in the domain of international human rights law, for whom we observe weak agree- ment. We study their disagreements and build two-level task-independent taxonomy, supple- mented with COC-specific subcategories. We quantitatively assess different taxonomy cate- gories and find that disagreements mainly stem from underspecificttio of the legal context, which poses challenges given the typically lim- ited gramularity and noise in COC metadata. To knowledge, this is the first work in the legal NLP that facuses on building a taxonomy over human label variation. plainablility of state-of-the-e COC models on RAVE and observe limited agreement between models and experts. Overall, our case study re- veals hitherto underappreciated complexities in creating benchmark datasets in legal NLP that revolve around identifying aspects of case's facts supposedly relevant to its outcome.
Expert 1
Expert 2
Alleged Article(s): Art. 3, Art. 8, Art.1 THE FACTS:
applicants did apply join water supply system. According to them, while living at previous localion. they had been denied access to the group weseer-dstributtin connection by their neighbours, who hadd nol allowed Ihemn to lay waler pipe under "their" land
Model's explanation
Figure I: The disagreement between experts' annotation and the misalignment of model and experts'.
into patterns, they can be used to index legal ana- lytics databases that inform litigation strategy. In other words, the case outcome signal can serve proxy for legal relevance of certain parts of the fact deseription.
A major obstacle in realizing this vision is the misalignment between what models predict from with what experts consider relevant. For instance, prior work by Chalkidis et al. 2021 demonstrates that the alignment between models and experts rationale is fairly limited in the jurisprudence of the European Court of Human Rights (ECtHR). Previous work (Santosh et al. 2022) shows that these models are drawn shallow spurious predic- tors if not deconfounded. Alignment is typically evaluated by deriving saliency//mportance map from the model's predictions on the input text and comparing it with the ground truth labelled by, in most cases, a single expert. Aforementioned previ- ous works have derived and evaluated the explana- tion rationales at the fairly coarse paragraph level, which may over-estimate alignment.
further assess ex-
1
Introduction
The task of case outcome classification (COC) in- volves classifying the outcome of a case from a textual description of its facts. While high clas- sification performance is of course desirable, its greatest potential utility for legal experts lies in the identification of aspects involved in the case that contribute to the outcome prediction. For exam- ple, these extracted aspects could be grouped
In light of this, introduce RAVE, novel ex- planation dataset of COC rationales from two legal experts much more granular and challenging token level. In addition, while many cases involve multiple allegations, previous works typically col-
'Our guidelines, dataset and code available at https: //github. cow htmm/TuMLegllTcchhheee T 'These authors sontributed equally to this work

---

lect only one annotation per case. contrast, our approach involves collecting annotations for ev- ery alleged article under each case, enabling us to capture a more nuanced decision-making process of the annotators. However, we observed weak inter-annotator agreement (IAA) between experts' rationale annotations, indicating the challenge of marking up rationales consistently and emphasiz- ing the perspectivist nature of the legal decision- making process (see Figure 1).
ment mining (Mochales and Moens, 2008; Haber- nal et al., 2023; Poudyal et al., 2019, 2020), event extraction (Filtz et al., 2020; Navas-Loro and Rodriguez-Doncee 2022) and vulnerability type classification (Xu et al., 2023). In this work, we focus mainly on COC task using ECIHR corpus studying the sources of disagreement/HLV in COC rationale. to contribute to the field of explainable legal NLP.
There is a growing body of work in main- stream NLP highlighting the presence of irrecon- cilable variations among annotations, and finding that such disagreement is abundant and plausible (Pavlick and Kwiatkowski, 2019; Uma al., 2021; Sap et al., 2022). Researchers further argue that we should embrace such Human Label Variation (HLV) (Plank, 2022) signal which provides rich information signal, and not as noise which should be discarded (Aroyo and Welty, 2015; de Marneffe et al., 2012; Basile et al., 2021). This motivates us to study the disagreements we encountered be- tween our experts on rationales for COC.
2.2 Explainable COC
Prior work refers to COC as Legal Judgment Pre- diction (LJP). Under that term, has gained wide attention in various jurisdictions (e.g.. Aletras et al. 2016, Katz et al. 2017, Yue et al. 2021). In the legal domain, explainability is of crucial impor- tance to make models trustworthy. Chalkidis et al. 2021 investigated the explainability of ECIHR al- legation prediction an annotated dataset of 50 cases by identifying the allegation-relevant para- graphs of judgment facts sections. Also working at the paragraph-level, Santosh et al. 2022 used deconfounding to improve model alignment with expert rationales. Malik et al. 2021 introduced an Indian jurisdiction LJP corpus with 56 cases anno- tated with explanations at sentence-level by experts. Different from the above works, RaVE contains tionales at the more granular word-level for each case-article pair and systematically investigates the possible sources disagreement between experts.
To the best of our knowledge, this is the first work in the legal domain systematically study HLV in explanation datasets. We propose a two- level task-independent taxonomy disagreement sources and introduce COC task-specific subcat- egories. We also quantitatively assess the effects of different subcategories on experts' annotation agreement. The results indicate that disagreements are mainly due the underspecified information about the legal context in the annotation task, high- lighting the importance of developing better an- notation guidelines and schema in legal domain. Further, we assess the state-of-the-art COC mod- els on alignment with respect to obtained experts token-level rationales in various settings. Despite differences in COC prediction performance, we ob- serve consistently low alignment scores between models and experts rationales in different settings (Fig 1), suggesting need for further investigation on how to make models align its focus better with legal expert rationales.
2.3 Annotation Disagreement
Annotation disagreement has been found in a wide range of NLP tasks, especially not only in highly subjective tasks such as natural language inference (NLI) (Pavlick and Kwiatkowski, 2019), toxic lan- guage detection (Sap et al., 2022), but also in seem- ingly objective tasks such as part-of-speech tag- ging (Plank al., 2014). The most similar work to ours LIVENLI (Jiang et al., 2023), which sys- tematically studies the variations explanations of IL. task. LIVENLI collects 122 English NLI items, each with at least 10 annotations from crowd- workers. Unlike NLI, our task requires annotators to have deep understanding of ECHR jurisprudence to identify the rationales. Hence, we obtain 76 case-allegation article pairs, each annotated by 2 experts. Despite the lower numbers of annotators. we found low agreement in the rationale annotation, prompting us to study the sources of disagreement systematically.
2 Related Work
2.1 Tasks on ECIHR Corpora
Previous works involving ECIHR corpus has dealt with COC (Aletras al., 2016; Chalkidis al.,
2019; Valvoda et al., 2023; Santosh et al., 2022, 2023; Tyss et al., 2023; Blas et al., 2023), argu-

---

The Dala
The Annotator
Genuine Ambiguity
Text Production
Annotation
Artefact
Context
Noise
Subjectivity
Longitudinal
Normative Unceriang nly
Narrative Unclarity
Underspecified Allegation
Information
Sloppiness
Interface Limilaion
Subjectivity
Individual Behavior Change
Judge Spllt
KeyCasa
Number
Subsidiary
Allegation Cranuality
Difter ent Familiaity
Different Prediction ction
Paragraphs
Allegation
Figure 2: Taxonomy of disagreement sources: Macro Categories (Yellow), Fine-grained Categories (Green), COC
rationale annotation specific (Pink), Proxy Variahles (Blue).
2.4 Categorization of Disagreement
2022). Subsequently, the court reviews the case and determines whether violation has occurred, align- ing with Task A (violation prediction) in LexGLUE. When ruling on a case, the ECtHR typically exam- ines only the specific articles alleged by the appli- cant, which corresponds to Task AIB (violation pre- diction given allegation) as introduced by Santosh et al. 2022. In this study, we evaluate the explain- ability of COC models for Task AIB, as it mimics the real legal process. We also incorporate ecolog- ical validity into the construction of our rationale dataset. Inspired by Jiang et al. 2023, we define a rationale explanation in COC as ecologically valid if the annotator (1) provides both an explanation and a case outcome prediction and (2) to each spe- cific article in allegation. Previous legal rationale datasets, such as Chalkidis al. 2021 and Santosh et al. 2022, required experts to highlight rationales in the case facts that support the 'violation' of 'any' of the alleged articles. However, we consider this approach not ecologically valid because: Firstly, ex- perts may predict contrary to the given ground truth and thus exhibit bias in explanations when they dis- agree with the givenn violation label (Wiegreffe et al. 2022; Jiang et al. 2023). Thus, we disclose the allegedly violated articles to the experts, ask them to predict the outcome, and highlight ratio- nales to support their prediction. Secondly, since many cases in the ECHR involve multiple alleged articles, rationales in previous works could justify one or more alleged articles collectively. In con- trast, RAVE contains rationales for each alleged article in each case separately, thereby reflecting experts' nuanced decision makinng process.
Recently, several studies have proposed tax- onomies to identify the possible sources of dis- agreement in various natural language processing tasks. Basile al. 2021 outlines three main sources disagreement: the annotator personal opinions and judgment; the data ambiguity of the text, com- plexity of the task and the context changes in the subjects' mental state over time. Uma et al. 2021 expanded the list by adding the categories such as: annotator's error and imprecise annotation scheme. Jiang and Marneffe 2022 introduced lask-specific taxonomy for NLI, consisting of 10 categories that which can be broadly classified into three overar- ching classes. More recently, Sandri et al. 2023 in- troduced a taxonomy of disagreement in offensive language detection, which consists of 13 subcate- gories grouped in four main categories. As pointed out by Sandri et al. 2023, there many high-level overlaps between these taxonomies, whereas the lower categories, are rather task-specific. Building on the meta-analysis of the existing taxonomies, we generalize the taxonomies of disagreement to a two-layered categories and introduce task-specific categorics and proxy variables for COC rationale annotation as in Fig 2. Additionally, we map and unify existing taxonomies into our proposed com- mon categorization, providing an overview of the existing landscape (See Table 4 in App A).
3 Ecological Validity
"Ecological validity" refers to the extent to which the experimental situation is similar to wolldworld condition (Aronson and Lindzey, 1968). At the EC- tHR, the application process begins with the appli- cants lodging their accusation, alleging one or more violations of articles in the ECHR. This process cor- responds to Task B (allegation prediction) in the widely used LexGLUE benchmark (Chalkidis et al.,

---

4 Dataset Creation
4.1 Case Sampling
We sampled 50 cases from the ECtHR's public database HUDOC" between 2019-2022. Refer to App B for sampling criterion. Out of the 50 cases, we allocate 10 cases for pilot study which are used to co-ordinate between experts and refine the guidelines. The remaining 40 were kept as our test set to measure agreement, which contain a total of 76 case-alleged article pairs (hereafter pairs).
UMA MALappaion
Figure 3: Distribution of the IAA Kappa Scores
4.3 The Rationale Dataset and IAA
4.2 Annotation Procedure
Table presents the key statistics of our rationale dataset. The dataset comprises 40 cases, with an average of 2764 words per case. On average, each case involves 1.9 allegations, resulting in a total of 76 case-allegation pairs. Among these 76 pairs, annotator did not annotate 6 pairs, she did not find any relevant segment in the case facts for the paired article. Similarly, annotator 2 did not annotate pairs, which were subset of the afore- mentioned cases. Annotator 1 tended to provide longer rationales, averaging 485 words per pair; while annotator 2 has average of 373 words per pair. Aligning with a shorter annotation is more challenging it necessitates capturing the critical aspects of the case text with greater precision. We measure IAA using Cohen's kappa coefficient x (Cohen, 1960) between the two experts' annotated token-level markups for every pair. We cast expert annotated markup for every pair into a binary vec- tor with of length equal to the number words a given case, with value of if the word is marked up by in annotator and is not an NLTK (Bird et al.. 2009) stop word, or 0 otherwise. Figure displays the distribution of IAA kappa scores alleged article-case pairs falling in the range of 0.03, 0.56), with mean of 0.24. There are 3 out of 76 pairs whose score fall in the negative agree- ment category, indicating no agreement: 22 out 76 fall into [0.0,0.2], indicating slight agreement; 38 cases have socres between [0.2,0.4], indicating fair agreement, and only 13 scores fall into [0.4.0.6]. which deemed to be moderate agreement.
The rationale annotation process was done using the GLOSS annotation tool (App E) (Savelka and Ashley, 2018). Annotators' were given the fact statements of the case, along with information about the alleged article(s). For each alleged ar- ticle, they were asked to highlight relevant text segments of case fact, which increase the likeli- hood of them reaching the prediction that a certain article has been violated or not. The full Annota- tion Guidelines are available in App F. We allow the annotators to highlight the same text span as rationales for multiple articles alleged under that case. We also ask the experts to record their predic- tions for the outcome each pair, and mark cases they were already familiar with.
Inconsistent A Allegation Information in HUDOC During the pilot annotation phase, experts reported occasional incompleteness of the provided allega- tion information, which we fetched from the HU- DOC database. As result, we recollected the al- legation information semi-manually from the case judgment documents for our dataset. For detailed information on this collection process, please refer to App C. We manually inspected all our 50 sam- pled cases, and identified that such inconsistency exists in approximately 25% of them. Our ECHR experts posit that this issue stems from the fact that the HUDOC primarily registers only the main allegation in the metadata while omitting the sub- sidiary allegation. As a result, there is a mismatch the allegation information obtained from HU- DOC metadata and perceived by the experts from case text. While it was possible for us to manually curate and correct the metadata involved in this case study, our finding of this HUDOC metadata incon- sistency has wider implications for benchmarking efforts derived from the raw database.
the 76
Disagreement between Experts
5.1 Disagreement Taxonomy
To understand the reasons behind experts' disagree- ment, we present taxonomy of disagreement sources. The task eakiindependent two-level taxonomy constructed through a meta-analysis of prior stud-
"http::/hudoc.ee www.uddd...cc..coom int
"ee App D for annotators' background and expertise

---

Case fact
ing subcategories such as Rhetorical Questions and Sarcasm. In COC rationale annotation, genuine ambiguity can be traced back to Normative Un- certainty, which arises when multiple legal source interpretation and argumentation is possible to jus- tify an outcome by the court. Its occurrence is not uncommon in ECtHR judgments due to the deliber- ale drafting of its convention in relatively abstract manner, to allow interpretation adaptable to a wide range of scenarios.
cases
40
Avg. # allegations per case # case-allegation pair
1.9
76
Avg. length per case
2764 1170 words
Avg. # paragraph per case
40 24 paragraphs
Rationales from annotator
485 398 words Avg. length case case-allegation pair
Rationales from annotator
Avg. length case-allegation pair
373 + 331 words
Table I: Statistics for the dataset.
Text Production Artefacts pertain to inconsisten- cies, incompleteness or introduced biases during text production. The only one category relevant in previous work is Not Complete in Sandri et al.. which refers to incomplete texts resulting from er- rors in extracting social media threads. However, in the legal domain, where texts are often lengthy and complex, there is greater possibility of encoun- tering artefacts during the text production process. Additionally, the facts section of an ECtHR judg- ment not an entirely objective account, but the product of registrars' subjective summarization of framing the facts based on the information pre- sented before the Court and their interpretation. Further, case document production often involves multiple court registrars, each may have different narrative style for framing the cases. This can give rise artefacts such redundant inconsistent framing of facts. Consequently, experts may en- counter difficulties in interpreting the case due to such Narrative Unclarity.
ies on disagreement categorization across different tasks. Additionally, we propose subcategories spe- cific to COC rationale annotation developed to- gether with the two legal experts. We also select proxy variables to quantitatively study the effects of these categories experts TAA score. The tax- onomy was developed by the first author who holds a master's degree in linguistics with expertise in linguistic typology and experience in NLP projects on ECtHR data. To ensure the appropriateness of the proposed subcategories, a discussion and adju- dication phase was conducted with the experts to refine and adjust items. Fig 2 offers overview of the taxonomy and the proxy variables. In the fol- lowing sections, we provide detailed explanations for each category.
5.1.1 The Data
Disagreements within NLP annotation tasks can arise from various aspects of the textual data, which include genuine ambiguity within the text itself, potential artifacts introduced during the text pro- duction process or incomplete information within the annotation context.
Context Sensitivity encompasses instances where annotators may disagree due to a lack of context or additional information needed in the annota- tion guidelines to interpret text unambiguously. The Missing Information category in Sandri et al. 2023, Underspecification in Guidelines in Jiang and Marneffe 2022, and Item difficulty and Anno- tation Scheme in Uma et al. 2021 relate this category. our task, the context category arises from Underspecified Allegation Information. comparison to previous studies that only provided experts with factual text, we incorporate additional information regarding the alleged articles. How- ever, feedback from the experts highlights their requirement for more specific allegation informa- tion, particularly in cases involving multiple alle- gation articles. Specifically, they have identified two types of missing information that significantly impact their interpretation of the case: (1) Allega- tion approach: The distinction between main and
Genuine Ambiguity arises when an expression (a word, phrase, or sentence) can be interpreted in multiple ways. This ambiguity can stem from various language features, such as homonyms and figurative language. Categories from prior works, such as the Ambiguity in Uma et al. 2021 and Sandri et al. 2023, and the Uncertainty in sentence mean- ing in Jiang and Marneffe 2022 can be mapped to this category. However, it is important to note that the subcategories are specific to particular tasks. For instance, in NLI, Jiang and Marneffe primarily focus on semantic ambiguity, including subcate- gories such as Lexical and Implicatures. Alter- natively, in offensive language detection, Sandri et al. emphasizes pragmatic ambiguity, encompass-

---

subsidiary allegations has a substantial influence on their comprehension and analysis of the case. (2) Allegation gramularity: A more detailed allega- tion scheme is necessary. For instance, Article 6 of the ECHR comprises multiple sub-parts. Merely annotating for Article 6 as a whole does not enable experts to provide precise and optimal rationale annotations.
their choice of rationales. Longitudinal consists of disagreements made by the annotator at different times of testing. While Basile et al. 2021's 's taxonomy is the only one to consider the temporal aspect of disagreement re- search, our category is distinct from their Context category in two ways. First, Contexi includes at- tention slips of annotator, which we categorize noise. Second, not only consider Individuals Behavior Change of annotators, but also the atti- tude change of the whole human society towards certain phenomena over time, including changes in laws, policies, and cultural norms. For example, racial segregation used to be institutionalized in many countries, but it has since been legally abol- ished and is now widely condemned as violation of human rights. The longitudinal aspect is par- ticularly relevant in the legal domain, where case law evolves over time to reflect changing societal attitudes and values (Case Law Change). Lawyers must adapt their legal strategies and reasoning rela- tive to this development. Due to the limited scope of this work, we unfortunately cannot include a longitudinal study and leave it for future research.
5.1.2 The Annotator
Disagreements can arise due to variances in anno- tators' behavior. We consider following categories: Noise covers errors due to annotator's Sloppy An- notation Interface Limitation. Sloppy Annota- tion may occur frequently in crowd-sourcing plat- forms, where annotators may be recruited without proper training and their annotation quality may not be monitored. Examples that fall under this cat- egory include Uma et al. 2021's Error and Sandri et 2023's ' Sloppy Annotation. Because our an- notation process involved close collaboration with legal experts, we expect negligence-relate related noise be minimal. While the annotators were actively involved in producing the guidelines during the pi- lot study, in the main annotation phase these guide- lines were tested by new fact patterns and questions about how relevance was to be annotated in individ- ual cases. Limitations of coverage in our guidelines hence contribute to the disagreement we observe in this experiment. We plan to improve on this aspect in future work, but note that case facts can be highly specific and a refinement/extension with explicit instructions for writing styles and narrative patterns from development case examples would consume considerable resources. Additionally, Uma et al. 2021 notes that disagreements in annotation may also stem from Interface limitations (i.e., where the capture of relevant information is obstructed by interface quirks). In our case, such limitation of the GLOSS tool have been brought to our atten- tion.
5.2 Proxy Variables
To quantitatively investigate how the agreement of experts' rationale annotation is influenced by different taxonomic categories, derive proxy variables together with experts. We assess statis- tical association between these variables and ex- pert's TAA score. We hypothesize that these proxy variables correspond to lower agreement among experts.
Normative Uncertainty enables multiple possible legal interpretations. It is subjective and abstract concept that is challenging to measure directly from text alone. Therefore, we select the following vari- ables as proxy:
JudgeSplir: In cases where multiple interpretations are possible, the judges may fail to achieve an agreement and produce a split voting on the out- come, reflecting the intrinsic ambiguity in the case. We fetch the judges voting record from HUDOC and binarize as 1 if it is non-unanimous voting, and 0 otherwisc.
Subjectivity Annotation tasks often involve per- sonal opinions and biases, which are shaped by individual experiences that are inherently private. For instance Sap et al. 2022 demonstrated how an- notators' demographic identities and attitudes can influence how they label toxicity in text. In the legal domain, experts' interpretations of cases are inevitably influenced by personal biases. Further- more, variations in training backgrounds and prior familiarity with cases can also lead to different lc- gal interpretations of case and hence influence
KeyCase: The ECtHR annually chooses set of sig- nificant cases, known "key cases". These often deal with complex and novel legal issues. Given the absence of established legal standards for inter- preting them, they often generate controversy and

---

disagreement among experts. We retrieve relevant information from the HUDOC and categorize case as I if is listed as key case and 0 otherwise. Narrative Unclarity refers to the lack of clarity in the text resulting from artefacts during the text production process. Experts have emphasised in their feedback that an excessive amount of detail and repetitive descriptions of events or redundant events pose challenges in annotation. In ECtHR judgments, case facts are organized into short para- graphs, each generally representing distinct fac- tual event or aspect. Consequently, we utilize num- ber of paragraphs (NumPara) as a proxy for the level of factual detail.
Proxy
mean IAA
t-value
p-value
0
JudgeSplit
0.218
0.344
3.096
0.002*
KeyCase
0.242
0.249
-0.183
0.856
OmitAlleg
0.278
0.086
-4,800
8c-6*
Article6
0.260
0.150
2.378
0.020*
DiffPred
0.225
0.341
-2.105 105
0.038*
DiffFam
0.239
0.260
-0.454
0.651
NumPara
= -0.269
0.019
Table 2: Associations of proxy varaibles and IAA scores.
(<: p 0.05)
5.3 Results and Discussion
For each binary proxy variables, we compute the mean of the IAA scores among the group of in- stances exhibiting that proxy variable (value 1) and the rest (value 0). Then, we perform indepen- dent t-test to compare the mean TAA scores be- tween these two groups. For the ordinal variable NumPara, we calculated Pearson's 's correlation co- efficient r. The results are presented in Table 2. Our results show that experts have significantly lower agreement in their annotation for OmitAl- leg and Article 6. This indicates that under- specification allegation information leads to un- certainty in their annotations, validating our hypoth- esis which is further confirmed by experts' feed- back. We also find significant association between the experts' IAA and NumPara. This indicates that the level of detail in the case facts adds to the dif- ficulty. Surprisingly, we found that experts have higher agreement on DiffPred. According to ex- perts' feedback, they might have selected the same relevant facts to consider, but their interpretations differ, which can lead to different predictions about the outcome. This was echoed by Jiang et al. 2023 in the context of NLI explanation and entailment labels. Our hypothesis regarding DiffFam does not hold and can be attributed to the fact that agree- ment is more influenced by relative familiarity de- rived from analogous case fact patterns rather than absolute familiarity with a specific case. When approaching a new case, experts often draw upon their experience with similar cases and apply simi- lar legal principles during annotation. As result, diffFam may not necessarily lead to low agreement. We also observe significant association for Key- Case. This can be attributed to the fact that, while the legal issue might have been complex and novel at the time the case was designated as a Key Case, by the time of annotation, such cases may have transformed into established legal precedents. In
Incomplete Allegation Information encompasses disagreement due to the lack of precise allegation information during annotation, such the allega- tion approach and coarse allegation granularity. We select the following two proxy variables:
OmitAlleg (Omitted Allegation): As stated in Sec- tion 4.2, the ECtHR often omits subsidiary alle- gations in HUDOC metadata. Here we use these omitted allegations the allegations present in our curated allegation list but absent in the original HUDOC metadata as proxy for Subsidiary Alle- gation.
Article 6: According to the experts feedback, cases involving Article 6 (Fair Trial), which includes many sub-parts, lead to considerable uncertainty on which of its aspects was alleged, contributing to their low agreement.
Subjectivity pertains to the disagreement due to annotators' personal opinions and bias. We select the following two proxy variables for this category: DiffPred (Difference in Outcome Prediction): Ex- perts may have different outcome predictions on the same case, and further choose different ratio- nales to support their predictions;
DiffFam (Difference in Case Familiarity): Prior knowledge of a specific case enables an expert to be aware of its detailed legal and factual details, as well as the court's 's opinions. Consequently, diver- gent levels of case familiarity can result in different interpretations.
During annotation, we request experts indicate their outcome prediction and prior familiarity with each case. We binarize the variable as if both experts have the same prediction or familiarity, re- spectively, and 0 otherwise.

---

other words, what was uncertain at that time was decided has become settled by now. This temporal change of case law may also explain the counter- intuitive high IAA scores on JudgeSplit: the once complex legal issue caused judges split voting in the past may have stabilised by the time of annota- tion. This highlights the importance of considering the longitude dimension in future studies.
6.2 Article-aware Classification Models
Article-aware prediction for Task AIB takes the case fact text, the text of convention article, and binary value indicating whether the given arti- cle has been alleged as input. classifies whether the article has been deemed violated or not by the court, thereby capturing the interplay between case facts and the wording of the convention. We adapt the article-aware architecture from (Santosh et al., 2023) which uses a pre-trained BERT model to ob- tain pre-interaction token encodings for each token in every packet, which are then aggregated form packet representations using a token-level attention layer. A bidirectional GRU then contextualizes the packet representations. This procedure is done sep- arately for the text of both facts and article. The interaction component computes the dot product attention between packet representations of case facts and article. We concatenate different combina- tions of pre-interaction and post-interaction aware representations through concatenation, difference and element-wise product for both article and fact reprsentations. We then merge both branches by conditioning the final representation of case facts on the final article representation obtained from the packet GRU and packet attention layer. This condi- tioning is done by initializing bidirectional GRU layer's s hidden state of the case facts branch with the post-interaction representation of the article. This guides the GRU model to extract and refine the representations of case facts conditioned on the current article. We concatenate the obtained con- ditioned case facts representation with the binary label of allegation of that article and use two fully connected layers obtain the violation outcome. We train the model end-to-end using binary cross- entropy loss. See App G for model details.
6 Expert-Model Misalignment
We evaluate the SOTA COC models' alignment with the experts' rationales in RAVE across two paradigms of models: (i) Fact-only classification models which rely on case facts description as sole input for the violation outcome (Santosh et al., 2022; Chalkidis et al., 2022), and (ii) Article-aware prediction models which take case facts descrip- tion along with article text to predict the binary outcome of the case with respect to that article (Santosh al., 2023).
6.1 Fact-only Classification Models
As adopted from Santosh et al. 2022, we employ a BERT variant of the hierarchical attention model (Yang et al., 2016), which takes case facts descrip- tion and the set of articles claimed to be alleged by the applicant as a multi-hot vector as the input and predicts the set of articles that are deemed violated by the court as multi-hot vector. We a greedy input packing strategy where we merge multiple paragraphs into one packet until reaches the max- imum of 512 tokens given by BERT models. In this work we experiment with three BERT variants: BERT "bert-base-uncased (Kenton and Toutanova, 2019), CaselawBERT " "casehold//egalberr (Zheng et al.. 2021), LegalBERT hwtppueb/leegl-bbetbbbeeee uncased" (Chalkidis et al., 2020). It consists of a pre-trained BERT encoder to obtain token-level representations for every packet. Then these token- level representations are aggregated into packet representation using a token-attention layer. These packet representations are contextualized using a GRU layer and further aggregated into final case fact representation using pecket-level attention. Then the final representation is concatenated with the multi-hot allegation vector and passed through two fully connected layers to classify the violation outcome. We train the model using a binary cross- entropy loss against the multi-hot target. See App G for model details.
6.3 Dataset, & Evaluation Metrics
We use the Task A and Task B datasets from LexGLUE (Chalkidis et al., 2022) and merge them to form Task AIB (i.e., predict convention viola- tions given information about allegedly violated articles) as described in (Santosh et al., 2022). It comprises 1 1 case fact descriptions chronologi- cally split into train (2001-2016, 9k), validation (2016-2017, 1k) and test sets (2017-2019, 1k). For article-aware classification, we use the expanded dataset from Santosh et al. 2023 with the texts of the 10 articles used in LexGLUE.
We use Integrated Gradient scores to obtain

---

Explanation Agreement
Classification Performance
Model
Kappa Al
Kappa A2
micro-F1
macro-Fl
hard-macro-Fi Fl
BERT
11.48 (3.62)
9.22 (3.29)
65.52
74.97
57.97
Fact-only
LegalBERT
11.37 (3.66)
9.18 (2.94)
66.03
76.49
58.42
Caselaw BERT
11.59 (3.71)
9.37 (3.58)
65.84
75.35
58.17
BERT
11.75 (3.64)
9.57 (3.44)
74,43
80.35
61.22
Article-aware
LegalBERT
11.53 (3.51)
9.31 (3.53)
74.58
80.83
61.35
CasclawBERT
11.82 (3.66)
9.79 (3.22)
74.49
80.95
61.38
Table Agreement and prediction scores of Fact-only and Article-aware classification. We report Kappa score with standard error for explanation agreement, and Fis for classification performance.
token-level importance from the model with re- spect of violation outcome of every alleged article (Sundararajan et al., 2017). We max pool over sub-words to convert token-level IG scores into word-level scores, followed by threshold-based binarization. We report averaged ς between the models' focus and the experts' annotations across the cusc-allegation-artiel pairs, for each of the two experts. We also report prediction performance in terms of micro-IFl (mic-FF),, macro-Fi (mac-F1) and Hard-macro-Fl (hmF1), which the mean F1- score computed for each article where cases with that article having been violated are considered as positive instances, and cases with that article be- ing alleged but not found to have been violated as negative instances following Santosh et al. 2022.
7 Conclusion
In this paper introduce RAVE, novel cor- pus for COC annotated with rationale variation provided by multiple legal experts, that accounts for variation in rationales between annotators, We study the human label variation and propose a hi- erarchical, two-layer taxonomy of sources of dis- agreement. Our analysis reveals that annotator dis- crepancies mainly arise from the underspecified allegation information during annotation. We eval- uate the interpretability of current state-of-the-art COC models using RAVE and find that there is limited consensus between the models and domain experts. Our results have important implications, highlighting the intricate nature of COC and em- phasize the necessity for interdisciplinary collabo- ration between legal experts and ML researchers to integrate expertise into trustworthy and explainable COC systems that aligned with legal principles.
6.4 Discussion & Analysis
Acknowledgments
Table 3 presents classification (F1 scores) and alignment (k scores) performance of fact-only and article-aware classification. We make the follow- ing observations: (i) All models across both the settings slightly better aligned with annotator A1, instead of A2, which may due artefact that Al tends have longer annotation markup than A2. (ii) Yet overall, agreement between models and low (iii) Though Article- aware models outperform the fact-only variants in all three cetisificaatio-rllaate metrics, demonstrat- ing the effectiveness of incorporating article infor- mation, their alignment with the experts remain similarly low, In other words, the models might be right for the wrong reasons. Overall, our results show that there need for further investigation and more datasets on how to make models align its focus better with legal expert rationales.
We are grateful Jaromir Savelka for the ability to use the GLOSS annotation tool (Savelka and Ashley, 2018). We would also like to thank the members of the MaiNLP/CIS reading group for the inspiring discussions on Human Label Variation. We also thank the anonymous reviewers for valu- able comments. BP is funded by ERC Consolidator Grant DIALECT 101043235.
human annotators very
Limitations
While our two-level taxonomy of disagreement is designed to be task-independent and covers the an- notation of various subjective tasks, the specific subcategories introduced in this study are tailored to the COC rationale annotation task. Future in- vestigation is needed to assess the applicability these subcategories to other tasks within the legal

---

domain. Similarly, the limited scope of this an- notation effort restricts the generalizability of our findings, which remain to be confirmed on a larger dataset with more than two annotators, possibly with different experience profiles. The inclusion of a longitudinal category in Section 5.1 encourages future research to conduct experiments and delve deeper into exploring the temporal aspect within the legal domain.
portance of trustworthy and explainable legal NLP. In this study, we focus on obtaining the token- level rationale dataset annotated by experts with an overarching goal to improve the alignment between what models and expert deem relevant. Further- more, our focus is on investigating the sources of disagreements among lawyers in their assessment of case facts. We have carefully considered the ethical implications of our research and do not an- ticipate any specific risks associated with it. On the contrary, our analysis of different types of disagree- ment aims to promote the acceptance of human label variation within the legal NLP community. By acknowledging and understanding various per- spectives, interpretations and biases of legal pro- fessionals, we contribute to a more comprehensive and inclusive discourse within the field.
We focus on in-text rationales for COC, i.c. spans of text in the facts statement marked up by multiple annotators. However, when annotators chose the same rationale elements, they may have had different reasons to do so. Further, in-text ratio- nales only provide evidence for the label without conveying the mechanisms for how the evidence leads to the label. A potentially better way to al- leviate both these limitations would be to obtain free-text rationales (Tan, 2022).
References
Additionally, while our findings of model align- ment with experts are specific to the ECtHR do- main and datasets, comparable experiments in other domains will see variation based on the nature of the fact input used and the legal issues to mod- cled. Nevertheless, all derivation of insight from legal case data comes with modeling assumptions and data-related limitations. We consider this case- study to be prototypical for the kinds of challenges that will have to be tackled to develop future data- driven legal NLP models that are explainable and work in synergy with experts.
Nikolaos Aletras, Dimitrios Tsarapatsanis. Daniel Preofiuc-Pietro, and Vasileios Lampos. 2016. Pre- dicting judicial decisions of the european court of human rights: A natural language processing per- spective. PeerJ Computer Science, 2:e93.
Elliot Aronson and Gardner Lindzey. 1968. The hand- book of social psychology. Addison- Addison-Wesley Publish- ing Company.
Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. Al Magazime, 36( 36(1):15-24.
Isabelle Augenstein, Tim Rocktäschel, Andreas Vla- chos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceed- ings of EMNLP 2016, pages 876-885.
Ethics Statement
Our dataset are retrieved from a publicly available dataset of ECHHR decisions, sourced from the pub- lic court database HUDOC. While these decisions include real names and are not anonymized, we do not anticipate any harm arising from our periments beyond the availability of this informa- tion. The task of COC/LJIP raises significant eth- ical and legal concerns, both in a general sense and specifically regarding the European Court of Human Rights (Medvedeva et al., 2020). However, it is important to clarify that we do not advocate for the practical implementation of COC within courts. Previous work (Santosh et al., 2022) has demonstrated that these systems heavily rely on superficial and statistically predictive signals that lack legal relevance. This underscores the potential risks associated with employing predictive systems in critical domains like law, and highlights the im-
Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement evaluation. In Proceedings of the Ist Workshop on Benchmarking: Pust, Present and Future, pages 15-21, Online. Association for Com- putational Linguistics.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Nat- ural language processing with Python: analyzing text with the natural language toolkit. "O'Reill "O'Reilly".
Marcel Perez San Blas, Phillip Kemper, Matthias Grab- mair, et al. 2023. Leveraging task dependency and contrastive learning for legal judgement prediction on the curopean court of huraan rights. arXiv preprint 2302.0300000763
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Ale- tras. 2019. Neural legal judgment prediction in glish. In Proceedings of ACL 2019, pages 4317- 4323.

---

Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and lon Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. In Findings of EMNLP 2020, pages 2898- 2904.
Masha Medvedeva, Michel Vols, and Martijn Wieling. 2020. Using machine learning to predict decisions of the european court of human rights. Artificial Intelligence and Law, 28(2):237-266.
Raquel Mochales and Parie-Francine Moens. 2008. Study the structure of argumentation in case law. In Proceedings of the 2008 conference on legal knowl- edge and information systems, pages 11-20.
Ilas Chalkidis, Manos Fergadiotis, Dimitrios Tsarapat- sanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis. 2021. Paragraph-Ievel ratio- nale extraction through regularization: A case study on european court of human rights cases. In Proceed- ings of the NAACL-HLT 2021. pages 226-241.
Maria Eavas-Loro and Victor Rodriguez Donnzel 2022. Whenthefact: Extracting events from european le- gal decisions. In Legal Knowledge and Information Systems, pages 219-224, IOS Press.
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, lon Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. Lexglue: A benchmark dataset for legal language understanding in english. In Proceedings of ACL 2022, pages 4330-4330.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- rions of the Association for Computational Linguis- tics, 7:677-694.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37 46.
Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10671- Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Marie-Catherin de Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did it happen? the prag- matic complexity of veridicality assessment. Compu- tational Linguistics, 38(2):301-333.
Erwin Filtz. María Navas-Loro, Cristiana Santos, Axel Polleres, and Sabrina Kirrane. 2020. Events matter: Extraction of events from court decisions. Legal Knowledge and Information Systems, pages 33-42.
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 501-511, Baltimore. Maryland. Association for Computational Linguistics.
Ivan Habernal Daniel Faber, Nicola Recchia, Sebastian Breuthuer, Tryna Gurevych, Indra Spiecker genannt Döhmann, and Christoph Burchard. 2023. Mining legal arguments in court decisions. Artificial Intelli- gence and Law, pages 1-38.
Prakash Poudyal. Teresa Gonçalves, and Paulo Quaresma. 2019. Using clustering techniques to iden- tify arguments in legal documents. ASAIL.@ ICAIL, 2385.
Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating reasons for disagreement in natu- ral language inference. Transactions of the Associa- tion for Computational Linguistics, 10:1357-1374.
Prakash Poudyal, Jaromír Šavelka, Aagje leven, Marie Francine Moens, Teresa Goncalves, and Paulo Quaresma. 2020. Echr: Legal corpus for argument mining. In Proceedings of the 7th Workshop on Ar- gument Mining, pages 67-75.
Nan-Jiang Jiang, Chenhao Tan, and Marie-Catherinn de Mamefffe 2023. Understanding and predict- ing human label variation in natural language inference through explanation. arXiv preprint arXiv:2304. 12443.
Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elis- abetta Jezek. 2023. Why don't you do it right? analysing annotators' disagreement in subjective tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- tational Linguistics, pages 2428-2441, Dubrovnik, Croatia. Association for Computational Linguistics.
Daniel Martin Katz, Michael J Bommarito, and Josh Blackman. 2017. A general approach for predicting the behavior of the supreme court of the united states. PloS one, 12(4))00174698.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
T. S. Santosh, Oana Ichim, and Matthias Grab- mair. 2023. Zero shot transfer of legal judge- ment prediction article-aware entailment for the european court of human rights. arXiv preprint arXiv:2302.0000,
Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripabandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. 2021. Ilde for cipe: Indian legal documents corpus for court judg- ment prediction and explanation. In Proceedings of ACL-LICNLP 2021, pages 4046-4062.
T.y.s.s Santosh, Shanshan Xu, Oana Ichim, and Matthias Grabmair. 2022. Deconfounding legal judgment pre- diction for European court of human rights cases to- wards better alignment with experts. In Proceedings of EMNLP 2022.

---

Maarten Sap, Swabha Swayamdipta, Laura Vianna. Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection, In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association, for Computational Linguistics: Human Language Technologies, pages 1884-5906, Seattle, United States. Association for Computational Linguistics.
Conference on Research and Development Infor- mation Retrieval, pages 973-982.
Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of TCAIL. 2027, pages 159- 168.
Jaromir Savelka and Kevin D Ashley. 2018. Segmenting court decisions into functional and issue specific parts. In JURIX, pages 111-120.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In ICML, pages 3319-3328. PMLR.
Chenbao Tan. 2022. On the diversity and limits of human explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2173-2188.
Santosh Tyss, Marcel Perez San Blas, Phillip Kemper, and Matthias Grabmair. 2023. Leveraging task de- pendency and contrastive learning for outcome classification on european court of human rights cases. In Proceedings of the 17th Conference of the European Chapter of the Association for Compu- tational Linguistics. pages 100--103.
Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Sil- viu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385-1470.
Josef Valvoda, Ryan Cotterell, and Simone Teufel. 2023. On the role of negative precedent in legal outcome prediction. TACL. 11:34 48.
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing human-Al collaboration for generating free-text ex- planations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 632-658, Seattle, United States. Association for Computational Linguistics.
Shanshan Xu, Leon Staufer, Santosh T.Y.S.S, Oana Ichim, Corina Heri, and Matthias Grabmair. 2023. Vechr: A dataset for explainable and robust classifi- cation of vulnerability type the european court of human rights. arXiv preprint arXiv:2310.1 1368.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NATC pages 1480-1489.
Linan Yue, Liu, Binbin Jin, Han Wu. Kai Zhang. Yanqing An, Mingyue Cheng, Biao Yin, and Day- ong Wu. 2021. Neurjudge: cireuretance-aware neural framework for legal judgment prediction. In Proceedings of the 44th International ACM SIGIR

---

A Disagreement Taxonomy
B Case Sampling Criterion
We map and unify existing taxonomies into our proposed common categorization in Table to offer an overview of the taxonomy landscape.
For our rationale dataset, we sampled 50 cases from the ECtHR's public database HUDOC between 2019-22, excluding 'inter-state' cases as suggested by the legal experts. We extract the facts section of the judgement document using regular expres- sions. Following Chalkidis et al. 2022, we focus on the 10 convention articles that make up the largest share of the court's jurisprudence. We sample 50 cases based on the following criterion: each article should be represented by a minimum in which is claimed to be violated, and out which is deemed to be violated by the court at least cases, and not found so by the court in at least 2 cases. We use 10 cases as a development set, sampled such that each article is represented by minimum of case to develop the annotation guidelines.
cases
For the qualitative study, we select five cases based on the following criteria: case which experts have the the highest agreement, case with the low- est agreement, case where two experts have the same outcome prediction but lower rationale agrec- ment, case with different outcome prediction but high in rationale agreement, and case with polar- ized agreement scores among different allegation articles.
C Manual Curation of Allegations
To obtain a corrected set of allegation articles, we first parse the conclusions section in the case judge- ment document. After noticing that some alleged articles being mentioned in the texts were missing in the conclusion section, to improve quality, we also parsed section headers under the law section in the judgement. We did manual inspection of all 50 cases sampled for annotation where allegations seemed mismatch, which amounted to about 25% of the sample. While our test set has been manually curated, we leave an examination of this limitation of HUDOC metadata for future work.
D Annotator Background & Expertise
Annotator (the third author) is a Post dost-doctoral Researcher at a European Research Centre. She worked at the European Court of Human Rights in various roles, including Programme Adviser the Council of Europe. Directorate General of Human Rights and Legal Affairs. Annotator 2 (the fourth author) vis- iting professor at a European Law Faculty. She
case lawyer and

---

Change
Case
Law
Longitudinal
Individual Behavior Change
the Context
Error and interface problem
The Annotator
Sloppy annotation
Slopiness
Noise
Interface
Limitation
the Annotator
Subjectivity
Subjectivity
Annotator
Subjectivity
Subjectivity
Behavior
annotation scheme
Underspeciiication in guidelines
Item difficulty:
Annotation Context
information
Incomplete
Allegation
Information
Missing
the Data
Text Production Artefact
Narrative Unclarity
Ambiguity Uncertainty in Sentence Meaning
Ambiguity
the Data
Normative Uncertainty
Genuine
ambiguity
Jiang and Marneffe 2022
Basile 2021
Uma et al. 2021
Sandri al. al. 2023
our work
level I
level 2
Table Landscape of existing taxonomies of disagre eeggreement eeeeeeeentttouuue sources r

---

spent three months at the European Court of Hu- man Rights a trainee. Her research interests focus on the supervisory architecture of the Euro- pean Convention on Human Rights. She served as an expert to the Steering Committee for Human Rights (CDDH) within the Council of Europe.
the mouse cursor as you would in regular text pro- cessing environment. Once you release the mouse button, drop down with the available types will appear for you to select from.
You can annotate the same text with multiple types, but DO NOT annotation the same text with the same type multiple times.
E GLOSS Annotation Tool
Always annotate the maximal contiguous span. For example: If you want annotate three consecutive words with type T, then annotate them as one span. DO NOT separately annotate sub-segments of contiguous spans with the same type. Always start annotations at the beginning of word and end at the end of word.
The task of LJP rationale annotation was done us- ing the GLOSS annotation tool (Savelka and Ash- ley, 2018). Figure 4 shows screenshot of the GLOSS annotation interface.
F Annotation Guidelines
Task Description: Annotation of Facts Indicat- ing Convention Violations
Typical content types to be annotated are:
What looking for is annotate text seg-
Language segments describing specific facts or fact patterns. Examples: purchased the property with joint mortgage.', 'She had been interviewed by an investigator from the military prosecutor's office and taken into cus- tody"
ments reflecting specific facts that increase the like- lihood of the court reaching the conclusion that certain convention article has been violated or not, respectively. Typically this will be because the fact in question relates to requirement of a legal rule that is either part of the convention articles, the law developed in ECtHR's jurisprudence, or other applicable legal source material. The goals of the project include:
Single- or multiword expressions indicating relevant legal concepts. Examples: margin of appreciation, positive obligation etc.
To study what degree ECHR legal experts agree on what elements of the facts are rele- vant for the violations of which articles, and
Mentions of domestic laws regulations (c.g. well-known instruments involved in multiple cases) Examples: 'On 3 October 2001 the ap- plicant lodged application with the Rome Court of Appeal under Law no. 89 of 24 March 2001, known as the "Pinto Act", com- plaining the excessive length the above- described proceedings'
develop machine learning models that, given a case fact description, can identify text segments that contain factual information rel- evant for the determination about whether a given article has been violated. Achieving this will be incremental step towards build- ing advanced systems that can support lawyers in analyzing ECtHR jurisprudence and case docket material.
Specific sequences of events that have a bear- ing on the finding of a violation that could justify discarding the applicant's allegation of violation Examples: "The applicant's allega- tions were rejected based on classified note received from the Moldovan secret service, according to which the applicants presented a threat to national security. The decisions did not give any details to the content of the note, not even the date on which it had been issued' 'However, the court could not see that these measures had had much effect her ability to provide care.'
The text to be annotated the FACTS section an ECUHR judgment along with information about which convention articles have been alleged to have been violated. The task is to annotate the portions of the text that are relevant for a determination of each alleged article based on this goal. To this end, the GLOSS annotation tool provides one annotation type per article.
Technical Instructions
When the ECHR relies the FACT section extensively on domestic courts' decisions or
Please annotate text spans by clicking and dragging

---

Types
10
Armotations
35
Allaged Articlafs): Art.2, Art.14
Article
THE FACT
Article 11
in and lives Unterwagram. childran, bom in 2004, and 12.0n 10 Juy 2010 the applicant called the police because her husband had beaten hei. In her statement to the police she alleged that she had problems with her husband and that he had been beating her for years. In the preceding months the situation had worsened because had gambling addiction, was heavlyy debt had lost job.
appicant
She married E. 2028. They
2005.
Article
Article
had always supported him financially, but also lost job thersfore could langer pay his dabls. The police noted that applicent showed signs of injuries, namely haematomas her slbow and upper arm, which she stated she had sustained through bealinge by her hueband.
Article
Article
13, Pursuant to section 38a the Security Police Act (Sichehhittpolizeieeeet see paragraph 48 below), the police handed the applicant leafot informing her, emong other things, the possiblitty seeking temporery restraining order (sinstwellgge Vertügung) agsinet husband under sections 382b 382e the Enforcement Act (see peragrachs 54 seq. below).
Article
Article
14.When confronted with the all allegations by the police, E. stated that he did not have problems with his wife, but that had had fight with his brother the right before and had sustained Injuries his face. There were no indications that was possession of weapon. A barring and protection Gewalt) accordance with 38a the Security
Article
Article P1-1
(Betretungsverbot Wegwelsung zum
Figure 4: Screenshot of the GLOSS annotation interface
law, we should only annotate facts that are rel- evant for the alleged violation and not general statements regarding the interpretation of that article in domestic law or even in light of the Convention. We are only looking facts that would be taken into account into the ECHR's s proportionality assessment (the "weighing ma- terial" of the national court decision thus annotated even if the legal evaluation of that material differs from the national court)
Distinction: What NOT to annotate
We are not annotating spurious facts, i.e., words, phrases facts that are not relevant to a finding on the merits regarding the presence or absence of a violation. Redundant/Arrellvaat facts should not be annotated.
What do with repetitive phrases?
repetitive phrase occurs when the same fact pat- tern is written about multiple times in the text. This mainly happens for three reasons, each of which lead to different treatment during annotation: To illustrate the regularity of some event, where one such event would suffice for the legal assessment. In such cases, only one example mention needs to be annotated. We are only looking for facts which reveal the domestic authorities' diligence in dealing with the applicant's s allegation (typically the one mentioned first) Example: Cases of pro- longed detention in which domestic courts only repeat the initial reasons for arrest without reasess- ing the applicant's 's situation Because the presence of repetitiveness is legally decisive, in which case all instances to be annotated If fact pattern changes over time or is presented in contradictory perspectives (e-g., differing diagnosis of a mental illness) AND the diversity of information/viewt will be decisive for the court's 's assessment (e.g., in determining discharge of positive obligations by the respondent state), then all instances should be annotated. Example:
Specific dates are to annotated only in cases where those dates disclose events that are rele- vant for the assessment of the substance of the article Examples: In cases where the arrest is supposed to happen only for a limited time (24 hours or 3 days) and the courts lack the the diligence review the application to release the applicant
Focal criterion is the relevance of facts within the particular context of the judgment. Annotation should be conducted by looking through the context find how facts fit together in order to justify the violation/non inlatioonnon-vvolaaioonn. It is the specific composition of facts and laws in the context that that should
makes that violation/non-violation be reflected in the annotation. It is important to remember that ECtHR fact descriptions are not objective. They have already been trimmed by the Registry and do not represent the applicant's initial/original allegations, there has also been an exchange between the Government and the applicant and thus the final version on HUDOC is drafted after those facts have been 'set straight'.
For example, if the number of meetings might lead to reasonable delay violation (Articles 6, 5

---

about the promptness of review) to how many times an investigation was delayed (Articles 2, 3), or how many exams an applicant had to undergo (Articles 8, 3) etc. The underlying rationale is to flesh out only relevant factual patterns and avoid unnecessary and rhetorical language.
Token encoding layer We use BERT encoders to obtain token-level representations 2
[ A12. Zin for each token in every packet 2, = { F22 T22,..... Fin
Token attention layer We obtain the representa- tion for each packet by aggregating their token- level representations using attention follows:
How to annotate special legal vocabulary? Given that we are only dealing with the factual part of judgments which contains information of both the respondent Government and of the applicant, terms like "margin of appreciation" and "positive obligation" etc will be annotated only once (see previous rule) and placed in relation with the facts that are relevant for the alleged violation because what wc interested in is to follow how the Court will later use (in dealing with the merits) these rhetorical devices in relation the facts.
U4 = tanhh 2it + bur)
(1)
exp(uitui)) Olit Σi exp(ur(ww
n
& ʃ: Σ
ait
(2)
where W ww and the are trainable parameters and Ore represents the importance of tth token in the 7th packet. Thus we obtain the representations for the input packets as f {fi, f2....,fmh Packet encoding Given the packet vectors f from token attention layer, we pass them through a bi- directional GRU to obtain context-aware packet representations g [91, 1,1,922.... ..2....:.00 9m..
How to annotate segments relevant for multiple articles?
Packet attention layer Finally, we aggregate packet representations g into case representation e using attention similar to Eq. 2.
In case of conjoint articles (for example 8 and 14, 13 and 8) and in cases where the alleged violations are reflected in the same set of facts (private life and inhumane treatment, property and discrimina- tion) given that the Court might decide for instance that the fact give rise to a violation under the pro- cedural aspect of Article 8 and ignore Article 13, please annotate the same set of facts (with the dif- ferent colours) for the two articles. This will lead to slight color change as the markup colorings overlap.
Classification layer: We concatenate the multi- hot b with the obtained feature representation and pass it through two fully connected layers classify the violation outcome. We again train the model using a binary cross-entropy loss against the multi-hot target.
G.2 Article-aware Classification
After greedy packing, we obtain case facts descrip- tion x {aı, .2...., Fm and article text s {8), 82, ... 86} where I; = {2s1, T22. }in] and 8; = {s
G Model Architectures
Fig. illustrates the detailed architecture of both models.
}- x, (8,), miy (s1,). m (k) and (p) denote 7th packet, jth token in 7th packet, number of packets, number of tokens in 7th packet of the case facts description / article respec- tively. Here and are binary variables indicating whether article has been alleged and violated, respectively.
G.1 Fact-only Classification
We follow the greedy packing strategy and obtain inputs case fact description as {x1, 22.,, } where z; = (an Ti2...., )in). Te, Tiy, m and n denote the th packet, the jth to- ken in the tth packet, number of packets, number of tokens in the 4t packet, respectively. b and a are provided in form of (0, 1)*k, where & denotes the total number of modeled convention articles. 1 and O indicate positive and negative instance under allegation (violation) of that corresponding article in (a)."
Pre-interaction encoding This layer is applied independently to case facts and article text. We use a pre-trained BERT encoder to obtain token-level representations 2; (211, 212,.... zin} for packet Zi. Packet representations f/ are computed via an attention mechanism similar to Eq. 2. Then the packet-repreesentation are passed through a bi-directional GRU to obtain their context-aware
"Wo use the terms 'taken" and 'packet" instead of 'word' and 'sentence" as used (Yang et al., 2016) as we use BERT as
hackbone which operates on wor-word/toenn representations. We also merge paragraphs into packets rather than sentences.

---

binary
Violation
cotton Layer
Cassilication Layer
Pool Infir aodoraadoonn Encoding Layar
Allegatoon birery
Packict
Packet Ameention
Contiration
Packeet Attenton
Allegation Vector
9
9n
Packee Encoding Layer
Combination
Interaction Layer
Dg spen Da
9, Si Sig 9.
Pro Intoraction Encoding Layer
Token Encoding Layer
Packet
Tosan Allintion
Token Attentio
Xy Xgr Pie
X
Xg Bg eusss Aya
Big eoc.o 4 Articla
Case Facts
Case Facts
(a) Fact only Classification Variant for Legal Judgement Prediction
(b) Article-aware prediction Variant for Legal Judgement Prediction
Figure 5: Base Model Architectures
representations as (11,, (1u,,h,..... ... hm Anal- ogously, we obtain context-aware representations g = [91 12. ...... 99k for the article text.
Initially, we concatenate pre-interaction and fact- aware packet representations of the article as fol- lows:
Interaction: This layer captures the interaction between the packets of the case facts description and the article. We compute the article-aware rep- resentation of the case facts and fact-aware repre- sentations of the article based on relevant content from the packets using the dot product attention mechanism as follows:
P: = [gt, 9.. 9: g 9. 9/]
(5)
where @ denotes the element wise product. This representation is intended to capture higher order interactions between the pre- and post-interaction representations through concatenation, difference and element-wise product. The final packet rep- resentations p= ............ pp) are passed over non-linear projection and bi-directional GRU to obtain the context-aware representations P/1.P2..... p/- All packet representations are aggregated by an attention mechanism to obtain the final article representation S similar to Eq. 2. (4)
k
exp(erj)
Ciy = hf
h4 Σ
Oj
Ehe exp(ea)
exp(cij) 14
1f D exp(et))
The final fact representations are computed anal- ogously by combining the pre- and post-interaction representations, and then passing them through a non-linear projection and bi-directional GRU. To condition them on the article text, we initialize the GRU hidden state with the final representation of the given article (Augenstein et al., 2016). This guides the GRU model to extract and refine the representations of case facts not only based on the neighbouring packet contexts but also based on the article text. Then all the packet representations aggregated to obtain the final representation of case facts C using the same attention computation described in Eq. 2.
where Eij represents the dot product interaction score between the context-aware representations of the 7th packet of case facts and the j"h packet of the article. M/ and o'y represent the article-aware representation corresponding to the ,th packet of the case facts and fact-aware representation corre- sponding to the jth packet of the article, respec- tively. Thus we obtain interaction- packet representations of the case facts and the article as N' (M, ,/ and g' 1g..g........... . respectively.
Post-interaction Encoding: This layer uses the interaction-aware representations of both the case- facts and the article to derive the final representa- tion of case facts conditioned on the article text.
Classification Layer: We concatenate the above obtained case facts representation C with the bi-

---

nary label of allegation articles and use two fully connected layers to obtain the violation outcome. We calculate a binary cross-entropy loss over each article in the outcome vector.


Reviews:
[{"review_id": "rt66J5c9wh", "reviewer": "Reviewer_UWAY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n\n- (1a) XAI method: We agree that comparing various XAI methods like attention scores or occlusion-based approaches would strengthen the argument, but the reliability and informativeness of these techniques remain an ongoing research challenge. We chose Integrated Gradients due to its prevalent use, notably demonstrated in previous research by Santosh et al. (2022) on ECHR tasks and suitability for token-level evaluation. Occlusion approaches as in Chalkidis et al. are intuitive at paragraph or sentence levels but do not immediately translate to token-level granularity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written, clear, and easy to follow.\n* The dataset and the taxonomy are useful resources and can be a good first step towards incorporating disagreement when tackling Legal NLP tasks which could potentially lead to improved models.\n* The discussion of the disagreement is engaging and gives valuable insights concerning the difficulties of Case Outcome Classification and Legal NLP in general.", "reasons_to_reject": "* My only concern has to do with the methodology when studying the misalignment between experts and models. In particular, the authors use only Integrated Gradients to extract rationals. On its own, this is not strong evidence that the models are misaligned with the experts. This claim could be better supported by examining more explainability methods including explainability by construction, similarly to Chalkidis et al.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies the disagreement between human experts when extracting rationales from ECtHR cases to support their predictions of the outcome of the case. Given a case and the allegedly violated articles, the experts are asked to predict the outcome of the case (which articles, if any, of the case were violated) and annotate the tokens of the facts that support their predictions.\n\nThe resulting dataset, RAVE, is used to measure the agreement between the experts which is found to be low. To better analyze the disagreements the authors propose a new taxonomy containing potential disagreement reasons. Both the dataset and the taxonomy, along with the guidelines will be made publicly available.\n\nFinally, the authors measure the alignment of SotA models with the rationales of the experts. The authors use two types of models. The first consumes only the facts of the case to predict the allegedly violated articles. The second consumes both the facts and an allegedly violated article and decides if the article was violated. To extract rationales they use Integrated Gradient scores and find that the models do not align with the rationales of the human experts.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "891KFrbDyI", "reviewer": "Reviewer_7qoD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. \n- (2a) Difficulty of Getting Experts: We understand your concerns regarding the number of experts. As you mentioned, our legal rational annotation demands deep understanding of ECtHR jurisprudence and extensive reading of legal material, making it very challenging to engage a larger number of domain experts. Nevertheless, in order to uphold the integrity of our proposed taxonomy, we took measures to enhance its quality. We conducted a pilot study, engaged in several rounds of discussions including the experts, and finally also employed an adjudication process in collaboration with the experts.\n\n- (2b) Disagreement Taxonomy: We completely agree with decoupling domain subjectivity and individual annotator bias and have in fact decoupled them by the separate categories “subjectivity” under Annotator (Line 419, representing bias of the individual annotator) and “Normative Uncertainty” under Data (Line 329 capturing with subjectivity of the field - the legal norms).\n\n- (2c) Inconsistent Allegation Information in HUDOC: We appreciate the reviewer agreeing that the HUDOC metadata inconsistencies is an important finding in itself (as prior approaches in this field extensively relied on this metadata). To clarify: We have remedied the problem for data in our experiments, and hence our results are not compromised. We included detailed information on the corrective measures in App C, but will introduce explanations in the main of the paper. In a nutshell, we manually curated/corrected the metadata entries from the text of the decisions, including basing dataset stratification on corrected metadata.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The dataset introduced seems to be a good contribution to the field of explainable-NLP.\n2. The annotation process is designed in such a manner that experts would be made to be specific about the impact of specific text to COC, which is a significant benefit.\n3. While the author/s do adopt a in-text annotation approach to the task of rationale identification and acknowledge it as a limitation, the approach taken to gauge the impact of certain words and phrases to COC is an interesting one and merits further investigation along the lines proposed.\n4. While I have some questions regarding the taxonomy proposed by the authors to gauge annotator disagreement, it is a good contribution and could be helpful in making the subjective nature of COC more objective.", "reasons_to_reject": "1. The findings of the study seem to indicate more research is needed into understanding why there is disagreement between experts and models in terms of COC. Possibly due to the appreciable difficulty in getting experts to invest significant time and effort into annotating vast legal corpuses, the study involves just two experts annotating the case-allegation pairs, and the disagreement observed between them could be a result of bias inherent in the individual, unrelated to the field's subjectivity. The taxonomy that the author/s propose combines the subjectivity of the field with individual bias which in my opinion we would benefit from decoupling. More experts being involved in the annotation process can also help investigate this claim. [ADDRESSED]\n2. In section 4.2, the incompleteness of allegations identified by the author/s in the HUDOC dataset is a significant finding in and of itself, but the fact that it may have resulted in inconsistent information for 25% of cases in the pilot dataset is slightly worrying. Additional information on the corrective measures applied by the author/s may be required in this section. [ADDRESSED]", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper contributes a novel dataset to the task of Case Outcome Classification. This dataset is built with the involvement of two experts on Human Rights Law. Further, the author/s analyse the disagreement between the experts by introducing a taxonomy augmented with COC-specific terms. The data collected is then tested on reasonable baselines and the author/s observe poor correlation between the model results and the expert annotations.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m5FhZBnyDQ", "reviewer": "Reviewer_CY3m", "summary": "", "questions": "1. Can u please elaborate on the role of the model performance in Table 2. What is the objective of this table? Is it to show that Article based classification is better than fact only classification?\n\n2. What is the role of  section Dataset, Models & Evaluation Metrics, how are using the LexGlue dataset here. Are u comparing your dataset with LexGlue, this is not entirely clear?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the suggestions. We would like to politely clarify that our paper’s contributions go beyond our fine-grained annotation dataset and include the following \n(i) a detailed explainability-to-agreement study and \n(ii) conceptual taxonomy of disagreement sources in the domain of law.\n\n- (3a) Regarding table 2: We will add an additional explanatory table row to make the two insights easier to recognize. The main takeaway from table 2 is although the article-aware model is performing better than the fact-only in overall prediction performance (indicated by f1 scores), this improvement does not bring higher explainability agreement performance as indicated by kappa scores. That means models are predicting better for the wrong reasons and hence they should be further scrutinised. The broader implication for case outcome classification research is that solely relying on prediction performance is not the way forward if the field is to strive for aligning models with experts.\n\n- (3b) Regarding role of LexGLUE: Our agreement annotation dataset extends a subset of cases from the testset of LexGLUE’s ECtHR subtask. We hence evaluate prediction performance and model agreement on those cases by training models on the whole LexGLUE ECtHR task data. In other words, we use the structurally compatible LexGLUE training data to develop models and evaluate them on the annotated and curated test dataset we contribute in this paper.\n\n- (3c) For dataset statistics, please refer to Table 4 in Appendix G where they are provided. We ask understanding that given limited space we use the appendices for general dataset statistics.\n\n- (3d) We thank the reviewer for suggesting a better mnemonic for our task The “A|B” notation is adopted from Santosh et al. 2022. It is highly contextual, though, and we will replace it with a more intuitive name.\n\n- (3e) Model details : \n              - (i) Greedy input packing strategy : we merge multiple paragraphs into one packet until it reaches the maximum of 512 tokens given by BERT models.\n              - (ii) Packet: refers to multiple paragraphs packed into one block of 512 tokens to be sent to bert model\n              - (iii) Contextualization: We follow a hierarchical model where first we obtain each representation of a packet and then those packets are sent through the GRU layer to derive context from neighbouring sentences.\n              - Our model details completely with mathematical equations are contained in Appendix I. Again, we ask for the reader’s understanding of using the appendix for general architectural information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Dataset papers are always useful, in addition this dataset contains fine grained annotation of facts that leads to a specific case outcome.", "reasons_to_reject": "1. The paper presentation can significantly be improved. It is very hard to understand, even though the paper potentially contains useful facts.", "questions_to_authors": "1. Can u please elaborate on the role of the model performance in Table 2. What is the objective of this table? Is it to show that Article based classification is better than fact only classification?\n\n2. What is the role of  section Dataset, Models & Evaluation Metrics, how are using the LexGlue dataset here. Are u comparing your dataset with LexGlue, this is not entirely clear?", "paper_topic_and_main_contributions": "The paper creates a new dataset that annotates the facts that lead to a specific legal case outcome .", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Please present the dataset characteristics in a table to include the following stats, how many facts per case are annotated, what is the case class outcome distribution.\n\n2. Line 184 - 187: Instead of notation Task A|B, use a meaningful mnemonic based notation, e.g., Outcome | Allegation\n\n3. Line 596 -  what is \"greedy input packing strategy\"?\n\n4. Line 598 - What is the context of \"one packet\" , what does packet refer to?\n\n5. Line 604  - can you please clarify what this means \"These packet representations are contextualized using a GRU layer and further aggregated into final case facts representation using packet-level attention.\""}}]

Author rebuttal:

Meta-review: The paper examines label variation/human disagreement in case outcome classification.

Reviewers generally agree that the paper presents a valuable contribution, especially as it provides important insight into human subjectivity and limitations of current models. They appreciate the thoughtfulness of the fine-grained dataset and the taxonomy of disagreement sources. One reviewer also notes the methodology as one that opens up opportunities for additional investigation.
Decision: Accept-Main