Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement
Authors: Rosamond Elizabeth Thalken
Edward Stiglitz
David Mimno
Matthew Wilkens
Venue: EMNLP 2023 Main (2023)
Keywords: language modeling
annotation
legal reasoning
United States Supreme Court

Abstract: Generative language models (LMs) are increasingly used for document class-prediction tasks and promise enormous improvements in cost and efficiency. Existing research often examines simple classification tasks, but the capability of LMs to classify on complex or specialized tasks is less well understood. We consider a highly complex task that is challenging even for humans: the classification of legal reasoning according to jurisprudential philosophy. Using a novel dataset of historical United States Supreme Court opinions annotated by a team of domain experts, we systematically test the performance of a variety of LMs. We find that generative models perform poorly when given instructions (i.e. prompts) equal to the instructions presented to human annotators through our codebook. Our strongest results derive from fine-tuning models on the annotated dataset; the best performing model is an in-domain model, LEGAL-BERT. We apply predictions from this fine-tuned model to study historical trends in jurisprudence, an exercise that both aligns with prominent qualitative historical accounts and points to areas of possible refinement in those accounts. Our findings generally sound a note of caution in the use of generative LMs on complex tasks without fine-tuning and point to the continued relevance of human annotation-intensive classification methods.

Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement
Rosamond Thalken
Edward H. Stiglitz 2 David Mimno
Matthew Wilkens
Cornell University 2 Cornell Law School
{ret85, js2758, mimno, wi wilkens)@cornell.edd edu
Abstract
writer reflects a particular school of thought. These questions may be difficult even describe to a trained human annotator, much less apply. It is un- clear if generative LMs perform well on this type of complex and specialized task.
Generative language models (LMs) are increas- ingly used for document class-prediction tasks and promise enormous improvements in cost and efficiency. Existing research often exam- ines simple classification tasks, but the capa- bility of LMs classify on complex or spe- cialized tasks is less well understood. We con- sider a highly complex task that is challeng- ing even for humans: the classification of le- gal reasoning according to jurisprudential phi- losophy. Using novel dataset historical United States Supreme Court opinions anno- tated by team of domain experts, we system- atically test the performance of variety of LMs. We find that generative models perform poorly when given instructions (i.e. prompts) equal the instructions presented human notators through our codebook. Our strongest results derive from fine-tuning models on the annotated dataset; the best performing model is an in-domain model, LEGAL-BERT. We ap- ply predictions from this fine-tune tne-tuned model to study historical trends in jurisprudence, an exer- cise that both aligns with prominent qualitative historical accounts and points areas of possi- ble refinement in those accounts. Our findings generally sound a note caution the use of generative LMs complex tasks without fine- tuning and point to the continued relevance of human annotation-intensive classification meth- ods.
In this study we systematically examine the abil- ity of large LMs to parse a construct that is difficult even for highly trained annotators: modes of legal reasoning. We consider two prominent modes of legal reasoning that judges employ as identified by legal historians, in addition a null non- interpretative class. Although the classes of legal reasoning identified by historians reflect relatively well-defined concepts, determining whether par- ticular document reflects a mode of reasoning can be exceptionally challenging. We suspect this is common to many high-value but specialized tasks, such as classifying complex emotional states or de- tecting indirect racial or gender bias. These tasks often require both abstract reasoning and special- ized knowledge.
Legal reasoning is suitable setting for examin- ing model performance highly complex classi- fication task. The foundation of our research is a new dataset of thousands of paragraphs of histor- ical Supreme Court opinions annotated by a team of upper-year students at a highly selective law school. We find that even the largest models per- form poorly at the task without fine-tuning, even when using similar instructions as those given to human annotators. This finding suggests that LMs, even as augmented through few-shot or chain-of- thought prompting, may not be well-suited com- plex or specialized classification lasks without task- specific fine-tuning For such tasks, substantial annotation by domain experts remains a critical component.
1 Introduction
Academia and industry increasingly use generative language models (LMs) for document annotation and class-prediction tasks, which promise enor- mous improvements in cost and efficiency. How- ever, rescarch tends to focus relatively simple and generic annotation contexts, such topic or query-keywon relevance (Gilardi et al., 2023; He al., 2023; Törnberg, 2023). But many potential applications call for annotation or prediction of complex or specialized concepts, such as whether a
To demonstrate this point, we examine the per- formance of established to cutting-edge L.Ss when fine-tuned on our annotated data. Our results show strong performance for many of these fine-tuned

---

models. Our analysis explores various approaches model structure, such as a multi-class task ver- sus serialized binary tasks, but we find that using an in-domain pre-trained model, LEGAL-BERT (Chalkidis et al., 2020), results in the highest perfor- mance for a task that requires specialized domain knowledge.
high-quality human-annotated data, recent work has suggested that annotation tasks previously per- formed by students, domain experts, or crowd- sourced workers could be replicated with equal performance by LMs. Generative models perform well on query guery-keyword relevance tasks (He et al., 2023); on topic detection in tweets (Gilardi et al., 2023); on detecting political affiliation in tweets (Törnberg, 2023). Burnham (2023) suggests that zero-shot and few-shot models are a legitimate al- ternative for stance detection because of the unreli- ability of human annotators due to the vast contex- tual information annotators may or may not draw from. In the legal domain, scholars examine clas- sification performance of generative LMs on the type of case (e.g., contracts, immigration, etc) (Liv- ermore and Rockmore, 2023), the court's use of a specific canon in statutory construction (Choi, 2023).
The primary contributions of this paper as follows:
1. We develop a new dataset domain-exper annotations in a complex area.
2. We find that SOTA in-context generative mod- els perform poorly on this task.
3. We show that various fine-tuned models have relatively strong performance.
4. We study the relationship between our best- performing model's predictions and the con- sensus historical periodization of judicial rea- soning, finding both substantial convergence and opportunities for refinement in the histori- cal accounts.
The range of applications for which generative LMs might adequately perform is an open ques- tion. We have found limited work that requires specialized knowledge in addition the use of ab- stract reasoning skills. In this study, we ask the models to engage in precisely this form of reason- ing, which is challenging even for domain expert humans. What distinguishes this form of reason- ing is that it requires the analyst to conceptualize abstract principles and determine whether a spe- cialized, domain-specific example fits one of those concepts. This difficulty contrasts with simpler tasks, which may key off well-established associ- ations in training data between concepts, such political affiliation and word usage.
In sum, our paper shows that a complex and specialized domain, without fine-tuning, current generative models exhibit serious limitations; there is a continued need for domain-expert annotation, which can be effectively leveraged to unseen in- stances through fine-tuned models.!
2 Related Work
Researchers have developed strategies to guide LMs to perform complex tasks without the time and infrastructure costs of fine-tuning, often by breaking decisions down into multiple steps of rea- soming. Wei et al. (2022) use few-shot chain-of- thought (CoT) prompting to provide a model with examples of intermediary logic before making de- cision. An alternative, zero-shot CoT also results in improved performance in certain tasks, as LMs are prompted break down their reasoning (e.g. "let's think step by step"") (Kojima al., 2023). Another procedure, Hlan-and-Solvy (PS) prompting, asks a model to devise and execute a plan for reasoning through a problem (Wang et al., 2023).
Legal Reasoning
Our focus is on legal reasoning involving statu- tory interpretation. 2 In the United States, Congress writes statutes, but determining how statutes ap- ply in individual cases is often left to the courts. Every year, the Supreme Court decides numerous cases of statutory interpretation, ranging from ques- tions such as whether a tomato is a "vegetable" or a "fruit" for the purposes of import tariffs as in Nix v. Hedden, 3 to whether the Clean Air Act authorizes the Environmental Protection Agency to regulate greenhouse gases as in West Virginia v. EPA.
At certain tasks and with these prompting strate- classification
gies, LMs perform annotation tasks the level of humans. Given the high costs (e.g., time, money, logistics) of collecting
'Elsewhere, some of examine jurisprudence more broadly (Stiglitz and Thalken, 2023).
'Code is available at: https://github... com/ rosthalken/legal- haailemm/eegga--nnnerreettttoo
$149 U.S. 304 (1893). *142 S. Ct. 2587 (2022).

---

Jurists adopt wide range of approaches to in- terpreting statules and engaging in legal reasoning more generally. A classic distinction is berween what 20th century legal scholar Karl Llewellyn referred to as "formal" and "grand" styles of rea- soning (Llewellyn, 1960). Grand reasoning refers to a form of legal reasoning that respects precedent but is characterized by "the on-going production and improvement of rules which make sense on their face" (Llewellyn, 1960, p. 38). On interpre- tive questions, it therefore privileges work-abiling future orientation, and common-sense understand- ability. By contrast, formalism focuses not on the "policy" considerations of a law's consequences, but instead on its more mechanical application: "the rules of law are to decide the cases; policy is for the legislature, not for the courts... Opinions run in deductive form with an air or expression of single-line inevitability" (Llewellyn, 1960, p. 39).
Class
Definition
Formal
A legal decision made according to rule, often viewing the law closed and mechanical system. It screens the decision-maker off from the political, social, and economic choices involved in the decision.
Grand
A legal decision that views the law as an open-ended and on-going enter- prise for the production and improve- ment of decisions that make sense on their face and in light of political, so- cial, and economic factors.
None
A passage or mode of reasoning that does not reflect either the Grand or Formal approaches. Note that this cod- ing would include areas of substantive law outside of statutory interpretation, including procedural matters.
Llewellyn's modes of legal reasoning apply more broadly than statutory interpretation. With respect to statutory interpretation specifically, under the grand style of reasoning "case-law statutes were construed 'freely' to implement their purpose, the court commonly accepting the legislature's 's choice of policy and setting to work to implement it" (Llewellyn, 1950, p. 400); showing his sympathies, under the formal style, Llewellyn wrote, "statutes tended to be limited or even eviscerated by wooden and literal reading, in a sort of long-drawn battle between a balky, stiff-necked, wrong-headed court and a legislature which had only words with which to drive that court" (Llewellyn, 1950, 400).
Table 1: Codebook definition for each class of legal reasoning.
of thought (Horwitz, 1992, 18).
Our contribution focuses on this broad consen- around key distinction in the modes of legal reasoning. On the one hand, mode of reasoning that innovative, open- open-ended, and oriented so- cial, political, and economic consequences of law; on the other hand, a mechanical, logic-oriented ap- proach that conceives of the law as a closed and deductive system of reasoning. Though scholars differ on terminology, we follow Llewellyn and re- fer to these schools as Grand and Formal (Table 1).
Though their terminology does not always fol- low Llewellyn, other legal scholars identify simi- lar primary distinction in legal reasoning. Horwitz, for instance, centers discussion legal "ortho- doxy," which seeks separate law from conse- quences and elevate "logical inexorability" (Hor- witz, 1992). Against orthodoxy, Horwitz identified a progressive critique, which "represented a broad attack on claims of Classical Legal Thought to be natural, neutral, and apolitical" (Horwitz, 1992, 189). Other prominent accounts follow Llewellyn's distinctions more explicitly (Gilmore, 2014). Oper- ative doctrines in important areas of law, moreover, reflect the broad schools of thought: e.g., the "rule of reason" in anti-trust, which involves holistically examining the pros and cons of conduct rather than a rule-like test under the Sherman Act, may be un- derstood to reflect the socially-awaz aware grand school
Not only does this basic conceptual consensus exist, but there is also rough consensus on peri- odization: that is, the periods of history in which each school was dominant. The "conventional" (Kennedy, 2006) view is that the pre-Civil War period, the grand style dominated; in the period between the Civil War and World War I, the for- mal style dominated; the Grand school then domi- nated for much of the twenticth century (Llewellyn, 1960). The standard view is that currently live in a period of formalism (Eskridge, 1990; Grey, 1999).
We use this periodization to validate our mea- sure; but also use the measure to provide a nuanced account of historical trends in legal reasoning.

---

4 Data
ity of the output space does not imply that the task is easy. In fact, it took weeks for highly trained upper-year law students to reach level of expertise at which they were able to reach consistent results.
We use a dataset of 15,860 historical United States Supreme Court opinions likely involving statutory interpretation and issued between 1870 and 2014, The raw data come from Harvard's Caselaw Ac- cess Project. 6 Opinion text underwent minimal pre-processing, but all case citations were removed to reduce cognitive workload for the annotators."
Flow chart introduced
0.8
To create the dataset for annotation, we included only opinions that conduct statutory interpretation and then upsampled paragraphs likely to use for- mal or grand reasoning. The seed terms and details about this sampling procedure are included in Ap- pendix A. In the final collection, 25% of paragraphs include at least one formal seed, 25% include at least one grand seed, and the remaining 50% are randomly sampled.
Kopperddettts attre
06
044
0.2
Figure I: Weekly progression of Krippendorff's alpha for paragraphs and sentences assigned the formal, grand, and none labels. Decision chart added before Week
5 Human Annotations for Legal Reasoning
A team of domain experts, four upper-year law stu- dents at highly selective law school, annotated selections from court opinions as formal, grand, or lacking statutory interpretation. This team col- laboratively developed and tested codebook (in- cluded in Appendix D) by iteratively annotating court opinions and calculating inter-rater reliability on a weekly basis over the spring 2023 semester.
Inter-rater reliability increased after the introduc- tion of a decision chart (Figure 1), which broke down decisions about each of the classes into series of guided questions (Appendix B). For each paragraph assigned the low confidence label, the team deliberated over possible labels until reaching a group decision through a majority vote. Training and evaluation data includes this resulting label for low confidence paragraphs; not the initial label.
The annotation task asked each annotator to assign one of three labels, "formal," " "grand," or "none," to each paragraph. A fourth label, "low confidence," could be added in addition to one of the three core labels if the type of reasoning was ambiguous. We calculated inter-rater reliability using Krippendorff's alpha evaluate agreement between the four labelers and across the three main classes. This coefficient was calculated weekly and guided the decision of when to start collecting data for training. Paragraphs with high disagreement were discussed in depth and these discussions led to the revision of our codebook.
In total, excluding paragraphs prior decent inter-rater reliability, 2748 paragraphs were labeled and included in the training and evaluation data, Even with the upsampling of legal interpretation based on seed terms, paragraphs that did not en- gage in legal interpretation or interpreted some- thing other than statute, our "none" class, made up 68% of the data (Table 2). Grand reasoning was the second most common label, and formal the least common. Only 101 of these paragraphs ceived the additional low confidence label, and the formal class was the most common class receive the low confidence label.
We note that while this annotation is formally a three-way classification task, the low dimensional-
Automated Annotation with LMs
"This set of cases may include decisions on the merits and orders. See Appendix A for details procedure.
opinion selection
Though each member of the annotation team an upper-year law student who had completed highly relevant coursework, the task remained dif- ficult for the human annotators, as reflected in the mid-range inter-rater reliability (0.63 Krippen- dorff's alpha). The abstract concepts of the modes
'The raw data can be accessed here: https::/ //caee. Law/. The Casclaw Access Project is not open-access but it grants unrestricted access to researchers.
"To do this, we used the eyecite Python library to identify the occurrence of case citations and replace them with the token /[CITE/J' (Project, 2023).

---

Class
# Ident'd d
# LC
LC %
and Llama-2-Chat (7B) (Touvron al., 2023). We created five random splits of the annotated data with 75% of the data the training set and 25% of the data in the test segment. Models that were fine-tuned were fine-tuned over three epochs, with 50 warm-up up steps, learning rate of 2c-5, with a weight decay of 0.01.
Formal
329
37
11.2%
Grand
551
33
6.0%
None
1869
31
1.7%
Total 2748
101
Table 2: Number of paragraphs fitting each class identi- fied by annotators, and number assigned the low confi- dence (I.C) class by its initial annotator.
6.2 Identifying Legal Reasoning
As a slightly simplified initial task, we begin by considering whether a model can detect instances in which some form of legal reasoning occurs (regard- less of formal or grand reasoning). This remains challenging task but is comparatively less complex than identifying the mode of reasoning. We con- sider any paragraph annotated as either formal or grand as being a paragraph where legal reasoning is present; this is binary classification problem. We compare two procedures for identifying legal reasoning in text:
of legal reasoning were clear, but determining whether specific instances reflected one mode other required specialized knowledge and an ability map those abstract concepts to the incomplete evidence in the paragraphs.
The complexity of this task makes it challeng- ing for generative model prompted in-context or with CoT reasoning. As initial experiment, begin with slightly simplified task: identifying whether a passage involves some form of legal rea- soning (regardless of class). We then compare a larger variety of models on the primary task of in- terest: identifying instances of formal and grand legal reasoning.
In-context generative identification based on a description of legal reasoning (prompt in- cluded in Appendix C).
Fine-tuned binary classification based on hand-labeled annotations.
For both tasks, we compare the performance of in-context and fine-tuned models, with the ex- pectation that identifying legal reasoning more achievable for in-context models than identifying the specific formal or grand classes. Here, we test thresholds of task complexity, to better identify the point at which annotated dataset for fine-tuning is needed; not just carefully crafted prompt.
All fine-tuned models perform relatively well on distinguishing paragraphs with legal reasoning from paragraphs without legal reasoning (Table 3), In comparison to these models, the zero-shot mod- els prompted with a description of legal reasoning perform worse, and either over- or under- identify legal reasoning (e.g. high recall for the reasoning class but low precision). However, these models perform surprisingly well given the comparative workload behind each method: our fine-tuned mod- els are built upon weeks of extensive labeling and discussion; the in-context models, only a prompt.
6.1 Model Training and Evaluation
In both tasks, we compare the performance of set of fine-tuned models to a set of prompted mod- els. Models were chosen based established us- age, popularity, and accessibility (i.e. model size), since applied NLP researchers may be less likely to have access to the computing power needed for extremely large models. The fine-tuned models in- clude BERT-base (Devlin et al., 2019), DistilBERT (Sanh et al., 2020), and m5llmall and T5-base (Raf- fel et al., 2020). We include one in-domain model, LEGAL-BERT-bas.. that scratch on United States and European Union legal corpora, including United States Supreme Court cases (Chalkidis al., 2020). Models prompted to identify legal reasoning include GPT-4 (Ope- nAI, 2023), FLAN-T5-large (Chung et al., 2022),
6.3 Identifying Types of Legal Reasoning
The primary task requires additional specialized knowledge in the identification of specific classes of reasoning, formal and grand. This task also re- quires the identification of imbalanced classes, as formal reasoning was only identified in 11% all annotated paragraphs. We test various assemblies of models and compare fine-tuning with prompt- ing for identifying legal reasoning in text. Our approaches prompting include the following:
pre-trained from
In Context, Descriptions: An in-context prompt that provides the model with descrip- tions of the legal reasoning classes before

---

Macro
Interpretation
None
Model
F1
P
R
F1
P
R
F1
P
R
Fine-Tuned
BERT
0.80
0.80
0.80
0.73
0.72
0.75
0.87
0.88
0.86
LEGAL-BERT
0.82
0.81
0.83
0.76
0.73
0.81
0.88
0.90
0.86
DistilBERT
0.80
0.80
0.80
0.73
0.72
0.74
0.87
0.88
0.86
TS-base
0.71
0.70
0.72
0.76
0.72
0.80
0.87
0.90
0.85
T5-small
0.75
0.75
0.75
0.67
0.66
0.67
0.84
0.84
0.83
In-Context
GPT-4
0.46
0.52
0.52
0.58
0.43
0.89
0.57
0.89
0.42
FLAN-T5
0.42
0.54
0.50
0.04
0.41
0.02
0.80
0.68
0.98
Llama-22 M-aaa
0.16
0.34
0.56
0.44
0.33
0.67
0.03
0.68
0.02
Table 3: Model performance for binary interpretation averaged over 5 train test splits. Macro averages represent averages unweighted by class.
asking for inference on new paragraphs (Fig- ure 2). The descriptions used this prompt are the same presented to the annotation team in the codebook.
cation based on hand-labeled annotations.
Nested: An assembly of models that breaks the classification task into nested binary stages. One model fine-tuned to identify in- terpretation and another model to distinguish between grand and formal classes. The results from the first model used by the second.
In Context, Examples: An in-context prompt that provides the model with examples of the legal reasoning classes before asking for infer- ence new paragraphs (Appendix C). The examples used in this prompt are the same presented to the annotation team the code- book.
Model Iuputt
Some pargraahh in court cases interpret statutes. Within Interpretation, thene are two GRAND and FORMAL.
Chain-of-Thougle A CoT prompt that pro- vides steps of reasoning to follow prior to determining the class of legal reasoning (Ap- pendix C). The steps used in this prompt de- rive from the decision chart provided to anno- tators.
PORMAL theory is logal deciston made arcording rult, aften viewing law disesion-man malurr off from the politizal, sutidl, and ecomomic choisass involved decision.
system.
GRAND theory a legal decision that views law open-enced and prodaction improvement decisions
going enterpric that make sense on their face and in light political social, and
Each prompting strategy is derived from our codebook (see Appendix D), which guided hu- man annotators through data annotation. We do not exhaustively explore prompts beyond our code- book. Instead, consider whether reasonable prompt that is successful for humans works well for model. While it is possible that another, as- yet-unknown, prompt could have provided better results, we know that the language in our codebook is sufficient to describe the task and the desired results.
NONE
passage
reasontnng
etther the Grand oc Formal appreaches. Note that this cocing would include of cuberantive low outtide statstory interpreaaionn, including geu peocoduraa matters.
legal interpretation Ruturn a single choise from GRAND, FORMAL, or NOKE.
passage
ITSXY FOR IMPERENCER
Model Output
FORMAL.
Figure 2: In-context prompt for identifying whether and which type of interpretation occurs.
We contrast the results of the prompted genera- tive models with the results from fine-tuned models. These models were fine-tuned with variety of ap- proaches, including:
7 Model Performance
We test the performance of all models on the same five test splits data and find that the fine-tuned models consistently outperform the in-context mod-
Multi- Clasi- fine-tuned multi-class classifi-

---

Macro
Grand
Formal
None
Model
F1
P
R
F1
P
R
F1
P
R
F1
P
R
Nested
BERT-base
0.67
0.67
0.68
0.65
0.62
0.68
0.49
0.50
0.48
0.87
0.88
0.86
LEGAL-BERY
0.69
0.68
0.71
0.67
0.64
0.71
0.54
0.51
0.57
0.88
0.90
0.85
DistiBBERT
0.67
0.67
0.68
0.65
0.63
0.68
0.50
0.50
0.50
0.87
0.88
0.86
TS-base
0.67
0.67
0.68
0.66
0.59
0.75
0.47
0.52
0.43
0.87
0.90
0.85
T5-small
0.54
0.56
0.55
0.57
0.50
0.66
0.21
0.34
0.15
0.84
0.84
0.83
Multi-Class
BERT-base
0.68
0.68
0.68
0.67
0.65
0.69
0.50
0.52
0.48
0.88
0.88
0.87
LEGAL-BERT
0.70
0.70
0.71
0.68
0.65
0.72
0.55
0.55
0.55
0.88
0.89
0.87
DistilBERT
0.67
0.67
0.66
0.66
0.64
0.68
0.47
0.51
0.44
0.87
0.86
0.87
T5-base
0.64
0.64
0.65
0.66
0.63
0.69
0.51
0.52
0.50
0.87
0.88
0.86
T5-small
0.48
0.62
0.49
0.57
0.66
0.51
0.02
0.43
0.01
0.84
0.76
0.94
In- Context, Descriptions
GPT4
0.22
0,24
0.27
0.44
0.38
0.51
0.36
0.23
0.81
0.58
0.92
0.42
FLAN-T5
0.19
0.36
0.36
0.36
0.24
0.77
0.16
0.11
0.28
0.04
0.72
0.02
Llama-2-Chat
0.20
0.34
0.36
0.35
0.22
0.92
0.00
0.00
0.00
0.24
0.82
0.14
In- In-Contexi, Examples
GPT-4
0.45
0.47
0.54
0.45
0.38
0.57
0.36
0.25
0.65
0.62
0.86
0.48
FLAN-T5
0.08
0.28
0.23
0.15
0.28
0.10
0.20
0.12
0.90
0.01
0.82
0.01
Llama-2 Mama--2-Chat
0.10
0.31
0.50
0.35
0.21
0.96
0.04
0.23
0.02
0.01
0.8
0.00
Chain-of Thoughh
GPT-4
0.34
0.37
0.37
0.25
0.50
0.17
0.43
0.32
0.67
0.78
0.80
0.76
FLAN-T5
0.08
0.33
0.34
0.00
0.00
0.00
0.21
0.12
1.00
0.03
0.86
0.02
Llama-2-Chat
0.08
0.55
0.44
0.32
0.20
0.74
0.00
1.00
0.00
0.00
1.00
0.00
Table 4: Model performance averaged over five train test splits. Macro averages represent averages unweighted by class.
els (Table 4). Our results suggest that even state-of- the-art LMs may not be a suitable replacement for human annotation on highly complex and special- ized classification tasks.
recall for class artificially high.
Also notable, the performance of the generative models on this more complex task low compared to the simpler task of identifying whether some type of relevant legal reasoning occurs (Table 3). This is true both in absolute terms and relative to the in-domain, fine-tuned models. For instance, the macro F1 for GPT-4 on the simpler task is 0.46, 0.36 lower than the corresponding Fl for the in- domain, fine-tuned model. On this more complex task, the macro Fl for GPT-4 with descriptions 0.22, 0.48 lower than the F1 for the in-domain, fine-tuned model.
Of all training or prompting procedures, models fine-tuned to perform multi-class classification tend to result in the highest performance. Out of all mod- cls, the best performing model is LEGAL-BERT, the one in-domain model included in this analysis. GPT-4 performs worse than all fine-tuned models, but has much better performance than Llama-2- Chat or FLAN-TS. Llama-2-Chaa and FLAN-TS greatly over-predict one of the three classes, and rarely predict the other two classes, making the
*Our reported results employ a user-role and the default temperature on GPT-4. We experimented with zero-ed out temperature setting and with adding system prompt, but the results did not improve substantially. Additionally, for the Llama-2-Chal models we used the same prompts as the other models hut added the Llama-2-CHn 2aae--2-hhhaeese Chat-specific formatting that is necessary for instructing this model.
We inspect the generated text from these models and find that FLAN-T5 15 and GPT-4 often over-predict certain classes, but these models rarely hallucinate or retum text beyond the requested class (e.g. "grand"). Unsurprisingly, Llama-2- 2hhhat often returns additional text beyond the class label: we extract the class lahel (if it occurs) from the text and use that as the label for evaluation.

---

8 Application: Periods of Legal Reasoning
that involve interpretation, and code paragraphs classified as "formal" with 1 and paragraphs clas- sified as "grand" with O. These yearly averages, therefore, reveal the proportion of interpretive para- graphs that classify as formal as opposed to grand. Figure 3 plots the yearly averages over our series: the left panel (panel a.) shows the yearly average at the paragraph level, and the right panel (panel b.) aggregates paragraphs within documents to show the yearly average at the opinion level.
The conventional wisdom among legal observers is that we currently live in a period in which the formal style of reasoning predominates (Eskridge, 1990; Grey, 1999). Yet has not always been this way: in other historical periods, the grand style of reasoning prevailed. Indeed, there is rough consensus in the legal literature regarding historical periodization (Kennedy, 2006).
Writing in the mid-twentieth century, Llewellyn identified three periods of legal reasoning. Prior to the Civil War, the grand style of reasoning pre- dominated; from the Civil War to World War I, the formal style of reasoning prevailed; and from World War I onward, courts again operated under the grand style of reasoning. More recently, schol- ars identify the 1980s critical point of transi- tion towards formalism (Eskridge, 1990). Other scholars identify fundamentally similar periodiza- tions (Horwitz, 1992; Gilmore, 2014), and though differences exist, it is possible to speak of a "con- ventional" view (Kennedy, 2006). These histori- cal characterizations arise from leading scholars reading judicial opinions and forming judgments through the use of their full faculties about the pre- vailing style of reasoning.
Mann Panugrun flod loodd harmury Scare
Mean Openam- Cormully Score
1880
1500
1500
1040
1000
1080
2009
1000
1000
1900
1040
1960
1000
2000
Our data starts at Reconstruction (the period fol- lowing the US Civil War) and allows us to examine the convergence between the scholarly consensus historical periodization and the historical periodiza- tion implied by our LM-derived results. We can also our predictions to offer more granular sessments of the periods and potentially adju- dicate differences among the views of prominent scholars. This latter analysis is preliminary, in part, because carlier scholars examined judicial reason- ing broadly, whereas our current analysis considers only Supreme Court opinions involving statutory interpretation."
Figure 3: Historical trends in interpretive style at the paragraph (panel a.) and document (panel b.) levels, by year. Coded so that I=formal and O=grand.
Broadly understood, the historical trends in our predictions converge with the views of Llewellyn and other legal observers. That is, the period after the Civil War and before World War I was character- ized by formal judicial reasoning; the mid-century was characterized by grand legal reasoning; and we live in period of formalist resurgence. The story is essentially the same at the paragraph the document level.
But our predictions also allow for a more gran- ular assessment of the historical periods. To illus- trate this, use dashed vertical lines in Figure 3 to denote important historical events: in 1905, the Supreme Court decided Lochner New York," which some observers note as highwater point for formalism; in 1937, a year of "judicial revolution," in which the Supreme Court is widely viewed to have shifted its jurisprudence from opposition to ac-
For this exercise, we study historical trends in the predictions from the highest performing model, multi-class, fine-tuned LEGAL-BERT. We exam- ine yearly averages at both the paragraph level and the opinion level." 11 We focus only on paragraphs
We screen opinions these predictions using statu- tory interpretation filter identified in Appendix A. Stiglitz Thalken (2023) provide more comprehensive LM analysis historical trends in jurisprudence.
"nn opinion-level prediction represents the average of para- graphs in that opinion. If the number of paragraphs in opinions is not time invariant, historical trends in opinions may not he
the same as trends in paragraphs. 12988 U.S. 45 (1905).

---

ceptance of federal and state regulations:'3 and in 1981, President Reagan's judicial appointments started to take office and a possible marker for the formalist revival."*
10 Limitations
A limitation of this study is the relatively low inter- rater reliability between annotators even after exten- sive training and conversation. This relatively low reliability results from the difficulty of the task and the inevitable ambiguity of some passages, espe- cially when read out of case context. Another lim- itation relates to our prompting strategy: to make the in-context prompting more comparable to work- ing with the team of annotators, book descriptions and examples in the in-context prompts. Likely, these descriptions and examples could have been optimized for better model perfor- mance through additional prompt strategies, and our results for these models may depict lower per- formance than is possible.
To a striking degree, these historical markers correspond with changes in our metric of jurispru- dence. Consistent with those who see Lochner as a high-water point for formalism, the prevalence of formalist reasoning declines after 1905. Likewise, we see remarkable increase in the prevalence of grand reasoning in 1937. This pattern is consis- tent with a "judicial revolution" in jurisprudence accommodate regulatory programs, the type of which had been earlier invalidated under formalist regimes. Finally, our mcasures recover a sharp in- crease in formalism in the 1980s, again consistent with the views of legal observers.
the code-
These results represent some of the first long-run quantitative characterization of trends in jurispru- dential philosophies. They both broadly support the qualitative characterizations of legal scholars and provide opportunities for refinement of legal theory and historical accounts.
Acknowledgments
We thank our annotation team, including Houston Brown, Michael Demers, Sarah Engell, and Josiah Rutledge, for their extensive work creating this dataset. We also thank Zach Clopton, Vijay Karuna- murthy, Gregory Yauney, Andrea Wang, Feder- ica Bologna, Anna Choi, Rebecca Hicke, Kiara Liu, and participants at workshops at Cornell Law School and Cornell Computer Science Department for helpful comments. This work was supported by NSF DRMITF-2001311 and NSF #1652536.
9 Conclusion
We found that for task involving abstract rea- soning in addition specialized domain-specific knowledge, it remains essential have an anno- tated dataset created by domain experts. Although other work has shown that generative models are able to replicate annotation for complex tasks us- ing carefully crafted prompts, we demonstrate that models fine-tuned a sizable dataset of expert annotations perform better than models instructed perform the task through in-context and CoT prompts. We recommend that researchers use cau- tion when employing non-fine-tuned generative models to replicate complex tasks otherwise com- pleted by humans or with human supervision. Best practices would call for human validation of gen- erative model results and an assessment of cost- performance tradeoffs with respect to in-domain models.
References
Aaron-Andrew Bruhl. 2018. Statutory interpretation and the rest of the iceberg: Divergences between the lower federal courts and the supreme court. Duke Law Journal, 68( 17)33
Michael Burnham. 2023. Stance detection with super- vised, zero-shot, and few-shot applications.
John Calhoun. 2014. Measuring the fortress: Explain- ing trends in supreme court and circuit court dictio- nary use. Yale Law Journal, 124(2)444-5226
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school.
This revolution is also known as "switch in time that saved nine." referring the changed voting behavior Justice Owen Roberts response the running threat by President Roosevelt to pack the Court.
Jonathan H. Choi. 2020. An empirical study of statutory interpretation in tax law. New York University Law Review, 95(2):363
"Justice Scalia, for instance, was appointed by President Reagan to the Supreme Court in 1986, and is often viewed as the single most influential person in the rise of new formalism. For an account of that rise, see Eskridge (1990).
Jonathan H. Choi. 2023. How to use large language models for empirical legal research.

---

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros Ross, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le. and Jason Wei. 2022. Scaling instriction-finetunedd language models.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21( (1(14))11667
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
Nancy Staudt, Lee Epstein, Peter Wiedenbeck. Rene Lindstiidt, and Ryan J. Vander Wielen. 2005. Judging statutes: Interpretive regimes. Loyola Law Review, 38(5):109-11770
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Edward H. Stiglitz and Rosamond Thalken, 2023. His- torical trends in jurrppduriss:: A language model assessment.
William N. Eskridge. 1990. The new textualism. University of California Los Angeles Law Review, 37(4):621-692.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu. Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller. Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nic, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tumed chat models.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-worken for text- annotation tasks.
Gilbert Gilmore. 2014. The Ages of American Law. Yale University Press, New Haven.
Thomas C. Grey. 1999. The new formalism.
Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023. Annollm: Making large language models be better crowdsourced annotators.
Morton J. Horwitz 1992. The Transformation of Ameri- can Law, 1870-1960: The Crisis of Legal Orthodoxy. Oxford University Press, New York, NY.
Dunean Kennedy. 2006. The Rise and Fall of Classical Legal Thought. Beard Books, Washington, DC.
Petter Tömnherg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter mes- sages with sero-shot learning.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2023. Large lan- guage models are zero-shot reasoners.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Lee, and Ee-Peng Lim. 2023. Plan- and-solve prompting: Improving zero-shot chain-of- thought reasoning by large language models.
Michael A. Livermore and Daniel N. Rockmore. 2023. Language model interpretability and empirical legal studies.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny 7hou. 2022. Chain-of-thought prompting elicits reasoning large language models.
Karl N. Llewellyn. 1950. Remarks the theory of appellate decision and the rules or canons about how statutes are to be construed. Vanderbilt Law Review, 3(3):395 406.
Selection of Paragraphs for Labeling A
Karl N. Llewellyn. 1960. The Common Law Tradition: Deciding Appeals. W.S. Hein, Buffalo, NY.
We use the following seed terms to up-sample para- graphs that more likely reflect the interpretive ap- proaches of interest." 15
OpenAI. 2023. Gpt-4 technical report.
Free Law Project. 2023. eyecite, https://githur: comm/Freelawwprojeeteeeeec
Theee terms largely draw from carlier efforts by legal scholars to develop search terms that recover cases involving

---

Grand seeds: conference report, committee report, senate report, house report, assembly report, senate hearing, house hearing, assem- bly hearing, committee hearing, conference hearing, floor debate, legislative history, his- tory of the legislation, conference commit- tee. jee.joont committee, senate committee, house committee, assembly committe,, legislative purpose, congressional purpose, purpose of congress, purpose of the legislature, social, society
B Decision Chart
A decision chart was created and provided to an- notators between the fourth and fifth weeks of an- notations (Figure 4). Following the incorporation of this decision chart, we saw boosted inter-rater reliability and more consistent agreement between annotators.
C Prompts
We designed three prompting strategies to instruct LMs to identify legal interpretation and classes of legal interpretation in text. These prompts are included in Figures 2 and 5. All prompts modeled after our annotation codebook.
Formal seeds: dictionary, dictionarium, liguae britannicae, world book, funk & wag- nalls, expressio, expresio, inclusio, noscitur sociis, noscitur socis, ejusdem generis, last antecedent, plain language, whole act, whole- act, whole code, whole-code, in pari materia, meaningful variation, consistent usage, sur- plusage, superfluit, plain meaning, ordinary meaning, word
D Codebook
The codebook was iteratively created throughout the process of annotation to guide annotators. Ta- ble 5 includes the final definitions of each class alongside core examples of each class.
The selection of paragraphs to annotate occurred through a series of steps:
1. We include only opinions that perform statu- tory interpretation. We identify these opinions by finding opinions that include any of the tokens 'statute', 'legislation', or 'act', within 200 characters of the tokens 'mean', 'constru' (i.e. construct), 'interpret', 'reading', or 'un- derstand".
2. Opinions that pass the statutory interpretation filter were split into paragraphs. In each para- graph, we looked for the occurrence of differ- ent seed terms corresponding to either formal or grand reasoning.
3. Of the total number of paragraphs used for labeling, 25% included one more formal seeds, 25% included one or more grand seeds, and 50% included none of the seed terms. This proportion remained the same until the last two rounds of labeling when more exam- ples of formal or grand seeds were included. During those two rounds, the proportion of formal and grand seeds was increased 40% for both classes.
methods of statutory interpretation related to the formal and grand styles of jurisprudence. Choi (2020) helpfully collects many of those terms; see also related efforts (Standt et al., 2005: Calhoun, 2014; Bruhl, 2018).

---

Does the pussage endurse emplay rules, formal categories or view law closed and/or mechanical exercise?
Yes
Yes
Ves
Yos
Formal
Does appear to body of opinton?
Does it involve statute?
Is it engaged reasoning?
author's viewpoint discernible?
Yes
Interpretive Classes
Does passage endorse emplay open-ended reasoning considers works abrlitaillity and may refer political, economic, sacial factors?
No
No
No
No
Yos
Grand
None
Figure The decision chart provided to the annotation team.

---

Model Input
Model Input
Some parsgraphs
interprel statules, this typ paragraph, there is analysis of statute and claim made about its meaning.
Some paragraphi court interpeet ststutes. Within interpretation, there types: grand and formal.
Grand interprotation represents legal deciston that views law as an open-ended ended and on going enterprise for the production and improvement of decisions that make sense on their face and in light of political, sucial, and econamic factors.
In the following paragraph, determine if l'll Interpretation occurs. If yes, respond with "interpretation" and if not. respand with "no interpretation.
Formal Interpretation is a legal decision made according to rule, often viewing the law as a closed and mechanical system. It screens the decision maker off from the political, social, and coonomic choices involved in the decision.
Nevertheless, respondent urges the legislative purpose of the statute is best served by construing it permit same choice in determining the iength of the penalty period. In respondent's view, purpose of siatute is essentially remediul nod compensetorg und thus should
Let's analyze the following passage step-by stepb First, determine interprets a statute. Second, it interprets statute, determine whether Interpretation is grand or formal, The first word your response should label passage with "GRAND", "FORMAL', or "NONE" explain why you chose label.
interpreted literolly to produce monstory oward that for in exeess of any equitaible remedy to punizive.
Model Output
ITEX INFERENCE)
Interpretation
Model Output
FORMAL
b.
Model Input
Determine the legal interprotation used in the following passage. Return single choice from GRAND, FORMAL, or NONE. Here examples:
#*#
Text: [FORMAL CODEBOOK EXAMPLE]
FORMAL
Text: [GRAND CODEBOOK EXAMPLE]
GRAND
###
CODEDOOK EXAMPLE]
Text: [TEXT FOR INFERENCE]
Model Output
GRAND
Figure 5: In-context prompt for identifying whether interpretation occurs not. Prompt a is the prompt used for binary interpretation. Prompt is the prompt used for CoT reasoning and the classes of legal interpretation. Prompt is the prompt used for few-shot classification of the classes of legal interpretation.

---

Class
Definition
Example
Formal
A legal decision that views the law as open-ended and on- A legal decision made according to a rule, often viewing the law a closed and mechanical system. It screens the decision-make maker off from the political, social, and economic choices involved in the decision.
Accepting this point, too, for argument's 's sake, the question becomes: What did "discriminate" mean in 1964? As it turns out, it meant then roughly what it means today: "To make difference in treatment or favor (of one as compared with others)." Webster's New International Dictionary 745 (2d ed. 1954). To "discriminate against" a person, then, would seem to mean treating that individual worse than others who are similarly situated. [CITE]. In so-called "disparate treatment" cases like today's, 's this Court has also held that the difference in treatment based on sex must be intentional. See, e.g., [CITE]. So, taken together, an employer who intentionally treats a person worse because of sex-such as by firing the person for actions or attributes it would tolerate in individual of another sex discriminates against that person in violation of Title VII. Bostock Clayton County
Grand
Respondent's s argument is not without force. But it overlooks the significance of the fact that the Kaiser-USWA plan is affirmative
action plan voluntarily adopted by private parties to eliminate tra- ditional patterns of racial segregation. In this context respondent's reliance upon a literal construction of $ 703 (a) and (d) and upon McDonald is misplaced. See [CITE]. It is a "familiar rule, that a thing may be within the letter of the statute and yet not within the statute, because not within its spirit, nor within the intention of its makers." [CITE]. The prohibition against racial discrimination in $$ 703 (a) and (d) of Title VII must therefore be read against the background of the legislative history of Title VII and the historical context from which the Act arose. See [CITE]. Examination of those sources makes clear that an interpretation of the sections that for- bade all race-conscious affirmative action would "bring about an end completely at variance with the purpose of the statute" and must be rejected. [CITEJ. See [CITE]. Steelworkers v. Weber
going enterprise for
the production and improvement of deci- sions that make sense on their face and in light of political, so- cial, and economic factors.
None
A passage or mode of reasoning that does not reflect either the Grand or Formal ap- proaches. Note that this coding would in- clude areas of sub- stantive law outside of statutory interpre- tation, including pro- cedural matters.
The questions are, What is the form of an assignment, and how must it be evidenced? There is no precise form. It may be. by delivery. Briggs V. Dorr, CITE, citing numerous cases; Onion Paul, 1 Har. & Johns. 114; Dunn Snell, CITE; Titcomb Thomas, 5 Greenl. 282. True, it is said it must be on valuable consideration, with intent to transfer it. But these last are requisites in all assignments, or transfers of securities, negotiable or not. It may be by writing under seal, by writing without seal, by oral declarations, accompanied in all cases by delivery, and on a just consideration. The evidence may be by proof of handwriting and proof of. possession. It may be proved by proving the signature of the payee or obligec on the back, and possession by third person. 3 Gill & Johns. 218.
Table 5: Codebook definition and examples of each of the interpretive classes.


Reviews:
[{"review_id": "IPU1XLLdbE", "reviewer": "Reviewer_ikuS", "summary": "", "questions": "(perhaps these are questions for a follow up paper.)\n\nA:  The mean scores presented in figure 3 are interesting.  Is there any interesting time series in the cross-section?  For example, perhaps is a plot of variance (st.dev) across time.\nB:  In the time-series plot in Figure 3, is there a change a \"change in style\" or a \"change in court membership?\"", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review! \n\n*Concern: Is there any interesting time series in the cross-section? For example, perhaps is a plot of variance (st.dev) across time.*\n\nWe agree there are many substantive questions to pursue with the legal reasoning scores developed in this paper. Another working paper of ours looks at the “purity” in style over time, which is closely related to your point about the standard deviation. We find that the transitional periods tend to be less pure, as jurists adapt to new modes of thought. \n\n*Concern: In the time-series plot in Figure 3, is there a \"change in style\" or a \"change in court membership?\"*\n\nThis is another great question. We also have on our radar, but have not fully explored, the question of whether changes in average style reflect primarily changes in the composition of the court, or instead changes in style holding the court composition fixed. Our initial results suggest it is a mix of the two processes. For instance, the jump in style in 1937 corresponds to a large-scale change in court composition, with a host of new, more liberal justices. But there also appear to be changes in style even accounting for composition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The data here is very interesting.  The annotated data is a style of \"reasoning\" (in contrast to, say, sentiment).  This is a nice setting to test LLMs.\n- The comparison across models is comprehensive (i.e., not just a variety of models, but a variety in how the task is structured). I found this insightful and the conclusions compelling.\n- The quantitative social science component that quantifies the style shift in the court is both well done and interesting.", "reasons_to_reject": "- The models do not end up performing particularly well", "questions_to_authors": "(perhaps these are questions for a follow up paper.)\n\nA:  The mean scores presented in figure 3 are interesting.  Is there any interesting time series in the cross-section?  For example, perhaps is a plot of variance (st.dev) across time.\nB:  In the time-series plot in Figure 3, is there a change a \"change in style\" or a \"change in court membership?\"", "paper_topic_and_main_contributions": "This paper develops a large annotated database of legal reasoning (specifically, the data is examples of \"formal\" and \"grand\" styles of legal reasoning in court decisions.  Using this data, the paper then compares classification performance across several models.  The conclusion is that for current models, the classification task is challenging.  Most interesting (at least to me) was the time-series analysis of court cases that quantifies the shift in opinion style.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zYNi7uyHB9", "reviewer": "Reviewer_WWog", "summary": "", "questions": "A. Can you justify the reason for choosing BERT, DistilBERT as baselines, not more recent pre-trained SOTA models such as T5?\n\nB. Follow-up question, why choose FLAN-T5-Large instead of FLAN-T5-XL and XXL, is that due to computing resource limitations?\n\nC. Speaking of FLAN-T5-Large, the results are abnormally low. My personal experience with FLAN-T5 is that the model may generate text that is different from the prompted options, and you need to write some heuristics to map them back (for a faithful performance evaluation). Have you checked what FLAN-T5 is outputting?\n\nD. What is the human performance in this dataset?  If the inter-rater consensus is low, what did you do to verify the annotation? e.g., hiring more people for majority voting. In addition, if the human performance is lower than fine-tuned model performance, will it hurt the annotation quality and dataset validity?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review! We appreciate your concern for keeping all EMNLP submissions at a high quality and your recommendations for ways to improve our experimentation. \n\n*Concern: positioning of the work; “is this a resource paper or is this a benchmark paper?”* \n\nWe recognize that the resource paper and the model comparison paper are two common archetypes, but we view this paper as falling into another category of EMNLP submissions. In this case our research question is \"what happens when we apply LLMs to a classification task that highly-trained humans find difficult?\" As an NLP Applications paper, we believe that answering this question requires a hybrid approach. This approach includes the influence of domain-specific details on data preparation and the careful selection of models to rule out the possibility that observations hold for only one model. \n\n*Concern (A): the experimentation is not comprehensive enough; “Can you justify the reason for choosing BERT, DistilBERT as baselines, not more recent pre-trained SOTA models such as T5?”* \n\nWe agree with this concern, and will ensure that the quality of the model performance testing matches the quality of our dataset creation, including more examples of how various models perform on this task. We understand the concern about the lack of justification for the models selected, and will clarify our choices for each model. Our criteria for inclusion are that models should be commonly used for applied NLP research, that they can be fine-tuned for \"instruction\" style interaction, and we prioritized models that are either available through public APIs or small enough to be run by domain experts without access to their own GPU hardware.\n\nTo provide a more complete assessment of available LLMs, we have begun testing the performance of additional models (T5-base, T5-small, and Llama-2-7b-chat) noted in the table below. Model names that are bolded are new models; the rest of the results are the same as the submitted version. Thus far, the results from these models align with our prior experiments. We plan to test Llama-2-7b-chat in the chain-of-thought and few-shot setting and T5-base and T5-small with nested fine-tuning, but were unable to complete this before the rebuttal deadline due to time constraints. \n\n|                          | Macro Averages |          |          | Grand    |          |          | Formal   |          |          | None     |          |          |\n|--------------------------|----------------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|\n|                          | F1             | P        | R        | F1       | P        | R        | F1       | P        | R        | F1       | P        | R        |\n| Nested                   |                |          |          |          |          |          |          |          |          |          |          |          |\n| BERT-base                | 0.67           | 0.67     | 0.68     | 0.65     | 0.62     | 0.68     | 0.49     | 0.50     | 0.48     | 0.87     | 0.88     | 0.86     |\n| LEGAL-BERT-base          | 0.69           | 0.68     | 0.71     | 0.67     | 0.64     | 0.71     | 0.54     | 0.51     | 0.57     | 0.88     | 0.90 &   | 0.85     |\n| DistilBERT               | 0.67           | 0.67     | 0.68     | 0.65     | 0.63     | 0.68     | 0.50     | 0.50     | 0.50     | 0.87     | 0.88     | 0.86     |\n|                          |                |          |          |          |          |          |          |          |          |          |          |          |\n| Multi-Class              |                |          |          |          |          |          |          |          |          |          |          |          |\n| BERT-base                | 0.68           | 0.68     | 0.68     | 0.67     | 0.65     | 0.69     | 0.50     | 0.52     | 0.48     | 0.88     | 0.88     | 0.87     |\n| LEGAL-BERT-base          | **0.70**       | **0.70** | **0.71** | **0.68** | **0.65** | 0.72     | **0.55** | **0.55** | 0.55     | **0.88** | 0.89     | **0.87** |\n| DistilBERT               | 0.67           | 0.67     | 0.66     | 0.66     | 0.64     | 0.68     | 0.47     | 0.51     | 0.44     | 0.87     | 0.86     | 0.87     |\n| **T5-base**              | 0.64           | 0.64     | 0.65     | 0.66     | 0.63     | 0.69     | 0.51     | 0.52     | 0.50     | 0.87     | 0.88     | 0.86     |\n| **T5-small**             | 0.48           | 0.62     | 0.49     | 0.57     | 0.66     | 0.51     | 0.02     | 0.43     | 0.01     | 0.84     | 0.76     | 0.94     |\n|                          |                |          |          |          |          |          |          |          |          |          |          |          |\n| In-Context, Descriptions |                |          |          |          |          |          |          |          |          |          |          |          |\n| GPT-4                    | 0.22           | 0.24     | 0.27     | 0.44     | 0.38     | 0.51     | 0.36     | 0.23     | **0.81** | 0.58 &   | **0.92** | 0.42     |\n| FLAN-T5-large            | 0.19           | 0.36     | 0.36     | 0.36     | 0.24     | 0.77     | 0.16     | 0.11     | 0.28     | 0.04     | 0.72     | 0.02     |\n| **Llama-2-7b-chat**      | 0.2            | 0.34     | 0.36     | 0.35     | 0.22     | **0.92** | 0        | 0        | 0        | 0.24     | 0.82     | 0.14     |\n|                          |                |          |          |          |          |          |          |          |          |          |          |          |\n| In-Context, Examples     |                |          |          |          |          |          |          |          |          |          |          |          |\n| GPT-4                    | 0.45           | 0.47     | 0.54     | 0.45     | 0.38     | 0.57     | 0.36     | 0.25     | 0.65     | 0.62     | 0.86     | 0.48     |\n| FLAN-T5-large            | 0.08           | 0.30     | 0.23     | 0.14     | 0.27     | 0.10     | 0.20     | 0.11     | 0.90     | 0.01     | 0.92     | 0.01     |\n|                          |                |          |          |          |          |          |          |          |          |          |          |          |\n| Chain-of-Thought         |                |          |          |          |          |          |          |          |          |          |          |          |\n| GPT-4                    | 0.34           | 0.37     | 0.37     | 0.25     | 0.50     | 0.17     | 0.43     | 0.32     | 0.67     | 0.78     | 0.80     | 0.76     |\n| FLAN-T5-large            | 0.08           | 0.33     | 0.34     | 0.00     | 0.00     | 0.00     | 0.21     | 0.12     | 1.00     | 0.03     | 0.88     | 0.02     |\n\n\n*Concern (B): why choose FLAN-T5-Large instead of FLAN-T5-XL and XXL, is that due to computing resource limitations?*\n\nWe limit to a set of slightly smaller models due to resource constraints and our substantive criteria for model inclusion. While larger models like FLAN-T5-XL or XXL may have better results, these models are less accessible to the average academic researcher. \n\n*Concern: How do you justify the prompt you designed is making the best of the models?* \n\nWe agree that there may be  prompts that would improve the results of the generative models. As we observe more fully in our response to reviewer TywZ, we try to address this concern by exploring a range of prompt methodologies, by emphasizing that our interest is in the capacity of the generative models to replace human inputs in complex research tasks, and by being forthright about the limits of our study. \n\nOur general view is that the paper advances the debate but does not settle it. We will revisit the language in the paper to conform any errant language to reflect this viewpoint. For instance,  we can clarify that we are not evaluating a stronger hypothesis (\"there does not exist a prompt that is effective\") but rather a weaker hypothesis (\"a reasonable prompt that is successful for humans does not work as well\"). \n\n*Concern (C): Have you checked what FLAN-T5 is outputting?* \n\nYes, we have verified that these results are correct. For the descriptive prompt, where FLAN-T5 performs slightly better, FLAN-T5 rarely identifies a paragraph as “None” and overpredicts Grand and Formal. For the few-shot and chain-of-thought prompts, FLAN-T5 almost always predicts a paragraph as Formal. \n\n*Concern (D): What is the human performance in this dataset? If the inter-rater consensus is low, what did you do to verify the annotation? In addition, if the human performance is lower than fine-tuned model performance, will it hurt the annotation quality and dataset validity?* \n\nThis is a complex task, requiring extensive legal knowledge and careful analysis of each passage. We see the final inter-rater reliability (0.63 Krippendorff’s *alpha*) as an appropriate score, relative to task difficulty. \n\nWe verified the annotations in multiple ways. First, we leveraged the knowledge of the group of four law students who had completed the annotation. Each weekly batch of paragraphs for annotation contained a small overlapping set of paragraphs that were assigned to all four annotators. During our weekly discussions, each annotator justified the class assigned to difficult paragraphs in this shared sample. As a group, we then deliberated on that class, and made changes to the codebook based on that discussion. \n\nEven after extensive training, discussion, and deliberation, there were a set of paragraphs that remained ambiguous as to classification. However, we see these examples as highly important to include in the training data, and so we treated them with additional care. Each annotator could assign a “low confidence” score to a paragraph, which then resulted in a group deliberation and vote about what the paragraph’s correct assignment should be. The label that came from this majority vote was included in the training and validation sets. \n\nWe further undertook efforts to validate the predictions from our preferred model, including several exercises not reported in the current draft but that may be in the revised draft. For instance, our annotation codebook includes paragraph exemplars for each class. We examine the case-level predictions for the parents of those paragraphs, and find that the model predicts case-level classes as expected (e.g., the median formal example parent case contained about 60 percent formal reasoning in its paragraphs; the corresponding figure for the grand examples was about 25 percent). In another exercise, we examine the formality of opinions written by different authors and find, as expected, that jurists known for formal jurisprudence, such as Justice Scalia, return with high average scores.\n\nFinally, the current draft of the paper includes one of our validation efforts, the results reported in section 8 (“Periods of Legal Reasoning”). That section shows the yearly averages for formality in legal reasoning over a roughly 150 year period. While we can expect some degree of misalignment between qualitative narratives of historical trends in legal reasoning, the fundamental convergence between the qualitative historical consensus and our own findings strongly suggests that our annotations correspond with the target concepts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset seems of relatively good quality, which is domain-experts annotated. The experiment setup and results are valid.", "reasons_to_reject": "The primary concern is the novelty (or the position/framing of this paper). \n\n- More specifically, if this is a resource paper, The pro of this dataset is expert annotation and well-defined task, however, I would say the difficulty of this dataset is not hard enough, where a fine-tuned LEGAL-BERT can already reach over 80% accuracy, and I would assume a much larger model can perform even better when fine-tuned, so the space left for improvement is very limited. In addition, the authors spend a lot of space introducing and describing the task from the legal history perspective, I would suggest shortening that part in the main content and putting the story in the appendix. \n\n- If this is a paper focusing on the benchmarking of SOTA generic LMs, the experiments are not comprehensive to provide enough contribution. \n\nThe authors only conduct fine-tune experiments on relatively small (and a bit early) LMs such as BERT and DistilBERT, but do not include seq2seq models (e.g., BART, T5) and even larger models (e.g., LLaMA), which lead to the potential issue that the baselines are not SOTA. \n\nFor the no-tuning experiment, the authors only compared GPT-4 and FLAN-T5-large(why not FLAN-T5-XL and XXL?, or LLaMA and Alpaca?), and I do leave a question below for FLAN-T5's result here. The prompts in the Appendix are okay, but there are many questions left (if the author wants to comprehensively support the claim): \nHow to justify the prompt you designed is making the best of the models? \nHow do you determine the number of few-shot examples, and how do you select them? \nHow to justify the few-shot setting you use is making the best of the models?\n\nTo summarize, I appreciate the careful design of the dataset collection and experiment setup & analysis. However, I would suggest the authors sell the paper in a way that emphasizes all the pros (and avoid the cons), instead of trying to explore everything but not doing everything comprehensively. \nThis paper is of a relatively higher quality compared with all the papers I reviewed for EMNLP this year (maybe it's just my sampling bias). As a result, I do have higher expectations for the authors to make this work outstanding and solid. I may change my score depending on the rebuttal. \n\nps. I recently found one paper that is exploring using LLM in another specialized domain, and I would recommend the authors take a look at how extensive their experiments are to support their similar claim (LLMs are not suitable for specialized domains off the shelf).\nLet me make my point clear, I'm not suggesting the authors cite this paper (it was just submitted to Arxiv a couple of days ago and of a different domain), but I'd like to use it as an example to show the level of comprehensiveness you need, if you want to emphasize the contribution on the benchmarking experiment side. They explored a lot of different prompt designs for zero-shot, conducted few-shot experiments, conducted subsampling and transfer learning experiments for fine-tuning the LM, and used many SOTA models (Alpaca, Lora-finetuned Alpaca, GPT-3.5, in addition to BERT and Mental-RoBERTa) for comparison.\n- Leveraging Large Language Models for Mental Health Prediction via Online Text Data (https://arxiv.org/abs/2307.14385)", "questions_to_authors": "A. Can you justify the reason for choosing BERT, DistilBERT as baselines, not more recent pre-trained SOTA models such as T5?\n\nB. Follow-up question, why choose FLAN-T5-Large instead of FLAN-T5-XL and XXL, is that due to computing resource limitations?\n\nC. Speaking of FLAN-T5-Large, the results are abnormally low. My personal experience with FLAN-T5 is that the model may generate text that is different from the prompted options, and you need to write some heuristics to map them back (for a faithful performance evaluation). Have you checked what FLAN-T5 is outputting?\n\nD. What is the human performance in this dataset?  If the inter-rater consensus is low, what did you do to verify the annotation? e.g., hiring more people for majority voting. In addition, if the human performance is lower than fine-tuned model performance, will it hurt the annotation quality and dataset validity?", "paper_topic_and_main_contributions": "The authors proposed a legal classification dataset and benchmarked LMs task-solving and reasoning performance off the shelf. The results justified generic LMs have limited ability to solve specialized tasks without task-specific fine-tuning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 355, 'all models were tested with the same subset of training data', should it be 'testing data'?"}, "extra_text": {}}, {"review_id": "B7KSbrxrRU", "reviewer": "Reviewer_TywZ", "summary": "", "questions": "Question 1A: Why not distinguish between 'literal' versus 'purposive' interpretation instead of 'grand' versus 'formal'?\nQuestion 1B: If you had distinguished in that way, could the prompt provided have provided better results in the generative NLP experiments?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review! We appreciate your comments and suggestions on how to refine the paper.\n\n*Concern: Why not distinguish between 'literal' versus 'purposive' interpretation instead of 'grand' versus 'formal'?* \n\nThis is a great question – in our revisions, we will draw parallels to these modes of interpretation. Though the “literal” and “purposive” distinction is closely related to the “formal” and “grand” distinction, we believe that the formal and grand distinction encompasses a somewhat wider range of reasoning methods. For instance, one aspect of formal reasoning, according to Llewellyn and others, is that it is closed off to social, political, and economic factors that operate at the time of the legal decision. These factors often but not always relate to the purpose of the law at the time of passage. Our weekly discussions with the annotation team were not uncommonly related to the degree to which a legal passage embraced social considerations, and we likewise attempted to integrate that factor into our prompts. \n\n*Concern: If you had distinguished in that way, could the prompt provided have provided better results in the generative NLP experiments?*\n\nA challenge with this research project is that we, in effect, find a “null” result for the generative language models. That is, we find that they do not perform adequately. A generic concern with null results is that they reflect an aspect of the study design rather than the data generating process: for instance, the study may be under-powered, or perhaps it focuses on the wrong place or time for an effect. Applied to our study, a concern is that the generative language models might perform better with different prompts—that is, perhaps our null is a function of our prompts rather than the generative models. \n\nWe attempt to address this concern in three ways. First, we explore a range of prompt methodologies, including zero shot, few shot, and chain-of-thought approaches. None of these approaches results in strong performance. Second, thematically, our interest is in the ability of the generative models to replace human inputs in complex research tasks. Our general approach, therefore, is to provide the models with similar types of instructional context that we provided to our annotation team. Finally, we acknowledge that we cannot completely dispel the idea that generative models, with different prompts, would perform better. Viewed broadly, we see our paper as advancing a debate rather than settling it. In revision, we adapt language to reflect these points and limits to the study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are several reasons to accept the paper. First, it makes a contribution in a previously underexplored area. Second, it assesses various different ways of resolving the task that it sets out and compares these well. Third, it contains a helpful (but see below) conceptual framework distinguishing between different approaches to statutory interpretation.", "reasons_to_reject": "There are no real reasons to reject the paper, but merely a suggestion that its conceptual framework could have been somewhat tighter. First, the authors' idea of 'intepretation' is noit sufficiently rigorous and clear (from the point of view of a legal scholar like myself). Whether an item falls within the extension of a legal concept is usually called a question of 'application' not of 'interpretation'. Interpretation means simply spelling out the linguistic meaning of an item (a statutory provisionsay) bu using paraphrases in clauses such as 'X means Y'. Second, and more importantly, it is not clear at all that Llewellyn's distinction between 'grand' and 'formal' styles captures something specific. In fact, it seems to me that it would have been much better to simply distinguish, as we do in some civil law systems, between 'literal' and 'purposive' interpretation of legal texts (such as statutes), the first being mainly geared towards the conventional linguistic meaning of the expression and the second towards identifying the purposes of the provision, i.e. the goals that the rule is supposed to serve. But again, these are not reasons to reject the paper, merely suggestions about strengthening it.", "questions_to_authors": "Question 1A: Why not distinguish between 'literal' versus 'purposive' interpretation instead of 'grand' versus 'formal'?\nQuestion 1B: If you had distinguished in that way, could the prompt provided have provided better results in the generative NLP experiments?", "paper_topic_and_main_contributions": "The paper makes an important contribution to the topic of automatically retrieving types of legal argument regarding statutory interpretation (mainly in the context of US Supreme Court judgments). It tests a range oif techniques and compes up with innovative ways of zooming in on a particularly efficient one such range of techniques.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None that I can think of.", "Typos_Grammar_Style_and_Presentation_Improvements": "Style is up to standard and clear."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes an important contribution by rigorously evaluating the performance of language models on a complex legal reasoning classification task. The key technical innovation is a high-quality dataset annotated by legal experts categorizing passages of US Supreme Court opinions as either formal/textualist or grand/purposivist interpretive reasoning. This granular labelling required extensive domain knowledge and deliberation, even for human annotators.

The core experiments compare several model architectures, including BERT, DistilBERT, GPT-3, and FLAN-T5. The striking finding is that most models struggle to match human performance levels absent fine-tuning on this in-domain labelled data. Even a strong BERT model fine-tuned on the annotations reaches only ~80% accuracy, indicating inherent challenges in this reasoning style classification task.

In general, reviewers praise the novelty of the dataset and approach, but identify areas for improvement:

Additional SOTA models could be tested to comprehensively benchmark performance. The authors sufficiently address this by including results for models like LLaMA.
Connections could be drawn to the literal vs. purposive interpretation distinction used in some legal frameworks. Of course, I note that the authors already indicate they will incorporate such parallels to situate the conceptual framing.
Lastly, further analysis into changes in court membership vs. judicial philosophy over time was suggested. 

I agree with the reviewers that the key contribution here is the introduction of the dataset. However, the task appears to me to be quite trivial - a classification task with 3 labels! Tasks like these are already well solved even in the legal AI field. Moreover, I find it surprising that a finetuned version of a BERT variant, i.e., LEGAL-BERT, even though confers some domain specificity will be able to match the performance of recent LLMs. I am not convinced and I think the authors need to do more exploratory studies with proper prompting techniques - be it in-context or few/zero-shot learning employing newer/larger models.
Decision: Accept-Main