Lawma: The Power of Specialization for Legal Annotation
Authors: Ricardo Dominguez-Olmedo
Vedant Nanda
Rediet Abebe
Stefan Bechtold
Christoph Engel
Jens Frankenreiter
Krishna P. Gummadi
Moritz Hardt
Michael Livermore
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
legal classification tasks
benchmarks
Primary area: datasets and benchmarks
TL;DR: A lightly finetuned Llama-3 model outperforms GPT-4.5 and Claude 3.7 Sonnet on 260 new legal classification tasks

Abstract: Annotation and classification of legal text are central components of empirical legal research. Traditionally, these tasks are often delegated to trained research assistants. Motivated by the advances in language modeling, empirical legal scholars are increasingly turning to commercial models, hoping that it will alleviate the significant cost of human annotation. In this work, we present a comprehensive analysis of large language models’ current abilities to perform legal annotation tasks. To do so, we construct CaselawQA, a benchmark comprising 260 legal text classification tasks, nearly all new to the machine learning community. We demonstrate that commercial models, such as GPT-4.5 and Claude 3.7 Sonnet, achieve non-trivial accuracy but generally fall short of the performance required for legal work. We then demonstrate that small, lightly fine-tuned models vastly outperform commercial models. A few dozen to a few hundred labeled examples are usually enough to achieve higher accuracy. Our work points to a viable alternative to the predominant practice of prompting commercial models. For concrete legal annotation tasks with some available labeled data, researchers are likely better off using a fine-tuned open-source model. Code, datasets, and fine-tuned models are available at https://github.com/socialfoundations/lawma.

Published as a conference paper at ICLR 2025
LAWMA: THE POWER OF SPECIALIZATION FOR LEGAL ANNOTATION
Ricardo Dominguez-Olmedoo Vedant Nanda,, Redict Abe Stefan Bechtold Christoph Engel" Jens Frankenreiter Krishna Gummadi Moritz Hardt and Michael Livermore*
Max Planck Institute for Intelligent Systems. Tübingen, and Tübingen Al Center Max Planck Institute for Software Systems, Saarbrücken 'ELLIS Institute, Tübingen *ETH Zurich 'Max Planck Institute for Research on Collective Goods, Bonn "Wahhingoon University in St. Louis School of Law UUniversity of Virginia School of Law
ABSTRACT
Annotation and classification of legal text are central components of empirical legal research. Traditionally, these tasks are often delegated to trained research assistants. Motivated by the advances in language modeling, empirical legal schol- ars increasingly turning to commercial models, hoping that it will alleviate the significant cost of human annotation. In this work, we present a comprehensive analysis of large language models' current abilities to perform legal annotation tasks. To do so, we construct Case CaselawCa lawQA, a benchmark comprising 260 legal text classification tasks, nearly all new to the machine learning community. We demonstrate that commercial models, such as GPT-4.5 and Claude 3.7 Sonnet, achieve non-trivial accuracy but generally fall short of the performance required for legal work. We then demonstrate that small, lightly fine-tuned models vastly outperform commercial models. A few dozen to a few hundred labeled examples are usually enough to achieve higher accuracy. Our work points to a viable alter- native to the predominant practice of prompting commercial models. For concrete legal annotation tasks with some available labeled data, researchers are likely bet- ter off using a fine-tuned open- model. Code, datasets, and fine-tuned mod- els are available at http:://gthbbccoooooccaluuuddtdd
INTRODUCTION
The legal system generates staggering volume of complex documents. United States federal courts alone process hundreds of thousands of cases year, each having substantial case files. Much empir- ical legal research involves the systematic collection and analysis of such data in order to understand how laws function in practice and what impact they have on society. What limits researchers across the board is the cost of annotating and classifying legal documents. Legal classification tasks vary in complexity, but often require substantial expertise and effort. Employing trained research assistants stretches to few thousand documents at time, but is no match for the sheer scale of legal data.
There has long been an interest by empirical legal scholars in NLP tools for feature extraction (i.e., annotation) in lieu of human annotators (Livermore & Rockmore 2019). Starting from sentiment analysis and topic models, to now large language models. The costs and error of existing methods the single most important bottleneck in the empirical legal studies pipeline. Yet, the of large language models to annotate legal text remains a critically understudied area.
Nonetheless, motivated by the rapid advances in language models, law scholars increasingly try commercial models, such as GPT-4, on variety of legal tasks, hoping to boost the efficiency of legal research. The underlying assumption is that large models such GPT-4 provide the best solution to the problem that is currently available. In this work, we critically examine this assumption.
Alphahetical order.

---

Published as a conference paper at ICLR 2025
CaselawQAl accuracy
1.0
0.87
0.8
0.74
0.76
0.76
0.77
0.78
0.70
0.71
0.71
0.6
Liama 3.1 BB
Qwen 2.5 72B IT
Liama 3.3 70B
o3 mini (high)
GPT-4o
4058
GPT-4.5 Preview
DeepSeek R1
Claude 3.7 Sonnet
Lawma 8B
Liama
Figure 1: The cost of generality: Performance of various language models on the CaselawQA bench- mark for legal annotation. Lawma 8B, specialized for legal annotation, outperforms all other models.
CaselawQA accuracy
1.0
0.82
0.85
0.87
0.8
0.76
0.78
0..7
0.6
0.4
Llama 3 3.4088
Claude 3.7 Sonnet
Lawma 135M
Lawma 360M
Lawma 18
Lawma 38
Lawma BB
Figure 2: Performance of the Lawma models. The smallest Lawma model, Lawma 135M, com- petitive with the best-performing commercial model, Claude 3.7 Sonnet.
1.1 OUR CONTRIBUTIONS
We introduce and study collection of 260 legal classification tasks, nearly all to the ma- chine learning community. The tasks we introduce are actual legal annotation tasks based on the U.S. Supreme Court (Spaeth et al. 2023) and Court of Appeals (Songer) databases. These databases offer rich annotations for court cases, which we utilize as labels to create challenging multi-class classification tasks. We aggregate these tasks into easy-to-use benchmark, which call Case We detail in Section 2 the process used to construct this benchmark.
Our primary finding is that small, fine-tuned models substantially outperform large commercial mod- els (Figure D.. Specifically, we fine-tune series of small language models, ranging from 135M to 8B parameters, which we collectively refer to as the Lawma models. Our Lawma 8B model achieves 87 % accuracy on CaselawQA. outperforming all commercial models by at least 9 percentage points, with the best-performing commercial model, Claude 3.7 Sonnet, attaining 78% accuracy. Although it is expected that fine-tuning helps, the superiority of fine-tuning open-weights model at much smaller scale surprising. After all, commercial models are orders of magnitude larger. Our re- sults demonstrate that, for legal annotation, researchers are better off using small specialized models rather than large general-purpose LLMs.
We conduct various large-scale fine-tuning experiments that further demonstrate the benefits and practicality of specializing models for legal annotation:
Larger models respond better to fine-tuning than smaller models. Accuracy of the Lawma models increases steadily with model size (Figure However, observe signs of dimin- ishing returns. This suggests that, in the future, major improvements may not come from model scale alone.
Fine-tuning is data efficient. few hundred examples typically suffice to achieve higher accuracy than commercial models (Section 4.2 Figure [O). This is crucial, since labeling a few hundred data points is often financially feasible for many legal scholars, whereas labeling many thousands may not.
2

---

Published as a conference paper at ICLR 2025
Fine-tuning generalizes to unseen tasks. Fine-tuning Llama 8B Inst only the Court of Appeals tasks improves its average accuracy on Supreme Court tasks by 18.8 accuracy points (Appendix 4.3 Figure ITp.
We can simultaneously fine-tune on all 260 tasks. There is not large loss compared with fine-tuning on a specific task (Section D. Figure 13). This is desirable in practice, as obviates the need train and maintain separate model for each task.
We contextualize veals task heterogeneity in the relationship between model accuracy and intercoder agree- ment (Appendix C
accuracy numbers with intercoder agreement rates. Our analysis
Our results speak to the power of specialization for legal annotation. Our insights suggest that the empirical legal community should invest in an ecosystem of fine-tuned models for relevant anno- tation tasks. Such an ecosystem could radically expand the capacity of legal scholars to engage in quantitative work.
From a benchmarking perspective, the tasks presented in this work are of independent interest. They are challenging multi-class classification problems that require some amount of legal expertise. The best models achieve non-trivial, but modest performance. And even fine-tuned models don't 't reach intercoder agreement rates. These legal classification tasks are diverse, non-trivial evaluation tasks for future model advances.
Finally, our work challenges the prevailing narrative about the suitability of "generalist" models. In commercial APIs, users are generally limited to prompting generalist models, as fine-tuning is costly for the model provider. But as we show, generalist models are neither sufficiently good nor best possible for many practical tasks. Specializing models to concrete tasks of interests, even with relatively small base models and few labeled examples, can provide a simple, practical, and far more accurate solution.
1.2 RELATED WORK
Benchmarks for legal tasks. LegalBench (Guha al. 20233 a recent multi-task benehmark for natural language understanding in legal domains. As of writing, LegalBench consists of 162 tasks
gathered from 40 contributors. LegalBench draws on numerous earlier benchmarking efforts in dif-
ferent legal domains, specifically, inference contracts (Koreeda & Manning, [021] Hendrycks jet al., 202T), merger agreement understanding (Wang et al. 2023), identifying the legal holding
of a case (Zheng et al. (021). statutory reasoning (Holzenberger & Van Durme, 202 1). privacy
compliance and policy (Wilson et al. 2016) Zimmeck et al. Ravichhhherr et al. (211)). and
identifying unfair clauses in terms of service (Lippi et al. (019). BBhambhoria al. (2024) evaluate the performance of general-purpose models on legal question-answert development of open-source models tailored to the legal domain. We extend and strengthen these valuable efforts to benchmark large language models in legal settings, We focus on core legal classi- fication tasks based on the U.S. Supreme Court Database (Spaeth al., 2023) and the U.S. Courts of Appeals database (Songer). Our evaluation suite measures the performance of models in annotating court opinions. focusing on tasks that are of interest to the field of empirical legal studies. The tasks we study are complementary to those in LegalBench. We do not evaluate our model on LegahBench, since our model is specialized to the Supreme Court and Appeals Court data.
tasks and advocate for the
Large language models for the legal domain. Gerpere-purpose language models are likely be trained on a substantial amount of legal data because much of this data is publicly available on the internet. For example, the FreeLaw dataset includes a large collection of court opinions (Gao al. 2021). Legal-BERT (Chalkidis et al., (020) is BERT-like transformer model that was pretrained on few hundred thousand legal documents. The more recent SaulLM models (Colombo et al., 2024b)a) a) adapt the open-weights Mistral (Tiang et al. 2023 2024) models to the legal domain both by continual pretraining and instruction-tuning on legal text. In contrast to Lawma, we consider SaulLM to be a general-purpose model for the legal domain, not tailored to any specific legal task. Our approach differs significantly; we focus on developing models specialized for annotation tasks of practical interest to empirical legal studies. We demonstrate that specialization is highly effective, with our Lawma models significantly outperforming all other evaluated LLMs. For a discussion on the adoption of large language models in the legal community, refer to Appendix A
3

---

Published as a conference paper at ICLR 2025
what follows is an opinion from the Supreme Court of the United States. Your task is to identify whether the opinion effectively says that the decision in this case "overruled" one or more of the Court\'s own precedents. Alteration also extends to language in the majority opinion that states that a precedent of the Supreme Court has been "disapproved," or is "no longer good law". Note, however, that alteration does not apply to cases in which the Court "distinguishes" precedent.
[COURT OPINION]
Question: Did the the decision of the court overrule one more of the Court's own precedents?
A. Yes
B. No
Think step by step. At the end, respond with "The final answer is [final_answer]", where [final_answer] is either single uppercase letter (A-Z) or a numerical value (e.g., 9, 121).
Figure Example task corresponding the Supreme Court "precedent alteration" variable.
Data annotation and labeling. Hall & Wright 2008) provide an overview the use of human annotators in empirical legal studies. Student coders have been deployed to extract wide variety of features from legal data. Although student researchers are much less expensive than private attorneys, the costs can quickly become prohibitive. Depending the size of the document and the complexity of the task, research assistants can label roughly dozens of examples per hour. Projects involving the labeling of hundreds of documents are financially feasible for many legal scholars, but projects involving many thousands of documents are largely impractical. In an example of larger annotation effort, |Frankenreiter et al. (202T) employed human coders to annotate several thousands of corporate charters. Using ChatGPT for a similar task, Frankenreiter & Talley (2024) estimated that employing human coders would have been approximately len times more costly.
Data annotation and labeling also play major role in machine learning benchmarks and applica- tions, see, e.g., Aroyo Welty (2015); :: Gray & Suri (2019); Hardt & Recht (2022) for background. Dorner & Hardr (2024) give an extended discussion about label quality and annotator disagreement in the context of machine learning benchmarks.
1.3 LIMITATIONS
While fine-tuned models substantially outperform commercial models, emphasize that fine-tuned models are still far from perfect, and the variance in accuracy across tasks remains high. Although our work meets the ethical and technical recommendations Kapoor al. (2024) for "de- velopers of legal AI", we maintain caution about the use of large language models for consequential legal tasks. To which extent these models are suitable for use specific applications requires addi- tional substantive investigation. We add that the legal documents we consider are exclusively from either the U.S. Supreme Court or appellate courts the United States. We cannot speak how these results may change for tasks in other legal domains within the United States or legal systems in other countries.
2 CASELAWQA
In this work, we focus on legal classification tasks. Legal classification tasks range in complexity, from extremely simple tasks that require little specialized knowledge, to highly sophisticated tasks that involve specific legal knowledge, familiarity with legal principles or discourse, and the ability to engage in nuanced analogical or conceptual reasoning. For example, labeling the ideological valence of a decision requires the annotator to understand how specific legal issues map onto contemporary political debates. Labeling the standard of review applied by appellate court requires detailed knowledge of these standards as well as the ability to parse procedural history. Many legal doctrines quite complicated, involving multipart tests, nuanced exceptions, and balancing inquiries.
4

---

Published as a conference paper at ICLR 2025
Our reasons to study legal classification tasks are both technical and substantive. From technical machine learning perspective, these tasks provide highly non-trivial classification problems where even the best models leave much room for improvement. From substantive legal perspective, effi- cient solutions to such classification problems have rich and important applications in legal research, Appendix A. for detailed discussion.
2.1
DATA SOURCES
Central to our study are the U.S. Supreme Court Database (Spaeth et al. 2023) (SCDB) and the U.S. Courts of Appeals database (Songer) (USCAD). The SCDB compiles comprehensive infor- mation on U.S. Supreme Court decisions from 1946 onward, and includes variables such as case outcomes, issue areas, legal provisions, and vote counts. The USCAD contains detailed information about decisions made by the U.S. Courts of Appeals from 1925 to 1988. It includes data on judi- cial decisions, panel compositions, and case characteristics. Both databases provide essential tools for scholars conducting quantitative analyses of the judicial system, decision-making, ideological trends, and the impact of various factors case outcomes.
The SCDB and USCAD have been instrumental in advancing research on judicial decision making within the fields of political science and empirical legal studies (Epstein al. 2013 Segal & Spacth, 2002 Martin & Quinn 2002). These datasets have been used to drive a substantial research program by allowing scholars to systematically analyze large numbers of court cases, uncovering patterns, trends, and factors influencing judicial outcomes. By providing detailed information on case char- acteristics, judge attributes, and decision outcomes, these databases have enabled researchers to test theories of judicial behavior, examine the impact of ideology on court decisions, and explore the dynamics of judicial docision-makinn at different levels of the court system. The insights gained from research using these databases have had significant implications for legal practitioners, poli- cymakers, and the broader legal community, contributing to a better understanding of how courts operate and how legal outcomes are shaped.
2.2 CONSTRUCTION OF THE CLASSINCATION TASKS
We use the variables of the USDB and the USCAD to construct a set of classification tasks. We construct a total of 260 distinct classification tasks, of them corresponding to the Supreme Court database and 232 to the U.S. Court of Appeals. The annotations in the USDB and USCAD serve as labels for these classification tasks. For each task, we additionally construct a prompt template consisting of a general description of the task, followed by a multiple choice question containing each of the possible variable codes. We formulate the task description, question, and answer choices by closely following the databases' variable descriptions. See Figure for an example task.
For every case contained in the USDB and the USCAD, we use the provided case citations to search for its corresponding majority opinion of the court on the the Caselaw Access Project| database of dig- itized court opinions. We match a total of 24,9 44 916 court cases, which we divide into 11%/10%/20%% train/validationn/esst split. That is, models may not train on any of the court cases used for evaluation.
Since many of the classification tasks contain heavily imbalanced classes, we subsample the majority class such that there are at most as many task examples in the majority class as task examples in all other classes combined. As a result, a constant classifier that outputs the majority class label will never achieve more than 50% accuracy on any individual task. This results measure of model performance, as models cannot attain high accuracy simply because a task is heavily imbalanced. We report in Appendix results without subsampling of the majority class.
more honest
We plot some statistics of the tasks in Figure First, court opinions tend to be long, with 12% having above 8,000 tokens, the typical maximum context size for current state-of-the-an models, such Llama 3. Second, some tasks have a large number of classes, with 28% of tasks having more than 10 classes. Third, there is a large variability in terms of the number of task examples, ranging from couple dozen to 18500 task examples. Our final dataset comprises 718.971 task examples.
To reduce the compute required for evaluating the benchmark, we select at random 5,000 examples from the Supreme Court tasks and 5,000 examples from the Court of Appeals tasks. We include only court cases where the court opinion. including the head matter, contains at least 2,000 characters, ensuring the opinion is at least few sentences long. These 10,000 task examples comprise the test
5

---

Published as a conference paper at ICLR 2025
Number of opinions
Number of taskks
60
Number of tasks
20
40
500
20
0
10
10a
10a
10"
10a
10
10a
10a
10a
10
10a
10a
10a
Number of tokens in the opinion (GPT-4)
Number of classes in the task
Number of examples in the task
Figure 4: General statistics of the court opinions and legal classification tasks considered.
SC Area
SC Case Source
Disposition
Direction
Disposition
1.00
0.75
0.50
0.25
Songer
Songer Origin
Songer
Songer
1.00
0.50
0.25
Constant baseline
Liama 3.3 70B
Claude 3.7 Sonnet
Lawma 8B
Figure 5: Accuracy of models on ten different legal classification tasks of particular interest.
set of CaselawgA. We nonetheless make available 143,635 task examples corresponding to the test court cases, which we call the extended test set. Evaluating on the extended test set is 14x as expensive, but provides much more fine-grained information on models' performance across all 260 legal classification tasks, rather than simply aggregate measure of model performance. In this work, we report accuracy on the extended test set, unless otherwise stated.
2.3
EVALUATION METHODOLOGY
We evaluate models using prompt template identical to the for the MMLU bench- mark (Hendrycks et al.) 2020). Since many popular benchmarks are phrased as multiple-choice questions, recent models tend do well for them (Dominguez-Olmedo et 2024 Due diverse set of models and large number of tasks under consideration, we perform no prompt tuning.
We use accuracy as the evaluation metric. Given that the tasks we consider involve vastly differing numbers of answer choices, accuracy provides an interpretable measure of performance. Addition- ally, accuracy the standard metric used in knowledge-testine L.MM benchmarks. For completeness, we also report balanced accuracy and macro-averaged F1 scores in Appendix
When reporting aggregate performance across multiple tasks (e.g., all Supreme Court tasks), we compute the average accuracy across all task examples. Intuitively, we can visualize the Supreme Court database as a large table with dimensions corresponding to the number of court cases (rows) and the number of tasks (columns). The aggregate accuracy, in this case, represents the fraction of entries in this table that the model correctly predicts. For completeness, we also report mean task accuracy (i.c., macro- macro-averaging rather than micro-averaging across tasks) in Appendix
3 EVALUATION BASELINES
We evaluate the performance of various large language models on CaselawQA, our legal annota- tion benchmark. Among models with open weights, we select for evaluation the prominent Llama 3 (MetaAl 2024) and Qwen 2.5 YYang al. 2024) instruct model families, and the recently leased DeepSeek RI (Guo et al. 2025) reasoning model. Among commercial models, we evaluate GPT-4o 2024-08-06 (Hurst al. 2024), 03- 3-mini 2025-01-31 (OpenAl 2025b) (high reasoning ef- fort), GPT-4.5 Preview 2025-02-27 (OpenAl 2025a) and Claude 3.7 Sonnet (Anthropic (002)). We also report the performance of the constant classifier that always predicts the majority class for each

---

Published as conference paper at ICLR 2025
Llama 3.3 70B Instruct
Lawma 8B
Number of tasks
Number of tasks
40
40
20
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Task accuracy
Task accuracy
Figure 6: Distribution of task performance for Llama 3.3 70B Instruct and Lawma 8B.
CaselawQA Accurac)
1.00
CaselawOA Accuracy
0.47
O-shat Direct QA
0.43
15-shat Direct QA
0.4
0.50
Chain of Thought
0.24
0.2
0.25
0.00
0.0
Liama 3.2 18
Liama 3.2 38
Liama 3.1 8B
Liama 3.3 708 IT
Constant
LegalBERT
Mixtral 8x78 IT
SauiLM 54B
Figure General purposs models for the legal prompting outperforms few-shot QA prompting. domain perform poorly on CaselawQA.
Figure 7: For large models, Chain of Thought
task, regardless of the Court opinion being labeled. This simple classifier serves bascline for non-trivial performance and achieves an accuracy 40%.
Figure 1 shows the performance of the largest evaluated models, each with at least 70B total pa- rameters. Their accuracy ranges from 70% to 78%, with Claude 3.7 Sonnet achieving the highest performance. All large models outperform the constant classifier baseline by wide margin. We highlight in Figure the performance Llama 3.3 70B Instruct and Claude 3.7 Sonnet on ten dif- ferent tasks of particular importance empirical legal research. See Appendix for description of these highlighted tasks. We observe that performance can be modest even relatively simple tasks (e.g., < 75% accuracy for the SC Issue Area task). For more complex tasks, Claude 3.7 Sonnet may perform no better than the constant classifier baseline (e.g., for SC LC Direction, that is, identifying the ideological direction of the lower Court's decision).
More broadly, we observe large variance in models' performance across tasks. We plot in Figure left the distribution of tasks' accuracies for Llama 3.3 70B Instruct. While its average accu- racy is 71%, it exhibits a reasonably long tail of tasks for which performance is very poor. In fact, for 88 of the tasks (34% of all tasks), Ilaaa 3.3 70B Instruct does not perform significantly better than the trivial constant classifier.
Our evaluations indicate that, while large models generally exhibit non-trivial legal annotation per- formance, their performance across tasks is highly varied and can be modest even for simple tasks.
Few-shot and Chain of Thought prompting. The predominant alternative to CoT prompting is MMLU-style direct question answering ("Direct QA") prompting, where the model is expected to directly output an answer label (e.g.. "A" or "B") without first producing a reasoning chain. One benefit of Direct QA is that it is straightforward to include examples in-context. In contrast, few- shot CoT prompting requires collecting reasoning traces for each of the in-context examples.
We compare in Figure 7 the following prompting strategies: zero-shot direct QA, few-shot direct QA, and zero-shot COT. We consider the Llama 3 Instruct family of models. For the Llama models, can typically fit 15 examples in-context, since their maximum context window is 128k tokens and each task example is at most 8k tokens. We observe that for the smaller models (i.e., <3B parameters), few-shot Direct QA performs best. In contrast, for the larger models (i.e., >8B parameters), Chain of Thought is superior. In fact, the largest model evaluated few-shot, Llama 3.3 70B Instruct, does not benefit from including examples in-context.

---

Published as a conference paper at ICLR 2025
CaselawOA Accuracy
70B
405B
0.75
8B
0.50
38
18
0.25
Llama Instruct
0.00
1023
10
1025
Pre-training Compute (FLOPs)
Figure 9: CaselawQA performance as function pre-training compute for Llama models of varying scales. Performance improves monotonically with compute, with diminishing returns.
Our results indicate that few-shot prompting is not fruitful strategy to adapt large models to the legal classification tasks at hand. Since Court opinions tend to be rather lengthy, few examples may fit in-context, potentially preventing the model from improving over the zero-shot baseline.
Language models for the legal domain. We additionally evaluate two prominent LLMs adapted to the legal domain: LegalBERT (Chalkidis et al., 2020), a small BERT-style model pre-trained on legal documents, and MaulL.M 54B (Colombo et al. 2024a), a Mixtral 7x8B (Jiang et al. 2024) model adapted to the legal domain both by continual pretraining and instruction-tuning legal text.
We report their CaselawQA QA performance in Figure 8 We observe that LegalBERT performs poorly, substantially underperforming the constant classifier baseline. This is unsurprising, as LegalBERT a very small model by today's 's standards, with only 110M parameters and a context window of 512 tokens, which most of our Court opinions exceed. Regarding SauILM 54B, find that it improves upon its base mixel-Mixtral 8x7B Instruct- by 4 accuracy points. Nonetheless, its legal annotation performance is poor, and lags that of smaller, generalist models such Llama 3.1 8B Instruct.
The efficacy of scaling generalist models. crease with pre-training compute (Wei et al. 2022; MetaAl 2024 Gadre et al. [0244] Dominguez- Olmedo et al. 2024). We plot in Figure the performance the Ilama 3 Instruct family of models against their pre-training compute. Similarly to Kaplan et al. (2020), we approximate pre-training D, where N is model size and is the number tokens.
Downstream benchmark performance tends in-
compute in FLOPs as C R3
We find that accuracy improves monotonically with pre-training compute. However, observe signs diminishing returns. Therefore, further scaling pre-training compute will likely only yield moderate improvements in performance, with great financial cost (c.g., the cost of training LLama 405B is the order of tens of millions U.S. dollars). Further evidence the limitations of continuing scale generalist models is that state-of-the-art commercial models such as GPT-4.5 and Claude 3.7 Sonnet show minimal improvements over Llama 3 405B Instruct, see Figure
FINE-TUNING AND THE POWER OF SPECIALIZATION
In this section, we present a detailed analysis of how models can specialized for legal classifica- tion tasks. We start by fine-tuning five different models, ranging in size from 135M parameters to 8B parameters, on all 260 legal annotation tasks simultancously. resulting in our Lawma family mod- cls. We then perform additional fine-tuning experiments highlighting different aspects, its sample efficiency, its generalization to unseen tasks and Courts, and the effect of single task specialization.
4.1 THE LAWMA MODELS
We first fine-tune all tasks simultaneously. We fine-tune the following models: HuggingFace's SmolLM2 (Allal et al. 2025) 135M and 360M Instruct, Llama 3.2 (MetaAl 2024) 1B and 3B Instruct, and Llama 3.1 8B Instruct. We refer to these models Lawma 135M, Lawma 360M, Lawma IB, Lawma 3B, and Lawma 8B, respectively. We fine-tune on the 260 classification tasks comprising CaselawQA. The fine-tuning dataset contains total of 1.96B tokens. We fine-tune for
8

---

Published as a conference paper at ICLR 2025
Fine-tuned models
Claude 3.7 Sonnet
SC Issue Area
SC Direction
SC Disposition
SC Case Source
SC LC Disposition
Accuracy
0.8
0.75
0.7
0.7
0.6
0.90
0.25
Songer Gen. Issue
Songer Direction
Songer Treatment
Songer Case Origin
SC LC Direction
Accuracy
0.90
0.8
0.88
0.7
0.55
0.50
0.85
0.6
0.85
0.6
0.45
0.4
250
500
1000
5.00
1000
1000
250
500
1000
Train examples
Train examples
Train examples
Train examples
Train examples
Figure 10: Sample efficiency of fine-tuning Llama 3.1 8B Instruct on a single task. Hundreds of task examples are typically enough to match the zero-shot performance of Claude 3.7 Sonnet. Dashed blue line indicates the accuracy of the fine-tuned model function of the number of training ex- amples. The shaded area indicates the 95% confidence interval over the randomly sampled training examples (5 random seeds)
epochs. We find that additional epochs do not significantly improve performance. See Appendix for additional details regarding the model training.
We compare in Figure 2 the performance of the Lawma models with the largest Llama 3 model, Llama 405B, and the best-performing commercial model, Claude 3.7 Sonnet. We observe that performance of the specialized models improves with model size. Remarkably, the smallest special- ized model, Lawma 135M, is competitive with both Llama 3 405B and Claude 3.7 Sonnet, while being several orders of magnitude smaller. In addition, the largest Lawma model, Lawma 8B, sub- stantially outperforms Claude 3.7 Sonnet, achieving 87% accuracy on CaselawQA. Moreover, for the 10 legal annotation tasks highlighted in Figure 5 Lawma 8B outperforms Claude 3.7 Sonnet in 7 of them, and matches its performance on the remaining mance improvements are very large. For example, Lawma 8B outperforms Claude 3.7 Sonnet by 30 accuracy points on SC LC Direction and 20 accuracy points on Songer Case Origin.
For some of the tasks, the perfor-
We plot the distribution of task accuracies of Lawma 8B in Figure 6 right. While generalist models can exhibit a long tail of tasks with very poor performance, Lawma 8B achieves not only higher performance but also smaller variance in its task accuracies, which generally lie in the 75% 95% accuracy range. Nonetheless, the variance in accuracy across tasks remains reasonably high, and the model performs poorly for substantial number of tasks.
4.2 SAMPLE EFFICIENCY
We study how task accuracy scales as models fine-tune on more training examples. We consider the 10 tasks highlighted in Section B We fine-tune Llama 3.1 8B Instruct on each task independently, rather than on all tasks simultaneously as in the previous experiments. For each task, we fine-tune on 10, 50, 100, 250, 500, and 1000 task examples. We select task examples uniformly random, and train 5 different models corresponding to different random seeds on the examples selected for training. We therefore fine-tune and evaluate a total of 10 6- 5 300 models. We fine-tune for maximum of 20 epochs and early stop when validation loss increases for consecutive epochs.
Figure 10 shows how accuracy improves with the number training examples. We additionally plot the accuracy of Claude 3.7 Sonnet, the best-performing commercial model. Due the high cost of evaluating Claude 3.7 Sonnet, we only evaluate 100 examples per task. We observe that hundreds of training examples out of the 10 highlighted tasks. This is crucial, since labeling a few hundred data points often financially feasible for many legal scholars (Hall & Wright 2008). With relative few labelled task
enough match or beat the Claude 3.7 Sonnet baseline for

---

Published as a conference paper at ICLR 2025
0.8
Accuracy
0.7
Court of Appeals database
Supreme Court database
0.5
0.4
10-2
10-1
10
Fraction of Songer data seen
Figure 11: Training on the Court of Appeals tasks improves accuracy on Supreme Court tasks.
examples, fine-tuning reasonably small publicly available models can be competitive with state-of- the-art closed models. Moreover, accuracy continues to improve with additional examples.
4.3 GENERALIZATION TO UNSEEN DATABASES
We now investigate whether fine-tuning only on the Songer Appeals Court database allows us to generalize to the Supreme Court database. We fine-tune Llama 8B Inst for one epoch on all Songer tasks simultaneously. We plot in Figure [1] the mean accuracy for Court of Appeals tasks and Supreme Court tasks at intermediate checkpoints. As expected, performance on Court of Ap- peals tasks improves monotonically with the number of training examples seen. More interestingly, we observe that mean task accuracy for the Supreme Court also improves substantially, by up to 18.8 accuracy points at 20% of the training steps. Thereafter, performance degrades, scemingly plateauing at 11 1..3 accuracy points above the non-finetuned performance of Llama 3 8B Inst.
Our findings indicate that, since there is Supreme Court tasks, fine-tuning on the former transfers to the latter. This suggests that Lawma might be of practical use beyond the Supreme Court and Court of Appeals tasks it was trained on.
degree of overlap between Court of Appeal and
Note, however, that fine-tuning only on the Court of Appeals database results in a mean case ac- curacy of 51.6%, compared 82.4% for Lawma 8B. That is, not fine-tuning Supreme Court cases results in a 30.9 accuracy points decrease in performance. These results again highlight the importance of fine-tuning precisely on the target tasks of interest.
DISCUSSION
The cost of human annotators represents considerable bottleneck for the field of empirical legal studies. In many scientific disciplines, the advent of low-cost and flexible tools for data extraction can lead to tremendous boosts in scholarly productivity and knowledge production. For example, the falling cost of genetic sequencing led to a paradigm shift across the biological sciences, as genetic data became increasingly available in fields disparate public health and entomology (Köser et al. 2012, Ballare et al. 2019). A flexible automated feature extraction tool for legal texts holds similar potential for empirical legal studies, as a large realm of conceivable but impracticably ex- pensive research projects becomes accessible.
The generalist abilities of large language models are vital for commercial APIs, where users largely restricted to prompting. But as we show, generalist models are neither sufficiently good nor best possible for classification tasks that arise in empirical legal work. Lightly fine-tuned special purpose models achieve significantly higher accuracy from relatively few labeled examples. Label- ing a few hundred cases often financially feasible. This suggests a simple and practical strategy for solving legal classification tasks: Obtain few hundred labeled examples, fine-tune an-open weights model. and use the fine-tuned model to annotate the remaining cases.
The tasks we introduce are also interesting from benchmarking perspective. The accuracy num- bers are neither too low nor too high. The best models achieve non-trivial, but modest zero-shot performance. And even fine-tuned models don't reach intercoder agreement rates. This situation suggests that these legal classification tasks may be good test cases for future model advances. As such, we hope extend and strengthen existing evaluation efforts.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guitherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydliček, Agustín Piqueres Lajarin, Vaibhav Srivastav, et al. Smollm2: When smol goes big- big-data-centric training of small language model. arXiv preprint arXiv:2502.02237, 2025.
Anthropic. Claude 3.7 sonnet system card, 2025.
Lora Aroyo and Chris Welty. Truth is lie: Crowd truth and the seven myths of human annotation. Al Magazine, 36(1):15-24, 2015.
Kimberly M Ballare, Nathaniel S Pope. Antonio R Castilla, Sarah Cusser, Richard P Metz, and Shalene Jha. Utilizing field collected insects for next generation sequencing: Effects of sampling, storage, and dna extraction methods. Ecology and Evolution, 9(24): 13690-13755, 2019.
Rohan Bhambhoria, Samuel Dahan, Jonathan Li, and Xiaodan Zhu. Evaluating ai for law: Bridging the gap with open-source solutions. arXiv preprint arXiv:2404.1233 12349, 2024.
Caselaw Access Project. Caselaw access project. URL: https:/caae.law/
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion An- droutsopoulos. LEGAL-BERT: the muppets straight out of law school. arXiv preprint arXiv:2010.02559, 2020.
Jonathan H Choi. How to use large language models for empirical legal research. Journal of Institutional and Theoretical Economics (Forthcoming), 2023.
Jonathan H Choi and Daniel Schwarcz. Al assistance legal analysis: An empirical study, Available at SSRN 4539836, 2023.
Jonathan H Choi, Kristin E Hickman, Amy B Monahan, and Daniel Schwarcz. ChatGPT goes to law school. Legal Educ., 71:387, 2023.
Axolotl AI Cloud. Axolotl, 2024. URL btttes//ggtaab.cccm//oooolllaaad Accessed: 2024-07-17.
axolot1
Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Sofia Morgado, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, and Michael Desa. Saullm-54b saullm- 141b: Scaling up domain adaptation for the legal domain. arXiv preprint arXiv:2407. 19584, 2024a.
Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, An- dre FT Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, et al. Saullm-7h: A pioneering large language model for law. arXiv preprint arXiv:2403. 033883 2024b.
Ricardo Dominguez-Olmedo, Florian Dorner, and Moritz Hardt. Training on the test task con- founds evaluation and emergence. arXiv preprint arXiv:2407.07890 2024.
Florian E Dorner and Moritz Hardt. Don't label twice: Quantity beats quality when comparing binary classifiers on a budget. In International Conference on Machine Learning. 2024.
Christoph Engel and Richard H Mcadams. Asking GPT for the ordinary meaning of statutory terms. MPI Collective Goods Discussion Paper, (2024/5). 2024.
Lee Epstein, William Landes, and Richard Posner. The behavior of jed a theoretical and empirical study of rational choice. Harvard University Press, 2013.
Jens Frankenreiter and Eric I. Talley. Sticky charters? the surprisingly tepid embrace of officer- protecting waivers in delaware. The Surprisingly Tepid Embrace of Officer-Protectingg Waivers in Delaware (March 19, 2024). European Corporate Governance Institute- Working Paper, (762), 2024.
Jens Frankenreiter, Cathy Hwang, Yaron Nili, and Eric Talley. Cleaning corporate governance. U. Pa. I. Rev., 170:1, 2021.
11

---

Published as a conference paper at ICLR 2025
Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Kch, Rui Xin, Marianna Nezhu- rina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and downstream tasks. arXiv preprint, 08540]
arxiv. http:://aaaxiv...oaaa/////e/////
I. Gao, S Biderman, S Black, I. Golding, T' Hoppe, C Foster, J Phang, H He, A Thite, N Nabeshima, et al. The pile: an 800gb dataset of diverse text for language modeling 2020. arXiv preprint arXiv:2/01.00027, 2021.
Mary L Gray and Siddharth Suri. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books, 2019.
Morgan A Gray, Jaromir Savelka, Wesley M Oliver, and Kevin D Ashley. Empirical legal anal- ysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors. Philosophical Transactions of the Royal Society A, 382(2270): 2024.
Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Shohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe Nils Holzen- berger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. LegalBench: A collaboratively built bench- mark for measuring legal reasoning in large language models, 2023.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Mark A Hall and Ronald F Wright. Systematic content analysis of judicial opinions. Calif. L. Rev., 96:63, 2008
Moritz Hardt and Benjamin Recht. Patterns, predictions, and actions: Foundations of machine learning. Princeton University Press, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. Cuad: An expert- nlp dataset for legal contract review, arXiv preprint 103.:2103300668, 2021.
Nils Holzenberger and Benjamin Van Durme. Factoring statutory reasoning as language understand- ing challenges. arXiv preprint arXiv:2105.07903: 2021.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint urXiv:240J.04000 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
12

---

Published as a conference paper at ICLR 2025
Sayash Kapoor, Peter Henderson, and Arvind Narayanan. Promises and pitfalls of artificial intelli- gence for legal applications. arXiv preprint arXiv:2402.01656, 2024.
Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. GPT-4 passes the Bar Exam. Philosophical Trunsactions of the Royal Society A, 382222700)20232554, 2024.
Yuta Koreeda and Christopher D Manning. ContractNLI: A dataset for document-level natural language inference for contracts. arXiv preprini 1r.iv:2210.01770 2021.
Claudio U Köser, Matthew J Ellington, Edward JP Cartwright, Stephen H Gillespie, Nicholas M Brown, Mark Farrington, Matthew TG Holden, Gordon Dougan, Stephen D Bentley, Julian Parkhill, et al. Routine use of microbial whole genome sequencing diagnostic and public health microbiology. 2012.
LexisNexis. LexisNexis Launches Lexis+ AI, a Generative Al Solution with Linked Hallucination- Free Legal Citations. News Release, Oct. 25 2023.
Marco Lippi. Przemyslaw Palka, Giuseppe Contissa, Francesca Lagioia, Hans- Mans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni. CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law, 27:117-139, 2019.
Michael A. Livermore and Daniel N. Rockmore (eds.). Law Data: Computation, Text, & the Future of Legal Analysis. Santa Fe Institute Press, 2019.
Michael A Livermore, Felix Herron, and Daniel Rockmore. Language model interpretability and empirical legal studies. Virginia Public Law and Legal Theory Research Paper, (2023-69), 2023.
Andrew D Martin and Kevin M Quinn. Dynamic ideal point estimation via markov chain monte carlo for the us supreme court, 1953-1999. Political analysis, 10(2):134-133, 2002.
MetaAL, Llama 3: Advancing open foundation models, 2024. URL https://ai meta com/ blog/ / 1ama-33
John Nay, David Karamardian, Sarah B Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H Choi, and Jungo Kasai. Large language models tax attorneys: a case study legal capabilities emergence. Philosophical Transactions of the Royal Society A, 382(2270):20231153 2024.
OpenAl. Openai gpt-4.5 system card, 2025a
OpenAL. Openai 03-mini system card, 2025b.
Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadeh. Ques- tion answering for privacy policies: Combining computational and legal perspectives. arXiv preprint arXiv: 190::004110 2019.
Jaromir Savelka and Kevin Ashley, The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts. Frontiers in Artificial Intelligence, 6, 2023.
Jeffrey A Segal and Harold J Spaeth. The Supreme Court and the attitudinal model revisited. Cam- bridge University Press, 2002.
Donald R. Songer. U.S. Courts of Appeals databases
URL http://www:
jongerproject wrggus--curtees of--opealalppppp--- alaaessss html
Harold J. Spaeth, Lee Epstein, Andrew D. Martin, Jeffrey A. Segal, Theodore J. Ruger, and
Sara C. Benesh. 2023 supreme court database, version 2023 release 01, 2023. URL http:
Steven H Wang. Antoine Scardigli, Leonard Tang, Wei Chen, Dimitry Levkin, Anya Chen, Spencer Ball, Thomas Woodside, Oliver Zhang, and Dan Hendrycks. MAUD: An expert-annotated legal nip dataset for merger agreement understanding. arXiv preprint arXiv:2301.0000 2023.
13

---

Published as a conference paper at ICLR 2025
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07666 2022.
Kyle Wiggers. Harvey, which uses Al to answer legal questions, lands cash from OpenAl. TechCruch, Nov. 23 2022.
Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N Cameron Russell, al. The creation and analysis of website privacy policy corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 1330-1340, 2016.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings the the eighteenth international conference on artificial intelligence and law. pp. 159 168, 2021.
Sebastian Zimmeck, Peter Story, Daniel Smullen, Abhilasha Ravichander, Ziqi Wang, Joel R Rei- denberg, N Cameron Russell, and Norman Sadeh. Maps: Scaling privacy compliance analysis to million apps. Proc. Priv. Enhancing Tech., 2019:66, 2019.
A ADOPTION OF LARGE LANGUAGE MODELS IN THE LEGAL COMMUNITY.
The legal community has moved relatively quickly in adopting GPT models. Several startups have begun using incorporating large language models, including GPT, into legal products (Wiggers) 2022). Lexis Nexis, a major commercial provider of law-relater services, has partnered with Open AI and Anthropic to offer legal text generation (exissNexis) 2023). Legal scholars have evaluated GPT's performance on the bar exam (Katz et al. (2024) well as law school exam (Choi et al.) 2023 Choi & Schwarez (2023) examined how GPT-4 can improve student performance on law school exams. Nay et al. 2024) examined how LLMs perform on answering multiple choice ques- tions related to tax law. Gray et al.] (2024) used GPT models to extract information from cases concerning the factors that predict the constitutionality of police stops. Choi (2023) used GPT-4 to extract information concerning interpretative techniques from U.S. Supreme Court decisions. Liv- ermore al. 2023) tested the performance of GPT models for categorizing cases by issue areas and in recommending citations based on case similarity. Saavlka & Ashley (2023) evaluate the zero-shot performance of GPT-4 on a variety of semantic legal annotation tasks. Engell & Mcadams (2024) ask GPT for the ordinary meaning of statutory terms. In the area of corporate law, Frankenreiter & Talley 2024) use GPT-4 to extract information about the contents of corporate charters.
A.1 POTENTIAL APPLICATIONS OF EFFICIENT SOLUTIONS TO LEGAL CLASSIFICATION TASKS
More efficient ways solve legal classification tasks would be tremendously useful in practice. A well functioning system to automatically extract relevant features from legal texts could, in particu- lar, facilitate empirical legal study across a wide range of domains. This research could include not only social scientific study of the causes or consequences of judicial decisions, but also more tradi- tional research modalities based on doctrinal interpretation (Livermore & Rockmore 2019). There is an almost unlimited variety of features that legal scholars could study, ranging from the factors cited by judges when deciding the outcomes of property law disputes to the relationship between the party affiliation of judges and their use of different interpretative styles. With the digitization of legal texts at the U.S. state level and outside the U.S., low-cost and flexible featurization can also boost efforts show the geographic diffusion of legal concepts.
14

---

Published as a conference paper at ICLR 2025
B HIGHLIGHTED TASKS
Throughout this paper, as in Figure S we provide detailed results for ten tasks. Six of these tasks are from the SCDB, and four from the USCAD. We selected tasks that we believe are particularly relevant to the legal community and chose tasks with varying levels of complexity, ranging from rel- atively simple (e.g., determining the issue area) to more complex (e.g.. determining the ideological "direction" of the court decision).
Four tasks from the USCAD and all tasks from the SCDB were selected to form pairs, with each pair consisting of one task from the USCAD and one from the SCDB that capture similar concepts. is important note that, despite capturing broadly similar concepts, the precise formulation of the tasks might differ between the USCAD and the SCDB, making them less than perfectly comparable. In addition the four pairs, we include two tasks from the SCDB that involve determining features of the decision reviewed by the Supreme Court on the basis of the Supreme Court opinion. The following is description of the task pairs:
SC Issue Area / Songer Gen Issue: These tasks capture the case's 's issue area, requiring determination of whether the case belongs to one of several broadly defined categories, such as criminal cases or First Amendment cases. These tasks are expected be of relatively low complexity.
SC Case Source/ / Songer Case Origin:
These tasks require identifying the court or adjudi- cation body where the case originally initiated before moving up the judicial hierarchy. Like the previous pair, these tasks are expected to be of relatively low complexity.
SC Disposition / Songer Treatment: These tasks involve determining how the deciding court treated the lower court opinion it reviewed, such as whether it affirmed or reversed the opinion. We consider these tasks to be of relatively low complexity.
SC Direction / Songer Direction: These tasks involve determining the ideological 'direc- tion' of the decision, specifically whether the decision supports a "conservative" or "liberal" outcome. We consider these tasks be comparably complex.
SC LC Disposition / SC LC Direction: These tasks involve determining the disposition and ideological 'direction' of the decision reviewed by the Supreme Court. As these tasks require analyzing features of another decision based on the text of the Supreme Court de- cision, we consider these tasks to be comparably complex.
C INTERCODER AGREEMENT ANALYSIS
The Songer Appeals Court database provides intercoder agreement rates for subset of the vari- ables. These intercoder agreement rates provide valuable context for the performance of our model. Specifically, intercoder agreement gives us information about the inherent label noise in the anno- tation procedure. In particular, the intercoder agreement rate gives natural upper bound on model performance, as we cannot expect the model to perform well when the label is uncertain or subject to interpretation.
However, we cannot directly compare intercoder agreement rates with the accuracy numbers we report. The reason is that in each task we subsampled the majority class to be larger than the union of all other classes. This is a design choice we made to account for class imbalance. In this section, we map our model's 's accuracy to adjusted accuracy numbers that undo the subsampling step. This results in accuracy numbers that are commensurate with the intercoder agreement rate.
Table considers several tasks from the Appeals Court database, including the selected highlighted in various figures. Each row corresponds to one task and provides the intercoder agree- ment rate, adjusted (and unadjusted) accuracy achieved by Lawma 8B, and the fraction of samples we retained in the majority class. A fraction of 100% means that we kept all samples. The smaller the fraction the larger the majority class relative to the other classes.
The table contains several interesting insights:
The adjusted accuracy of Lawma 8B is generally within single digit percentage points of the intercoder agreement rate for easy tasks such as general issue classification (GENISS).
15

---

Published as a conference paper at ICLR 2025
Name
IC Agreement
Adj accuracy
(unadjusted)
Keep
WEIGHTEV (songer weightev)
76
78.7%
(77.2%)
28.72%
PROCEDUR (songer procedur)
78
75.2%
(73.9%)
83.08%
ORIGIN (songer_origin)
83.2
80.1 %
(77.7%)
53.13%
DIRECT2 (songe))
85.6
67.5%
(67.5%)
100.00%
DIRECTI loongerddirecto
94
80.5%
(80.5%)
100.00%
TREAT (songer_treat .treat)
95.2
91.1%
(90.1%)
71.26%
GENISS boongerggenise
97.6
93.2%
(92.9%)
84.77%
CIRCUIT (songer circuit)
100
93.2%
(93.2%)
100.00%
COMMENT (songer_comment)
100
100.0%
(91.7%)
0.13%
Table I: Intercoder agreement rates, Lawma accuracies, and fraction of the majority class retained in our sample. Rows sorted in increasing order of agreement rate.
Lawma 8B accuracy vs inter-coder agreement
100
Correlation line, con: 0.52
Adjusted accuracy (%)
90
70
95
100
Inter-coder agreement
Figure 12: Lawma task accuracy against inter-coder agreement. Lawma
Lawma 8B is surprisingly close on the two tasks with the lowest intercoder reliability, i.e., WEIGHTEV and PROCEDUR. This shows that high intercoder reliability is prereque- site for the model to perform well, i.e., close to the agreement rate.
On harder tasks, like identifying the ideological valence of a decision (DIRECTI and DI- RECT2), Lawma 8B is below the agreement rate by double digit percentage points.
Tasks with very high agreement rate (e.g., CIRCUIT and COMMENT) are not all alike. Some of them (e.g., COMMENT) correspond to task with extreme class imbalance. Here, the model reaches the agreement rate. Other tasks (e.g., CIRCUIT) have perfect agreement rate, no class imbalance, and yet Lawma is far from the agreement rate.
These findings speak the task heterogeneity and the non-trivial nature of the task suite classi- fication benchmark.
D SPECIALIZING FOR SINGLE TASKS
We now study how much accuracy we stand to gain by fine-tuning on a single task. We specialize models for each of the 10 tasks highlighted in Section B We specialize the follow models: Llama 8B Inst, Llama 3 8B Inst fine-tuned for one epoch on tasks, and Lawma 8B (i.c., Llama 8B Inst fine-tuned for three epochs on all tasks). For each task, fine-tune for maximum of 20 epochs and early stop when validation loss increases for 3 consecutive evaluation stpes, each corresponding to one tenth of epoch.
Figure 13 shows the results of specialization to single tasks. First, we observe that, for out of 10 tasks, Llama 3 8B Inst fine-tuned on all tasks for epoch (yellow) outperforms Llama 3 8B
16

---

Published as a conference paper at ICLR 2025
Accuracy
0.8
Songer Case Origin
SC LC Disposition
Songer Direction
SC Issue Area
SC Direction
SC
Songer Treatment
SC Case Source
Songer Gen. Issue
SC Disposition
LC Direction
Llama 8B Inst + specialized for single task
Llama 8B Inst 1 epoch everything
specialized for single task specialized for a single task
Lawma (3 epochs on everything)
Figure 13: Specializing Lawma 8B to individual tasks can yield small improvements in accuracy.
Inst specialized for a single task (blue). That is, there is value to fine-tuning on our entire dataset rather than overspecializing for single task. One explanation is that there is substantial cross-task overlap, and fine-tuning on the entire dataset amounts training on many more examples -even if on average these examples are less relevant.
Secondly, we observe that after fine-tuning on all 260 tasks for epoch (yellow), further specializing for single task (green) improves performance on all cases. Importantly, the latter outperforms the specialized Llama 3 8B Inst (blue) in all tasks. That is, model that is fine-tuned on everything provides a "better" foundation from which to then "overspecializing" for single task.
Thirdly fine-tuning everything for three epochs (i.e., Lawma 8B, in red) again improves over the specialized models (i.e., green). Lastly, "overspecializing" Lawma 8B for a single task results in small single digit improvements for 3 out of the 10 tasks. However, specializing Lawma 8B for most (7/10) of the tasks These results show that we don't 't leave much accuracy the table by fine-tuning single model for all tasks. This is practically quite appealing, since it obviates the need to maintain a separate model for each task. A single model suffices.
observe benefits from
E ADDITIONAL. PERFORMANCE RESULTS
E. BALANCED ACCURACY AND MACRO-F1
See Figure 14 and Figure 15 for evaluation results using mean balanced accuracy and mean macro- Fl as the evaluation metric, respectively.
E.2 RESULTS WITHOUT SUBSAMPLING THE MAJORITY CLASS
Figure presents the evaluation results when not subsampling the majority class. Models achieve very hight accuracy on many tasks simply because they correctly identify the majority class.
E.3
AVERAGE TASK ACCURACY RESULTS
Figure presents the results when using mean task accuracy across tasks as the evaluation metric.
E.4 COMPARING LLAMA 70B INSTRUCT AND GPT-4 TO THE CONSTANT CLASSIFIER
Figure 18 illustrates the difference in performance across tasks between GPT-4 and Llama 3 70B Instruct, and the majority class classifier. GPT-4 and Llama 70B Instruct perform worse than the constant classifier for dozens of tasks.
'There is small decrease in performance for SC Issue Area. This is because early stopping is performed with respect loss the validation set, hut models evaluated for accuracy on the
17

---

Published as a conference paper at ICLR 2025
All tasks
Supreme Court tasks
Court of Appeals tasks
0.75
Mean
0.25
SC Issue Area
SC Direction
SC Disposition
SC Case Sourte
SC LC Disposition
Balanced ed
0.75
0.50
0.00
Songer Gen. Issue
Songer Direction
Songer Treatment Greatmen
Songer Case Origin
SC LC Direction
Dolanced
accuracy
0.75
0.50
0.00
Majority classifier
Saul 7B Inst
Llama 70B Inst
LegaiBert
Mixtral 8x7B Inst
GPT4
Mistral 78 Inst
Llama 88 Inst
Lawma 88
Figure 14: Evaluation results using balanced accuracy as the evaluation metric.
tasks
Supreme Court tasks
Court Appeals tasks
1.00
0.50
Mean
0.25
0.00
SC Issue Area
SC Direction
SC Dispasition
SC Case Source
SC LC Disposition
0.75
Macro
0.50
0.25
0.00
Songer Gen, Issue
Songer Direction
Songer Treatment
Songer Case Origin
SC LC Direction
1.00
0.75
Macro
0.25
0.00
Majority classifier
Saul 7B Inst
Liama 70B Inst
LogalBert
Mistral
Inst
Figure 15: Evaluation results using mean macro-F1 the evaluation metric.
18

---

Published as a conference paper at ICLR 2025
Supreme
Appeals
1.20
Edea See
0.50
0.25
0.30
SC Ksue Area
SC Case Source
SC Disposition
Direction
SC LC Disposition
Sanger Con, Issue
Songer Case Origin
Songer Treatment
Songer Direction
SC LC Direction
Accurayy
Majontt
LegalBert
Saul 7B Inst
Liama 3 9B Inst
GP74
Lawma 70B
Figure 16: Evaluation results without subsampling the majority class.
All tasks
Supreme Court tasks
Court of Appeals tasks
Mcan task accuracy
1.00
6,99 5:92
0,89
0.1
0.75
0.58
0.40
0.50
3.42
0.4 0.43
0.49
0.42
0.25
0:12
0.14
2.20
6:21
5.26
0.00
Majority classifier LegalBert
Saul 7B Inst
Llama 8B Inst
GPT4
Lawma 70B
Figure 17: Evaluation results when using mean task accuracy across tasks as the evaluation metric.
Difference in Accuracy
Supreme Court
Court of Appeals
0.5
0.0
GPT4 is better
Majority classifier is better
Difference in Accuracy
Supreme Court
Court of Appeals
0.5
0.5
Llama 3 70B Inst is better
Majority classifier is better
Figure 18: Difference in zero-shot accuracy between GPT4, Llama 3 70B Instruct, and the majority classifier. Each vertical bar represents the accuracy difference on one task, sorted in ascending order.
E.5 CHAIN OF THOUGHT EVALUATION
We follow the standard methodology of eliciting CoT by appending to the prompt "Let's think step by step." Since CoT requires two orders of magnitude more compute for evaluation than the standard QA approach, only evaluate Llama 3 8B Instruct and Llama 70B Instruct. This required over
19

---

Published as a conference paper at ICLR 2025
Impact of Chain Thought (CoT)
accuracy
0,855
Accuracy
0.623
0.640
0.6
0,503
0.529
0.4
0.2
0.0
Liama 3 8B Instruct
Liama 8B Instruct + CoT
Llama 70B Instruct
Llama 3 70B Instruct + CoT
Lawma 8B
Figure 19: Performance improvements using Chain of Thought (CoT).
500 HI00 GPU hours. We observe that CoT leads to modest improvements of performance for both the 8B and 70B model, average of to accuracy points, Figure 19 Nonetheless, Lawma 8B still strongly outperforms Llama 70B, by over 20 accuracy points.
F FINE-TUNING DETAILS
Compute requirements. We fine-tune on a cluster consisting of NVIDIA H100 GPUs. Fine- tuning all tasks simultaneously required approximately 600 H100 hours for the 8B model and 1600 GPU hours for the 70B model. In total, the experiments presented the paper required approximately 8000 H100 GPU hours.
F.1 LAWMA
We fine-tuning with maximum sequence length of 8192 tokens. We use the AdamW optimizer with full precision, 31 = 0.9, & 0.95, 10-8 We use a peak learning rate of 2 10-6 We use cosine learning rate schedule, with 180 warm-up steps (approx. 4% of a full epoch) and decay to 10% of the peak learning rate. We use weight decay of 0.1. We clip gradient to 1.0 max norm. We pack samples using the axolotl library (Cloud 2024). which improves training efficiency by approximately 40%. For Lawma 8B, fine-tune Llama 3 8B Instruct for 3 epochs. We train a node H100s using DeepSpeed Zero 2, with a global batch size 56. For Lawma 70B, we fine-tune Llama 70B Instruct for epoch. We train 8 nodes of 8 H100s each using DeepSpeed Zero 3, with global batch size 64. We find that additional epochs hurt average task performance, although performance continues to improve for some of the tasks.
F.2 ADDITIONAL FINE-TUNING EXPERIMENTS
The hyperparameters are identical to those used for Lawma unless otherwise specified.
Scaling experiments. We fine-tune the Pythia and Llama 2 models with peak learning rate of 2. 10-3, which we find to be result in higher performance than peak learning rate of 10-6 For the Llama 3 models, we use a learning rate 2 10-6, which we find to be perform better than 2 10-5 We fine-tune for single epoch. We use batch size 64. We fine-tune models with their pretraining max sequence length, that is, 2k tokens for Pythia, 4k tokens for Llama 2, and 8k tokens for Llama 3. We use
up ratio of 0.03. Due to the costs associated with training the 70B model, we simply take Lawma 70B rather than re-training the model with these slightly different training hyperparameters.
20

---

Published as a conference paper at ICLR 2025
Sample efficiency and specialization We fine-tune for up 20 epochs. We evaluate the loss a separate validation set and early stop the loss increases for consecutive evaluation steps. For the sample efficiency experiments, experiments, we evaluate every 0.1 epochs. We decay the learning rate to 10% of the peak learning rate over the 20 epochs. We fine-tune with batch size of 64. For the specialization experiments, we train models both with and without learning rate warm up, and report the accuracy of the best model. We use the AdamW AndBAtsss 8-bit optimizer, allowing single H100 GPU.
evaluate at the end of every epoch. For the specialization
fine-tune the models in
Generalization We fine-tune only on the Songer Court of Appeals tasks. We fine-tune with batch size 64. We fine-tune for one epoch and checkpoint models 10, 30, 60, 100, 300, 600, 1000, 2000, and 3000 training steps. full epoch on the Songer Court of Appeal tasks corresponds to 3096 training steps.
G LIST OF ALL TASKS
Variable sc_adminaction
Question
Sample answer choices Army and Air Force Exchange Ser- vice, Atomic Energy Commission,
What is the agency involved in administrative action?
Secretary or administrative unit or per- sonnel of the U.S. Air Force No, Yes
adminaction is
Did administrative action occur context the case?
sc aa.amanncttiontttt
What is the state of the state agency associated with the ad- ministrative action?
Alabama, Alaska, American Samoa
sc.authoritydecision
What is the basis of the Supreme Court's decision?
judicial review (national level). ju- dicial review (state level), Supreme Court supervision of lower federal courts or original jurisdiction stay, petition, or motion granted, af- firmed (includes modified), reversed
sc_casedisposition
What the disposition of the case, that is, the treatment the Supreme Court accorded the court whose decision viewed?
sc_caseorigin
What is the court in which the case originated?
U.S. Court of Customs and Patent Appeals, U.S. Court of International Trade, U.S. Court of Claims, Court of Federal Claims
sc.ccseoriginntate
What is the state of the court in which the case originated? What the court whose decision the Supreme Court reviewed?
Alabama, Alaska, American Samoa
sc_casesource
U.S. Court Customs Patent Appeals, U.S. Court of International Trade, U.S. Court of Claims, Court of Federal Claims
sc_casesourcestate
What the state of the court whose decision the Supreme Court reviewed?
Alabama, Alaska, American Samoa
sc.certreason
What reason, any. does the court give granting the peti- tion for certiorari?
did not arise granted, federal court conflict, federal court conflict and to resolve important or significant question Conservative, Liberal, Unspecifiable
not
deddeissondiiection
What is the ideological direction of decision?
se ae_decisiontype
What type of decision did the court make?
opinion of the court (orally argued), per curiam (no oral argument), decrees
21

---

Published as a conference paper at ICLR 2025
se.declarationuncon
Did the Court declare unconsti- tutional an act of Congress; a state or territorial statute, regula- tion, constitutional provision; or municipal or other local or- dinance?
declaration of unconstitutionaltty Act Congress declared unconstitu- tional, State or territorial law, regula- tion, or constitutional provision uncon- stitutional
sc_issue_|
What is the issuc of the decision?
subconstitutional fair procedure: fugi- tive from justice, seff-incrimination, immunity from prosecution, cruel and unusual punishment, death penalty (cf. extra legal jury influence, death penalty)
___ssuee100
What the issue of the decision?
federal pre-emption of state legislation or regulation. cf. state regulation business. rarely involves union activ- ity. Does not involve constitutional interpretation unless the Court says it does., federal pre-emption of state leg- islation or regulation. ef. state regula- tion of business. rarely involves union activity. Docs involve constitu- tional interpretation unless the Court says does., national supremacy: pub- lic utilities (cf. federal public utilities regulation)
oc_issue_11
What the issue of the decision?
non-real property dispute between states, non-real property dispute be- tween states, boundary dispute be- tween states
issue. 12
What the issue of the decision?
federal taxation, typically under pro- visions the Internal Revenue Code, federal taxation, typically under pro- visions of the Internal Revenue Code, federal taxation of gifts, personal, busi- ness, professional expenses
sc_issue_2
What is the issue of the decision?
sex discrimination (excluding sex dis- crimination in employment). Voting Rights Act of 1965, plus amendments, juveniles (cf. rights illegitimates) libel, privacy: true and false light in- vasions of privacy, parochiaid: govern- ment aid religious schools, reli- gious requirements
sc.issue_3
What is the issue of the decision?
public schools, First Amendment, miscellancous (cf. comity: First Amendment)
sc_issue_4
What the issue of the decision?
due process: takings clause, other non-constitttional govermmental ing of property, due process: miscel- laneous (cf. loyalty oath), the residual code, due process: miscellaneous (cf. loyalty oath), the residual code Freedom of Information Act and re- lated foderal or state statutes or regu- lations, abortion: including contracep- tives, abortion: including contracep- lives
sc_issuc_5
What the issue of the decision?
SC. sc_issuc_6
What is the issue of the decision?
attorneys and governmental employ- ees' officials" fees compensation licenses, commercial speech, attor- neys (cf. commercial speech), attor- neys' and governmental employees' or officials' fees or compensation or li- censes
22

---

Published as a conference paper at ICLR 2025
sc_issue_7
What the issue of decision?
labor management disputes: right organize, union-union member dis- pute (except as pertains to union or closed shop), sabor-management dis- putes: employec discharge
sc_issuc__
What the issuc of the decision?
natural resources environmental pro- tection (cf. national supremacy: natu- ral resources, national supremacy: pol- lution), Employee Retirement Income Security Act (cf. union trust funds), election remedies: legal remedies available to injured persons or things standing sue: private implied cause of action, judicial administra- tion: review of non-final order, judicial administration: jurisdiction or author- ity of federal district courts territo- rial courts
co_issue_9
What the issue of the decision?
sc issuearea
What is the issue area of the de- cision?
Criminal Procedure, Civil Rights, First Amendment
jurisdiction
What the manner which the Court took jurisdiction?
cert, appeal, bail
sc_ lo.ecdiagrremment
Docs the court opinion mention that
Yes, No
more of the mem- bers of the court whose decision Supreme Court reviewed dis- sented?
lddispooitioon
What treatment did the court whose decision the Supreme Court reviewed accorded the de- cision of the court it reviewed? What is the ideological direction of the decision reviewed by the
stay, petition, motion granted, af- firmed, reversed
sc.ledispositiondrrettto
Conservative, Liberal, Unspecitiable
Supreme Court?
sc.partywinning
Consider that the petitioning party lost if the Supreme Court affirmed dismissed the case, denied the petition. Consider that the petitioning party won in part or in full if the Supreme Court reversed, reversed and remanded, vacated and remanded, affirmed and reversed in part, affirmed and reversed in part and remanded, or vacated the case. Did the peti- tioning win the case?
Yes. No
sc.petitioner
Who is the petitioner of the case?
attorney general of the United States, or his office, specified state board or department of education, city, town, township, village, borough govern- ment or governmental
sc.petitionerstate
What state is associated with the petitioner?
Alahama, Alaska, American Samoa
se se-precedentaleeration
Did the decision overrule one or more of the
court
Yes. No
Court's 's own precodents?
sc.respondent
Who is the respondent of the case?
attorney general of the United States, or his office. specified state board department education, city, town, township. village, or borough govern- ment governmental unit Alabama, Alaska, American Samoa
reresondentttttee
What associated with the respondent?
23

---

Published as a conference paper at ICLR 2025
se.threjjudeede
Was case heard by three-
Yes. No
judge federal district court?
songer abusedis
Did the court conclude that it should defer agency discre- tion? For example, the action was committed to agency discre- tion.
No. Yes. Mixed answer
songer.adminrev
What federal agency's decision was reviewed by the court ap- peals?
Benefits Review Board, Civil Aero- nautics Board, Civil Service Commis- sion No, Yes, Mixed answer
songer.agen.age
Did the court rule for the govern- in issue related agency acquisition of information (e.g. physical inspections, searches, subpoenas, records, etc)?
songer alj
Did the court support the de- cision an administrative law judge?
No. Yes, Mixed answer
songer altdisp
Did the court's ruling arising of alternative dispute resolution process (ADR, settlement conference, role of mediator or arbitrator, etc.) favor the appellant?
is-
No, Yes, Mixed answer
songer.amicus
Was there any amicus participa- tion before the court of appeals?
no amicus participation on either side, separate amicus brief was filed, 2 separate amicus briefs were filed not, Alabama, Alaska
songer.app-stid
What the state of the first listed state or local government agency that is an appellant?
songer.appbus
What is the total number of ap- pellants in the that fall into the category "private business and its executives")) Answer with number.
N/A
songer appell 2
This question concerns the first listed appellant.
local, neither local national, tional multi-national
nature of this litigant falls into the cate- gory "private business (including criminal enterprises)". What is the scope this business?
songer.appell.13
This question concerns the first
agriculture, mining, construction
listed appellant. The nature of this litigant falls into the cate- gory "private business (including criminal enterprises)". What cat- egory of business best describes the area of activity of this litigant which involved in this case? first
songer.appell. appp
This question concerns listed appellant. The nature of this litigant falls into the cate- gory "private business (including criminal enterprises)". specifi- cally "agriculture" What subcat- egory of business best describes this litigant?
single family farm, commercial farm, agri-business, farm other
24

---

Published as a conference paper at ICLR 2025
songer_appell
This question concerns first listed appellant. The nature of this litigant falls into the cate- gory "private organization sociation". ". What category of pri- vale associations best describes this litigant?
business, trade, professional. or union (BTPU), other
songer.appell .23
This question concerns the first listed appellant. The nature of this litigant falls into the category "private organization or asso- ciation". specifically "business, trade, professional, or union (BTPU)". What subcategory of private association best describes this litigant?
Business trade association, utili- ties co-ops, Professional association other than or medicine
songer_appell. 3.2
This question concerns the first listed appellant. The nature of this litigant falls into the category "fedcral government (including DC)". Which category of federal government agencies and activi- ties best describes litigant?
cabinet level department, courts leg- islative, agency whose word "fedcral"
songer_appell 3.3
This question concerns the first listed appellant. The nature of this litigant falls into the category "federal government (including DC)", specifically "cabinet level department". Which specific fed- eral government agency best de- scribes this litigant?
Department of Agriculture, Depart- ment of Commerce, Department of Defense (includes War Department and Navy Department)
songer appell 2
This question concerns the first listed appellant. The nature of this litigant falls into the cate- gory "sub-state government (c.g., county, local, special district)". Which category of substate gov- ernment best describes this liti- gant?
legislative, executive/adminisstrtiv, bureaucracy providing services
songer.appell .4.3
This question concerns the first listed appellant. The nature of this litigant falls into the category "sub-state government (e.g.. county, local, special dis- trict)", specifically "legislative". Which specific substate govern- ment agency hest describes this litigant?
City/county council, School Board, board of trustees for college or junior
college, Other legislative body
songer.appell 5.2
This question concerns the first listed appellant. The nature of this litigant falls into the cate- gory "state government (includes territories commonwcalths)". Which category of state govern- ment best describes this litigant?
legislative, executive/admimiitrttiv, bureaucracy providing services
songer_appell .53
This question concerns first listed appellant.
Legislature or separate house as an or-
nature this litigant falls into the cate- gory "state government (includes territories commonwealths)". specifically "legislative". Which specific state government agency best describes this litigant?
ganization, Legislative Committee or Commission, Other Legislative Unit
25

---

Published as a conference paper at ICLR 2025
songer_appell.2 _7.2
This question concerns first listed appellant. The nature of this litigant falls into the cate- gory "natural person (excludes persons named their official capacity who appear because of a role in a private organiza- tion)". What is the gender of this litigant"Use names classify the party's s sex only if there little ambiguity.
not ascertained. male indication opinion (e.g., use of masculine pro- noun), male assumed because of name
songer appell 773
This question concerns the first listed appellant. The nature of this litigant falls into the cate- gory "natural person (excludes persons named in their official capacity who appear because of role in a private organiza- tion)". What is race or ethnic identity of this litigant as identi- fied in the opinion?
not ascertained, caucasian specific in- dication in opinion, black specific in- dication in opinion
songer songer.appell. 7.4
This question concerns the first listed appellant.
ascertained, US citizen, alien
nature of this litigant falls into the cate- gory "natural person (excludes persons named in their official appear because of a role in a private organiza- tion)". What is the citizenship of this litigant as indicated in the opinion?
capacity
songer_appell _7.5
This question concerns the first listed appellant. The nature of this litigant falls into the cate- gory "natural person (excludes persons named their official capacity or who appear because of a role in a private organiza- tion)". Which of these categories best describes the income the litigant?
not ascertained, poor + wards state, presumed poor
songer_appell 88_2
This question concerns the first listed appellant. The nature of this litigant falls into the cate- gory "miscellaneous". Which of the following categories hest de- scribes the litigant?
fiduciary, executor, trustee, other, nature of the litigant not ascertained
songer_appell 883
This question concerns the first listed appellant. The nature of this litigant falls into the cat- egory "miscellaneous". specif- ically "fiduciary, executor, trustee". Which the following specific subcategories best de- scribes the litigant?
in bankruptcy institution, trustee bankruptcy individual, ex- ecutor or administrator of estate in- stitution
songer.appel2.11
the ond listed appellant. The nature of this litigant falls into the cat- egory "private business (includ- ing criminal enterprises)". What the scope of this business?
This question
local, neither local national, tional multi-nations national
26

---

Published as a conference paper at ICLR 2025
songer.appel2.113
This question concerns sec- ond listed appellant. The nature of this litigant falls into the cate- gory "private business (including criminal enterprises)". What cat- egory business best describes the area of activity of this litigant which is involved in this case?
agriculture, mining, construction
songer.appar apperr.ppel2... _.p
This question concerns the ond listed appellant.
single family larm, commercial farm, agri-business, farm other
na- ture of this litigant falls into the category "private business (including criminal enterprises)", specifically "agriculture". What subcategory business best de- serihes this litigant?
songer_appeI2 2.2
This question concerns the nature of this litigant falls into the cate- gory "private organization or as- sociation". What category of pri- vate associations best describes this litigant?
business, trade, professional, union (BTPU), other
ond listed appellant.
songer.appel2..1 2.3
This question concerns the sec- ond listed appellant. The na- of this litigant falls into category "private organiza- tion or association". specifically "business, trade, professional, or union (BTPU)". What subcate- gory of private association best describes this litigant?
Business or trade association, utili- ties co-ops, Professional association other than law medicine
songer.appel2.33
This question concerns the sec- ond listed appellant. The nature of this litigant falls into the egory "federal government (in- cluding DC)". Which category of federal government agencies and activities best describes this liti- gant?
cabinet level department, courts or leg- islative, agency whose first word "federal"
songer_appe annger.appe223311 __33
This question concerns the sec- ond listed appellant. The na- of this litigant falls into category "federal govern- ment (including DCY", specifi- cally "cabinet level department", Which specific federal govern- ment agency best descrihes this litigant?
Department of Agriculture, Depart- ment of Commerce, Department of Defense (includes War Department and Navy Department)
songer.appel2..4 ..2
This question concerns the sec- ond listed appellant. The nature of this litigant falls into the cate- gory "sub-state government (c.g., county, local, special district)". Which category of substate gov- ernment best describes this liti- gant?
legislative, executiveladministtaive,, burcaucracy providing services
27

---

Published as a conference paper at ICLR 2025
songer.appel2.433
This question concerns sec- ond listed appellant. The na- ture of this litigant falls into the category "sub-state government (c.g., county, local, special dis- trict)", specifically "legislative". Which specific substate govern- ment agency best describes this litigant?
City/county council, School Board. board of trustees for college or junior college, Other legislative body
songer.appel2.5.
This question concerns see- ond listed appellant. The nature of this litigant falls into the cate- gory "state government (includes territories commonwealths)" Which category state govern- ment best describes this litigant?
legislative, exccutivefadmiinistaaive,, bureaucracy providing services
songer_appeI2.5.
This question concerns the nature
Legislature separate house
ond listed appellant. of this litigant falls into the cate- gory "state government (includes territories & commonwealths)", specifically "legislative". Which specific state government agency best describes this litigant?
ganization, Legislative Committee or Commission, Other Legislative Unit
Lungr_appel2..1. 2.2
This question concerns the sec- ond listed appellant. The nature of this litigant falls into the cat- egory "natural person (excludes persons named in their official capacity who appear because of role in a private orgamiza- tion)". What is gender of this litigantUUse names to classify the party's 's only if there is little ambiguity.
not ascertained, male indication opinion (e.g., of masculine pro- noun), male assumed because name
appell..
This question concerns the see- ond listed appellant. The nature of this litigant falls into the cat- egory "natural person (excludes persons named
not ascertained, caucasian specific in- dication in opinion, black specific in- dication in opinion
their official capacity or who appear because of a role in a private organiza- tion)". What is the identity of this litigant identi- fied in opinion?
ethnic
songer_appel2.7..4
This question concerns the sec- ond listed appellant. The nature of this litigant falls into the egory "natural person (excludes persons named in their official capacity who appear because of role in private organiza- tion)". What of this litigant as indicated the opinion?
not ascertained, US citizen, alien
citizenship
28

---

Published as a conference paper at ICLR 2025
5onger.appel2.775
This question concerns sec- ond listed appellant. The nature of this litigant falls into the cat- egory "natural person (excludes persons named their official capacity who appear because of a role in a private organiza- tion)". Which of these categories best describes the income of the litigant?
not ascertained, poor + wards state, presumed poor
songer appel2 8.2
This question concerns the sec- ond listed appellant. The nature of this litigant falls into the cate- gory "miscellaneous". Which of following categories hest de- scrihes the litigant?
fiduciary, executor, or trustee, other, nature of the litigant not ascertained
songer_appel2.8.
This question concerns the nature
trustee in bankruptcy institution, trustee bankruptey individual, ex- ecutor or administrator estate in- stitution
ond listed appellant. of this litigant falls into the cat- egory "miscellancous", specif- ically "fiduciary, executor, or trustee". Which of the following specific subcategories best de- scribes the litigant?
aapler.appled
What is the total number of ap- pellants in the case that fall into category "the federal gov- ermment, its agencies, and offi- cialss""! Answer with number. What is the total number of ap- pellants the category "fiduciaries"" An-
N/A
songer appfiduc
N/A
case
into
swer with number.
songer applirom
What the type of district court decision or judgment appealed from (i.e., the nature of the deci- sion below in the district court)? What is the total number of ap- pellants
Trial (either jury hench trial), In- junction or denial injunction or stay of injunction, Summary judgment or denial of summary judgment N/A
songer appnnatr
case fall into the category "natural persons"?
Answer with number.
appernp
What is the total number of ap- pellants
N/A
into the category "groups and associ- ations"? Answer with number. What is the total number of ap- pellants
case
songer_appstate
N/A
case fall into the category "state govermments, their agencies, and officials""
Answer with number.
songer appsubst
What is the total number of ap- pellants in the case that fall into the category "sub-state govern- their agencies, and offi- cials"? Answer with number.
N/A
songer.attyfee
Did the court's ruling on attor- neys' fees favor the appellant?
No, Yes, Mixed answer
songer songer.bank.cgg www.bannkk... appl
the first listed appellant Yes, No
bankrupt?
songer songer.bank.co app2
Is the second listed appellant Yes, No
bankrupt?
songer_bor songer_bank.rl
the first listed respondent Yes, No bankrupt?
29

---

Published as a conference paper at ICLR 2025
songer.bank.r2
the second listed respondent bankrupt?
Yes. No
songer capric
Did the courts's s use or inteppre- tation of the arbitrary and capri- cious standard support the gov- crmment? Note that APA allows courts to overturn agency actions decmed to be arbitrary or capri- cious, abuse of discretion, otherwise not in accordance law. Overton Park emphasized this is narrow standard, and one must prove that agency's action without a rational basis. This also includes the "substantial jus- tification" doctrine.
No. Yes. Mixed answer
songer.casetypl.112
What is the specific issue in the case within the general category of "issue"?
federal offense, state offense, not de- termined whether state federal of- fense murder, rape, arson
songer.casetypl...3.
What is the specific issue in the
within the general category of "issuc"?
songer caseet.cssetyp.... 1-3-
What is the specific issue in the case within the general category of "issue"?
murder, rape, arson
2
songer konger.caselypl_11.
What is specific issue the case within the general category of "issue"?
murder, rape, arson
songer_casee songer_casetypl 2-2
What is the specific issue in the
civil rights claims by prisoners and those accused of crimes, voting rights, race discrimination, sex discrimina- tion, other civil rights
case within the general category of "issue"?
songer_casetypl 2-3-
What is the specific issue in the case within the general category of "issue""
suit for damages for false confinement, cruel and unusual pun- ishment, due process rights in prison voting rights reapportionment & dis- tricting, participation rights rights of candidates or groups to fully partici- pate in the political process; access to ballot, voting rights other (includes discrimination in voting)
false
songer_casetypl 2-3-
What is the specitic issue in the case within the general category of "issue"?
2
songer casgerl 223-
What is the specific issue in the case within the general category of "issue"?
alien petitions - (includes disputes over attempts at deportation). indian rights and law, juveniles
songer_casetypn
What is specific issue in the case within the general category of "issue"?
religion, press, commercial, speech and other expression
songer.casetypl..4
What is the specific issue in the case within the general category of "issue"?
commercial speech, libel, slander, defamation, free exercise of religion
songer.casetypl..35
What is the specitic issue in the case within the general category of "issue""
obscenity, association, federal internal security and communist control acts, loyalty oaths, security risks denial of fair hearing or notice gov- ernment employees (includes claims of terminated government workers), denial of hearing or notice non- employment context, taking clause (i.e,, denial of due process under the "taking" clause of the 5th 14th Amendments)
2
songer.casetypl.44
What is the specific issue in the case within the general category of "issue"?
30

---

Published as a conference paper at ICLR 2025
songer.casetypl_.3
What is the specific issue the case within the general category of "issue"?
abortion rights, homosexual rights where privacy claim raised, contracep- tion and other privacy claims related to marital relations sexual behavior (not 501 502)
songer.casetypl..n L6-3
What is the specific issue in the case within the general category of "issuc"?
union organizing, unfair labor prac- tices, Fair Labor Standards Act issues
songer.casetypl.74 .7-2
What is the specific issue the case within the general category of "issue"?
taxes, patents, copyright, torts, com- mercial disputes
songer.casetypl.775
What is the specific issue in the case within the general category of "issue"?
local tax, federal taxation individual income tax (includes taxes of individuals, fiduciaries, & estates), federal tax business income tax (in- cludes corporate and parnership) motor vehicle, airplane, product liabil- ity
songer casetypl 7-3.
What is specific issue the case within the general category of "issue"?
2
songer casetypl 7-3-
What is the specific issue in the case within the general category of "issue"?
contract disputes-genen (private par- ties) (includes breach of contract, dis- putes over meaning of contracts, suits for specific performance, disputes over whether contract fulfilled, claims that money owed contract) (Note: this category not used when dispute fits one of the more specific categories below), disputes over government con- tracts, insurance disputes
songer.casetypl
What is the specific issue the case within the general category of "issue"?
bankruptcy private individual (e.g.. chapter 7), bankruptcy business re- organization (e.g., chapter 11). other bankruptcy
4
songer casetypl
What is the specific issue the case within the general category of "issue"?
social security benefits (including SS disability payments), other govern- ment benefit programs (e.g.. welfare, RR retirement, veterans benefits, risk insurance, food stamps). state or local cconomic regulation
5
songer.casetypl.....
What is the specific issue in the case within the general category of "issue"?
disputes over real property (private), eminent domain and disputes with government over real property, land- lord tenant disputes
6
songer casetyyl .9-3
What is the specific issue in the case within the general category of "issue"?
miscellaneous interstate conflict, other federalism issue (only code issue if opinion explicitly discusses federalism as an important issue or if opinion ex- plicity discusses conflict of state power federal power), attorneys (dishar- ment; etc)
songer.casetyp2-eenis
What is the second general issue in the case, other than mainissuc?
criminal, civil rights, First. Amendment
songer circuit
What is the circuit of the court that decided the case?
First Circuit, Second Circuit, Third Circuit
songer civproe1
What is the most frequently cited federal rule of civil procedure in the headnotes to this case? An-
N/A
swer with number.
songer songer_civproe2
What is the second most fre- quently cited federal rule of civil procedure in the headnotes to this case? Answer with a number.
N/A
31

---

Published as a conference paper at ICLR 2025
songer.classact
case described the opin- No, Yes ion as a class action suit??
songer comment
Did the agency give proper op-
No, Yes. Mixed answer
portunity to comment?
songer concur
What the number judges
0, 1,
who concurred in the result but not in the opinion of the court?
songer.confer
Did the court conclude that confession or an incriminating statement was improperly admit- ted? Consider only incriminat- ing statements made by the de- fendant.
No, Yes, Yes, but error was harmless
songer_constl
What is the most frequently cited provision of the U.S. Constitu- tion in the headnotes to this case? one of the original arti- cles of the constitution, code the numher of the article preceeded by two zeros. If it is an amend- the constitution, code the number of the amendment (zero filled to two places) preceeded by "1". Examples: 001 Article of the original constitution, 101 Amendment, 114 14th
N/A
Amendment.
songer.const2
What is the second most fre- quently cited provision of the U.S. Constitution notes to this case? it is one of the original articles of the consti- tution, code the number of the ticle preceeded by zeros. If an amendment stitution, code the number of the amendment (zero filled to two places) preceeded by a "1", Ex- amples: 001 Article I of the original constitution, 101 1st Amendment, 114 = 14th Amend- ment.
N/A
the head-
the con-
songer_constit
Did court's conclusion about the constitutionality law or administrative action favor the appellant?
Issue discussed, The issue cussed in the opinion and the reso- lution of the issue hy the court fa- vored the respondent, The issue discussed in the opinion and the lution issue by the court favored the appellant
songer counsel
Did the court rule that the defen- dant had inadequate counsel?
No, Yes, Yes, but error was harmless
songer_counsell
What is the nature of the counsel
none (pro se), court appointed, legal aid or public defender (pro se), court appointed, legal aid public defender N/A
for the appellant?
songer.counsel2
What the nature of the counsel for the respondent?
songer.crmproel
What is the most frequently cited federal rule of criminal proce- dure in the headnotes to this case? Answer with a number.
songer.crmproe2
What is the second most fre- quently cited federal rule of crim- inal procedure in the headnotes to this case? Answer with number.
N/A
32

---

Published as a conference paper at ICLR 2025
songer_crossapp
Were there cross appeals from the decision below to the court of
No, Yes. Not ascertained
appeals that were consolidated in the present case?
songer.deathpen
Did the court conclude that the
No, Yes, Yes, but error was harmless
death penalty was improperly im-
posed? Consider only the valid- ity of the sentence, rather than whether the conviction proper.
songer.decuncon
Did the court declare any statute administrative action uncon- stitutional?
no declarations of unconstituiionaliy, act of Congress declared unconsti- tutional (facial invalidity), interpreta- tion/application of federal law invalid No, Yes, Mixed answer
songer_denovo
Did the court's use of the stan- dard of review, "de novo facts" support the government? The courts generally recognize that de novo review is impracti- cal for the bulk of agency deci- sions the substantial evidence standard helps provide middle course. Consider the de novo re- view of administrative action, not de novo review of trial court by appeals court.
songer_directl
What the idcological direction- ality of the court of appeals deci- sion?
conservative, liberal, mixed
songer_direct2
What the ideological direction- ality of the court of appeals deci- sion?
conservative, liberal, mixed
songer discover
Did the court's interpretation of rules relating to discovery or other issues related to obtaining evidence favor the appellant?
No, Yes, Mixed answer
songer_dissent
What is the number of judges majority?
0, 1.2
who dissented
songer district
From which district in the state was this case appealed?
Not applicable, Eastern, Western
songer
songer_diverse diverse
Did the court conclude that the
No, Yes, Mixed answer
parties were truly diverse?
songer_dueprro
Did the interpretation of the re- quirements of due process by the court favor the appellant?
No, Yes, Mixed answer
songer_entrap
Did the court rule that the defen- dant was the victim of illegal en- trapment?
No, Yes, Yes, but error was harmless
songer.erro
Did court's use erroncous standard support the government? That is. a some- what narrower standard than sub- stantial evidence, ignoring usual agency standards.
clearly
No, Yes, Mixed answer
songer_execord
Did the interpretation of execu- tive order administrative reg- ulation by the court favor the appellant? This does include whether or not an executive order was lawful.
No, Yes. Mixed answer
33

---

Published as a conference paper at ICLR 2025
songer_exhaust
Did the court determine that would not hear the appeal for
No, Yes. Mixed answer
one of the following reasons: a) administrative remedies had not been exhausted; b) the issue was not ripe for judicial action?
songer_fedlaw
Did the interpretation of federal statute the court favor the ap- pellant?
No, Yes, Mixed answer
songer_fedvst
Did the court rule that federal law should take precedence over state involving
No, Yes, Mixed answer
local laws in the conflict of laws (i.e, which laws or rules apply)?
songer_foreign
Did the court rule that domes- tic law (federal, state local) for-
No. Yes, Mixed answer
should take precedence eign law in case involving the conflict of laws (i.e., which laws or rules apply- foreign country vs federal, state, local)?
songer freeinfo
Did court rule favor the government when the adminis- trative action in question related the agency's providing infor- mation to those who request For example, Freedom of Infor- mation, issues of governmental confidentiality, or "government in the sunshine".
No, Yes, Mixed answer
songer_frivapp
Did the court conclude that it could not reach the merits of the case because the motion ap- peal frivolous raised only trivial issues and was therefore not suitable for appellate review?
No, Yes, Mixed answer
songer frivol
Did the court conclude that either the original case was frivolous or raised only trivial issues and therefore was not suitable for ac- tions on the merits?
No, Yes, Mixed answer
Longer_genapell
What is the listed appellant?
of the first
private business (including criminal enterprises), private organization as- sociation, federal government (includ- ing DC)
songer.genapel2
What is the ond listed appellant whose de- tailed code is not identical to the code for the first listed appellant? What is the general issue the case?
of the
private business (including criminal enterprises), private organization as- sociation, federal government (includ- ing DC)
songer geniss
criminal, civil rights, First Amendment
songer lengrss
What is the nature of the first listed respondent?
private business (including criminal enterprises), private organization sociation, federal govermment (includ- ing DC)
songer_genresp2
What is the nature of the sccond listed respondent whose detailed code is not identical the code first listed respondent?
private business (including criminal enterprises), private organization sociation, federal govermment (includ- ing DC)
34

---

Published as a conference paper at ICLR 2025
songer.genstand
Did agency articulate ap- propriate general standard? This question includes whether the agency interpreted the statute "correetly". The courts often fer here rational basis test, plain meaning, reasonable con- struction of the statute, congres- sional intent, also includes question which law applies or whether amended law vs law before amendment ap- plies.
No. Yes, Mixed answer
This issue
songer_habeas
Was case an appeal de- cision by district court on petition for habeas corpus?
no, yes, state habeas corpus (criminal), yes, federal habeas corpus (criminal)
songer.immunity
Did the court refuse reach the merits the appeal because concluded the defendant had immunity?
No, Yes, Mixed answer
songer_improper
Did the court conclude that there was improper influence the
No. Yes, Yes, but harmless
jury? For example, include jury tampering or failure to shield jury from prejudicial media accounts. Exclude prejudicial conduct by
the prosecutor.
songer_indict
Did the court rule that the indict- ment defective?
No, Yes, Yes, but error was harmless
songer_indigent
Did court rule that the defen- dant's 's rights as an indigent were violated?
No, Yes, Yes, but error was harmless
songer initiate
What party initiated the appeal?
Original plaintiff. Original defendant,
Federal agency representing plaintiff'
songer injunct
Did the court's ruling on the va-
No, Yes, Mixed answer
lidity of injunction or the de- stay of injunction favor appellant?
nial of injunction
songer insane
Did the court below err in not
No, Yes, Yes, but error was harmless
permitting insanity defense?
songer_imt law
Did the court rule favor the
No, Yes, Mixed answer
appellant on an issuc related to the interpretation of treaty or in-
ternational law?
songer_interven
Did one or more individuals
no intervenor case, intervenor ap- pellant, intervenor respondent
groups seek to formally intervene in the appeals court consideration of the case?
songer.judgdist
Did the court's ruling the abuse of discretion by the judge favor the appellant? This includes the issue of whether the
No, Yes, Mixed answer
judge actually had the authority for the action taken, include questions of discretion of administrative law judges.
does not
35

---

Published as a conference paper at ICLR 2025
songer-judrev
Did court conclude deci- sion was subject to judicial re- view? While questions of fact are subject to limited review, ques- tions of law subject full view. The problem becomes de- termining which are clear ques- tions of law or fact as they are of- ten "mixed".
No, Yes, Mixed answer
songer.jurissie jungerjurisdictton
Did the court determine that had jurisdiction to hear this case?
No, Yes, Mixed answer
songer-juryinst
Did the court conclude that the jury instructions were improper?
No. Yes, Yes, but was harmless
songer_late
Did the court refuse to decide the appeal because the appellant failed to comply with some rule relating timeliness of the ap- peal?
No, Yes, Mixed answer
mojeer_mavvotes
What the number of judges who voted in favor of the dispo- sition favored by the majority?
0, 1,2
songer_method
What the nature of the proceed- ing in the court of appeals for this casc?
decided by panel for first time (no in- dication of re-hcaring or remand), de- cided by panel after re-hearing (sccond time this case has been heard by this same panel), decided by panel after re- mand from Supreme Court No, Yes, Mixed answer
mongersmootness
Did the court conclude that an is- moot?
songer noticr
Decisions that affect life, liberty, or property must be preceded by adequate notice and opportu- nity for fair hearing. Did the
No, Yes, Mixed answer
agency give proper notice?
songer_numappel
What is the total number of ap- pellants in the case? Answer with number.
N/A
songer_numresp
What is the total number of re- spondents in the case? Answer with number.
N/A
songer.opinstal
the opinion writer identified in the opinion, or was the opinion curiam?
Signed, with reasons, curiam. with reasons, Not ascertained
ornger
What type court made the orig- inal decision?
Federal district court (single judge). 3 judge district court, State court
songer_othadme.
Did the court rule that some ev- idence, other than confession made by the defendant gal search and seizure, was inad- missibile (or did ruling on appro- priateness of evidentary hearing benefit the defendant)?
No, Yes, Yes, but error was harmless
ille-
songer othappth
Did the court refuse to the merits of the appeal because of some threshhold issue other than timeliness frivolousness that relevant on appeal but not at the original trial?
No, Yes, Mixed answer
36

---

Published as a conference paper at ICLR 2025
songer_otherim
Did court for the de- fendant on grounds other than procedural grounds? For exam- ple, right speedy trial, double jeopardy, confrontation, tivity, self defense. This includes the question of whether the de- fendant waived the right raise some claim.
No, Yes. Yes, but error was harmless
songer.othjury
Did the court conclude that jury composition or selection was invalid or that the jury was biased tampered with?
No, Yes, Yes, but error was harmless
songer songer.oththres. oththres
Did the court refuse to rule on the merits of the appeal hecause of threshhold issue other than lack of jurisdiction, standing, moot- ness, failure to state claim, ex- haustion, timeliness, immunity, frivolousness, or nonjusticiable political question?
No, Yes, Mixed answer
songer_plca
Did the court rule for the de-
No, Yes, Yes, but error was harmless
fendant on an issue related to plea bargaining? Plca bargain in- cludes all challenges plea.
songer.polquest
Did the court refuse to rule the
No, Yes, Mixed answer
merits of the case because it was considered to be nonjusticiable "political question""?
songer.post.trl
Did the court's ruling on some post-trial procedure or motion (e.g,, allocating court costs or post award relief) favor the ap- pellant? This doe not include toreys'' fees, does include motions to set aside jury ver- dict.
No, Yes, Mixed answer
songer songer.prejud
Was there prejudicial conduct prosecution?
No, Yes, Yes, but error was harmless
songer pretrial
Did the court's rulings on pre- trial procedure favor the appel- lant? This includes whether there a right jury trial, whether the case should be certi- fied as a class action, or whether prospective party has right to intervene the case, does not include rulings on motions for summary judgment.
No, Yes, Mixed answer
songer-proedis
Did the court uphold the dis- missal by district court proce- dural grounds?
No. Yes, Yes, but harmless
songer-procedle
Did the interpretation of federal rule of procedures. judicial doc- trinc, case law by the court fa- vor the appellant?
No, Yes. Mixed answer
songer.r.bus
What is the total number of re- spondents in the case that fall into the category "private busi- ness and its executives") ? Answer with a number.
N/A
37

---

Published as a conference paper at ICLR 2025
songer_r_fed
What is the total number re- spondents in the case that fall into the category "the federal gov-
N/A
emmment, its agencies, and offi- cialss"? Answer with number.
songer_r_fidue
What is the total number of re- spondents in the case that fall into
N/A
the category "fiduciaries"" An-
with number.
songer_rr_aapp
What is the total number of re- spondents in the case that fall into
N/A
the category "natural persons"?
Answer with number.
songer_r_nonp
What is the total number of re- spondents in the case that fall into the category "groups and associ- ations"? Answer with number.
N/A
songer_r_state
What is the total number of re- N/A spondents in the case that fall
into the category "state govern- ments, their agencies, and offi cials"? Answer with number. What is the state of the first listed not, Alabama, Alaska state or local government agency that a respondent?
songer_r_stid
songer.r.subst
What is the total number of re- spondents in the case that fall into the category "sub-state govern- ments, their agencies, and offi- cials"? Answer with number. Are the formally listed appellants in the the "real parties", that is, are they the parties whose real interests are most directly at stake?
N/A
songer_realapp
both 1st and 2nd listed appellants are real parties (or only appellant, and that appellant is real party), the 1st appellant is not real party, 2nd ap- pellant is not a real party
songer_realresp
Are the formally listed respon- dents in the case the "real par- ties", that is, are they parties whose real interests are most di- rectly at stake?
both 1st and 2nd listed respondents are real parties (or only one respondent, and that respondent is real party), the 1st respondent is not real party, the 2nd respondent is not a real party No, Ycs, Mixed answer
songer_reee
Did agency fail develop adequate record? For example, the court was unable to deter- used for
mine what doctrine the decision unable to deter- mine the basis of the decision. This question concerns the first listed respondent. The nature of this litigant falls into the cate- gory "private business (including criminal enterprises)". What is the scope of this business?
songer.respondl..1 _1,2
local, neither local nor national, na- tional multi-national
songer respondl
This question concerns the first listed respondent. The nature of this litigant falls into the cate- gory "private business (including criminal enterprises)". What cat- egory business best describes the area activity of this litigant which is involved in this case?
agriculture, mining, construction
38

---

Published as a conference paper at ICLR 2025
songer.respondl..
This question concerns first listed respondent. The nature of
single family farm, commercial farm, agri-business, farm other
this litigant falls into the cate- gory "private business (including criminal enterprises)", specifi- cally "agriculture". What subcat- egory of business best describes this litigant?
songer.respond 2.2
the lirst listed respondent. The nature this litigant falls into the cate- gory "private organization sociation". What category of pri- vate associations best describes this litigant?
This question
business, trade, professional, (BTPU), other
as-
songer respondl 2 3
This question concerns the first listed respondent. The litigant falls into the category "private organiza- tion or association", specifically "business, trade, professional, or union (BTPU)". What subcate- gory private association best describes this litigant?
Business or trade association, utili- ties co-ops, Professional association other than law or medicine
ture
songer_respar ponger.responl.. 3.2
This question concerns the first listed respondent. The of this litigant falls into the cat- egory "federal government (in- cluding DC)". Which category of federal government agencies and activities best describes this liti- gant?
cabinet level department, courts leg- islative, agency whose first word "federal"
songer.respondl .3.3
This question concerns the first listed respondent. The nature of this litigant falls into the category "fedcral government (including DC)", specifically "cabinet level department". Which specific fed- eral government agency best de- scribes litigant?
Department of Agriculture, Depart- ment of Commerce, Department of Defense (includes War Department and Navy Department)
songer rengerrrespond...m .42
This question concerns the first listed respondent. The nature of
legislative, executive/admiiistraiiv bureaucracy providing services
this litigant falls into the cate- gory "sub-state government (e.g.. county, local, special district)". Which category of substate gov- ermment best deseribes this liti- gant?
songer.respondl..4
This question concerns the first listed respondent. The nature of this litigant falls into the category "sub-state government (c.g., county, local, special dis- trict)", specifically "legislative". Which specific substate govern- agency best describes this litigant?
City/county council, School Board, board of trustees for college junior college, Other legislative body
39

---

Published as a conference paper at ICLR 2025
songer.respondl .52
This question concerns first listed respondent. The nature of
legislative, exccutive/admiinsttaiive,,
bureaucracy providing services
this litigant falls into the cate- gory "state government (includes territories commonwealths).. Which category state govern- ment best describes this litigant?
songer.respondl.5. _5..3
This question concerns the first listed respondent. The nature of this litigant
Legislature separate house
ganization, Legislative Committee Commission, Other Legislative Unit
into the cate- gory "state government (includes territories & commonwealths)", specifically "legislative". Which specific state government agency best describes this litigant?
songer respondl 7 2
This question concerns the first listed respondent. The nature of this litigant falls into the cat- egory "natural person (excludes persons named in their official capacity who appear because of role in private organiza- tion)". What is gender of this litigant?Use names to classify the party's s sex only if there is little ambiguity.
not ascertained, male indication in opinion (e.g.. of masculine pro- noun), male assumed because name
songer ongerrrespond! .7.3
This question concerns first listed respondent. The nature of this litigant falls into the cat- egory "natural person (excludes persons named in their official capacity or who appear because of a role in a private organiza-
not ascertained, caucasian specific in- dication in opinion, black specific in- dication in opinion
tion)". What is the identity of this litigant identi- fied in opinion?
ethnic
songer_responte _7.4
This question concerns the first listed respondent. The nature of this litigant falls into the cat- egory "natural person (excludes persons named in their official capacity who appear because of role in private organiza- tion)". What the citizenship this litigant as indicated the opinion?
not ascertained, US citizen, alien
songer _7_5
This question concerns the first listed respondent. The nature of this litigant falls into the cat- egory "natural person (excludes persons named in their official capacity who appear because of a role in a private organiza- tion)". Which of these categories best describes the income of the litigant?
not ascertained, poor + wards of state, presumed poor
songer
rewpondd ronger.
8.2
This question concerns first listed respondent. The nature of this litigant falls into the cate- gory "miscellaneous". Which of the following categories hest de- scribes the litigant?
fiduciary, executor, trustee, other. nature of the litigant not ascertained
40

---

Published as a conference paper at ICLR 2025
songer.respondl..
This question concerns first listed respondent. The nature of this litigant falls into the cat- egory "miscellancous", specif- ically "fiduciary, executor, trustee". Which the following specific subcategories best de- scribes the litigant?
trustee bankruptcy institution, trustee in bankruptcy individual, ex- ecutor or administrator of estate in- stitution
songer.respondz 1_2
This question concerns the ond listed respondent. The nature of this litigant falls into the cate- gory "private business (including criminal enterprises)". What is scope this business?
local, neither local national, tional multi-national
songer respond2 1.3
This question concerns the sec- ond listed respondent. The nature of this litigant falls into the cate- gory "private business (including criminal enterprises)". What cat- egory of business best describes the area of activity of this litigant which involved in this case?
agriculture, mining, construction
songer.respond2.1.4
This question concerns the see- ond listed respondent. The na- ture of this litigant falls into the category "private business (including criminal enterprises)", specifically "agriculture". What subcategory of business best de- scribes this litigant?
single family farm, commercial farm, agri-business, farm other
songer respond2
This question concerns see- ond listed respondent. The nature of this litigant falls into the cate- gory "private organization sociation". What category of pri- vate associations best describes this litigant?
business, trade, professional, or union (BTPU), other
songer.respond2.2. 22...
This question concerns the ond listed respondent. The na- ture of this litigant falls into the category "private orgamiza- tion association", specifically "business, trade, professional, union (BTPU)" What subcate- gory of private association best describes this litigant?
Business trade association, utili- ties co-ops, Professional association other than law or medicine
songer respond2 3.2
This question concerns the ond listed respondent. The nature of this litigant falls into the cat- egory "federal government (in- cluding DC)". Which category of federal government agencies and activities best describes this liti- gant?
cabinct level department, courts leg- islative, agency whose word "federal"
songer_respond2.3.3
This question concerns the ond listed respondent. The na- litigant falls into the category "federal govern- ment (including DC)", specifi- cally "cabinet level department", Which specific federal govern- ment agency best describes this litigant?
Department of Agriculture, Depart- ment of Commerce, Department Defense (includes War Department and Navy Department)
ture
41

---

Published as a conference paper at ICLR 2025
songer.respond2.4.
This question concerns sec- ond listed respondent. The nature of this litigant falls into the cate- gory "sub-state government (e.g., county, local, special district)", Which category substate gov- ermment best describes this liti- gant?
legislative, exccutive/admiinsttaiive,,
bureaucracy providing services
songer_respond2. 4_3
This question ond listed respondent. The na- ture of this litigant falls into the category "sub-state government (e.g.. county, local, special dis- trict)", specifically "legislative", Which specific substate govern- ment agency hest deserihes this litigant?
the
City/county council, School Board, board trustees college junior
college, Other legislative body
songer.respond2.5..
This question concerns sec- ond listed respondent. The nature of this litigant falls into the cate- gory "state government (includes territories commonwealths).. Which category state govern- ment best describes this litigant?
legislative, exccutivefadmiinsttative burcaucracy providing services
Sungerrespood22.3.
This question concerns the sec- ond listed respondent. The nature of this litigant falls into the cate- gory "state government (includes territories & commonwealths)", specifically "legislative". Which specific state government ageney best describes this litigant?
Legislature or separate house as an or- ganization, Legislative Committee Commission, Other Legislative Unit
songer_.espond2...22
This question concerns the sec- ond listed respondent. The nature of this litigant falls into the cat- egory "natural person (excludes persons named in their official capacity who appear because of role in private organiza- tion)". What is gender of this litigant?Use names to classify the party's s sex only if there is little ambiguity.
not ascertained, male indication in opinion (e.g.. of masculine pro- noun), male assumed because of name
songer respond2.7. 7.3
This question concerns sec- ond listed respondent. The nature of this litigant falls into the cat- egory "natural person (excludes persons named in their official capacity or who appear because of a role in a private organiza- tion)". What is the race or ethnic identity of this litigant identi- fied in opinion?
not ascertained, caucasian specific in- dication in opinion, black specific in- dication in opinion
songer 1nngerrrespond2.77 7,44
This question concerns the sec- ond listed respondent. The nature of this litigant falls into the cat- egory "natural person (excludes persons named in their official capacity who appear because of role in a private organiza- tion)". What is the citizenship this litigant as indicated the opinion?
not ascertained, US citizen, alien
42

---

Published as a conference paper at ICLR 2025
songer.respon22...5
This question concerns sec- ond listed respondent. The nature of this litigant falls into the cat- egory "natural person (excludes persons named their official capacity who appear because of a role in a private organiza- tion)". Which of these categories best describes the income of the litigant?
not ascertained, poor + wards state, presumed poor
songer renger.espoon2.86. 8.2
This question concerns the sec- ond listed respondent. The nature of this litigant falls into the cate- gory "miscellaneous". Which of following categories hest de- scrihes the litigant?
fiduciary, executor, or trustee, other, nature of the litigant not ascertained
songer.respond2...
This question concerns the ond listed respondent. The nature of this litigant falls into the cat- egory "miscellancous", specif- ically "fiduciary, executor, or trustee". Which of the following specific subcategories best de- scribes the litigant?
trustee in bankruptcy institution, trustee bankruptey individual, ex- ecutor or administrator estate in- stitution
songer_rtcoun
Did the court rule that the defen- dant's right counsel vio- (for some reason other inadequate counsel)?
No, Yes, Yes, but error was harmless
songer_search
Did the court below improperly No, Yes, Yes, but error was harmless rule for the prosecution
sue related to an alleged illegal search and seizure?
songer_sentence
Did the court conclude that some penalty, excluding the death penalty, was improperly im- posed?
No, Yes, Yes, but error was harmless
songer.source source
What forum hcard this case im- mediately before the case came to court appeals?
Federal district court (single judge), 3 judge district court, State court
songer_st_v.st
Did the court rule in favor of the appellant the issue of flict of laws which laws rules apply other than federal v state or foreign domestic (e.g., one state vs second state)?
No, Yes, Mixed answer
songer_standing
Did the court determine that the parties had standing?
No, Yes, Mixed answer
songer_state
In what state or territory was the case first heard?
not, Alabama, Alaska
songer_stateclaim
Did court dismiss the case be- cause of the failure of the plain-
No, Yes, Mixed answer
tiff to state a claim upon which relief could be granted?
songer.stpolicy
Did the interpretation of state or local law, executive order, ad- ministrative regulation, doctrine, rule of procedure by the court favor the appellant?
No, Yes. Mixed answer
43

---

Published as a conference paper at ICLR 2025
songer.subevid
Did the court's interpretation of the substantial evidence rule sup- port the government? For exam- plc, "such evidence as reason- able mind might accept ad- equate support conclusion" or "more than a mere scintilla". This issue is present only when the court indicates that is using this doctrine, rather than when the court merely discussing the evidence to determine whether the evidence supports the posi- tion of the appellant respon- dent.
No, Yes, Mixed answer
songer_suffic
Did the court rule that there was insufficient evidence for convic- tion?
No, Yes, Yes, but error was harmless
songer_summary
Did the court's ruling on the ap- propriateness of summary judg- ment or the denial of summary judgment favor the appellant?
No, Yes, Mixed answer
songer_timely
Did the court conclude that it could not reach the merits of the case because the litigants had not complied with some rule relating to timeliness, a filing fee, or be- cause statute of limitations had expired?
No, Yes, Mixed answer
songer
What the disposition the decision
stay, petition, motion granted, af- firmed; or affirmed and petition nied, reversed (include reversed & va- cated)
court of appeals of the court or agency below?
songer trialpro
Did the court's ruling on pro- cedure at favor appel- lant? This includes jury instruc- tions and motions for directed verdicts made during trial.
No, Yes, Mixed answer
songer songer.typeiss
two (wo issues
Are there two issues in case? What is the general category of issues discussed in the opinion of the court?
no, yes
criminal and prisoner petitions, civil government, diversity of citizenship
songer_usel
What is the most frequently cited title of the U.S. Code in the head- notes this case? Answer with number.
N/A
songer.uselsees
What is the number of the see- tion from the title of the most frequently cited title of the U.S. Code in the headnotes to this case, is, usc1? Answer with a number.
N/A
songer_use2
The most frequently cited title of
N/A
the U.S. Code the headnotes this case is uscl. What the sec- ond most frequently cited title of
this U.S. Code in the headnotes to this case? Answer with num- ber.
44

---

Published as a conference paper at ICLR 2025
songer.uss22ect
What is the number the sec- tion from the title of the second most frequently cited title of the
N/A
U.S. Code in the hcadnotes to
this case, that is, title usc2? An- swer with number.
songer_weightev
Did the factual interpretation by No, Yes, Mixed answer the court or its conclusions (c.g., regarding the weight of evidence or sufficiency of evidence) vor the appellant?
songer_whlaws
Did the court's discussion of which state's laws should control their ruling the case support the position taken by the appel- lant?
No. Yes, Mixed answer
45


Reviews:
[{"review_id": "IQxw7ARYU8", "reviewer": "Reviewer_2w3m", "summary": "This paper produces a novel dataset, CaselawQA, of broad-ranging text classification tasks, based on the annotations of two US Supreme Court and Court of Appeals datasets (for a wide range of issues, everything from whether the defense attorney was a legal aid or public defender to whether a previous precedent was being overturned) and then investigates the success of current LLMs on these tasks. They find that prompting typical LLMs produces very poor results, often below a most frequent answer baseline, prompting GPT-4 or Llama 3 70B produces moderately good results, but that much better results can be produced by using a Llama 3 8B model fine-tuned for the various tasks at issue here. The result is not only a useful new benchmark but a strong result showing that in various distant-from-the-web domains, a fine-tuned small LLM can perform much better than a state-of-the-art huge LLM.", "questions": "[none of these are important fundamental questions wrt the paper]\n\nline 129: Would not it actually be useful to show performance on LegalBench? It would be a useful test of transfer, beyond section xx, and give people a better indication of how useful Lawma would be in general for legal tasks with/without doing further fine-tuning?\n\nline 246: I think this form of data sampling is questionable. It seems particularly questionable for binary tasks, since it ends up making the two classes balanced, which is the easiest case. Historically, it has usually been argued that text classification should be done as an unbalanced task, because that is the setting that has ecological validity, and the falsely high numbers that accuracy then gives can be dealt with by not using accuracy but macro F1 for evaluation. Of course, I saw that you have all the other settings in the appendix, so I'm not unhappy, just questioning the choice of setting for the main paper. \n\nline 290: Similarly, here, using micro-averaging not macro-averaging seems questionable, but you provide the opposite in the appendix.\n\nline 411: While this graph is useful, the single axis for FLOPs is really unsystematically mixing scaling training data size and model size in a way that I think can cause as much fog as light. This isn't like a chart of Chinchilla-optimally scaled models for which we are comparing performance for different amounts of compute. Rather, if I have everything right (the paper doesn't give the details), the 6 Pythia models on the left are all trained on the same amount of data but are progressively larger models. You might conclude that models larger 410M make little difference for the benchmark here. Conversely, the 3 models 2nd, 3rd, and 4th from the right (Pythia 6.9B, Llama 2 7B, and Llama 3 8B) are all approximately the same size but differ by scaling the training the scaling data (and by instruction post-training of Llama 3 70B and Llama 3 perhaps just being better done than Llama 2). These results indicate that more pre-training data still really helps (well, at least very clearly on the Supreme Court tasks). The rightmost 2 data points return to a comparison of model size (it doesn't help much at large sizes). This suggests that around line 426 that suggesting more, better pre-training data is probably also a good source for improvements. Though I think all the big LLM companies are increasingly aware of this.\n\nFigure 8: Given that most of the curves are still point up steeply, it would be lovely to also see results for 2500 train examples. But I realize you've already expended a lot of H100 hours on this paper. On the other hand, it seems like the graphs would be closer to a logarithmic scale and the steep curve between the first two points would be avoided if you also included points for 25 main examples, and that would cost much less to add.\n\n\nTypos:\n\n111 interests > interest;\n317 were > where;\n340 unfeasible > infeasible;\n343 compared > compared to;\nLots of things need capitalization in the references, e.g. on line 546.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The strengths include:\n- Carefully and thoroughly done experimentation. Even when I disagreed with the choices they made for how to measure things in the main paper, the way I would have done things is usually available in the lengthy appendices. The paper reads as comprehensive, not rushed.\n- A valuable new benchmark dataset, with nearly all new tasks rather than just collating existing tasks.\n- The tasks of the dataset are derived from a pre-existing database by programmatic means. This is a strength since they are annotations that have been built up by lawyers and political scientists and so have ecological validity, but a weakness in that they were in a sense pre-existing rather than this being a major contribution of new labeling.\n- The paper is clearly and well written.", "weaknesses": "- The tasks of the dataset are derived from a pre-existing database by programmatic means. This is a strength since they are annotations that have been built up by lawyers and political scientists and so have ecological validity, but a weakness in that they were in a sense pre-existing rather than this being a major contribution of new labeling.\n- The paper is not very original: There is no new machine learning, there are several pre-existing benchmark legal datasets to which this adds another one, and not with a new type of task (all of them have many text classification tasks), and the central result that fine-tuning can outperform prompting has appeared in many places (as well as sometimes the opposite; it depends on various factors including model and data scale and the distance of the tasks from what appears in the pre-training and post-training data). E.g., https://www.semanticscholar.org/paper/Prompt-Engineering-or-Fine-Tuning-A-Case-Study-on-Trad-Chehab/505e4a7bedadab7f6de006c3c1e1144e272f4695, https://arxiv.org/abs/2408.01346, https://arxiv.org/abs/2402.17193, https://reglab.github.io/racialcovenants/ vs. the opposite in https://arxiv.org/abs/2309.01715 .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U67xNDMZUe", "reviewer": "Reviewer_WAb5", "summary": "The authors introduce a dataset of 260 legal text classification tasks, by extracting case text and labels from US Supreme Court and Court of Appeals. They evaluate different approaches such as prompt engineering, few-shot learning, and fine-tuning. It is not clear how much engagement the authors had with true legal experts in the creation of this dataset.", "questions": "- What process did you use to determine the relative difficulty of these tasks and whether they are comprehensive wrt the legal activities of humans? For instance the paper would benefit significantly from a legal expert's classification of the tasks by difficulty, followed by an analysis of Lawma according to those tasks\n- What does \"mixed answer\" mean in Appendix G?\n- Will you release the model weights and code?\n- Is there any transferability across tasks—i.e., if you train a model on a subset of the task, how does it perform on the held-out tasks? Do you anticipate that in the future all subtasks of legal reasoning need to be spelled out, or is there a critical mass of legal subtasks that accrue towards \"legal AGI\" ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Legal text processing has economic value but is difficult for most state of the art LLMs.\n- The authors assemble a large dataset of real-world legal documents (via querying 3rd party services), annotated with diverse question-answering tasks.\n- They show the effectiveness of fine-tuning on this dataset, over few-shot learning. They report performance of various tuning configurations.", "weaknesses": "- There is little exploration of the relative difficulty of the different types of tasks they introduce—it would be useful to know more about which tasks that GPT-4 outperformed on, and where fine-tuning had minimal vs. substantial gains.\n- The paper is awkwardly organized, such as “limitations” abruptly inserted before the main contributions are outlined.\n- “The costs and error of existing methods is the single most important bottleneck in the empirical legal studies pipeline.” (39-40) is vague and needs a citation.\n- Despite the significant contribution of the dataset and language modeling performance, there is little methodological novelty to their approach. This begs the question of why ICLR is the appropriate venue for this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YuDkDQqNE6", "reviewer": "Reviewer_VqTU", "summary": "This paper proposes a new dataset of 260 classification tasks, derived from the USDB and USCAD resources. These tasks each require fairly long contexts, and can have a large number of classes (e.g. multi-label classification). They combine these tasks into an evaluation benchmark and show results for a variety of legal and otherwise SoTA models.  They find that most models struggle, although performance follows a monotonically increasing pattern based on pre-training compute.\n\nThey then propose to fine-tune the model on these law tasks, and show that fine-tuning them (which they call Lawma) results is significantly improved performance, across a wide range of tasks.", "questions": "From Weakness 1: How did you make the prompts for the questions? \nQ2: What license does this fall under and will it be publicly released?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The data proposed seems very useful and covers a broad range\n- The analysis evaluates a wide range of models and situations\n- The fine-tuning experiments show large gains can be had with domain-specific specialization.\n- The appendix has a lot of good information about where the data came from and their inter annotator agreements", "weaknesses": "- It is somewhat unclear how the authors created these tasks, in terms of how the questions were designed. E.g. who wrote the explanation for each of the legal variables provided by USCAD? How are the authors sure that these are accurate representations of the classification?\n- [Minor] some of these tasks are pretty niceh/easy (\"What state is associated with the respondent\") but again this comes from using the variables in some schema.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5IwknLh8PZ", "reviewer": "Reviewer_LXJY", "summary": "The paper explores the use of specialized language models for legal text classification, introducing CaselawQA, a dataset with 260 legal classification tasks based on U.S. Supreme Court and Court of Appeals cases. The authors evaluate various models, including GPT-4 and a fine-tuned LLaMA model (Lawma), to test whether specialization improves performance. They find that Lawma, trained specifically on CaselawQA, outperforms GPT-4 in accuracy by up to 20 percentage points. This performance boost highlights that fine-tuned, domain-specific models can handle nuanced legal tasks more effectively than general-purpose models. Additionally, the authors demonstrate that Lawma achieves high accuracy with limited labeled data, making it a feasible and cost-effective option for empirical legal research.", "questions": "- **Generalization to Other Jurisdictions:** Can the authors provide insights or preliminary findings on how well the Lawma models might transfer to legal data from jurisdictions outside the U.S. or to other branches and tasks of law?\n- **Impact of Fine-tuning on Error Patterns:** What are some examples of typical errors Lawma makes, especially on complex tasks? Understanding these errors could help refine the approach and better inform future model improvements.\n- **Exploration of Advanced Prompting Techniques:** The paper mentions that few-shot prompting did not improve GPT-4’s performance significantly. Were other advanced prompting methods, such as chain-of-thought or other reasoning prompts, considered? This could be relevant, as legal tasks often benefit from reasoning-style responses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Clarity and Quality of Writing:** The paper is well-written, with clear explanations of the methodology and findings.\n- **Dataset Contribution (CaselawQA):** CaselawQA is a valuable addition to the legal NLP field, featuring a comprehensive set of 260 legal classification tasks that reveal the limitations of general-purpose models in handling specialized legal contexts. Its diverse structure makes CaselawQA a beneficial resource for benchmarking capabilities of LLMs and advancing specialized model development.\n- **Empirical Validation of Fine-tuning:** The study demonstrates the advantages of fine-tuning open-source models for legal tasks, with Lawma, a fine-tuned Llama model, consistently outperforming larger general-purpose LLMs like GPT-4 across various legal tasks. This empirical evidence supports the claim that domain-specific fine-tuning can yield more effective models for niche tasks, particularly within the legal domain.\n- **Sample Efficiency Insights:** By analyzing performance across different sample sizes, the study provides valuable insights into the sample efficiency of fine-tuning, showing that Lawma can achieve high accuracy with limited labeled data. This practical approach underscores the feasibility of creating specialized models even with restricted datasets, a crucial benefit for resource-limited legal research settings.", "weaknesses": "- **Limited Novelty in Contribution:** Although the paper effectively demonstrates the value of task-specific fine-tuning, the concept itself is not novel. Prior work has already shown that fine-tuned, specialized models outperform general-purpose LLMs [1, 2, 3]. This paper reinforces existing ideas rather than offering new methodologies or innovations in model specialization.\n\n- **Overlooked Generalizability:** By evaluating Lawma exclusively on CaselawQA, a U.S.-specific dataset, the study limits insights into its effectiveness across other legal systems and domains [4, 5]. Without comparisons on datasets from different jurisdictions or areas of law, it’s challenging to assess the model’s broader applicability. Expanding evaluations to diverse legal datasets could strengthen the paper’s general claims about the effectiveness of fine-tuning for legal NLP tasks.\n\n- **Narrow Focus on Text Classification:** The paper’s focus is restricted to text classification tasks, a valuable but limited subset of legal NLP. Broader experimentation on tasks such as legal text summarization for real-world low-resource data and hardware scenarios [6] could enhance the impact and relevance of the Lawma contribution for the legal community.\n\n- **English-only Focus:** The experiments are conducted solely in English, overlooking the need for legal NLP solutions in low-resource languages and multilingual contexts [7, 8]. Legal research frequently requires cross-linguistic and multilingual analysis, and models that generalize across languages would have broader utility.\n\n**Minor Presentation Weaknesses:**\n- Typographical errors (e.g., \"we conclude **that that** the performance\") need correction.\n- Figure 1 lacks clarity and could benefit from improved labeling or explanation to make differences in model performance more interpretable.\n- The Limitations section would be more effective if positioned after the Conclusion, rather than within the Introduction, to ensure that findings are fully contextualized before limitations are addressed.\n\n\n**References:**\n1. UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition. ICLR 2024.\n2. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. ACL 2023.\n3. Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification. arXiv 2024.\n4. LAWSUIT: a LArge expert-Written SUmmarization dataset of ITalian constitutional court verdicts. Artificial Intelligence and Law 2024.\n5. Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization. Artificial Intelligence and Law 2024.\n6. Semantic Self-Segmentation for Abstractive Summarization of Long Documents in Low-Resource Regimes. AAAI 2022.\n7. MultiEURLEX - A Multi-Lingual and Multi-Label Legal Document Classification Dataset for Zero-Shot Cross-Lingual Transfer. EMNLP 2021.\n8. Multi-Language Transfer Learning for Low-Resource Legal Case Summarization. Artificial Intelligence and Law 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)