SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain
Authors: Pierre Colombo
Telmo Pires
Malik Boudiaf
Rui Melo
Gabriel Hautreux
Etienne Malaboeuf
Johanne Charpentier
Dominic Culver
Michael Desa
Venue: NeurIPS 2024 poster (2024)
Keywords: NLP; Deep Learning; Law
Primary area: natural_language_processing
TL;DR: LLM for Law via domain adaptation

Abstract: In this paper, we introduce SaulLM-medium and SaulLM-large, two large language models (LLMs) families tailored for the legal sector. These models, which feature architectures of 54 billion and 140 billion parameters, respectively, are based on the Mixtral architecture. The development of SaulLM-54B and SaulLM-140B is guided by large-scale domain adaptation,  divided into strategies: (1) the exploitation of continued pretaining involving a legal corpus that includes over $400$ billion tokens, (2) the implementation of a specialized legal instruction-following protocol, and (3) the alignment of model outputs with human preferences in legal interpretations. The integration of synthetically generated data in the second and third steps enhances the models' capabilities in interpreting and processing legal texts, effectively reaching state-of-the-art performance and outperforming all previous open-source models on LegalBench Instruct. This research thoroughly explores the trade-offs involved in domain-specific adaptation at this scale, offering insights that may inform future studies on domain adaptation using strong decoder models. Building upon SaulLM-7B, this study refines the approach to produce an LLM better equipped for legal tasks and domains. Additionally, we release base, instruct and aligned versions on top of SaulLM-medium and SaulLM-large under the MIT License to facilitate reuse and collaborative research.

SaulLM-54B & SaulLM-141B:: Scaling Up Domain Adaptation for the Legal Domain
equa!
equa
Pierre Colombo
Telmo Pires Equall
Malik Boudiaf Equall
Rui Melo Equall
Equall
MICS CentraleSupelec
equal
Dominic Culver Equall
Etienne Malaboeuf
Gabriel Hautreux
Johanne Charpentier CINES
CINES
CINES
Michael Desa
Equall
Abstract
this paper, we introduce SaulLM-54B and SaulLM- 141BB, two large language models (LLMs) tailored for the legal sector. These models, which feature architec- tures of 54 billion and 141 billion parameters, respectively. are based on the Mixtral architecture. The development of Sau1LM-54B and SaulLM-1411 141B is guided by large-scale domain adaptation, divided into three strategies: (1) the exploitation of continued pretraining involving a base corpus that includes over 540 billion of legal tokens, (2) the implementation of a specialized legal instruction-folloiing protocol, and (3) the alignment of model outputs with human preferences in le- gal interpretations. The integration of synthetically generated data in the second and third steps enhances the models' capabilities interpreting and processing legal texts, effectively reaching state-of-the-art performance and outperforming previous open-source models on SeegalBench-Insttree trade-offs involved in domain-specific adaptation at this scale, offering insights that may inform future studies on domain adaptation using strong decoder models. LLM
This work explores the
Building upon SaulLM-7B, this study refines the approach to produce better equipped for legal tasks. We are releasing base, instruct, and aligned versions top of SaulLM-B 54B and BaulLM-1418 under the MIT License to facilitate reuse and collaborative research.
Introduction
LLMs have demonstrated exceptional capabilities across various domains [1, 67, 60, 76, 39, 40, 77, 7, 28), excelling in tasks such as language translation [6]. medical diagnostics [16, 11, 12), and automated code generation [4, 42, 32], among others. These achievements highlight the potential for human-like communication through large language models (LLMs). Despite the significant potential benefits, the adaptation of most recent LLMs for legal tasks has not been extensively examined, with only two recent studies cited from [18, 55, 86), and its impact on society could be substantial. Indeed, time when legal systems many countries are overburdened [71], the development of robust and gigghperforming legal LLMs could provide critical support to lawyers and judicial systems [65, 10]. However, adapting LLMs to the legal domain presents unique challenges, particularly because vast scale involved, with hundreds of billions of existing legal data available.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Previous efforts to tailor LLMs to the legal sector have encountered significant challenges [18, 55, 86]: first, a limited model scale, capped at 7/12B parameters, which is considerably smaller than the largest open-source models [7, 40]; second, training datasets restricted to no more than 30 billion tokens, significantly fewer than potentially available tokens [30, 54]. Given the importance of scale and breadth in effectively adapting LLMs to new domains, this paper aims to answer the following research question:
How much can improve the specialization of generic LLMS for legal tasks by scaling up both model and corpus size?
In this paper, we present an empirical study on the scalability and domain adaptation of LLMs in the legal sector. Relying on a corpus exceeding 500B tokens and models up to 141B parameters, our research seeks to address the gaps in the examination of legal applications. our approach is the adaptation of large-scale Mixture of Experts (MoE) models with 54B and 141B parameters, which have gained significant traction recent months [93, 22, 45, 90, 87, 61]. Formally, this study makes two principal contributions:
novel aspect of
1. Comprehensive Analysis of Domain Adaptation Strategies for Legal LLMs Domain adaptation for legal LLMs remains a challenging and somewhat underexplored area. This work advances the field by specializing each step in the process of developing modern LLMs, from continued pretraining to instruction fine-tuning and alignment, relying fresh perspective on the efficacy of each step and its value for adapting to the legal domain, potentially guiding further research in the legal domain as well as in other expert domains.
both synthetic and real data. This paper offers
2. SaulLM-54B & SaulLM-144 -M11BBB Joining Maul to form a Family of Legal LLMs under Permissive License' We specialize general-pun pprrpos, large-scale LLMs for the law. This work represents an ambitious advancement in terms of scale and leveraging the increasingly popular MoE architecture. While this architecture is widely used, its specific applications within focused domains, particularly the legal sector, are still largely unexplored. By releasing these models, we aim to foster further research in legal NLP and contribute to unlocking the full potential of LLMs.
2 Related Work
2.1 Domain Specialization For Large Language Models
The process of domain specialization for LLMs has demonstrated promising results in areas such as medicine [16], science [74], translation [6, 5], or code [66, 42, 4]. Models like SciBERT [9], PubMedBERT [75], Galactica [74] and Meditron [16] have been specifically trained on domain- related corpora to enhance their performance. Studies have identified that both the scale of the model and the size of the in-domain data are crucial for achieving strong domain adaptation [16, 66).
In the legal domain, earlier models such as LegalBERT [15], InCasel LaaseeLaaBEER LawBERT [59], and MM [18), among others, while pioneering, have been constrained by their relatively small scale and the specificity of their training data, which covers limited number of documents and jurisdictions. Our work aims to build on these efforts by deploying LLMs at an unprecedented scale, utilizing models of up to 141B parameters and a base corpus exceeding 500 billion tokens to significantly enhance the depth and breadth of legal language comprehension and generation.
2.2 Legal Domain Adaptation for Modern LLM
The field of legal domain adaptation has traditionally concentrated on refining models through pretraining on specialized corpora [15, 18, 21J. Yet, in the current paradigm, pretraining represents just one aspect of the solution, as LLMs often utilize techniques like instruction fine-tuning and alignment, employing algorithms such as DPO [63], PPO [68] or RLHF [57, 44].
Recent domain adapted models, such as SaulLM or Legal-FLAN-T5 (a closed model), have tried to improve alignment with legal instructions. However, SaulLM is a smaller model, and Legal- FLAN-TS, is based on an outdated architecture and does not leverage the extreme scale pretraining that contemporary models do. Moreover, it not being publicly available stymies progress vital for advancing research and applications in the legal sector.
Model will be made available at https: hatilbbbeattttpss//ggggggg https::://ugggnggccc../////
2

---

We believe this work pioneers a holistic approach to domain adaptation by training modern LLMs specifically for the legal domain, from pretraining to instruction fine-tuning and legal preference alignment. We demonstrate that synthetic data can be effectively utilized for alignment, advancing beyond SaulLM--7B's use solely of instruction fine-tuning. The resulting models, SaulLM-54B and SaulLM-141B lay the groundwork for further research and development, and expand access to pegf--eerfrmmance legal LLMs.
Data Collection and Corpus Construction
This section outlines our approach to assembling and refining a comprehensive legal text corpus tailored for training large language models in the legal domain.
3.1 Pretraining Corpora
The diversity legal systems worldwide, from common law to civil law traditions, presents unique challenges and opportunities [54, 35]. To address this, compiled extensive English-language corpus from various jurisdictions including the U.S., Europe, Australia, and others [3, 33], which comprises 500 billion tokens before cleaning and deduplication.
3.1.1 Legal Sources
Our base corpus combines various legal datasets [53] with newly sourced public domain documents. includes significant collections such as the FreeLaw subset and the Multil Lggal Pile, augmented with extensive web-scraped content. Table summarizes the composition and scale of our dataset.
3.1.2 Other Sources
Replay Sources. To mitigate the risk of catastrophic forgetting during model training, we reintroduced data from earlier training distributions [48, 16, 73, 72, 25, 24]. This replay strategy incorporates general data sources such as Wikipedia, StackExchange, and GitHub, and makes up approximately 2% of the total training mix. These datasets are sampled from SlimPajama [69, 19, 70].
Additionally, we included 5% of math datasets in the pretraining mix using commercially available math sources. We found this approach usefull for retaining the reasoning performance of the final model and avoiding the weaker performance observed in previous research attempts like MauILM--B
Table 1: Sources of Legal Pretraining Data
Instruction Sources. We found that incorpo- rating conversational data during pretraining is advantagcous, drawing inspiration from re- cent breakthroughs in neural machine trans- lation [6]. Studies suggest that the enhanced translation capabilities of LLMs can be at- tributed to the presence of accidental parallel data within their training corpora. Accord- ingly. we have integrated the Super Natural Instruction [83] and FLAN collection [46] into our pretraining mix, enriching the dataset with diverse instructional content.
Source Name
Tokens (B)
Freelaaw Suhset from The Pile
15
EDGAR Database
English Multil-egai Pilc
50
English EuroParl
6
GovInfo Statutes, Opinions & Codes 11
Law Stack Exchange
0.019
Comm Open Australian Legal Corpus
0.5
EU Legislation
0.315
UK Legislation
0.190
Court Transcripts
0.350
UPSTO Database
4.7
Web Data (legal)
400
Data for Model Annealing, Model annealing is primarily achieved through a methodical re- duction of the learning rate [58, 38), known as learning rate annealing.
Other
30
Total
520
In our experiments, model annealing with high-quality, domain-relevant data significantly enhanced performance. Conversely, repetitive synthetic data from initial instruction fine-tuning harmed perfor- mance. Therefore, we used the commercial portion of the LawInstruct dataset for model annealing, These tindings also align with the high percentage of math and STEM in the pretraining mix from [56].

---

which proved more effective than for instruction finetuning. We also included UltraChat [26] as generic instructions during the annealing phase.
3.1.3 Data Preprocessing
Our data processing pipeline closely follows [18]. In particular, we do:
1. Text extraction: significant fraction of the collected data is in PDF format. We used Poppler to extract the text.
Data cleaning: extraction from PDF files creates some artifacts like page and line numbers in the middle of sentences, as well as broken lines of text, non-normalized Unicode characters, etc.
Text normalization. We normalize all text using the NFKC method, available through the uniedata Python package.
Rule-based filters. We created regex rules for filtering commonly undesirable but commonly recurring patterns, like page and line numbers in the middle of the text, HTML tags, etc. Following [18], found that some of the most common 10-grams in our dataset were repeated characters and whitespace and removed them.
Perplexity filtering. Similarly to [18] used KenLM [34] model trained small subset of carefully cleaned legal data to filter documents with high perplexity. Concretely, we filtered any document whose normalized perplexity larger than 1500.
Text deduplication: we used [50] remove duplicates and near-duplicate examples from the training set. We used default parameters except for the similarity threshold, which we set to 0.5.
Finally, we packed the individual documents together to build 8192 tokens-long training examples. Documents longer than this value were chunked into several examples.
3.2 Instruction Data
Instruction fine-tuning is essential for making an LLM follow instructions and optimize the perfor- mance of pretrained models across a variety tasks [81, 84, 17, 29, 26, 82]. To this end, we employ strategic mix of general and domain- speeiniic (legal) instructions, aimed at enhancing the model's s ability precisely interpret and execute commands, with a particular focus on legal scenarios. General Instructions Our methodology for sourcing general instructions involves the integration of a diverse array of datasets, each selected to augment different aspects of the model's 's capabilities across various domains [14, 92]:
General Instruction from UltraInteract [88]: UltraInteract an extensive, high-quality dataset designed to foster complex reasoning, featuring structured instructions that include preference trees, reasoning chains, and multi-turn interaction trajectories.
2. General Instruction from Dolphin : This dataset provides additional conversational data, further broadening the model's 's exposure to diverse communication styles and contexts.
Each dataset is subjected to rigorous filtering, deduplication, and curation processes, culminating in a refined compilation of approximately 1, 0000, 000 instructions meticulously prepared for the instruction fine-tuning phase.
Legal Instruction Construction For legal instructions, we synthesize dialogues and ques- tion/answer pairs that capture key legal concepts and document types to emulate legal analy- sis. accordance with the model scale, we used
for SaulLM-54p and
Mistral-1411 Instruct for SaulLM-1415 The generation follows the recipe from [18] and begins with a three-turn sequence: (1) a user inquires about a legal document, (2) the assistant reformulates this inquiry by integrating metadata such document type issue date, and (3) the user asks for further explanation of the assistant's reasoning. The dialogue progressively deepens, with the assistant methodically unpacking the legal reasoning in response increasingly nuanced questions from the user.
3.3 Preference Data
We enhance our models' adaptability and precision by incorporating preference data from both general and legal- segalssp sources [78, 62, 52, 80]. General datasets are UlraFeedback [20] and Orea.

---

M
Welb
Mash
Muttro--nstruc MIB
Logal
Instructions
Logal MR
Roploy
Legal Continue Pretroning
Legal anhonncad Instruction Finettninng
Lagel enhanced Preference Flimeturing
Mixtral Base
SoulL Base
SaulLM-Large
Figure Domain adaptation method model for turning Mixtral SaulLM- -1-1BB. Training involves different stages: legal domain pretraining, instruction filtering, and preference filtering.
For the legal domain, employ synthetic scenarios crafted to simulate complex legal reasoning and generate accepted/rejected responses. The Mixtral-142B, 110-----222IIItuuuttt model evaluates these responses based on factual accuracy, relevance, and logical coherence, selecting the most appropriate responses as preferred outcomes (similar to [91].
Implementation Details & Evaluation Protocol
4.1 Model Selection
We used Mixtral models [40], which are built on Transformer architecture [79] enhanced with Mixture of Experts to improve computational efficiency and adaptability for handling extensive contexts. The Mixtral-54B and Mixtral-144 -11BB architecture respectively consists of 32 (resp. 56) layers, model dimension of 4096 (resp. 6144), and hidden dimension of 14, 336 (resp. 16384). Although it supports context length of up 32, 768 (resp. 65536) tokens, we continue pretraining on 8, 192 tokens. Extending the context length beyond the scope of this paper. The MoE layers in Mixtral rely on 8 experts with 2 active experts selectively based the input, efficiently utilizing computational resources and providing significant model capacity. Interestingly, Mixtral is the only model available in dual configurations (Mixtral-543 and Mixtral-141B) allowing to evaluate the scalability of our domain adaptation approaches.
At the time of the training. Mixtral was the most powerful decoder in its class, surpassing all competitors including Llama [76, 77, 89], Yi, Qwen [7], and CroissantLLM [28] terms of both cost-effectiveness and performance.
4.2 Engineering Details
Codebase Configuration Our training framework uses PyTorch. The integration of DeepSpeed [64] (level 3) and Flash attention [23] mechanisms enhances our training efficiency and scalability. We make our models available through the Huggingface hub [85].
Compute Infrastructure The computational backbone for the continuous pretraining phase of our project consists 384 AMD MI250 GPUs. We can reach 40% GPU utilization with our implementation. For instruction fine-tuning and preference optimization, we rely on 64 AMD MI250 GPUs. Evaluation protocols are executed on a single node AMD MI250 GPU. Synthetic Data Generation For synthetic data generation, we used vLLM on node of NVIDIA- A100, primarily due to limited support libraries on MI250*.
4.3 Training Details
The model training process is divided into three distinct phases: continued pretraining, instruction finetuning (IFT), and preference alignment using domain-specific optimization (DPO). A full schema of the pipeline can be found in Figure 1.
Continued Pretraining For continued pretraining, we use the AdamW [41, 47, 8] optimizer with
'VLLM is not supported on AMD-M1250 and HuggingFace's text-generation inference had few bugs that prevented its use.

---

hyperparameters 51 = 0.99, 82 = 0.90, and learning rate of 2 x 10 10-5. We utilize entsoyy loss function optimize model predictions. The training protocol sets gradient accumulation to 4, with tailored batch sizes of 8 for SaulLM-54B and 4 for SaulLM- 141BB, optimizing both GPU utilization and training efficiency.
Instruction Fine-Tuning (IFT) IFT uses the Adam W optimizer (learning rate of 1 x 10-9), reinitial- ized to reset training states and maintain training stability. We limit this phase to a single training epoch, as our experiments suggest this maximizes performance gains.
Preference Training Using DPO We adjust the learning rate of the Adam optimizer to 1 x 10-6 during DPO. Our choice of DPO over IPO [13], KTO [27] or ORPO [37] was based on preliminary experiments.
4.4 Evaluation Protocol
[18], which refined the prompts from LegalBench [31] by climinating distracting elements and specifying response format enhance precision. Like LegalBench, it evaluates LLMs across six types of legal reasoning: issuc-spotting, rule-recall, rule-application, rule-conclusion, interpretation, and rhetorical understanding. Grounded in American legal frameworks but applicable globally. these categories provide a comprehensive evaluation of the models' legal reasoning capabilities. This structured approach helps in accurately assessing and guiding the enhancement of LLMs within and beyond American legal contexts. We follow previous work and rely on balanced accuracy as the primary metric across all tasks.
BesaaBeenchIInstuee We rely on LegalBench-Instee
Massive Multitask Language Understanding (MMLU) Previous works utilize MMLU [36], a recooy-rec benchmark, focusing on its legal-specific tasks in international law, professional law, and jurisprudence, with 120, 1500, and 110 examples respectively. These tasks are crucial for assessing our models' understanding and application of complex legal concepts, highlighting their proficiency in nuanced legal environments.
Choice of Baseline & Model Naming For our evaluation, aim for direct, aaples-to-apples comparison of models. is important to note that not all competing models are open source, and detailed information on their alignment procedures and instruction fine-tuning processes is not avail- able. This lack of transparency complicates the establishment of fully equitable baseline comparisons. In what follows, and the Instruct variants of Mixtral-543 54B, and Mixtral- Matrall141BB 141BB, Additionally, SaulLM-500 BAMLLLM-5BBIIIA IFT is the IFT version built on SaulLM- 54------B-beeee and SaulLM- i um for the DPO version based on SaulLM-54B-IPH SaulLM-larg large is the final version DPO and IFT based on SaulLM-14H 111BB
Al's GPT-4 of 10 May 2024, Meta's 's ama (the Instruct variant)
Experimental Results
5.1
Global Results
Figure 2 presents the results of SaulLM-large and SaulLM-medium on BegalBencc--Instruct, from which we make several observations:
80.0
77.5
75.4%
75.0
Our domain adaptation strategy is achiev- ing strong results. SaulLM-medium outper- forms Mixtral-54B, and similar findings are observed with SaulIM-large compared to dixtral-141B Interestingly, domain adapta- tion at both the instruction tuning stage and preference alignment enables smaller mod- els to outperform larger ones, such as GPT-4 and LLama3-70B. These results validate ap- proach and demonstrate that specializing the en- tire pipeline (i.e., from continued pretraining to preference alignment) is a promising direction for improving performance legal-related tasks.
725
70.0
67.5
82.2%
85.0
625
60.0
GP73.5
Mistra- MBBl
Liamaz
Sad-medim fd
Bauk-large
Model
Figure 2: Overall Results. Comparison of MargLM-large and Mamliu-medium with exist- ing models.

---

Mean Belanoed Accunao
0.75
0.70
Rhwlore
0.06
0.80
Miami
0.55
0.00
Mitere-aa
Mitreid 44 10P0
Sta S44
BautMM-ma Mend
Interrestaiin
Mintos IP
R9
Concustion
0.4
0.5
0.8
0.7
0.8
0.9
Model
Figure 3: Global Analysis. Role of continued pretraining.
Figure Category Analysis: Role of continue pretraining.
Domain adaptation works across scales and MoE models. The results from SaulLM-med um and MaLlLMM-arge confirm previous findings from [18] and confirm that domain adaptation is effective across different scales, including on MoE models. Interestingly, most of the data collected for this work comes from public sources, which were likely seen during the pretraining of the base models.
Path Towards Stronger Models. The results LLama3-70B and the scalability of our methods suggest that applying the same approach to the LLama3-70BB base model could lead to even better performance than our best model, SaulLM- 141B. It is worth noting that SaulLM- 141B has only 44B active parameters making it appealing for efficient serving.
Table Quantifying the role of DPO. We report the percentage of tasks where the difference in performance (A) between SaulLM and SaulLM- 54B- IFT is posi- tive (resp. negative).
Table Quantifying the role of scaling.
We report the percentage of tasks where the difference in performance (A) between SaulLM-medius and SaulLM- large is pos- itive (resp. negative).
Category A > 0
A < 0
Category
A 2 0
A <
0
Conclusion
37.5%
62.5%
Conclusion
18.2 18.2%
81.8%
Interpretation
65.1%
34.9%
Interpretation
23.7%
76.3%
Rhetoric
44.4%
55.6%
Rules
25.0%
75.0%
Rules 60.0%
40.0%
Issue
100.0%
Issue
100.0%
Rhetoric
100.0%
5.2 How much does continued pretraining help for the legal domain?
Previous works on domain adaptation via continued pretraining primarily focused on instruction finetuning [16, 18, 55]. In Figure 3 and Figure 4, report the performance of Mixtral-5AA trained with the IFT mix described in subsection 3.2 (Mixtral-54-IF") and subsequently aligned using the DPO dataset Satxxtra--4-IIF--D-- to the instruct version of Mixtral Mixtrall544B as outlined [40].
described in subsection 3.3. We also compare these results
Continuing pretraining significantly enhances model performance in the legal domain, benefit- ing both the IFT and DPO stages. From Figure 3, we observe that both IFT and DPO benefit from notable improvement (approximately +7%). Interestingly, this improvement is consistent across all five categories, as shown Figure 4.
Adding legal data to the IFT and DPO datasets improves the model's legal capabilities. By comparing the performance of Mixtral-64 IFT+DDO and Mixtral-54, we observe that mix used for IFT and DPO enhanced with legal data leads to stronger legal performance than that of Mixtral-54, which does not publicly describe the alignment methods used. This result aligns with findings reported in [55, 18].

---

5.3 How Much Does Legal Preference Alignment Help?
Our findings from Figure 3 indicate that alignment significantly improves the results. In particular, DPO improvement is mostly consistent across tasks and categories. As shown in Table 2, the aligmment version SaulLM-mediwu demonstrates significant improvements over the IFT version across most tasks, including conclusion, rhetoric, rules, and issue tasks. We observe, however, drop in performance in some interpretation tasks. Upon closer examination, we found that this decline is often due to the model becoming more verbose, which causes the evaluation process to fail in correctly parsing the answers i.c., this issue primarily arises from a benchmark limitation. Addressing model verbosity and the challenge of more reliable benchmarks is beyond the scope of this work, but it a well-known problem identified many concurrent studies [29]. Enhancing the evaluation process is one of the key improvements we plan to contribute to in the future.
1.30
Weig foed Mean Power Node: 1361,97
SaulLM-base-- 140B
Normalized Log Loss
1.18
Saull S-ule--bbse--55 54B
2250
Mean Power Node
2000
1.07
1730
6Pe
1500
0:00
0.97
1000
750
0.80
500
0.0
0.2
0.4
0.6
0.8
1.0
250
750
1000
1250
1750
Normalized Epochs
Job ID
Figure 5: Continue Pretraining Analysis. Training loss for MoulLM-141B-be and BaulLM-54B-bor over normalized epochs.
Figure 6: Energy Consumption Analysis. Mean Power per Node for training jobs on the ADASTRA Supercomputer.
5.4 Can We Achieve Further Improvements by Continuing Pretraining?
Training longer can potentially improve the results. Figure 5 illustrates the normalized log loss over normalized epochs for both model sizes, SaulLM- base and SaulLM- base.. The figure presents both the raw and smoothed loss curves, which exhibit clear downward trend throughout the training period, with no indication saturation.
This observation suggests that continuing the pretraining process beyond the current SaulLM SaulLM-base can lead to further improvements. The consistent decrease loss implies that the models have not yet reached their full potential and that additional pretraining could enhance their performance further, which is consistent with findings from other works [60, 51].
5.5 How Much Does Scaling Help?
Table 3 quantifies the impact of scaling the model and compares the performance between SaulLM-medium and SaulLM-large
The main takeaway is that scaling generally improves overall results, but we also observe inverse scaling some legal tasks [49, 43]. Unsurprisingy. for the majority of tasks across all categories, increasing the model size leads to improvements, but for tasks involving conclusion, interpretation, and rules, observe proportion of tasks (20%) that follow inverse scaling laws.
5.6 Energy Consumption
The training was conducted on Adastra, ranked 3rd in the Green500 since November 20225, as one of the world's most efficient machines in terms of performance per watt.
Experiments for training SaulLM were performed between February 20th and May 15th. Energy consumption for each job was meticulously tracked, and we calculated and displayed the average

---

power used per node for each job involved in this training in Figure 6%. The mean power usage ranged from 600W 2500W, reflecting the varying utilization of the GPUs. Each node contains four MI250X GPUs, which have theoretical Thermal Design Power (TDP) of 560W. This configuration explains the maximum consumption of 2500W during high-intensity GPU usage.
Overall, the project involves over 160, 000 hours of MI250 for debugging, continued pretraining, instruction finetuning and preference alignment. The total energy consumed was 65, 480.4K04KK Wh. Consumption is significantly lower than the typical energy requirements for full LLM training, showing that continued pretraining an effective strategy for specializing in new LLMs while optimizing energy efficiency.
Conclusion & Limitations
6.1
Conclusion
This study released two new legal L.MSs under the MIT license: SaulLM-54B and SaulLM-141B. They leverage the Mixtral architecture and continued pretraining on large legal corpus. Our findings show significant advancements in processing and understanding complex legal documents. Through continued pretraining, instruction fine-tuning, and preference alignment using domain- specific optimization, we have demonstrated substantial improvements compared to GPT-4, Llama3 and original Mixtral models measured LegalBeechhInsttee
6.2 Limitations
Our experiments suggest that the instruction finetuning and alignment processes utilized by Mixtral
Mearra-Innsien Mixtral-Insteen rely on proprietary datasets and significant computational resources that are not readily available in open-source frameworks. Although both SaulLM-54B and SaulLM-141B achieve stronger perfor- mances than Llama3 and Mixtral Instruct legal benchmarks, found that they are slightly weaker at following generic instructions.
Instruct and Llama3 are advanced and challenging to replicate. These processes often
Looking forward, we aim continue our work enhancing the SaulLM family, particularly focusing on integrating Llama3 and improving the alignment procedure. Our goal is to improve the alignment of these models with legal tasks, refining their ability process and understand legal language with even greater accuracy and relevance. This future work will strive to address the current limitations by developing more robust methods for instruction finetuning and alignment that accessible the broader research community.
Acknowledgments and Disclosure of Funding
This research was supported by computing grants from Adastra and Jeanzay. We extend our special thanks to Michael Robert, head of CINES, for his invaluable support and confidence in our work.
Our models have been trained on ADASTRA, with minor experimentation conducted on Jeanzay. The utilization of HPC resources made possible through the Jeanzay grants 101838, 103256, and 103298, as well as the Adastra grants C1615122, CAD14770, and CAD15031.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXiv:2303.08774 2023.
[2] Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, selection for language models. arXiv preprint wwXiv:2402.16827 2024.
survey on data
'The high number of jobs (over 1800) for this project arise from (i) the technical difficulties in efficiently using AMD GPUs, and (ii) numerous hardware failures when scaling up the number of nodes.

---

[3] Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preofiuc-Piete Pietro, and Vasileios Lampos. Pre- dicting judicial decisions of the european court of human rights: A natural language processing perspective. PeerJ computer science, 2:e93, 2016.
[4] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Car- los Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301 03988, 2023.
[5] Duarte M Alves, Nuno M Guerreiro, João Alves, José Pombal, Ricardo Rei, José GC de Souza, Pierre Colombo, and André FT Martins. Steering large language models for machine translation with finetuning and in-context learning. arXiv preprint arXiv:2310. 13448, 2023.
[6] Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters. Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. Tower: An open multilingual large language model for translation-related tasks. 2024.
[7] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023.
[8] Anas Barakat and Pascal Bianchi. Convergence and dynamical behavior of the adam algorithm for nonconvex stochastic optimization. SIAM Journal on Optimization, 31(1):244 22,44,-277,, 2021.
[9] Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv: 10676, 2019.
[10] Rohan Bhambhoria, Samuel Dahan, Jonathan Li, and Xiaodan Zhu. Evaluating ai for law: Bridging the gap with open-source solutions. arXiv preprint arXiv:2404.12349 2024.
[11] Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, et al. Biomedlm: A 2.7 b parameter language model trained on biomedical text. arXiv preprint arXiv:2403. 18421, 2024.
[12] Elliot Bolton, Betty Xiong, Vijaytha Muralidharan, Joel Schamroth, Vivek Muralidharan, Christopher D Manning, and Roxana Daneshjou. Assessing the potential of mid-sized language models for clinical qa. arXiv preprint
2024.
[13] Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al. Human aligmment of large language models through online preference optimisation. arXiv preprins arXiv:2403.08635 2024.
[14] Yihan Cao, Yanbin Kang, and Lichao Sun. Instruction mining: High-quality instruction data selection for large language models. arXiv preprint arXiv:2307.06290 2023.
[15] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androut- sopoulos. Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559, 2020.
[16] Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079. 2023.
[17] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et Scaling cestruction-finetumne language models. arXiv preprint arXiv:2210.11416 2022.
10

---

[18] Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and Michael Desa. Saullm-7b: A pioneering large language model for law. 2024.
[19] Together Computer. Redpajama: an open dataset for training large language models, 2023.
[20] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedbck:: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023.
[21] Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306. /6092, 2023.
[22] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization mixture-of-eeperts language models. arXiv preprint arXiv:2401 00000.00066..... 2024.
[23] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficiene exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16352 2022.
[24] Maxime Darrin, Pablo Piantanida, and Pierre Colombo. Rainproof: An umbrella to shield text generators from out-of-distribution data. arXiv preprint arXiv:2212.09171, 2022.
[25] Maxime Darrin, Guillaume Staerman, Eduardo Dadalto Câmara Gomes, Jackie CK Cheung, Pablo Piantanida, and Pierre Colombo. Unsupervised layer-wise score aggregation for textual ood detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38,
pages 17880-17888, 2024.
[26] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. Xrviv preprint arXiv:2305. 14233, 2023.
[27] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01366, 2024.
[28] Manuel Faysse, Patrick Fernandes, Nuno Guerreiro, António Loison, Duarte Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro Martins, et al. Croissantllm: A truly bilingual french-english language model. arXiv preprint arXiv:2400.0003 00780 2024.
[29] Manuel Faysse, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Revisiting instruction fine-tuned model evaluation to guide industrial applications. In Proceedings of the 2023 Confer- ence on Empirical Methods Naturul Language Processing. Association for Computational Linguistics, 2023.
[30] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. 2020.
[31] Neel Guha, Daniel E Ho, Julian Nyarko, and Christopher Ré. Legalbench: Prototyping collaborative benchmark for legal reasoning. arXiv preprint arXiv:2209.06/20 2022.
[32] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseck-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401. 2024.
[33] Asier Satiérrez-Fandiio, Jordi Ammenool-Estapl Aitor Gonzalez-Agirre, and Marta Villegas. Spanish legalese language model and corpora. arXiv preprint arXiv:2/10. 12201, 2021.
[34] Kenneth Heafield. KenLM: Faster and smaller language model queries. In Chris Callison-Burde Philipp Koehn, Christof Monz, and Omar F. Zaidan, editors, Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland, July 201 A Association for Computational Linguistics.

---

[35] Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. Pile of law: Learning responsible data filtering from the law and 256gb open- source legal dataset. Advances in Neural Information Processing Systems, 35:29217-222344 2022.
[36] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.088 0000000,, 2020.
[37] Jiwoo Hong, Noah Lee, and James Thome. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691, 2024.
[38] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang. Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06399 2024.
[39] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[40] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141i 6980, 2014.
[42] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.066 061606 2023.
[43] Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. Advances in Neural Information Processing Systems, 36, 2024.
[44] Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv preprint arXiv:2305.18438 2023.
[45] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401. 15947, 2024.
[46] Shayne Longpre, Le Hou, To Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688 2023.
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 2017.
[48] Michael McCloskey and Neal J. Cohen. Catastrophic interference connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pages 109-165, Academic Press, 1989.
[49] Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Fuan MeLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. Inverse scaling: When bigger isn't better. arXiv preprint arXiv:2306.0944 2023.
[50] Chenghao Mou, Chris Ha, Kenneth Enevoldsen, and Peiyuan Liu. Chenghaomoohhex..dddup Reference snapshot, September 2023.
12

---

[51] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus. Sampo Pyysalo. Thomas Wolf, and Colin Raffel. Scaling dota-constr language models. Advances in Neural Information Processing Systems, 36, 2024.
[52] Rémi Munos, Michal Valko. Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint arXiv:23/ 2200886 2023.
[53] Joel Niklaus and Daniele Giofré. Budgetlongformer:: Can we cheaply pretrain a sota legal language model from scratch? arXiv preprint arXiv:2211.17133, 2022.
[54] Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel E. Ho. Multilegalpile:
A 689gb multilingual legal corpus. 2023.
[55] Joel Niklaus, Lucia Zheng, Arya D McCarthy, Christopher Hahn, Brian M Rosen, Peter Henderson, Daniel E Ho, Garrett Honke, Percy Liang, and Christopher Manning. Flawn-t5: An empirical examination of effective instruction-tuning data mixtures for legal reasoning. arXiv preprint wrxiv:2444.022277 2024.
[56] Aitor Ormazabal, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Deyu Fu, Dono- van Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, et al. Reka core, flash, and edge: A series of powerful multimodal language models. arXiv preprint arXiv:2404 12387, 2024.
[57] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[58] Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subrama- nian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, et al. Nemotron-4 15b technical report. arXiv preprint arXiv:2402.16819 2024.
[59] Shounak Paul, Arpan Mandal, Pawan Goyal, and Saptarshi Ghosh. Pre-trained language models for the legal domain: A case study on indian law. In Proceedings of 19th International Conference on Artificial Intelligence and Law ICAIL 2023, 2023.
[60] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 16, 2023.
[61] Macicj Pióro. Kamil Ciebiera, Krystian Król, Jan Ludziejewski, and Sebastian Jaszczur. Moe-mamba: Efficient selective state space models with mixture of experts. arXiv preprint arXiv:2401.04088 2024.
[62] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv: 18290, 2023.
[63] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[64] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505-3506, 2020.
[65] Richard M Re and Alicia Solow-Niederman. Developing artificially intelligent justice, Ston. Tech. L. Rev., 22:242, 2019.
13

---

[66] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
[67] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 1ob--parameter open- access multilingual language model. arXiv preprint arXtv:2211.05100, 2022.
[68] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[69] Zhiqiang Shen, Tianhua Tao, Liqun Ma. Willie Neiswanger, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. Slimpajama-de Understanding data combinations for Ilm training. arXiv preprint arXiv:2309. 10818, 2023.
[70] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan
Dey. Slimpajama: A 627b token cleaned and deduplicated version of redpajama, 2023.
[71] Tania Sourdin, Bin Li, and Donna Marie McNamara. Court innovations and access to justice in times crisis. Health policy and technology, 9(4):447-453, 2020.
[72] Guillaume Staerman, Pavlo Mozharovskyi, Pierre Colombo, Stéphan Clémençon, and Florence d' AlAch Buc. A pseudo-metric between probability distributions based on depth-trimmed regions. arXiv preprint arXiv:2 12711, 2021.
[73] Jingyuan Sun, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Distill and replay for con- tinual language learning. In Proceedings of the 28th international conference on computational linguistics, pages 5569-3579 2020.
[74] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:221 21,22..,,00,...1 2022.
[75] Robert Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Fine-tuning large neural language models for biomedical natural language processing. Patterns, 4(4), 2023.
[76] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 3r31, 2023.
[77] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[78] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023.
[79] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
14

---

[80] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. btcps://githun 2020.
com con/huggingface/ton
[81] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources, arXiv preprint ar arXiv:2306.00 04.751 2023.
[82] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions, 2023.
[83] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. anpueartaturllintttuttonsss Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705, 2022.
[84] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
[85] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transform- ers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 2019.
[86] Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, and Furu Wei. Adapt-and-distile Developing small, fast and effective pretrained language models for domains. arXiv preprint arXiv:2106. 13474, 2021.
[87] Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via ootture-of-experts adapters. arXiv preprint 24Xi:2443.11144 2024.
[88] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan. Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees, 2024.
[89] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.022 023308 2024.
[90] Rongyu Zhang, Yulin Luo, Jiaming Liu, Huanrui Yang. Zhen Dong, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Yuan Du, et al. Efficient deweahter mixture-of-exppe -f-eeeerts with uncertainty-awar feature-wise linear modulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 16812-16820, 2024.
[91] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging alm-de- with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[92] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment, arXiv preprint arXiv:2305. 2023.
[93] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe Le, James Laudon, et al. Mixurre-ff-expert with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
Justification:
We propose a new domain adaptation recipe.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
Justification:
See limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
16

---

Justification:
No theoretical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer:
Justification:
Dataset and model are available with a list of hyperparameter in training
details.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer:
Justification:
Models are released under a commercial license and data are described.
Guidelines:
The answer NA means that the paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
wwwsssssslllloooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iips.co.o wttpccccpppccceeeeccecccccccccct wwwic//ggg////e/eeeoeeoeco /Coodeuumbie
/ cettubmmsssiii ssisnnsiinn seiinnyyicc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
Justification:
We relied on previous works that are cited.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
Justification:
We follow previous works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:
Justification:
See model details and energy analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: //wuurrrrs....// cc ht/ppbliii/tthhcccocueeeeeeeeell
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
Justification:
We fully comply.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
Justification:
See limitation.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:
Justification: MIT-liscence
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
20

---

Answer:
Justification:
We introduce new models that will be publically released under MIT liscence.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer:
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer:
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
A Energy Analysis
Figure 7, Figure 9, Figure 8, Figure 10 show different analysis of energy consumption while developing and training SaulLM-5A 54B and Mau1LM-141B.
21

---

100
400
-00
Number of Jobs
283
-30
200
CPU
160
20
100
20
25
Flansort Time (llenrc))
Kiumhar of Niviee
Figure Energy Analysis. GPU Load
Figure 8: Energy Analysis. Number of Jobs vs Number of Nodes.
Elapsed Time for Different Numbers of Nodes.
160
Log Scaled Total Power Consumption
Nodia
Consumee Energy (Joilee)
60
Nodes 28 NiG Nodes
80
-5
-50
Numbel of Nocss
$ Nodes
Nolles
40
80
Nodes
Nodea @0 Nodes
Flanser
Figure 9: Energy Analysis. Consumed En- engy Elapsed Time for Different Numbers of Nodes.
Figure 10: Energy Analysis. Log-Scaled Total Power Consumption GPU Load for Different Numbers of Nodes.
Rulves
Rudls
Rewforis
Rewisric
Model
Sau--lrge
Saul medium
GPT35
Mixrral 54
Saul-lagee
Midral
prototion
protation
Saul medium
Mitrai-5i
anclusion
0.5
0.6
077
0.8
0.9
0.6
0.8
0.9
Belanced Accunecv
Balanced Accumacv
Category LegalBench Instruct.
Figure 11: www.eennn---nnuttttt.mt
Analysis
Figure
12:
Category
Analysis
CeggalBench-Instaae
B Further Results on LegalBench
B.1 Per Category Analysis
B.2 Per Task Analysis
22

---

Cauppe Sh1r Iracs codoo
Dea- Bealdit)
Figure 13: Per Task Analysis: Quantifying the role of DPO w.r.t. instruction fine-tuning only.
23

---

Ganagy
conciusion_ _eriiusioon_ekksss
rhetoric. taaaks
rules, tasks
Interpretation. _taskon_tsksssss
issue tasks
12
86
Difference Perforance (Sau--arge- Saul-mdium)
Figure 14: Per Task Analysis: Quantifying the role of DPO w.r.t. instruction fine-tuning only.
24


Reviews:
[{"review_id": "sKBKap8RCG", "reviewer": "Reviewer_iZW5", "summary": "The paper introduces SaulLM-54B and SaulLM-141B, two large language models specifically designed for the legal sector. These models utilize the Mixtral architecture and are developed through extensive domain adaptation strategies, including continued pretraining on a large legal corpus, instruction fine-tuning, and preference alignment. The models incorporate synthetic data to enhance their capabilities in legal text interpretation and achieve state-of-the-art performance on LegalBench-Instruct. The study emphasizes the scalability of domain-specific adaptation and the potential benefits of larger model sizes for legal applications.", "questions": "On page 7, it is mentioned: \"The results of LLama3-70B and the scalability of our methods suggest that applying the same approach to the LLama3-70B base model could lead to even better performance than our best model, SaulLM-141B.\" Which figure's results support this illustration?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. SaulLM models are the first open-source legal LLMs with a larger model size, utilizing continual pretraining, instruction tuning, and RLHF.\n\n2. The release of these models under the MIT License facilitates reuse and collaborative research in legal NLP.", "weaknesses": "1. The techniques used, such as continual pretraining, instruction tuning, and preference alignment, are not novel compared to existing studies like SaulLM and LLMs in other domains.\n\n2. There are issues with the license, privacy, and quality of data. The paper should explicitly illustrate the license of each used data source. The quality of synthetic legal instruction tuning data and preference data is uncertain.\n\n3. The evaluation is not comprehensive. There is no ablation study to illustrate the effectiveness of continual pretraining, such as comparing the performance of Mixtral-54B and SaulLM-54B-base. It would be better to include other LLMs in the legal domain as baselines.\n\n4. In section 5.2, the explanation of \"How much does continued pretraining help for the legal domain\" is confusing. The continual pretraining previously referred to unsupervised continual pretraining with a next-token generation task on large-scale legal data, but in this section, it seems to mean IFT+preference alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CkkZ33X9Hp", "reviewer": "Reviewer_LHKy", "summary": "- The paper introduce two LLMs (at different sizes) specialized for law. These models have been adapted for law through continued pretraining, specialized legal instruction following, and a “legal alignment” process\n- The paper studies the tradeoffs of domain adaptation at this scale and presents results for these models.", "questions": "- The LegalBench website asks users of the benchmark to cite a collection of papers, not merely the original LegalBench paper. Since LegalBench-Instruct looks like a small modification to LegalBench, do you plan to cite that original collection of papers as well? Also the LegalBench citation seems wrong, and not to the actual LegalBench paper (https://arxiv.org/abs/2308.11462)\n- Do the authors plan on releasing these datasets?\n- Do the authors plan on releasing intermediate checkpoints for model training?]\n- Once concern in relying on LegalBench-Instruct is that the prompt format chosen favors Mixtral models over other types of models (e.g., Llama-3, GPT, etc.). Do the authors have a sense of how sensitive the Saul models are to the prompt format?\n- It would be very nice to see some examples of generations in the Appendix!\n\nI’ll increase my score if the additional details are added to the paper.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- To the best of my knowledge, this is the first study of domain adaptation for legal LLMs at this scale (model size and training corpora size). This makes it an extremely valuable contribution.\n- The paper is extremely clear and well-written. \n- The experiments seem thorough and well motivated.", "weaknesses": "- The paper is sparse on some key details related to the datasets used to train the model.\n    - For the preference data, what are the synthetic scenarios designed? How were they constructed? What were the Mixtral prompts used to evaluate them?\n    - For the legal instruction data, what were the legal documents initially chosen? What did the conversations look like?\n- Examples of both instruction and preference data in the Appendix would be very helpful to readers. \n- The ablations are extremely helpful but also difficult to read, in part because they’re mixed in with baselines. It would be nice if the authors could present a table showing the progression of performance between: Mixtral-54B, + continued pretraining on the legal corpora, + IFT, + legal alignment. I think some–but not all–of these in Figure 6?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vMq0ccHD2z", "reviewer": "Reviewer_pgvT", "summary": "The paper introduces two large language models specialized in the legal domain with instruction-following capabilities. These models are an extension of previous work, particularly Colombo et al.'s “SaulLM-7B: A Pioneering Large Language Model for Law,” by scaling up the corpus size and the number of trainable model parameters. The model parameters are updated through continued pretraining, instruction-tuning, and alignment (RLHF, DPO, or similar methods). Additionally, the authors gather and preprocess the largest legal dataset for pretraining from various sources. They examine the impact of both model size and dataset size on the effectiveness of adapting large language models (LLMs) to the legal domain.", "questions": "- What is “Score” on the y-axis of Figs. 3, 4, and 5? \n\n- Can the authors justify the usage of only “balanced accuracy” as the primary metric used in the results associated with performance (non-energy analysis related)? \n\n- Can the authors explain the process of preparing synthetic data for both instruction and preference fine-tuning in detail? Considering that generating synthetic data can lead to low-quality samples, did you quantitatively measure the quality of the generated data? Also, do you define any acceptance criteria for the generated data to avoid adding low-quality samples to the training corpus? \n\n- Considering that you included data from various jurisdictions, do you evaluate the performance of the model across different jurisdictions? \n\n- Based on the results presented in Figure 6, do you have any explanation or assumption as to why Mixtral-IFT is performing worse than Mixtral? \n\n- In the conclusion, you claim that “we have demonstrated substantial improvements compared to GPT-4.” The mean balanced average of GPT-4 and the proposed model are pretty close. Did you perform any significance tests to show that the difference is statistically significant? \n\n- The authors clearly indicated that the experiments are limited due to the proprietary nature of the datasets used to train Llama and Mixtral. Have the authors looked into open-source and not just open-weight models? For example, Olmo: https://arxiv.org/abs/2402.00838.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces the largest domain-specific legal language model to date, with parameters ranging from approximately 12B to 54B-141B. It curated the largest legal corpus for pretraining. \n\n- Their results demonstrate the superiority of their models compared to state-of-the-art general-purpose language models (both open-source and closed), such as Llama 3 and GPT-4. They perform a detailed analysis of their models' performance. \n\n- The process of training the model with all the parameters is discussed in detail. Additionally, the preprocessing of the curated corpus is clearly explained. They also depict their improvements through graphs and tables and study the impact of various factors on their results under separate headers. \n\n- They introduce the state-of-the-art legal-domain language model, which can provide critical support to lawyers and judicial systems. They also release their model, enabling future research in the field.", "weaknesses": "- This paper misses ablations on the three training methods (with and without continued pretraining), ± instruct tuning, and ± alignment training. I would have liked to see where the greatest improvement in results is obtained. For example, if we do not include continued pretraining at all but only perform instruct tuning and alignment. \n\n- The study does not include detailed results about the compared methods (Fig. 17 only compares Saul med with Saul large). Can these results be shown for the top-performing models for each training method and models of different architecture (e.g., GPT-4 and Saul large)? In the results section, they do not compare their proposed model with existing domain-specific language models like Saul-7B and legal-Flan-T5. \n\n- The paper does not explain the process of generating synthetic data clearly. A detailed explanation of what prompts have been used to generate the data, what model has been utilized, and which generation parameters have been used is required. \n\n- The paper discusses adding a math portion to the pretraining dataset to enhance reasoning capabilities, but the source of the data is unclear, and they don’t provide any supporting experiments to back up their claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b8OG3dQz1A", "reviewer": "Reviewer_jvqv", "summary": "The paper reports on two new legal-specific LLMs based on Mixtral. The primary contribution is to train larger law LLMs than previously reported, using (1) an extensive dataset of legal materials, and (2) a variety of best practices in pre-training. The results indicate incremental but significant improvements on legal benchmarks.", "questions": "(See above.)", "limitations": "Yes, the paper is well scoped to describe what it does and doesn't do.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper reports notable improvments in the state of the art in using LLMs for legal tasks. Most of these advantages are due to greater model size, more extensive training data for legal specialization, and engineering decisions. The results show that these techniques continue to bear fruit and have not yet reached a wall of fundamental architectural limitations. I found section 5 particularly illuminating, because it successful isolates some of the individual technical improvements to analyze which of them contribute to the performance improvement.", "weaknesses": "The paper's strength is also its weakness. It generates very little generalizable knowledge for understanding legal tasks, only a series of engineering improvements. There are some reasons to think that transformer-based LLMs will not be able to carry out successful human-level work without fundamental improvements in the underlying model architecture. But the research still gets us closer to the frontiers of what current LLM achitectures can do, and it is possible that similar continued improvements are all that are necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents SaulLM-54B and SaulLM-141B, two large language models tailored for legal applications. Built on the Mixtral architecture, these models are refined using comprehensive domain adaptation techniques, such as continued pretraining on a vast legal text corpus, instruction fine-tuning, preference alignment and synthetic data generation. The models achieve state-of-the-art results on the LegalBench-Instruct benchmark. The research highlights the effectiveness of scaling domain-specific adaptations and the advantages of larger model sizes in enhancing legal language processing.

The main strength of the paper is in demonstrating strong domain adaptation results for a specific and important domain, for large-size LLMs. The main limitation, when it comes to a scientific conference, is the use of well-known and well-explored techniques (even though they have not been shown to provide such strong results for domain adaptaiton of LLMs of this size). Yet, the engineering result is strong and novel. The divergence in reviewer scores is mostly with respect to how they weigh the obvious pros and cons mentioned above. I am leaning towards accept.
Decision: Accept (poster)