SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain

Authors: pierre  colombo
equall  mics -centralesupelec
malik  boudiaf
rui  melo
dominic  culver
etienne  malaboeuf
gabriel  hautreux
johanne  charpentier
michael  desa equall
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:12:42

Abstract: In this paper, we introduce SaulLM-medium and SaulLM-large, two large language models (LLMs) families tailored for the legal sector. These models, which feature architectures of 54 billion and 140 billion parameters, respectively, are based on the Mixtral architecture. The development of SaulLM-54B and SaulLM-140B is guided by large-scale domain adaptation,  divided into strategies: (1) the exploitation of continued pretaining involving a legal corpus that includes over $400$ billion tokens, (2) the implementation of a specialized legal instruction-following protocol, and (3) the alignment of model outputs with human preferences in legal interpretations. The integration of synthetically generated data in the second and third steps enhances the models' capabilities in interpreting and processing legal texts, effectively reaching state-of-the-art performance and outperforming all previous open-source models on LegalBench Instruct. This research thoroughly explores the trade-offs involved in domain-specific adaptation at this scale, offering insights that may inform future studies on domain adaptation using strong decoder models. Building upon SaulLM-7B, this study refines the approach to produce an LLM better equipped for legal tasks and domains. Additionally, we release base, instruct and aligned versions on top of SaulLM-medium and SaulLM-large under the MIT License to facilitate reuse and collaborative research.

Main paper: 
 introduction: 
 LLMs have demonstrated exceptional capabilities across various domains [1, 67, 60, 76, 39, 40, 77, 7, 28] , excelling in tasks such as language translation [6] , medical diagnostics [16, 11, 12] , and automated code generation [4, 42, 32] , among others. These achievements highlight the potential for human-like communication through large language models (LLMs). Despite the significant potential benefits, the adaptation of most recent LLMs for legal tasks has not been extensively examined, with only two recent studies cited from [18, 55, 86] , and its impact on society could be substantial. Indeed, at a time when legal systems in many countries are overburdened [71] , the development of robust and high-performing legal LLMs could provide critical support to lawyers and judicial systems [65, 10] . However, adapting LLMs to the legal domain presents unique challenges, particularly because of the vast scale involved, with hundreds of billions of existing legal data available.Previous efforts to tailor LLMs to the legal sector have encountered significant challenges [18, 55, 86] : first, a limited model scale, capped at 7/12B parameters, which is considerably smaller than the largest open-source models [7, 40] ; second, training datasets restricted to no more than 30 billion tokens, significantly fewer than potentially available tokens [30, 54] . Given the importance of scale and breadth in effectively adapting LLMs to new domains, this paper aims to answer the following research question:How much can we improve the specialization of generic LLMs for legal tasks by scaling up both model and corpus size?In this paper, we present an empirical study on the scalability and domain adaptation of LLMs in the legal sector. Relying on a corpus exceeding 500B tokens and models up to 141B parameters, our research seeks to address the gaps in the examination of legal applications. A novel aspect of our approach is the adaptation of large-scale Mixture of Experts (MoE) models with 54B and 141B parameters, which have gained significant traction in recent months [93, 22, 45, 90, 87, 61] . Formally, this study makes two principal contributions:1. Comprehensive Analysis of Domain Adaptation Strategies for Legal LLMs Domain adaptation for legal LLMs remains a challenging and somewhat underexplored area. This work advances the field by specializing each step in the process of developing modern LLMs, from continued pretraining to instruction fine-tuning and alignment, relying on both synthetic and real data. This paper offers a fresh perspective on the efficacy of each step and its value for adapting to the legal domain, potentially guiding further research in the legal domain as well as in other expert domains.2. SaulLM-54B & SaulLM-141B: Joining SaulLM-7b to form a Family of Legal LLMs under Permissive License 1 We specialize general-purpose, large-scale LLMs for the law. This work represents an ambitious advancement in terms of scale and leveraging the increasingly popular MoE architecture. While this architecture is widely used, its specific applications within focused domains, particularly the legal sector, are still largely unexplored. By releasing these models, we aim to foster further research in legal NLP and contribute to unlocking the full potential of LLMs.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer:Justification: We propose a new domain adaptation recipe.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
domain specialization for large language models: 
 The process of domain specialization for LLMs has demonstrated promising results in areas such as medicine [16] , science [74] , translation [6, 5] , or code [66, 42, 4] . Models like SciBERT [9] , PubMedBERT [75] , Galactica [74] and Meditron [16] have been specifically trained on domainrelated corpora to enhance their performance. Studies have identified that both the scale of the model and the size of the in-domain data are crucial for achieving strong domain adaptation [16, 66] .In the legal domain, earlier models such as LegalBERT [15] , InCaseLawBERT [59] , and SaulLM-7B [18] , among others, while pioneering, have been constrained by their relatively small scale and the specificity of their training data, which covers a limited number of documents and jurisdictions. Our work aims to build on these efforts by deploying LLMs at an unprecedented scale, utilizing models of up to 141B parameters and a base corpus exceeding 500 billion tokens to significantly enhance the depth and breadth of legal language comprehension and generation.The field of legal domain adaptation has traditionally concentrated on refining models through pretraining on specialized corpora [15, 18, 21] . Yet, in the current paradigm, pretraining represents just one aspect of the solution, as LLMs often utilize techniques like instruction fine-tuning and alignment, employing algorithms such as DPO [63] , PPO [68] or RLHF [57, 44] .Recent domain-adapted models, such as SaulLM or Legal-FLAN-T5 (a closed model), have tried to improve alignment with legal instructions. However, SaulLM is a smaller model, and Legal-FLAN-T5, is based on an outdated architecture and does not leverage the extreme scale pretraining that contemporary models do. Moreover, it not being publicly available stymies progress vital for advancing research and applications in the legal sector.We believe this work pioneers a holistic approach to domain adaptation by training modern LLMs specifically for the legal domain, from pretraining to instruction fine-tuning and legal preference alignment. We demonstrate that synthetic data can be effectively utilized for alignment, advancing beyond SaulLM-7B's use solely of instruction fine-tuning. The resulting models, SaulLM-54B and SaulLM-141B, lay the groundwork for further research and development, and expand access to high-performance legal LLMs.Question: Does the paper discuss the limitations of the work performed by the authors?Answer:Justification: See limitations.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
data collection and corpus construction: 
 This section outlines our approach to assembling and refining a comprehensive legal text corpus tailored for training large language models in the legal domain.The diversity of legal systems worldwide, from common law to civil law traditions, presents unique challenges and opportunities [54, 35] . To address this, we compiled an extensive English-language corpus from various jurisdictions including the U.S., Europe, Australia, and others [3, 33] , which comprises 500 billion tokens before cleaning and deduplication.Our base corpus combines various legal datasets [53] with newly sourced public domain documents. It includes significant collections such as the FreeLaw subset and the MultiLegal Pile, augmented with extensive web-scraped content. Table 1 summarizes the composition and scale of our dataset.Replay Sources. To mitigate the risk of catastrophic forgetting during model training, we reintroduced data from earlier training distributions [48, 16, 73, 72, 25, 24] . This replay strategy incorporates general data sources such as Wikipedia, StackExchange, and GitHub, and makes up approximately 2% of the total training mix. These datasets are sampled from SlimPajama [69, 19, 70] .Additionally, we included 5% of math datasets in the pretraining mix using commercially available math sources. We found this approach usefull for retaining the reasoning performance of the final model and avoiding the weaker performance observed in previous research attempts like SaulLM-7B Total 520Instruction Sources. We found that incorporating conversational data during pretraining is advantageous, drawing inspiration from recent breakthroughs in neural machine translation [6] . Studies suggest that the enhanced translation capabilities of LLMs can be attributed to the presence of accidental parallel data within their training corpora. Accordingly, we have integrated the Super Natural Instruction [83] and FLAN collection [46] into our pretraining mix, enriching the dataset with diverse instructional content. Data for Model Annealing. Model annealing is primarily achieved through a methodical reduction of the learning rate [58, 38] , known as learning rate annealing.In our experiments, model annealing with high-quality, domain-relevant data significantly enhanced performance. Conversely, repetitive synthetic data from initial instruction fine-tuning harmed performance. Therefore, we used the commercial portion of the LawInstruct dataset for model annealing, which proved more effective than for instruction finetuning. We also included UltraChat [26] as generic instructions during the annealing phase.Our data processing pipeline closely follows [18] . In particular, we do:1. Text extraction: a significant fraction of the collected data is in PDF format. We used Poppler to extract the text. 2. Data cleaning: extraction from PDF files creates some artifacts like page and line numbers in the middle of sentences, as well as broken lines of text, non-normalized Unicode characters, etc.• Text normalization. We normalize all text using the NFKC method, available through the unicodedata Python package. • Rule-based filters. We created regex rules for filtering commonly undesirable but commonly recurring patterns, like page and line numbers in the middle of the text, HTML tags, etc. Following [18] , we found that some of the most common 10-grams in our dataset were repeated characters and whitespace and removed them. • Perplexity filtering. Similarly to [18] we used a KenLM [34] model trained on a small subset of carefully cleaned legal data to filter documents with high perplexity. Concretely, we filtered any document whose normalized perplexity was larger than 1500.Text deduplication: we used [50] to remove duplicates and near-duplicate examples from the training set. We used default parameters except for the similarity threshold, which we set to 0.5.Finally, we packed the individual documents together to build 8192 tokens-long training examples. Documents longer than this value were chunked into several examples.Instruction fine-tuning is essential for making an LLM follow instructions and optimize the performance of pretrained models across a variety of tasks [81, 84, 17, 29, 26, 82] . To this end, we employ a strategic mix of general and domain-specific (legal) instructions, aimed at enhancing the model's ability to precisely interpret and execute commands, with a particular focus on legal scenarios. General Instructions Our methodology for sourcing general instructions involves the integration of a diverse array of datasets, each selected to augment different aspects of the model's capabilities across various domains [14, 92] : 1. General Instruction from UltraInteract [88] : UltraInteract is an extensive, high-quality dataset designed to foster complex reasoning, featuring structured instructions that include preference trees, reasoning chains, and multi-turn interaction trajectories. 2. General Instruction from Dolphin3 : This dataset provides additional conversational data, further broadening the model's exposure to diverse communication styles and contexts.Each dataset is subjected to rigorous filtering, deduplication, and curation processes, culminating in a refined compilation of approximately 1, 000, 000 instructions meticulously prepared for the instruction fine-tuning phase.Legal Instruction Construction For legal instructions, we synthesize dialogues and question/answer pairs that capture key legal concepts and document types to emulate legal analysis. In accordance with the model scale, we used Mistral-54B-Instruct for SaulLM-54B and Mistral-141B-Instruct for SaulLM-141B. The generation follows the recipe from [18] and begins with a three-turn sequence: (1) a user inquires about a legal document, (2) the assistant reformulates this inquiry by integrating metadata such as document type or issue date, and (3) the user asks for further explanation of the assistant's reasoning. The dialogue progressively deepens, with the assistant methodically unpacking the legal reasoning in response to increasingly nuanced questions from the user.We enhance our models' adaptability and precision by incorporating preference data from both general and legal-specific sources [78, 62, 52, 80] . General datasets are UltraFeedback [20] and Orca.Mixtral-Base 141B For the legal domain, we employ synthetic scenarios crafted to simulate complex legal reasoning and generate accepted/rejected responses. The Mixtral-142B-Instruct model evaluates these responses based on factual accuracy, relevance, and logical coherence, selecting the most appropriate responses as preferred outcomes (similar to [91] ).4 Implementation Details & Evaluation ProtocolQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer:Answer: Justification: Models are released under a commercial license and data are described. Guidelines:• The answer NA means that the paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
model selection: 
 We used Mixtral models [40] , which are built on a Transformer architecture [79] At the time of the training, Mixtral was the most powerful decoder in its class, surpassing all competitors including Llama [76, 77, 89] , Yi, Qwen [7] , and CroissantLLM [28] in terms of both cost-effectiveness and performance.Codebase Configuration Our training framework uses PyTorch. The integration of DeepSpeed [64] (level 3) and Flash attention [23] mechanisms enhances our training efficiency and scalability. We make our models available through the Huggingface hub [85] .Compute Infrastructure The computational backbone for the continuous pretraining phase of our project consists of 384 AMD MI250 GPUs. We can reach 40% GPU utilization with our implementation. For instruction fine-tuning and preference optimization, we rely on 64 AMD MI250 GPUs. Evaluation protocols are executed on a single node of AMD MI250 GPU. Synthetic Data Generation For synthetic data generation, we used vLLM on a node of NVIDIA-A100, primarily due to limited support of libraries on MI2504 .The model training process is divided into three distinct phases: continued pretraining, instruction finetuning (IFT), and preference alignment using domain-specific optimization (DPO). A full schema of the pipeline can be found in Figure 1 .Continued Pretraining For continued pretraining, we use the AdamW [41, 47, 8] optimizer with hyperparameters β 1 = 0.99, β 2 = 0.90, and a learning rate of 2 × 10 -5 . We utilize a cross-entropy loss function to optimize model predictions. The training protocol sets gradient accumulation to 4, with tailored batch sizes of 8 for SaulLM-54B and 4 for SaulLM-141B, optimizing both GPU utilization and training efficiency. Instruction Fine-Tuning (IFT) IFT uses the AdamW optimizer (learning rate of 1 × 10 -5 ), reinitialized to reset training states and maintain training stability. We limit this phase to a single training epoch, as our experiments suggest this maximizes performance gains.Preference Training Using DPO We adjust the learning rate of the AdamW optimizer to 1 × 10 -6 during DPO. Our choice of DPO over IPO [13] , KTO [27] or ORPO [37] was based on preliminary experiments.LegalBench-Instruct We rely on LegalBench-Instruct [18] , which refined the prompts from LegalBench [31] by eliminating distracting elements and specifying a response format to enhance precision. Like LegalBench, it evaluates LLMs across six types of legal reasoning: issue-spotting, rule-recall, rule-application, rule-conclusion, interpretation, and rhetorical understanding. Grounded in American legal frameworks but applicable globally, these categories provide a comprehensive evaluation of the models' legal reasoning capabilities. This structured approach helps in accurately assessing and guiding the enhancement of LLMs within and beyond American legal contexts. We follow previous work and rely on balanced accuracy as the primary metric across all tasks.Massive Multitask Language Understanding (MMLU) Previous works utilize MMLU [36] , a widely-recognized benchmark, focusing on its legal-specific tasks in international law, professional law, and jurisprudence, with 120, 1500, and 110 examples respectively. These tasks are crucial for assessing our models' understanding and application of complex legal concepts, highlighting their proficiency in nuanced legal environments.For our evaluation, we aim for a direct, apples-to-apples comparison of models. It is important to note that not all competing models are open source, and detailed information on their alignment procedures and instruction fine-tuning processes is not available. This lack of transparency complicates the establishment of fully equitable baseline comparisons.In what follows, we use OpenAI's GPT-4 as of 10 May 2024, Meta's Llama3 (the Instruct variant) and the Instruct variants of Mixtral-54B, and Mixtral-141B. Additionally, SaulLM-54B-IFT is the IFT version built on SaulLM-54B-base and SaulLM-medium for the DPO version based on SaulLM-54B-IFT. SaulLM-large is the final version DPO and IFT based on SaulLM-141B.5 Experimental Results presents the results of SaulLM-large and SaulLM-medium on LegalBench-Instruct, from which we make several observations: Our domain adaptation strategy is achieving strong results. SaulLM-medium outperforms Mixtral-54B, and similar findings are observed with SaulLM-large compared to Mixtral-141B. Interestingly, domain adaptation at both the instruction tuning stage and preference alignment enables our smaller models to outperform larger ones, such as GPT-4 and LLama3-70B. These results validate our approach and demonstrate that specializing the entire pipeline (i.e., from continued pretraining to preference alignment) is a promising direction for improving performance in legal-related tasks. Domain adaptation works across scales and MoE models. The results from SaulLM-medium and SaulLM-large confirm previous findings from [18] and confirm that domain adaptation is effective across different scales, including on MoE models. Interestingly, most of the data collected for this work comes from public sources, which were likely seen during the pretraining of the base models.A Path Towards Stronger Models. The results of LLama3-70B and the scalability of our methods suggest that applying the same approach to the LLama3-70B base model could lead to even better performance than our best model, SaulLM-141B. It is worth noting that SaulLM-141B has only 44B active parameters making it appealing for efficient serving. Previous works on domain adaptation via continued pretraining primarily focused on instruction finetuning [16, 18, 55] . In Figure 3 and Figure 4 , we report the performance of Mixtral-54B trained with the IFT mix described in subsection 3.2 (Mixtral-54-IFT) and subsequently aligned using the DPO dataset (Mixtral-54-IFT+DPO), as described in subsection 3.3. We also compare these results to the instruct version of Mixtral (Mixtral-54B), as outlined in [40] .Continuing pretraining significantly enhances model performance in the legal domain, benefiting both the IFT and DPO stages. From Figure 3 , we observe that both IFT and DPO benefit from a notable improvement (approximately +7%). Interestingly, this improvement is consistent across all five categories, as shown in Figure 4 .Adding legal data to the IFT and DPO datasets improves the model's legal capabilities. By comparing the performance of Mixtral-54-IFT+DPO and Mixtral-54, we observe that the mix used for IFT and DPO enhanced with legal data leads to stronger legal performance than that of Mixtral-54, which does not publicly describe the alignment methods used. This result aligns with findings reported in [55, 18] . 
how much does legal preference alignment help?: 
 Our findings from Figure 3 indicate that alignment significantly improves the results. In particular, DPO improvement is mostly consistent across tasks and categories. As shown in Table 2 , the alignment version SaulLM-medium demonstrates significant improvements over the IFT version across most tasks, including conclusion, rhetoric, rules, and issue tasks. We observe, however, a drop in performance in some interpretation tasks. Upon closer examination, we found that this decline is often due to the model becoming more verbose, which causes the evaluation process to fail in correctly parsing the answers , i.e., this issue primarily arises from a benchmark limitation.Addressing model verbosity and the challenge of more reliable benchmarks is beyond the scope of this work, but it is a well-known problem identified in many concurrent studies [29] . Enhancing the evaluation process is one of the key improvements we plan to contribute to in the future. SaulLM-base-140B SaulLM-base-54B This observation suggests that continuing the pretraining process beyond the current SaulLM-base can lead to further improvements. The consistent decrease in loss implies that the models have not yet reached their full potential and that additional pretraining could enhance their performance further, which is consistent with findings from other works [60, 2, 51] .Table 3 quantifies the impact of scaling the model and compares the performance between SaulLM-medium and SaulLM-large.The main takeaway is that scaling generally improves overall results, but we also observe inverse scaling on some legal tasks [49, 43] . Unsurprisingly, for the majority of tasks across all categories, increasing the model size leads to improvements, but for tasks involving conclusion, interpretation, and rules, we observe a proportion of tasks (20%) that follow inverse scaling laws.The training was conducted on Adastra, ranked 3rd in the Green500 since November 2022 5 , as one of the world's most efficient machines in terms of performance per watt.Experiments for training SaulLM were performed between February 20th and May 15th. Energy consumption for each job was meticulously tracked, and we calculated and displayed the average power used per node for each job involved in this training in Figure 6 6 . The mean power usage ranged from 600W to 2500W, reflecting the varying utilization of the GPUs. Each node contains four MI250X GPUs, which have a theoretical Thermal Design Power (TDP) of 560W. This configuration explains the maximum consumption of 2500W during high-intensity GPU usage.Overall, the project involves over 160, 000 hours of MI250 for debugging, continued pretraining, instruction finetuning and preference alignment. The total energy consumed was 65, 480.4kWh.Consumption is significantly lower than the typical energy requirements for full LLM training, showing that continued pretraining is an effective strategy for specializing in new LLMs while optimizing energy efficiency. 
conclusion & limitations 6.1 conclusion: 
 This study released two new legal LLMs under the MIT license: SaulLM-54B and SaulLM-141B.They leverage the Mixtral architecture and continued pretraining on a large legal corpus. Our findings show significant advancements in processing and understanding complex legal documents.Through continued pretraining, instruction fine-tuning, and preference alignment using domainspecific optimization, we have demonstrated substantial improvements compared to GPT-4, Llama3 and original Mixtral models as measured on LegalBench-Instruct.Our experiments suggest that the instruction finetuning and alignment processes utilized by Mixtral-Instruct and Llama3 are advanced and challenging to replicate. These processes often rely on proprietary datasets and significant computational resources that are not readily available in open-source frameworks. Although both SaulLM-54B and SaulLM-141B achieve stronger performances than Llama3 and Mixtral Instruct on legal benchmarks, we found that they are slightly weaker at following generic instructions.Looking forward, we aim to continue our work on enhancing the SaulLM family, particularly focusing on integrating Llama3 and improving the alignment procedure. Our goal is to improve the alignment of these models with legal tasks, refining their ability to process and understand legal language with even greater accuracy and relevance. This future work will strive to address the current limitations by developing more robust methods for instruction finetuning and alignment that are accessible to the broader research community.NeurIPS Paper Checklist 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: Justification: Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: Justification: MIT-liscence Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer:Justification: We introduce new models that will be publically released under MIT liscence.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer:Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer:Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.  
 
 Appendix: Model will be made available at https://huggingface.co/.These findings also align with the high percentage of math and STEM in the pretraining mix from[56].https://huggingface.co/datasets/cognitivecomputations/dolphinVLLM is not supported on AMD-MI250 and HuggingFace's text-generation-inference had a few bugs that prevented its use.https://top500.org/lists/green500/2023/11/The high number of jobs (over 1800) for this project arise from (i) the technical difficulties in efficiently using AMD GPUs, and (ii) numerous hardware failures when scaling up the number of nodes. 


Decision: The paper presents SaulLM-54B and SaulLM-141B, two large language models tailored for legal applications. Built on the Mixtral architecture, these models are refined using comprehensive domain adaptation techniques, such as continued pretraining on a vast legal text corpus, instruction fine-tuning, preference alignment and synthetic data generation. The models achieve state-of-the-art results on the LegalBench-Instruct benchmark. The research highlights the effectiveness of scaling domain-specific adaptations and the advantages of larger model sizes in enhancing legal language processing.

The main strength of the paper is in demonstrating strong domain adaptation results for a specific and important domain, for large-size LLMs. The main limitation, when it comes to a scientific conference, is the use of well-known and well-explored techniques (even though they have not been shown to provide such strong results for domain adaptaiton of LLMs of this size). Yet, the engineering result is strong and novel. The divergence in reviewer scores is mostly with respect to how they weigh the obvious pros and cons mentioned above. I am leaning towards accept.

Reviews: review: paper_summary: paper_summary: The paper introduces SaulLM-54B and SaulLM-141B, two large language models specifically designed for the legal sector. These models utilize the Mixtral architecture and are developed through extensive domain adaptation strategies, including continued pretraining on a large legal corpus, instruction fine-tuning, and preference alignment. The models incorporate synthetic data to enhance their capabilities in legal text interpretation and achieve state-of-the-art performance on LegalBench-Instruct. The study emphasizes the scalability of domain-specific adaptation and the potential benefits of larger model sizes for legal applications.
strength_weakness: strength_weakness: 1. The techniques used, such as continual pretraining, instruction tuning, and preference alignment, are not novel compared to existing studies like SaulLM and LLMs in other domains.

2. There are issues with the license, privacy, and quality of data. The paper should explicitly illustrate the license of each used data source. The quality of synthetic legal instruction tuning data and preference data is uncertain.

3. The evaluation is not comprehensive. There is no ablation study to illustrate the effectiveness of continual pretraining, such as comparing the performance of Mixtral-54B and SaulLM-54B-base. It would be better to include other LLMs in the legal domain as baselines.

4. In section 5.2, the explanation of "How much does continued pretraining help for the legal domain" is confusing. The continual pretraining previously referred to unsupervised continual pretraining with a next-token generation task on large-scale legal data, but in this section, it seems to mean IFT+preference alignment.
questions: questions: On page 7, it is mentioned: "The results of LLama3-70B and the scalability of our methods suggest that applying the same approach to the LLama3-70B base model could lead to even better performance than our best model, SaulLM-141B." Which figure's results support this illustration?
limitations: limitations: Yes
paper_summary: paper_summary: - The paper introduce two LLMs (at different sizes) specialized for law. These models have been adapted for law through continued pretraining, specialized legal instruction following, and a “legal alignment” process
- The paper studies the tradeoffs of domain adaptation at this scale and presents results for these models.
strength_weakness: strength_weakness: - The paper is sparse on some key details related to the datasets used to train the model.
    - For the preference data, what are the synthetic scenarios designed? How were they constructed? What were the Mixtral prompts used to evaluate them?
    - For the legal instruction data, what were the legal documents initially chosen? What did the conversations look like?
- Examples of both instruction and preference data in the Appendix would be very helpful to readers. 
- The ablations are extremely helpful but also difficult to read, in part because they’re mixed in with baselines. It would be nice if the authors could present a table showing the progression of performance between: Mixtral-54B, + continued pretraining on the legal corpora, + IFT, + legal alignment. I think some–but not all–of these in Figure 6?
questions: questions: - The LegalBench website asks users of the benchmark to cite a collection of papers, not merely the original LegalBench paper. Since LegalBench-Instruct looks like a small modification to LegalBench, do you plan to cite that original collection of papers as well? Also the LegalBench citation seems wrong, and not to the actual LegalBench paper (https://arxiv.org/abs/2308.11462)
- Do the authors plan on releasing these datasets?
- Do the authors plan on releasing intermediate checkpoints for model training?]
- Once concern in relying on LegalBench-Instruct is that the prompt format chosen favors Mixtral models over other types of models (e.g., Llama-3, GPT, etc.). Do the authors have a sense of how sensitive the Saul models are to the prompt format?
- It would be very nice to see some examples of generations in the Appendix!

I’ll increase my score if the additional details are added to the paper.
limitations: limitations: NA
paper_summary: paper_summary: The paper introduces two large language models specialized in the legal domain with instruction-following capabilities. These models are an extension of previous work, particularly Colombo et al.'s “SaulLM-7B: A Pioneering Large Language Model for Law,” by scaling up the corpus size and the number of trainable model parameters. The model parameters are updated through continued pretraining, instruction-tuning, and alignment (RLHF, DPO, or similar methods). Additionally, the authors gather and preprocess the largest legal dataset for pretraining from various sources. They examine the impact of both model size and dataset size on the effectiveness of adapting large language models (LLMs) to the legal domain.
strength_weakness: strength_weakness: - This paper misses ablations on the three training methods (with and without continued pretraining), ± instruct tuning, and ± alignment training. I would have liked to see where the greatest improvement in results is obtained. For example, if we do not include continued pretraining at all but only perform instruct tuning and alignment. 

- The study does not include detailed results about the compared methods (Fig. 17 only compares Saul med with Saul large). Can these results be shown for the top-performing models for each training method and models of different architecture (e.g., GPT-4 and Saul large)? In the results section, they do not compare their proposed model with existing domain-specific language models like Saul-7B and legal-Flan-T5. 

- The paper does not explain the process of generating synthetic data clearly. A detailed explanation of what prompts have been used to generate the data, what model has been utilized, and which generation parameters have been used is required. 

- The paper discusses adding a math portion to the pretraining dataset to enhance reasoning capabilities, but the source of the data is unclear, and they don’t provide any supporting experiments to back up their claim.
questions: questions: - What is “Score” on the y-axis of Figs. 3, 4, and 5? 

- Can the authors justify the usage of only “balanced accuracy” as the primary metric used in the results associated with performance (non-energy analysis related)? 

- Can the authors explain the process of preparing synthetic data for both instruction and preference fine-tuning in detail? Considering that generating synthetic data can lead to low-quality samples, did you quantitatively measure the quality of the generated data? Also, do you define any acceptance criteria for the generated data to avoid adding low-quality samples to the training corpus? 

- Considering that you included data from various jurisdictions, do you evaluate the performance of the model across different jurisdictions? 

- Based on the results presented in Figure 6, do you have any explanation or assumption as to why Mixtral-IFT is performing worse than Mixtral? 

- In the conclusion, you claim that “we have demonstrated substantial improvements compared to GPT-4.” The mean balanced average of GPT-4 and the proposed model are pretty close. Did you perform any significance tests to show that the difference is statistically significant? 

- The authors clearly indicated that the experiments are limited due to the proprietary nature of the datasets used to train Llama and Mixtral. Have the authors looked into open-source and not just open-weight models? For example, Olmo: https://arxiv.org/abs/2402.00838.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper reports on two new legal-specific LLMs based on Mixtral. The primary contribution is to train larger law LLMs than previously reported, using (1) an extensive dataset of legal materials, and (2) a variety of best practices in pre-training. The results indicate incremental but significant improvements on legal benchmarks.
strength_weakness: strength_weakness: The paper's strength is also its weakness. It generates very little generalizable knowledge for understanding legal tasks, only a series of engineering improvements. There are some reasons to think that transformer-based LLMs will not be able to carry out successful human-level work without fundamental improvements in the underlying model architecture. But the research still gets us closer to the frontiers of what current LLM achitectures can do, and it is possible that similar continued improvements are all that are necessary.
questions: questions: (See above.)
limitations: limitations: Yes, the paper is well scoped to describe what it does and doesn't do.

Comments: comment: Thanks for the additional details! I'll raise my score.
**We have indeed included the relevant licensing information in our rebuttal and will ensure it is incorporated into the camera-ready version of the paper and will make sure to reflect these information in the checklist.** Carefully checking the license has been a major focus in our data collection process, as we aim to release the LLM under the MIT license. **To guarantee compliance, we have also relied on professional external legal counsel, which we believe is best practice.**

We are happy to provide clarification if needed.
We have indeed included the relevant licensing information in our rebuttal and will ensure it is incorporated into the camera-ready version of the paper. Carrefully checking the license have been a major focus in our data collection process, as we aim to release the LLM under the MIT license. **To guarantee compliance, we have also relied on professional external legal counsel**.
**Thank you for your detailed and thoughtful feedback on our paper. We are pleased that you found our contributions significant and our paper technically strong.**

We understand your concern regarding the limited generalizable knowledge for understanding legal tasks. **This limitation is partly due to the constraints imposed by the available benchmarks.** The current benchmarks are limited because collecting detailed legal benchmarks requires significant investment, **as lawyer hours are quite expensive. We will address this in our paper by adding a section on the limitations of our benchmarks.**

Thank you once again for your valuable feedback.
We thank the reviewer for their review. We are glad they found our paper is well written and our contribution extremely valuable.

Below is the response to the concerns/questions:

**About the additional ablations**, they are already reported in the paper. See general comments. 

**Adding comparison to GPT-4.**

We acknowledge the request for more detailed comparisons similar to those depicted in Figures 17 and 18, which illustrate the relative improvement from DPO with respect to instruction fine-tuning and the effects of model scaling within the relying on the Mixtral backbone.
In the revised version, we extend these with a comparison to GPT-4, although emphasizing that GPT-4's size—1.5 trillion parameters—makes direct comparisons challenging. Our focus remains on domain adaptation of base models, ensuring apple-to-apple comparisons, rather than contrasting models of substantially different sizes. 
We have included results in Table 3 and Figure 5 and note that our models compare favorably with GPT4.

**On the addition of SaulLM-7B and Legal Flan T5:** See general comment

**Synthetic data generation:** see General Comment. 

**Addition of math data:** We performed an ablation study on a 7B model (not reported in the paper). Below are the results from the reviewers:

| Models               | Size | Results |
|----------------------|------|---------|
| Mistral 7B + pretraining (without math) + IFT   | 7B   | 0.612    |
|  Mistral 7B + pretraining (without math) + IFT    | 7B   | 0.628    |

In the revised version we did report the results in the appendix. 

**About the reviewer’s questions:**

**Score refers to balanced accuracy.** We will improve the label.

**Balanced accuracy** was originally used by the Legalbench authors (see section 5.1.3 of their paper). We reused their code for comparizon.

**See the general comment for the synthetic data.**

**We will add examples of generation to the appendix.** 

**Comparison on other jurisdictions.** For this study, we focused on LegalBench, which is emerging as a standard benchmark. While we plan to expand this benchmark to include other jurisdictions, such an extension requires significant time and financial resources; therefore, we did not evaluate other jurisdictions in this iteration. However, we conducted manual tests using concepts from European law, such as "imprevision", "basis of the bargain" or "abuse of dominant position", and found that the responses were significantly improved compared to the original model from Mixtral. We intend to explore this further and provide a detailed evaluation in a future paper. **We plan to add these examples in the revised version of the paper.**

**IFT v.s. Mixtral:** We believe that the Mixtral models align more closely with DPO, whereas the IFT model does not. This alignment discrepancy may represent a key difference between the models. Due to the limited information available about their training processes, we conducted an ablation study in Section 5.2 to further dissect and understand these different factors. The reviewer may notice that once aligned with legal synthetic data the IFT + DPO outperforms Mixtral.

**About the significance of the comparison with GPT4.** LegalBench reports over 90k samples by relying on the methodology of (D Card et al. EMNLP2020), we see a power of the test of over 95%. We have to restate that the goal of the paper is to study domain adaptation and thus the main comparison should be w.r.t. the mixtral models.

**About Olmo.** We are excited about the Olmo initiative and plan to reference it in our upcoming paper. Given our interest in working with models larger than 7 billion parameters, the Mixtral family was our preferred choice.

**We hope that we have addressed the reviewers’ concerns and hope they will increase their grades.**
**We thank the reviewer for their review we are glad they acknowledged that our paper is well written and find that our contribution is extremely valuable.**

Below is the response to the concerns/questions:

**About the generation of the post-training datasets and examples.** See the general comment.

**Ablation:** See general comment on section 5.2

**Missing citations:** have been identified as an oversight on our part and have now been corrected in the revised version. We will include references to LegalBench as well as all pertinent datasets, specifically citing Guha et al., 2023; Koreeda and Manning, 2021; Hendrycks et al., 2021; Wang et al., 2023; Wilson et al., 2016; Zheng et al., 2021; Zimmeck et al., 2019; Ravichander et al., 2019; Holzenberger and Van Durme, 2021; Lippi et al., 2019. 


**On the prompt format:** In our study, we adhered strictly to the original prompt formats from the LegalBench paper, relying solely on the prompts from the dataset itself for generating responses. It's important to clarify that our primary objective is to examine domain adaptation, which necessitates careful adjustments. While we also compare our results with GPT-4, it's important to note that this comparison may not be entirely equitable; GPT-4's model size is substantially larger—approximately 10 times that of a 140B model and 30 times that of a 54B model. Despite testing slight variations in prompt formatting, we observed minimal impact (less than 0.2%) on our models across approximately 100,000 total samples, leading us to report outcomes using the standard version.

**Regarding release artifacts, upon acceptance, we plan to release both the curated training dataset and the checkpoints along with optimizer states for seamless training resumption.**

**We hope that we have addressed the reviewers’ concerns and hope they will increase their grades.**
We thank the reviewer for their review we are glad they acknowledged the usefulness of our paper and our contribution to the community with these models.

Below are the answers to the reviewer's comments:
**There are no issues with the licenses:** The licenses permit commercial use for both the pretraining and instruction finetuning data. Below is a breakdown of the licenses:

| Source Name                                 | Tokens (B) | License                        |
|---------------------------------------------|------------|--------------------------------|
| FreeLaw Subset from The Pile                | 15         | MIT license                    |
| EDGAR Database                              | 5          | Apache 2                    |
| English MultiLegal Pile                     | 50         | CC-By-SA-4.0                   |
| English EuroParl                            | 6          | Public Domain                  |
| GovInfo Statutes, Opinions & Codes          | 11         | Open government license        |
| Law Stack Exchange                          | 0.019      | CC-By-SA-4.0                   |
| Comm Open Australian Legal Corpus           | 0.5        | CC-By-4.0                   |
| EU Legislation                              | 0.315      | CC-By-4.0                   |
| UK Legislation                              | 0.190      | Open government license        |
| Court Transcripts                           | 0.350      | CC-By-ND international 4.0       |
| UPSTO Database                              | 4.7        | CC-By-SA-4.0                   |
| Web Data (legal)                            | 400        | ODC-BY                   |

**Upon acceptance, we plan to include all the licenses in the appendix.**

**The doubt The quality of synthetic legal instruction tuning data and preference data is uncertain.** We believe that the quality of the post-training is demonstrated by the results.

**Including other Legal LLMs as a baseline:** See general comment.

**Clarification of section 5.2.**  See general comment.

**About the Claim:** On page 7, it is mentioned: "The results of LLama3-70B and the scalability of our methods suggest that applying the same approach to the LLama3-70B base model could lead to even better performance than our best model, SaulLM-141B." 

The intention behind this statement is to highlight that the LLama3-Instruct results demonstrate stronger performance compared to Mixtral 7X22B (see Fig. 2). Based on this, we believe that applying the same methods could yield even stronger results. We acknowledge that this sentence may be confusing, and we will remove it in the updated version of the paper to avoid any ambiguity.


**We hope that we have addressed the reviewers’ concerns and hope they will increase their grades.**
**About the post-training datasets.**

**For the instruction finetuning dataset:** We follow the SaulLM-7B recipe that relies on Ultrachat methodology as described in their original paper. Concretely, we generate detailed discussions on essential legal skills across various document types using a Mistral-140B-instruct to convert legal texts with metadata into structured dialogues (Ding et al., 2023). The conversation starts with three steps: (1) a user request about the document, (2) the assistant rephrases metadata like document type and date, and (3) the user prompts for deeper reasoning. Further exchanges involve the user asking targeted questions to understand the assistant's logic, with the assistant providing thorough explanations. We removed the 30% conversation that has the higher number of duplicated n-grams (computed independently for the user and assistant) which we found good enough. 

**Example:**

1. **User:**
   "Here is an extract from a legal contract: 'Except as otherwise set forth in this Debenture, the Company, for itself and its legal representatives, successors and assigns, expressly waives presentment, protest, demand, notice of dishonor, notice of nonpayment, notice of maturity, notice of protest, presentment for the purpose of accelerating maturity, and diligence in collection.' Tell us which label would you use to best describes this extract?”

2. **Assistant:**
   "The most relevant label would be 'Waivers'."

3. **User:**
   "Explain your reasoning."

[....]

8. **Assistant:**
   "These elements typically serve as enforcement mechanisms in debt repayment. Notices inform debtors of obligations or issues; demands formally request action or payment; protests document objections to non-payments; and presentments involve the physical delivery of financial documents. Waiving these steps simplifies and potentially expedites the debt handling process by removing formalities that might otherwise escalate disputes or delays."


**For the alignment dataset,** we start from a dataset where we ask the LLM to generate a question. From this question and the input sample, we generate 2 responses from the LLM. Then, we ask a judge to rank the two answers (see description in the paper: ). The first is the chosen and the second is the rejected. We rely on Mixtral 8x22B as described in Fig. 2. This is similar to the Nemotron paper (Nvidia Team 2024). 

Example:

**Question:** Which category best describes the following extract from a legal contract?

"Except as otherwise set forth in this Debenture, the Company, for itself and its legal representatives, successors and assigns, expressly waives presentment, protest, demand, notice of dishonor, notice of nonpayment, notice of maturity, notice of protest, presentment for the purpose of accelerating maturity, and diligence in collection."

LLM 1: Waivers

- **Reasoning:** This label is most appropriate because the [...]

LLM 2: Modifications

- **Reasoning:** One might also consider this as a  'Modifications' because the waiver of traditional debt collection rights [..]

LLM as a Judge: Waivers

Reasoning: The extract explicitly states that the company "expressly waives" several rights, [....]

**We have added more details in the paper’s appendix and we plan to release the datasets upon acceptance to clarify and support the reader. We will also include examples in the appendix.**
**Clarification on additional Baseline and the main claim of the papers.**

We appreciate the suggestion to include other legal LLMs as baselines. However, previous legal LLMs, such as SaulLM, are relatively small in scale (up to 7B parameters) and Legal-Flan-T5, is not available (not released by the authors) for comparison. For completeness, we will report the results of SaulLM:

| Models               | Size | Results |
|----------------------|------|---------|
| SaulLM 7B-Instruct   | 7B   | 0.61    |
| Mistral 7B v2        | 7B   | 0.52    |
| Mixtral 54B          | 54B  | 0.64    |
| SaulLM-Base          | 54B  | 0.73    |
| Mixtral 140B         | 140B | 0.65    |
| SaulLM-Large         | 140B | 0.75    |

**This comparison may not be entirely fair due to differences in model size and the pretraining regime of our paper.**
The reviewers positively recognized the significance of our paper (jvqv), noting its application in the legal domain (pgvt), scale (LHKy, pgvt), results (jvqv, pgvt), and openness (iZW5, pgvt). They also find the paper clear and well-written (jvqv).  Below, we answer the common questions of the reviewers about the choice of the baseline (pgvT, iZW5), about synthetic data (pgvT, LHKy) and about the ablation study (pgvT, LHKy).

**Ablation Study:  All results asked by the reviewers are already provided in Figures 6 and 7** 
5.2.1 allows quantifying the effect of pretraining, IFT and DPO. All results asked by the reviewers are already provided in Figures 6 and 7. Below is a mapping of the model name referenced in the paper as well as a model description:

| Model Description                                      | Reference Name in the paper       | Results |
|--------------------------------------------------------|----------------------|---------|
| Mixtral (original) + IFT (our IFT)                     | Mixtral 54-IFT       | 0.59    |
| Mixtral Instruct (original from Mixtral)               | Mixtral 54B          | 0.64    |
| Mixtral (original) + IFT (our IFT) + DPO (our DPO)     | Mixtral 54-IFT + DPO | 0.65    |
| Mixtral (original) + Continued Pretraining + IFT (our IFT) | SaulLM-med 54 IFT    | 0.66    |
| Mixtral (original) + Continued Pretraining + IFT (our IFT) + DPO (our DPO) | SaulLM-med | 0.73    

This allows the reviewer to quantify the impact of various improvements, including DPO, IFT, and continued pretraining. **A clarification has been made** in the revised version of the paper to avoid any ambiguity.