Teaching Natural Language Processing in Law School (2024)

Abstract: Fuelled by technical advances, the interest in Natural Language Processing in the legal domain has rapidly increased over the last months and years. The design, usage, and testing of domain-specific systems, but also assessing these systems from a legal perspective, needs competencies at the intersection of law and Natural Language Processing. While the demand for such competencies is high among students, only a few law schools, particularly in Europe, teach such competencies. In this paper, we present the design for a Natural Language Processing course for postgraduate law students that is based on the principle of constructive alignment and has proven to be successful over the last three years.

heading: 1 Introduction
text: Like most fields of occupation, the legal profession is undergoing a drastic change due to the introduction of Artificial Intelligence (AI) and particularly Natural Language Processing (NLP) in the workplace. While research on applications of AI in the legal domain has a long history going back to the 1970s (Rissland et al., 2003), the legal practice was, for a long and arguably more so than other fields, hesitant to adopt AI technology and the pace of adoption started to increase only very recently (Oskamp and Lauritsen, 2002; Araszkiewicz et al., 2022). While there have been many good reasons for this hesitance, including strict regulations and personal liability of legal professionals (Vladika et al., 2024), both the acceptance and use of AI technology are rapidly increasing within the legal profession in recent years (Weinstein, 2022).
This increase in usage also results in an increased need for legal professionals with technological expertise, both in traditional roles but also in specialised emerging roles like legal engineer or legal technologist. According to the “Future Ready Lawyer Report” by Wolters Kluwer, only 24%
of lawyers say that they understand “transformational technologies” like AI and big data (Weinstein, 2022). These numbers highlight that there is a significant demand for teaching technological essentials to (future) legal professionals. As language is the most important tool of lawyers and other legal professionals, among AI technologies, NLP technologies are arguably most relevant for them with a number of practical applications including contract generation, document management, legal decision-making, anonymization, and many other tasks (Vladika et al., 2024). Additionally, competencies in both NLP and law are not only needed for the application of NLP to legal problems but also for the application of law to NLP. With new regulations like the European Union’s AI Act, there will be an increased need for lawyers with a good understanding of AI technology who are able to counsel clients who develop, distribute, or use such systems.
While many US law schools have already included NLP in their curriculum (Johnson, 2023), in Europe, such offerings are still rather rare. Partially, this is caused by the fact that law degrees in most European countries are undergraduate degrees that often follow a highly regulated four to five year curriculum, unlike in the US, where law schools are part of postgraduate education. However, as consecutive Master of Laws (LL.M.) degrees are becoming more popular in Europe, law schools have started to offer such degrees particularly focused on Legal Technology (“Legal Tech”). This paper outlines the design of an introductory NLP course that is designed for this setting, i.e. as an elective course in postgraduate education for law students without prior knowledge of AI or computer science. The course has been taught in a very similar fashion for three years at an LL.M. program at a public German university. The aim of this design is to inform and inspire the development of similar courses.
85

heading: 2 Skills and Intended Learning Outcomes
text: Following the principle of constructive alignment (Biggs, 1996), the course was designed around the intended learning outcomes (ILOs) and more fundamentally the skills that are relevant for legal professionals. Given the context for which the course was designed, no relevant previous knowledge in AI or computer science can be expected. Therefore, an NLP course in a law school can clearly not go as deep into technological details as a traditional NLP course. After all, a law graduate with a specialisation in legal technology is, most likely, also not going to work as a programmer. Therefore, the most important question in designing an NLP course for law students is: What are the relevant skills that can help future legal professionals to work at the intersection of law and NLP?
It is worth emphasising that the course design outlined in this paper is not meant to be part of the general curriculum every law student has to go through, but rather as an elective for those that want to work at the intersection of law and NLP where a deepened interdisciplinary understanding is needed. We identified three different areas in which such interdisciplinary skills are mainly needed:
• Law practice: NLP tools will be part of the daily work routine of many legal professionals in the future. Understanding the fundamental principles these tools operate on will help legal professionals to make informed decisions on how and when to use which technology or tool.
• Technology development and implementation: Legal professionals will be involved in the development and implementation of legal tech tools in many different roles in which they will need a solid understanding of the underlying technology, like product manager or implementation consultant.
• Research: Whether as a legal data scientist or empirical legal scholar, legal research, both in academia and practice, will include many different NLP methods. Especially in interdisciplinary research projects, annotating legal corpora and preparing them to be used as training data will require legal expertise and an understanding of the requirements for training data for ML models.
Based on these areas, we derived ILOs that law
graduates would need to achieve to be prepared for such roles. The six derived ILOs are shown in Table 1. Each ILO is classified according to Bloom’s taxonomy (Anderson and Krathwohl, 2001). Since the goal of the course is to provide participants with practical NLP skills that are relevant in the legal domain, most of the ILOs are on the higher levels of Bloom’s taxonomy (create, evaluate, analyse, apply), i.e. go beyond acquiring just theoretical knowledge.

heading: 3 Course Plan
text: Based on the ILOs, we developed a plan for a course that runs over the span of a semester (∼15 weeks) and has a workload of 140 hours (or 5 points in the European Credit Transfer and Accumulation System, ECTS). Of those 140 hours, 56 are contact hours: 30 hours (or 2 hours per week) of lectures and 26 hours of tutorials (13 x 2 hours). The remaining 84 hours are reserved for self-study and project work (see Section 4.1). Particularly for the achievement of the ILOs that are related to the higher levels of Bloom’s taxonomy, practical experience, as mediated through tutorials and project work, is key to successful learning. Table 2 shows an overview of all educational activities and their content. While the lectures mainly focus on theoretical knowledge, the tutorials are designed as hands-on sessions, in which participants interact with different tools. Some of the course materials, particularly for the practical sessions, are available on GitHub1. It is important to stress that the designed course does not attempt to replace a computer science curriculum. Therefore, the lectures mainly stay on a conceptual level, without, e.g., going into the mathematical details or optimisation algorithms in ML. Similarly, in the tutorials, participants will not write complete programs but rather are provided with Jupyter notebooks that they can configure in a low-code fashion, so that they get a basic understanding of the process and structure that underlies the training of ML models, without necessarily having the learn how to program. Weeks 12 to 14 are reserved for guest lectures from practitioners. The idea is to invite representatives of different organisations that have a need for professionals with skills at the intersection of law and NLP to show the broad range of potential roles, not only within traditional law firms but
1https://github.com/DaBr01/ Teaching-NLP-in-Law-School
also within established companies or organisations and start-ups.

heading: 4 Assessment
text: The assessment of the course consists of an exam at the end of the course (that can be either written or orally), and a project on which students work in groups of two to three. Both assessment forms will determine 50% of the final grade each. While the exam is purely a formative assessment, the project contains summative and formative assessment moments. The goal of formative assessment is to assess the level of knowledge a student has at a certain point in time. In the context of the course, formative assessment is used to assess, at the end of the course, whether the ILOs have been achieved and to derive a grade. Formative assessment, on the other hand, is part of the teaching process and aims to provide the students, but also the teacher, with information about the current progress of the students, in order to allow for interventions and assure that, in the end, the ILOs will be achieved. (Garrison and Ehringhaus, 2007)

heading: 4.1 Project
text: For the project, each team will be provided with a corpus and a problem description. Each team will have to solve the following tasks based on their corpus and problem description:
1. Develop an annotation guideline.
2. Annotate the corpus based on the developed guideline.
3. Train a model that is able to extract information that is relevant in the context of the given problem description.
4. Write a project report of max. 4 pages that describes the results of your project.
5. Present the project results to your peers.
The summative assessment element of the project will be the final project grade that is based on the annotation guidelines (30%), the trained model (30%), the project report (30%), and the presentation (10%). In addition, participants hand in their annotation guidelines in week 4 for a formative assessment in which they receive feedback but also give feedback to other groups as part of a peer feedback process.

heading: 5 Literature
text: While there are many textbooks available on NLP (e.g. by Jurafsky and Martin (2008) or Hapke et al. (2019)), almost all of them are much more technical than is suited for the outlined course. The textbooks by Biemann et al. (2022) and Ignatow and Mihalcea (2017) present an introduction to NLP particularly targeted at the social sciences. As such, they include introductions to fundamental concepts that are also suitable for the context of the course but do not address any particularities of the processing of legal language or domain-specific tasks and tools. Particularly because of this lack of literature, we hope that the course design presented in this paper can inform and inspire the design of NLP courses in the context of law schools.
Type Title Content W ee k 8 Lecture Introduction to Natural Language Generation Data2Text / Image2Text / Text2Text; Templates; Rule-based NLG; Stochastic NLG; Hallucination; Abstractive and extractive summarisation; Evaluation and metrics (BLEU, ROUGE)
Tutorial Text Generation Rule-based text generation with SimpleNLG; Text generation with GPT-2; Exercise on hallucination
W ee
k 9
Lecture Document Processing and Automation eDiscovery; Document management, Legal document generation tools; Process automation Tutorial Document generation
Document generation with Word; Document generation with nocode platform from week 1; Document generation with specialised tool
W ee
k 10
Lecture Legal Case Outcome Prediction State of the research in legal case outcome prediction; Relevant features (particularly non-legal features like time, ...); Bias; Reflection about the difference between predicting the court decision and predicting the “right” decision Tutorial Legal Case Outcome Prediction Training of a simple model for the prediction of legal case outcomes based on a provided Jupyter notebook template Project Deadline Model and Jupyter notebook
W ee
k 11 Lecture Explainable AI Explainability; Interpretability; Adversarial attacks; Taxonomy of explainability (post hoc/ante hoc, global/local, model specific/model agnostics); Example-based approaches; Critical reflection of “explanations” generated by LLMs
Tutorial LIME Introduction of the LIME library for the generation of explanations
W ee
k 12
Lecture Guest lecture established organisation Show possible job profiles in established organisations, e.g. legal operations officer from large companies, publishing houses, software providers, NGOs, government Tutorial Project Support Before the final project submission, participants have time to ask questions and get support
W ee
k 13 Lecture Guest lecture startup Show possible job profiles in the startup world (e.g. legal tech startups, fintech, ...)
Tutorial Startup software Introduction to the product of the startup Project Deadline Report
W ee
k 14 Lecture Guest lecture law firm Job profiles in classical law forms
W ee
k 15 Lecture Project Presentation Teams present the results of their project work

heading: Future Work
text: Therefore, the most important question in designing an NLP course for law students is: What are the relevant skills that can help future legal professionals to work at the intersection of law and NLP?
It is worth emphasising that the course design outlined in this paper is not meant to be part of the general curriculum every law student has to go through, but rather as an elective for those that want to work at the intersection of law and NLP where a deepened interdisciplinary understanding is needed. We identified three different areas in which such interdisciplinary skills are mainly needed:
• Law practice: NLP tools will be part of the daily work routine of many legal professionals in the future. Understanding the fundamental principles these tools operate on will help legal professionals to make informed decisions on how and when to use which technology or tool. • Technology development and implementation: Legal professionals will be involved in the development and implementation of legal tech tools in many different roles in which they will need a solid understanding of the underlying technology, like product manager or implementation consultant. • Research: Whether as a legal data scientist or empirical legal scholar, legal research, both in academia and practice, will include many different NLP methods. Especially in interdisciplinary research projects, annotating legal corpora and preparing them to be used as training data will require legal expertise and an understanding of the requirements for training data for ML models. Based on these areas, we derived ILOs that law
graduates would need to achieve to be prepared for such roles. The six derived ILOs are shown in Table 1. Each ILO is classified according to Bloom’s taxonomy (Anderson and Krathwohl, 2001). Since the goal of the course is to provide participants with practical NLP skills that are relevant in the legal domain, most of the ILOs are on the higher levels of Bloom’s taxonomy (create, evaluate, analyse, apply), i. e. go beyond acquiring just theoretical knowledge