The Law and NLP: Bridging Disciplinary Disconnects
Authors: Robert Mahari
Dominik Stammbach
Elliott Ash
Alex Pentland
Venue: EMNLP 2023 Findings (2023)
Keywords: legal natural language processing
legal artificial intelligence
legal precedent retrieval
access to justice

Abstract: Legal practice is intrinsically rooted in the fabric of language, yet legal practitioners and scholars have been slow to adopt tools from natural language processing (NLP). At the same time, the legal system is experiencing an access to justice crisis, which could be partially alleviated with NLP. In this position paper, we argue that the slow uptake of NLP in legal practice is exacerbated by a disconnect between the needs of the legal community and the focus of NLP researchers. In a review of recent trends in the legal NLP literature, we find limited overlap between the legal NLP community and legal academia. Our interpretation is that some of the most popular legal NLP tasks fail to address the needs of legal practitioners. We discuss examples of legal NLP tasks that promise to bridge disciplinary disconnects and highlight interesting areas for legal NLP research that remain underexplored.

The Law and NLP: Bridging Disciplinary Disconnects
Robert Mahari
Dominik Stammbach
MIT and Harvard Law School rmahari@mi t. edu
ETH Zurich Cominsta@ethz.co
Elliott Ash
Alex 'Sandy' Pentland MIT pentlandemit. edu
ETH Zurich
ashe@ethz.ch ch
Abstract
texts. In the United States, around 92% of civil legal problems experienced by low-income Amer- icans receive no or inadcquate legal help (Slosar, 2022). In U.S. criminal cases, where criminal de- fendants generally have a right to legal counsel, public defenders are systematically overworked and under-resourced (Pace et al., 2023). Access to legal services is also limited for U.S. small busi- nesses (Baxter, 2022). The combination of unequal access to and high costs of legal services results in a troubling access to justice issue.
Legal practice is intrinsically rooted in fabric of language, yet legal practitioners and scholars have been slow to adopt tools from nat- ural language processing (NLP). At the same time, the legal system experiencing access to justice crisis, which could he partially allevi- ated with NLP. In this position paper, we argue that the slow uptake of NLP in legal practice is exacerbated by disconnect between the needs of the legal community and the focus of NLP researchers. In review of recent trends in the legal NL. literature, we find limited overlap between the legal NLP community and legal academia. Our interpretation is that some of the most popular legal NLP tasks fail to address the needs of legal practitioners. We discuss examples of legal NLP tasks that promise to bridge disciplinary disconnects and highlight interesting areas for legal NLP research that remain underexplored.
Internationally, there is tremendous variation in legal systems and challenges, but many opportu- nities where technology could help address legal inefficiencies and under-served communities have been identified globally (see e.g., Wilkins et al.. 2017; Cunha et al., 2018; 18: Bosio, 2023; World Jus- tice Project, 2019). Against this backdrop, numer- ous scholars have focused on legal technology as a path towards reducing the access to justice gap by extending the abilities of legal professionals and helping non-lawyers navigate legal processes (Bax- ter, 2022: Bommasani al., 2021; Cabral et al., 2012; Rhode, 2013; Katz et al., 2021).
1 Introduction
Rapid advances in NLP technology are already promising to transform socicty and the econ- omy e.g, OpenAI, 2023; Eloundou et al., 2023;
Bommasani et al., 2021), not least by their impact on many professions. Given that legal practice is embedded in written language, it stands to gain immensely from the application of NLP techniques. This promise has attracted research on NLP ap- plications related
Of course, technology is no panacea and many of the incquities and inefficiencies in jurisprudence are related deeply rooted social and cultural phe- nomena. Nonetheless, we view the responsible deployment of legal technology as crucial step towards improving the legal profession and broad- ening access to legal services.
wide array of legal tasks including legal research (Huang et al., 2021: Os- tendorff et al., 2021), legal reasoning (Guha et al., 2023; Mahari, 2021), contract review (Hendrycks et al., 2021; Leivaditi et al., 2020), statutory inter- pretation (Nyarko and Sanga, 2022; Savelka et al., 2019), document review (Yang et al., 2022; Zou and Kanoulas, 2020), and legal question answer- ing (Vold and Conrad, 2021; Khazaeli al., 2021; Martinez-Gil, 2023).
Despite their potential improve the provision of legal services, the legal industry has been slow to adopt computational techniques. The majority of legal services continue to rely heavily on manual work performed by highly trained human lawyers. This slow adoption may be partially attributed to risk aversion, misaligned incentives, and a lack of expertise within the legal community (Livermore and Rockmore, 2019; Fagan, 2020).
The current model of legal services is failing to address legal needs in several important con-
We argue that there another factor to blame,

---

rooted in legal practice but rather in legal NLP research: In short, legal NLP is failing to develop many applications that would useful for lawyers. Instead, legal NLP research tends to focus on generic NLP tasks and applies widely-used NLP methodologies to legal data, rather than developing new NLP tools and approaches that solve problems unique to the legal context.
Jabeling
recognision
network
charg
text
jud
g
ment
t
Knowl
edge
detectsor al
on
bert
inproiing
npp
case
retr
leve]
ee
datacet
cour
pewcest
t
arti
a
tritfiiiia
reciit
predictio coiiioonn
autona
ted
Just
Laggaanguue using
mulm
t
document en
aanguage
extractii
caned
proces
g
For example, NLP research might apply text clas- sification to predict the direction of a U.S. Supreme Court judgment based on portions of the judicial opinion. These types of models tend to be of lim- ited practical utility: First, the vast majority of lawyers and legal disputes will never reach the Supreme Court. Second, the legal reasoning ap- plied by the Supreme Court is unlikely to be rep- resentative of lower courts. And lastly, classifiers trained on published judgments may emulate judi- cial idiosyncrasics rather than modeling optimal legal reasoning. Meanwhile, there has been less research on systems to help lawyers identify rel- evant precedent for trial, on exploring automated summarization and generation of legal documents, leveraging NLP for online dispute resolution.
models
documen
roce
classirication
servicte
Figure 1: Word cloud of legal NLP paper titles.
legal practice but which remain underexplored.
Our work adds to growing number of recent position papers discussing the intersection of law and NLP (see e.g., Dale, 2019; Zhong et al., 2020; Tsarapatsanis and Aletras, 2021; de Oliveira al., 2022; Katz et al., 2023). The number of survey papers this domain might suggest some confu- sion about the state of legal NLP. In addition to offering descriptive findings, we aim to provide a normative argument. In light of the access to jus- tice issues highlighted above, we encourage legal NLP researchers pragmatically focus work that promises broaden access to justice. This objective helps advance a shared normative goal that does not neglect the 'legal' dimension legal NLP.
The work done by Livermore and Rockmore (2019); Katz et al. (2021); Cabral et al. (2012); Rhode (2013) and others takes an important step toward bridging disciplinary disconnects by pro- viding overviews of NLP and related methods to legal and multidisciplinary communities. We hope to build this work by encouraging the legal NLP community understand the needs of legal prac- titioners. Our paper offers some initial starting points for NLP research that are informed by prac- tical nceds.
To summarize, we make the following contribu- tions and recommendations.
(1) We review the current state of legal NLP.
(2) We discuss underexplored areas of legal NLP research.
(3) We propose the use of legal NLP to tackle the access to justice crisis shared normative goal.
We base our argument on review of recent le- gal NLP research, which identifies key themes in this literature (see Figure 1). We find that large portion of this rescarch focuses on tasks which we believe are discomnected from the needs of legal practitioners. We further observe that only a small fraction of citations to legal NLP publications stem from legal publications, providing evidence that NLP publications have not managed to rise to the attention of the legal community (see left panel of Figure 2). Grounded in this review, we seg- ment legal NLP tasks into three categories: applica- tions that could aid the provision of legal services; widespread NLP applications that have limited im- pact on practical legal issues; and areas of legal NLP research that could have significant impact on
(4) We advocate for more collaboration between NLP researchers and legal scholars and practi- tioners.
2 Literature Review
We conduct a rapid literature review via forward citation chasing over 171 papers, following rec- ommendations made by Khangura et al. (2012). Literature reviews have been employed in several position papers investigating the intersection of NLP and other disciplines (see e.g., Laureate al., 2023; Ricketts et al., 2023).
The starting point of our rapid review is Zhong

---

Paper Types
Paper Objectives
160
148.
20
20
02-
60
40-
10
22
Hattoral Longrage aaaa Precenving
foterd scativerr
Legal Rastemees Presiction
frde Aevettton futracaal
reformatio fio Ereraction
citvr
Deed Baiserr Coppua
Fraaiso FoiS
Langrigh Morkting
Decarmmt Clasvitren vir Summme lzacon
Fockow Fromr
daredton Acrweetigg imergverade saperr
Fameei
Proeeeriin Crassh Lasall antaiieee
Figure 2: (Left) Bar plot of paper categories in our literature review (y-axis is the total number of papers of each type). (Right) Bar plot of objectives in the reviewed papers (y-axis is the total number of papers for each objective).
et al. (2020), an overview paper about how legal NLP benefits the legal system. We selected this work our starting point because it is recent contribution in top NLP conference that has re- ceived reasonably high number of citations for us to review. Moreover, it provides (to the best of our knowledge) a fairly accurate overview about the current state of legal NLP. Our review includes all papers citing Zhong et al. (2020) on Google Scholar.
legal judgment prediction and question answering. We list all of these in Appendix A. If a paper de- scribes objective that cannot be classified into one of these categories, we define new category. After every 20 annotations, we review all categories assigned so far and decide whether to merge exist- ing categories (e.g., named entity recognition and semantic role labeling merge into information ex- traction, while similar case matching and semantic search merge into information retrieval) - this rep- resents the deductive step in our approach. After annotating all 171 papers, we review all categories again and perform final merge. Finally, for each paper. we determine whether it is a legal, NLP, or interdisciplinary publication. This categorization is based on the publication venue and the author's affiliations.
Zhong et al. (2020) aim identify popular legal NLP tasks and discuss these tasks from the perspec- tives of legal professionals. Tasks outlined in the paper include legal judgment prediction, question answering, and similar case matching.
We read and manually annotate all papers cit- ing Zhong et al. (2020), resulting in 171 annotated papers.' For each paper, we determine:
We display our main findings in Figure 2. We find that despite the inherently interdisciplinary nature of legal NLP, most work we reviewed is pro- duced and consumed by the NLP community. Only 10% of the papers citing Zhong et al. (2020) are pri- marily interdisciplinary, Perhaps more strikingly, only single law review article cites Zhong et al. (2020). In terms of paper objectives, we find that legal judgment prediction appears to be the most popular objective, with 20% of reviewed papers focusing on this task.
(1) Is the paper is mainly a NLP contribution, an interdisciplinary contribution, or a legal contri- bution?
(2) What are the paper's key objectives (e.g.. legal judgment prediction, summarization, position paper, introducing a benchmark)? A paper can have multiple objectives.
We follow hybrid dlluctive-deductive content analysis methodology, employed in Birhane al. (2022), and aim follow best practices and recom- mendations from the qualitative content analysis lit- erature (see e.g., Merriam and Grenier, 2019; Krip- pendorff, 2018; Hsieh and Shannon, 2005; Bengts- son, 2016).
3 Categorizing legal NLP applications
We start with set of ten objectives described in Zhong et al. (2020), such as relation extraction,
Based on our review, we categorize popular legal NLP objectives terms of their ability impact the practice of law or broaden access to justice. Subsequently, we identify potentially impactful ar- eas of research that remain underexplored.
't the time of writing, the paper had 182 citations in total. We discarded nine citations for which Google Scholar did not
provide a link.

---

3.1 Existing applications that promise to aid legal practitioners
to predict the associated judgment. This approach is problematic because the narrative presented by judges in their opinions is typically crafted with an outcome in mind, thereby precluding neutrality in the facts they present. As such, LJPP treats human- generated annotations as ground truths when in fact these annotations are based on confounding factors. Moreover, the automation of legal judg- ments is fraught with ethical challenges. Biased judgments would have grave social implications, not only for litigants directly affected by inaccurate legal judgments but also society at large auto- mated judgments undermine trust in the judicial system. LJP may have utility for low-stakes dis- putes that may not otherwise see a day in court, or it could be used to simulate a specific judge's or court's idiosyncrasies, which may be helpful strategic tool for potential litigants. Furthermore. LJP might also be useful to surface existing biases in judicial decision making. However, LJP is typi- cally framed as modeling the "correct" application of laws to facts. Due to its inherent risks, this ap- plication should be carefully considered and it is unlikely to materialize in the near future, if at all.
We identify three primary streams of legal NLP research that promise to benefit legal practice and improve access to justice.
Legal document generation and analysis. NLP technology can help speed up drafting, e.g., via ded- icated legal language models or general-purpose language models such as GPT-4. It can also help analyze legal documents, for example via informa- tion extraction or summarization (Galgani et al., 2012; Bommarito II et al., 2021). Based on a sur- vey of senior attorneys (see Appendix B), we note that document review and generation appear to be critical tasks from the perspective of many practi- tioners.
Semantic search. Legal arguments rely heavily on citations to statutes and precedential court opin- ions. Several scholars have thus focused on de- signing systems that aid attorneys in finding cita- tions to prior court decisions that support their ar- guments (Huang et al., 2021; Tang and Clematide, 2021).
It is important to underscore that other common legal NLP tasks may not directly aid legal practi- tioners, but nevertheless provide valuable resources and insights. These include detecting bias in legal language (Rice et al.. 2019) and legal NLP bench- marks which help measure the progress of NLP methods (Chalkidis et al., 2022: Guha et al., 2023).
A Accessibility of legal language. The translation of legal jargon into more accessible forms has been identified as important priority by legal schol- ars (Benson, 1984). Here, style transfer methods, legal summarization, question answering and infor- mation extraction methods can all prove helpful to make legal language more accessible and to sur- face key concepts (see e.g.. Farzindar and Lapalme, 2004; Manor and Li, 2019; Khazacli et al., 2021). These applications can help judges quickly under- stand filings submitted by attorneys, aid lawyers in gaining an overview of documents, and can help individuals better understand contracts, wills and other documents that may be relevant to them.
3.3 Underexplored applications that promise to aid legal practitioners
Understanding the nature of legal practice in more detail can help surface applications of NLP that would be useful to legal practitioners. Of course, this easier said than done as there are still lim- ited opportunities for legal NLP researchers and legal practitioners to exchange ideas. For this dis- cussion, we draw partially on survey conducted as part of Harvard Law School's s 2023 Leadership in Law Firms program (LLF). This survey asked over 50 senior attorneys from 17 different coun- tries to identify potentially impactful applications of NLP which would provide value in their firms (see Appendix B for an overview of responses).
3.2 Applications that fail to aid legal practitioners
Some legal NLP publications focus on tasks that are simply not part of legal practice or that use legal data in ways that do not fully account for how this data was generated. Other publications focus tasks with significant ethical implications that make them ill-suited for real-world deployment.
Legal judgment prediction (LJP), the most com- mon task identified in our review, suffers from both of these weaknesses. First, LJP typically extracts facts from court opinions and then uses the facts
We recognize that these responses are not representative of legal practice generally, but present them us a valuable example of how practitioners think about NI.P and as a starting point for idcation.

---

Persuasive legal reasoning. Litigation is least partially& a rhetorical exercise in which attornes seek to identify the most persuasive arguments while taking into account the presiding judge and, in some instances, the composition of the jury. The nature of litigation offers ample opportunity for the study of language in general, and the study of discourse and pragmatics specifically. Extrane- factors, like the presiding judge, have a sig- nificant impact on the persuasiveness of different arguments, and there already exists NLP research on context-aware argumentation (see e.g. Durmus et al., 2019) that could be applied to law.
evant records and, based on these retrieved records, generate new documents. Retrieval augmented generation (see e-g. Lewis et al., 2020; Borgeaud et al., 2022: Shi et al., 2023) is well suited to this task, however, it is critical that confidential records are not leaked to external parties or other public databases (Arora et al., 2023), and that generation is performed in an auditable fashion (Mahari et al., 2023).
Discussion
The disconnect between AI research on applica- tions and specific disciplines is not limited law (see e.g. Acosta et al., 2022). Law, however, is unique among disciplines in that it is a field built on language. Given the current state of legal practice, there is need for innovation to make legal services more affordable and to address the access to justice crisis. As such, law presents valuable opportunity for the NLP community to conduct research on ap- plications that could aid legal practitioners and that expand access to justice.
Practice-oriented legal research tools. Legal re- search and case analysis was one of the key areas identified in the LLF Survey. In common law juris- dictions, law develops organically through judicial opinions and higher courts may overturn or refine past court decisions. Legal research platforms la- bel whether case is considered "good law", that is whether it remains good basis for future ar- guments and thus current law. Current citation prediction work has largely ignored this aspect, cre- ating a risk that outdated or overturned opinions are recommended. NL. research techniques such as sentiment analysis could identify how a citation is used by judges to determine whether it remains good law.
Impactful legal NLP rescarch must be grounded in the needs of the legal community. The observed lack of cross-disciplinary citations in our review suggests that legal NLP researchers are largely dis- connected from the legal community. We encour- age legal NLP researchers identify tasks that are performed frequently by legal practitioners and that lend themselves to the application of NLP tech- niques. To aid NLP researchers in identifying these tasks, we urge them to consider closer interdisci- plinary collaborations with the legal community or at least to address legal issues identified in the legal literature.
related extension is the retrieval of individual legal passages. Judicial opinions are generally long documents and legal practitioners normally cite very specific passages. As a result, legal research platforms often present specific passages "head notes" or "key cites" to allow lawyers and judges identify the most important portions of opinions. Legal passage prediction (L.P) seeks to predict spe- cific passages, rather than entire judicial opinions, which is more closely aligned with the needs of legal professionals (Mahari, 2021). LPP may also be combined with extractive summarization (see e.g. Bauer et al., 2023), to identify passages from an opinion that are most likely to represent useful citations.
Conclusion
By leveraging a literature review, we find that the legal NLP community is largely disconnected from legal academia. We emphasize that certain popular legal NLP tasks are only of limited utility to legal practitioners. We thus urge legal NLP researchers to focus on access to justice as a shared norma- tive objective, to ground their work in the needs of the legal community, and to consider collabo- rating with lawyers to ensure that their research has applications in practice. NLP has the poten- tial to positively transform the practice of law and by extension society. However, this is impossible without cross-disciplinary understanding and col- laboration.
Retrieval augmented generation over private le- gal data. A more general opportunity for legal NLP is related to proprietary legal data. Law firms amass large knowledge banks from past cases that contain sensitive and confidential data. Practicing attorneys routinely build on their past work and experience. NLP tools could help them identify rel-

---

Limitations
broad and thoughtful discussion about the future of legal NLP.
Business applications. Reviews of NLP litera- ture provide insight into academic work, but they do not reveal business applications of NLP. While we observe disconnect between NLP research and law in academia, it is possible that there ex- ists unpublished work that is more attuned to the needs of the legal community. However, this work tends to focus on profitable applications, which are not always aligned with broadening access to jus- tice. The LLF survey provides a business-oriented perspective by surveying senior attorneys from an international group of law firms, however, more ex- haustive work is needed to fully understand where NLP tools provide value to law firms and to what degree these offerings also address access to justice issues.
Acknowledgements
We are grateful to the Harvard Law School Center on the Legal Profession, Scott Westfahl, and David Wilkins for helping us gain insight into the needs of legal practitioners; we appreciate all participants in the Law Firms Leadership Program for taking time share their perspectives with us. We thank Ron Dolin for his extremely helpful feedback on this paper.
References
Julián N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric Topol. 2022. Multimodal biomedical AI. Nature Medicine, 28(9):1773-1784,
Scope. We conduct a rapid review based on cita- tions to popular NLP paper. Our intuitions about the field lead us to believe that our findings extrap- olate to the field a whole. Contemporaneous work provides broader overview and identifies similar trends our review. For example, Katz al. (2023) find that classification is the most pop- ular objective in the legal NLP community (LJP
Simran Arora, Patrick Lewis, Angela Fan, Jacob Kahn, and Christopher Ré. 2023. Reasoning over public and private data in retrieval-baseee systems. Transac- tions of the Association for Computational Linguis- tics, 11:902-921.
Emmanuel Bauer, Dominik Stammbach, Nianlong Gu, and Elliott Ash. 2023. Legal extractive sum- marization of US court opinions. arXiv preprint arXiv:2305.08428.
represents a classification task). While our work is similar in spirit to Katz et al. (2023), we take a less descriptive but more normative approach.
Ralph Baxter. 2022. Dereliction of duty: State-bar inac- tion in response to america's 's access-to-justice crisis. Yale Law Journal Forum, 132:228.
Ethics Statement
Mariette Bengtsson. 2016. How to plan and perform qualitative study using content analysis. NursingPlus Open, 2:8-14.
If aligned with societal needs, legal NLP has tremendous potential to expand access to justice, to reduce biases in legal practice, and to create new ef- ficiencies in legal practice. At the same time, legal NLP deals with sensitive aspect of society. Poorly designed NLP tools could embed biases, remove human oversight, or undermine trust in the legal system. Our hope is that encouraging collaboration between NLP and legal researchers will also help identify and mitigate ethical challenges.
Robert W Benson. 1984, The end of legalese: The game is over. New York University Review of Law & Social Change, 13:519.
Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew. Ravit Dotan, and Michelle Bao. 2022. The values encoded in machine learning research. In Pro- ceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency. FAceT '22, page 173-184, New York, NY, USA, Association for Com- puting Machinery.
Broader Impact. This publication is a perspec- tive about the current state of legal NLP and its future directions, grounded in evidence about inter- disciplinary disconnects. Of course, the trajectory of an academic field ought to be based on delibera- tive discussions involving many stakeholders. We present our recommendations and visions about the future of legal NLP, which are, at least to some extent, subjective. We invite others to expand on and to critique our views and hope to contribute to
Michael J Bommarito II, Daniel Martin Katz, and Eric M Detterman. 2021. LexNLP: Natural language processing and information extraction for legal and regulatory texts. In Research Handbook on Big Data Law, pages 216-227. Edward Elgar Publishing.
Rishi Bommasani. Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, et al. 2021. On the opportuni- ties and risks of foundation models, arXiv preprint 108.v:2108.07258.

---

Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, -ean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Mag- giore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving. Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR.
Frank Fagan. 2020. Natural language processing for lawyers and judges. Michigan Law Review, 119:1399.
Atefeh Farzindar and Guy Lapalme, 2004, Legal text summarization by exploration of the thematic struc- ture and argumentative roles. In Text Summarization Branches Out, pages 27-34, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Filippo Galgani, Paul Compton, and Achim Hoffmann. 2012. Combining different summarization tech- niques for legal text. In Proceedings of the workshop on innovative hybrid approaches to the processing of textual data, pages 115-123.
Erica Bosio. 2023. A survey of judicial effectiveness: The last quarter century of empirical evidence. Tech- nical report, The World Bank.
Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas- Wood, Austin Peters. Brandon Waldon, Daniel N Rockmore, et al. 2023. Legalbench: A collabo- ratively built benchmark for measuring legal rea- soning in large language models. arXiv preprint ar arXiv:2308. 1462..
James E Cabral, Abhijeet Chavan, Thomas M Clarke, and John Greacen. 2012. Using technology to en- hance access to justice. Harvard Journal of Law & Technology, 26:241.
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated NLP dataset for legal contract review. arXiv preprint Ttt::2103.06666
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 43 1310-4330, Dublin, Ireland. Association for Computational Linguistics.
Hsiu-Fang Hsieh and Sarah E. Shannon. 2005. Three approaches to qualitative content analysis. Quali- tative Health Research, 15(9):1277-1288 PMID: 16204405.
Zihan Huang, Charles Low, Mengqiu Teng, Hongyi Zhang, Daniel E Ho, Mark S Krass, and Matthias Grabmair. 2021. Context-aware legal citation recom- mendation using deep learning. In Proceedings of the eighteenth international conference on artificial intelligence and law, pages 79-88.
Luciana Gross Cunha, Daniela Monteiro Gabbay. José Garcez Ghirardi. David M Trubek, and David Wilkins. 2018. The Brazilian Legal Profession in the Age of Globalization. Cambridge University Press.
Robert Dale. 2019. Law and word order: NLP legal tech. Natural Language Engineering.
Daniel Martin Katz, Ron Dolin, and Michael Bommar- ito. 2021. Legal informatics. Cambridge University Press.
25(1):211-217.
Leonardo Ferreira de Oliveira, Anderson da Silva Gomes, Yuri Enes, Thaíssa Velloso Castelo Branco, Raíssa Paiva Pires, Andrea Bolzon, and Gisela Demo. 2022. Path and future of artificial intelligence in the field of justice: systematic literature review and a research agenda. SN Social Sciences, 180)
Daniel Martin Katz, Dirk Hartung. Lauritz Gerlach, Abhik Jana, and Michael J. Bommarito II au2. 2023. Natural language processing in the legal domain.
Sara Khangura, Kristin Konnyu, Rob Cushman, Jeremy Grimshaw, and David Moher. 2012. Evidence sum- maries: the evolution of rapid review approach. Systematic Reviews, 1(1):00.
Esin Durmus, Faisal Ladhak, and Claire Cardie. 2019. The role of pragmatic and discourse context in de- termining argument impact. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNLPP
Soha Khazaeli, Janardhana Punuru, Chad Morris, San- jay Sharma, Bert Staub, Michael Cole, Sunny Chiu- Webster, and Dhruv Sakalley. 2021, A free format legal question answering system. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 107-113, Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. GPTs are GPTs: An early look the labor market impact potential of large language models.
Klaus Krippendorff 2018. Content analysis: An intro- duction to its methodology. Sage publications.

---

Caitlin Doogan Poet Laureate, Wray Buntine, and Henry Linger. 2023, systematic review of the use of topic models for short text social media analysis. Artificial Intelligence Review, pages 1-33.
Douglas Rice, Jesse H. Rhodes, and Tatishe Nteta. 2019. Racial bias in legal language. Research & Politics. 6(2):2053 (8066611190 18819994933
Jon Ricketts, David Barry, Weisi Guo, and Jonathan Pelham. 2023, A scoping literature review of natural language processing application to safety occurrence reports. Safety, 9(2):22.
Spyretta Leivaditi, Julien Rossi, and Evangelos Kanoulas. 2020. A benchmark for lease contract review. arviv preprint arXiv:2010.10386
Jaromir Savelka, Huihui Xu, and Kevin D Ashley. 2019. Improving sentence retrieval from case law for statu- tory interpretation. In Proceedings of the seventeenth international conference on artificial intelligence and law, pages 113 113-122.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin. Naman Goyal, Hein- rich Küttler, Mike Lewis. Wen-tau Yih, Tim Rock- täschel, al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459- 9474.
Weijia Shi. Sewon Min, Michihiro Yasunaga, Min- joon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiv preprint arXiv:2301. 12652.
Michael A. Livermore and Daniel N. Rockmore. 2019. Law Data: Compulation, Text, and the Future of Legal Analysis. The Santa Institute Press.
Mary C. Slosar. 2022. The justice gap: The unmet civil legal needs of low-income americans. Technical report, Legal Services Corporation.
Robert Mahari. 2021. Autolaw: Augmented legal rea- soning through legal precedent prediction. arXiv preprint arXiv: 100:::2106660:: 16034.
Li Tang and Simon Clematide. 2021. Searching for le- gal documents at paragraph level: Automating label generation and of an extended attention mask for boosting neural models of semantic similarity. In Proceedings of the Natural Legal Language Pro- cessing Workshop 2021, pages 1 114-122, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Robert Mahari, Tobin South, and Alex Pentland. 2023. Transparency by design for large language models. Computational Legal Futures, Network Law Review.
Laura Manor and Junyi Jessy Li. 2019. Plain English summarization of contracts. In Proceedings of the Natural Legal Language Processing Workshop 2019, pages 1-11, Minneapolis, Minnesota. Association for Computational Linguistics.
Dimitrios Tsarapatsanis and Nikolaos Aletras. 2021. On the ethical limits of natural language processing legal text. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages Online. Association for Computational Linguistics.
Jorge Martinez-Gil Gil. 2023. A survey on legal question- answering systems. Computer Science Review, 48:100552.
Sharan B. Merriam and Robin S. Grenier. 2019. Quali- tative Research in Practice: Examples for Discussion and Analysis. Wiley.
Andrew Vold and Jack G Conrad. 2021. Using trans- formers to improve answer retrieval for legal ques- tions. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pages 249-249.
Julian Nyarko and Sarath Sanga. 2022. A statistical test for legal interpretation: Theory and applications. The Journal of Law, Economics, and Organization, 38(2)5339-56.
David B Wilkins, Vikramaditya S Khanna, and David M Trubek. 2017. The Indian Legal Profession in the Age of Globalization. Cambridge University Press.
OpenAl. 2023. GPT-4 technical report.
World Justice Project. 2019. Global insights on access to justice: Findings from the world justice project general population poll in 101 countries.
Malte Ostendorff, Elliou Ash, Terry Ruas, Bela Gipp, Julian Moreno-Schneider, and Georg Rehm. 2021. Evaluating document representations for content- based legal literature recommendations. In Proceed- ings of the Eighteenth International Conference on Artificial Intelligence and Law, pages
Eugene Yang. Sean MacAvaney, David D Lewis, and Ophir Frieder. 2022. Goldilocks: Just-right tuning of BERT for technology -asssseed review. In European Conference Information Retrieval, pages 502-517. Springer.
Nicholas M. Pace, Malia N. Brink, Cynthia G. Lee, and Stephen F. Hanlon. 2023. National public defense workload study. Technical report, RAND Corpora- tion.
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun, 2020. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational
Deborah L Rhode. 2013. Access to justice: A roadmap for reform. Fordham Urhan Law Journal, 41:1227.

---

Linguistics. pages 5218-5230, Online, Association for Computational Linguistics.
(5) Simulating predicting legal outcomes (2/59)
Project and process management (2/59)
Jie Zou and Evangelos Kanoulas. 2020. Towards
Knowledge management (1/59)
question-basee bassod high-recall information retrieval: Lo-
Raw Responses
cating the last few relevant documents for technology- assisted reviews. ACM Transactions Information Systems (TOIS), 3-33).1-35.
Document review and summaries
2. Case review intake
3. Initial research from multiple sources create first draft
memo. For financial services regulatory
A Initial Objectives
Document review, precedents, process improvement
5. Time entry
Case analysis and statistics; usage in the discovery process Billing Process
Relation Extraction
8. Time entry
Event Timeline
Summarise an extensive document or prepare well sub- stantiated research
Element Detection
Legal Judgment Prediction
10. Financial/tax modelling. document review
11. Documentation review, regulatory review
Question Answering
12. Analysis of high volume procedural evidence
Similar Case Matching
13. Document review
Summarization
14. Time entry
15. Legal research (c.g.. finding relevant case law)
Embeddings
16. Time recording
Knowledge Graphs
17. Predictive tool for client outcomes
Language Models
18. Document review
19. Time entry
20. Documents review
Table 1: Set of initial objectives for the literature review.
21. Document and regulatory review
22. Document review in large, complex litigation
23. First draft of letters to media and social media
B Harvard Leadership in Law Firms:
24. Review of trading data in securities enforcement matters
25. Time entry and hilling
Survey Responses
26. Legal reseurch and regulatory option
27. Use of Al in the analysis of the firm's own data sets in
As part of Harvard Law School's Leadership in Law Firms program, the lead author of this pa- per conducted a survey of over 50 senior attorneys from 17 different countries on applications of NLP in legal practice. The survey asked: What is one degal-related task (e.g., document review, respond-
order to make use of expertise available in the firm as quickly and effectively as possible, such as previously prepared expert opinions topic. Furthermore, it should be possible to search external databases as effectively as possible 28. Document review
29. Document review
30. Dealing with AMI and KYC ohligatiins
31. Document review
ing motion), system (e.g., time entry or giving feedback), type of legal matter (deals, regulatory review) that you would LOVE for generative Al to make easier/more efficient?
32. Time recording
33. Admin related tasks around time entry, review of accounts
34. One reliable source with brief and regular updates
case-law, legislation and important developments including access in-depth information for the individual sources of law
While by no means representative of the legal industry as whole, the survey responses provide valuable insight into the priorities of practicing attorneys. As such, they serve starting point for some new avenues of legal NLP research and an example of how the research community can
35. Brief writing, time entry, legal research
36. Document Review and Legal Search time entry regula- tory review
37. Lease summaries
38. Document review
39. Time entry
40. Giving feedback
solicit insights from practitioners.
41. Automating tasks/workflows in the sense of having
spread sheet/document assistant
At a high level, the following application cate-
42. Document review in diligence processes
gories emerged from the survey:
43. Time keeping
(1) Legal and business document review and sum-
44. Document Review
45. Document review (both consulting and litigation)
marization (42/59)
46. Document review
(2) Time entry and billing, case intake, and review-
47. Document review
48. A system providing full and reliable overviews on legal topics by analyzing all relevant sources including legislation. legislation processes, case law and literature. This would help to often spend long time on getting certainty about being
ing invoices (14/59)
(3) Case law research and regulatory review
(11/59)
to date
(4) Legal document generation, creating multi- document summaries (3/59)
49. Document review
50. Comparative summary of publications and judgments

---

Horizon scanning for regulatory change
52. Document review
Analysis and comparison of different information sources,
Intranet. Internet, databases
Document review
counseling and litigation
Document review, regulatory review
56. Document Review
57. Legal-related task: document review and research; Sys-
tem: project management; Legal Matter: Due Diligence/deals
58. Administrative things like time entry or reviewing invoices
59. Summarise big volume of data


Reviews:
[{"review_id": "2k0AOBCEdg", "reviewer": "Reviewer_6D6N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for this constructive comment. To clarify our approach, we have taken the Zhong et al. 2020 paper as a starting point and then conducted a rapid literature review of the papers that cite the Zhong et al. paper, which, by construction, were all written in 2020 or later. From the 170 papers examined, roughly half are quite recent – from 2022 or 2023. Further, this set of papers includes (in our judgment) the most relevant new research in legal NLP, including (but not limited to) “LexGLUE” (Chalkidis et al. 2021), “GPT-4 passes the bar exam” (Katz et al 2023), and “LegalBench” (Guha et al. 2023). Hence, we feel that this set of papers fairly represents the legal NLP field. A more recent broad legal NLP review, “Natural Language Processing in the Legal Domain” (Katz et al. 2023), has only been cited seven times and those papers also cite the Zhong et al. paper. While it would be possible to further extend or restrict this list in various ways, this would be ad-hoc, subjective, and unlikely to be more representative of new research in the field. Our approach is based on best practices for conducting such systematic literature reviews.\n\nWe make these points more explicit in a revised version and hope that this remark somewhat alleviates your concerns about the validity of the conclusions, especially with respect to the more recent developments in NLP and AI for law.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The claims made are valid in many domains; however, in this case, they are based on a systematic review of over 170 scientific articles.", "reasons_to_reject": "The study is based on the articles reviewed in Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang 509 Zhang, Zhiyuan Liu, and Maosong Sun. 2020. In the past three years, however, NLP and AI have developed very quickly, and there is certainly a lot of new research in the field. This raises questions about the conclusions reached.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper makes the case that legal practising still involves a significant amount of human labour, in part because popular NLP tools being tailored to legal documents rather than new NLP tools being created to address the specific challenges faced by the legal profession.", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rRfbROiC13", "reviewer": "Reviewer_V1t3", "summary": "", "questions": "Question 1A: Do you think that there is some way in which predictive tasks could be somehow salvaged? I am thinking here of either prediction that uses metadata of judgments (e.g. judges' names) or other, potentially helpful, kinds of textual data.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very positive review! To address your question somewhat qualitatively: we believe that predictive tasks should for now mostly be used to investigate existing biases in the legal system, rather than as autonomous decision-making systems, which could have far-reaching negative externalities if deployed (see e.g., the ProPublica investigation of the COMPAS tool). Specifically, prediction tools that use features that ought to be irrelevant to a given decision (e.g., past decisions or judges’ public remarks) can help expose biases. To this extent, metadata on judgments can and should certainly be used, e.g., as fixed effects. That said, “Human Decisions and Machine Predictions\" (J Kleinberg et al. 2017) suggests that prediction could help judges render “correct” decisions, and given adequate safeguards it is possible that automated judgment prediction will one day augment judges. \n\nWe incorporate these recommendations in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper makes a significant contribution to oingoing debates, because (a) it summarises existing literature very well and (b) makes a clear (and, to this reviewer's mind, persuasive) case for a re-orientation of legal NLP. It achieves the above in a clear and concise way. It should therefore be accepted.", "reasons_to_reject": "I see no reasons to reject the paper. On the contrary, I think that it deserves a wide discussion.", "questions_to_authors": "Question 1A: Do you think that there is some way in which predictive tasks could be somehow salvaged? I am thinking here of either prediction that uses metadata of judgments (e.g. judges' names) or other, potentially helpful, kinds of textual data.", "paper_topic_and_main_contributions": "This is a position paper that very helpfully sets out ways in which 'legal NLP' could develop so as to help legal practitioners (and perhaps academics), in areas other than predictive tasks. The paper's main contribution is that it criticises very clearly extant research trends and also very clearly says which areas of future research would be most helpful for practitioners.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "There are no references that I can think of which are missing.", "Typos_Grammar_Style_and_Presentation_Improvements": "The style is up to standard and, above all, expression is rigorous."}, "extra_text": {}}, {"review_id": "xxBU3V1lmd", "reviewer": "Reviewer_aLn7", "summary": "", "questions": "\n- What is \"interdisplinary\" (line 159)? Can you specify what is meant here: e.g. the publication venue is not computer science/nlp, or authors of a paper are affilicated with different departements? \n \n- Figure 2, right side: what is in \"other\"? it is the second category but we don't have precisions on what is included  in there. \n\n- The paragraph starting line 285 suggests that NLP researcher create more opportunity to acces proprietary data. Can you specify? \n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the exhaustive and constructive review which is highly appreciated! \n- To clarify our approach: We have taken the Zhong et al. 2020 paper as a starting point and then conducted a rapid literature review of the papers that cite the Zhong et al. paper, which, by construction, were all written in 2020 or later. From the 170 papers examined, roughly half are quite recent – from 2022 or 2023. Further, this set of papers includes (in our judgment) the most relevant new research in legal NLP, including (but not limited to) “LexGLUE” (Chalkidis et al. 2021), “GPT-4 passes the bar exam” (Katz et al 2023), and “LegalBench” (Guha et al. 2023). Hence, we feel that this set of papers fairly represents the legal NLP field. A more recent broad legal NLP review, “Natural Language Processing in the Legal Domain” (Katz et al. 2023), has only been cited seven times and those papers also cite the Zhong et al. paper. While it would be possible to further extend or restrict this list in various ways, this would be ad-hoc, subjective, and unlikely to be more representative of new research in the field. Our approach is based on best practices for conducting such systematic literature reviews.\n- Lack of discussion of other barriers to applying NLP in law: Thank you for this comment! We expand on these restrictions in the final version of the paper, and also highlight some of the industry publications for a more fine-grained discussion, e.g., Norkute et al., 2021 \"Towards Explainable AI: Assessing the Usefulness and Impact of Added Explainability Features in Legal Document Summarization\"\n- Relevance of LJP to legal practice: Thank you for your comment, other reviewers have provided similar feedback. In our revised manuscript we include a more nuanced discussion about the relevance of LJP to legal practice, in particular, the potential of LJP to be used to surface biases in judicial decision-making. We underscore that there are merits to LJP and that future research should expand on these. However, we do not view LJP as “the holy grail” of legal NLP and feel that common motivations for LJP fail to address its real-world implications and limitations, for example, \"Legal judgment prediction (LJP) is a fundamental task in legal AI, which aims to assist the judge to hear the case and determine the judgment.\" (Wu et al. 2022). We are primarily advocating for more interaction between legal practitioners and legal NLP researchers, which, among other things, involves a discussion about the merits and limitations of LJP, especially within the legal NLP community. \n- Regarding ground-truth annotations: While ground-truth labels exist, they are not always straightforward. The underlying documents, e.g., facts in a judicial opinion, are presented in ways that support the ultimate judgment. If the judgment were different, the presentation of facts likely would be different too. In practice, judgments are subjective, and it is often unclear whether two judges would come to the same conclusions, making the “ground truth” more complex than it first appears. Instead of thinking in terms of ground truths, it can be helpful to think about adversarial perspectives, for example by finding agreement on facts between the two opposing legal briefs. As you point out, carefully curated multi-document approaches can be helpful, although we find that even when LJP approaches are based on multiple documents, they commonly fail to take full account of how the documents were generated and what biases they may inherently contain. We clarify these points in the final version and again advocate for more communication between legal practitioners and legal NLP researchers to avoid such sources of confusion in the future. \n\nAnswers to your questions:\n- “Interdisciplinary” is based on publication outlets and/or author affiliations. A paper is labeled as interdisciplinary either if it appears in an interdisciplinary venue or if it has authors from both law schools and computer science departments.\n- The “other” category consists of 27 singleton items, e.g., part of speech tagging in the legal domain. We include all of these in the Appendix in the final version of the paper.\n- Regarding opportunities to access proprietary legal data: As you also highlight in your review, one of the key challenges for legal NLP is access to (clean) data. In working with legal practitioners, we have found that researchers can gain access to high-quality proprietary data that enables fruitful legal NLP work. This poses a challenge to the community: While there are excellent justifications for norms against publications that rely on proprietary data, the legal NLP community may benefit from providing at least some opportunities to publish contributions that would be infeasible without the use of such data. More generally, our revised manuscript contains an expanded discussion of data availability as a major challenge for legal NLP, building on work done in this area such as “How to build a more open justice system” (Pah et al. 2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The purpose of the paper is very relevant and has the potential to trigger and frame future research.\n\n- Legal NLP to expose biases is very useful (line 235), as legal decisions and algorithmic legal decisions have been shown in the past to hold and perpetuate biases against marginalised communities and already vulnerable groups. It is pointed out by the authors that LPJ  is not the most useful task and there is good arguments. However it should be said in nuance that LPJ aim is not always to automate judgements, but to expose biases for example (Early Predictability of Asylum Court Decision, Dunn et al.)\n\n- The authors point out that 2 tasks that are under-studied yet the most useful for the legal work citations extraction, and legal passage prediction. There are good arguments as to why those 2 tasks are essential (line 276) from the legal practitioners. Similarly, access to justice is under-studied in legal NLP, although it may be one domain where NLP could really make a difference thanks to scalability. \n\n", "reasons_to_reject": "- The paper analysed literature that is collected if citing one specific paper (Zhong et al.) which sounds restrictive as mentioned in the limitations section by the authors themselves. \n\n- There are other restrictions to the application of NLP in law that are hardly mentioned, notably the access to clean data and the access to annotated data. There are few legal datasets, and especially there are few legal datasets that are curated for research work. There is a large amount of publicly accessible data published online that is not easy to process for research (OCR documents for example). \nThis is mentioned in the paragraph starting line 285 through the prism of proprietary data access. However, regarding this paragraph: there is legal NLP research in industry as well that is regularly published in NLP outlets. While the paragraph makes a valid point, the use of certain legal data causes commercial issues, IP issues, and potentially privacy issues that also need to be taken into account.\n\n- There is strong point being made on LPJ being irrelevant to legal work. While this may sometimes be true, it is interesting to distinguish areas of law: it may be useful for a company to know if it is worth filing a case or not, or to know the amount of work that you need to put in depending the chances of success. LPJ may not have been done on datasets that seem aligned with this goal (eg SCOTUS), but 1/ this is taking us back to few datasets available and 2/ once the methodology has been explore on one dataset, one could argue that is is easier to replicate. Therefore it could be interesting to nuance this position. \n\n- Annotations of eg. events, names, locations are ground truth. The authors make the point that despite being correctly annotated, in may be the case that facts presented are incomplete and partial. While this might be true in some cases, it does not mean necessarily that the annotations are incorrect and don't have ground truth. This also assume that the LPJ is based on one single report document, when it may also process several documents (multi-document) that constitute the file. Therefore this argument seems weak. \nOn the problem of the truth of the annotations, it would be worth mentioning that when there is in fact no ground truth (annotation the reasons for the judgment, a legal argument for instance) the lack of clarity results more often from human annotator disagreeing on the qualification of an argument/token. \n\nOverall the paper has a very interesting starting point and purpose that would hugely benefit the legal NLP community. However it lacks clarity, and would benefit from being more grounded in the legal nlp literature and should aim to be exhaustive both on the analysed literature and in support to the normative arguments of the authors. \n", "questions_to_authors": "\n- What is \"interdisplinary\" (line 159)? Can you specify what is meant here: e.g. the publication venue is not computer science/nlp, or authors of a paper are affilicated with different departements? \n \n- Figure 2, right side: what is in \"other\"? it is the second category but we don't have precisions on what is included  in there. \n\n- The paragraph starting line 285 suggests that NLP researcher create more opportunity to acces proprietary data. Can you specify? \n", "paper_topic_and_main_contributions": "The survey paper analyses past legal NLP papers and their topic, and identifies which tasks are the most relevant to legal practitioners. The problem that the authors identify is that there is often a gap between legal nlp tasks and concerns and the requirements of legal work. ", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"The Law and NLP – Bridging Disciplinary Disconnects\" is the title on the PDF, the title on OpenReview is: \"Legal Passage Retrieval: A pragmatic approach to legal AI\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a short paper containing a blend of a (somewhat) systematic literature review and a position paper.
There is quite a gap between the scores of the reviewers that is mostly due to how to value these contributions, while they seem to agree about the "facts". 

In my view, it would have been better to either write a more systematic literature review or a real position paper.
Decision: Accept-Findings