Information Extraction from Legal Wills: How Well Does GPT-4 Do?
Authors: Alice Saebom Kwak
Cheonkam Jeong
Gaetano Vincent Forte
Derek Bambauer
Clayton T Morrison
Mihai Surdeanu
Venue: EMNLP 2023 Findings (2023)
Keywords: Information Extraction
Legal Natural Language Processing
TL;DR: This work presents a manually annotated dataset for Information Extraction (IE) from legal wills, and relevant in-context learning experiments on the dataset.

Abstract: This work presents a manually annotated dataset for Information Extraction (IE) from legal wills, and relevant in-context learning experiments on the dataset. The dataset consists of entities, binary relations between the entities (e.g., relations between testator and beneficiary), and n-ary events (e.g., bequest) extracted from 45 legal wills from two US states. This dataset can serve as a foundation for downstream tasks in the legal domain. Another use case of this dataset is evaluating the performance of large language models (LLMs) on this IE task. We evaluated GPT-4 with our dataset to investigate its ability to extract information from legal wills. Our evaluation result demonstrates that the model is capable of handling the task reasonably well. When given instructions and examples as a prompt, GPT-4 shows decent performance for both entity extraction and relation extraction tasks. Nevertheless, the evaluation result also reveals that the model is not perfect. We observed inconsistent outputs (given a prompt) as well as prompt over-generalization.

Information Extraction from Legal Wills: How Well Does GPT-4 Do?
Alice Saebom Kwak', Cheonkam Jeong', Gaetano Vincent Forte?,
Derek E. Bambauer", Clayton T. Morrison', Mihai Surdeanu"
'Deparmeent of Linguistics, The University of Arizona 2 JJames E. Rogers College of Law, The University of Arizona School of Information, The University of Arizona *Department of Computer Science, The University of Arizona (alicekwak, cheonkamjeong, fortegv, derekbambauer, claytonm, msurdeanuj@@ar izona. edu
Abstract
Beyond social impact, will understanding is an important application of artificial intelligence (AI) for several reasons. First, will execution is excel- lent use case for the flourishing smart contract plat- forms (National Science Foundation, 2022). More- over, closer to natural language processing (NLP), information extraction from wills can serve as an important evaluation platform for large language models (LLMs). Lastly, given that most wills are not publicly available,' evaluating LLMs on IE from wills is less likely to suffer from contamina- tion (Sainz et al., 2023) than other NLP tasks.
This work presents a manually annotated dataset for Information Extraction (IE) from legal wills, and relevant in-context learning periments on the dataset. The dataset consists of entities, binary relations between the enti- ties (e.g., relations between testator and ben- eficiary), and n-ary events (e.g., bequest) ex- tracted from 45 legal wills from two US states. This dataset can serve as foundation for down- stream tasks the legal domain. Another use case of this dataset is evaluating the perfor- mance of large language models (LLMs) on this IE task. We evaluated GPT-4 with our dataset to investigate its ability to extract infor- mation from legal wills. Our evaluation result demonstrates that the model is capable of han- dling the task reasonably well. When given instructions and examples as prompt, GPT-4 shows decent performance for both entity traction and relation extraction tasks. Never- theless, the evaluation result also reveals that the model is not perfect. We observed inconsis- tent outputs (given a prompt) as well as prompt wvergennrralization.
To enable such use cases, we introduce dataset for IE from legal wills, and pertinent in-context learning experiments on this dataset. In particular, the main contributions of this work are as follows:
We introduce manually annotated dataset for IE from legal wills. Our dataset consists of 16,018 annotations of entities, relations, and events extracted from 45 legal wills from two US states: Tennessee and Idaho. The dataset can serve foundation for many downstream tasks concerning legal wills (e.g.. automatic will review, electronic will creation. etc.)
1 Introduction
Wills erucial legal documents that enable in- dividuals maintain control over their assets and ensure that their wishes are carried over after their death. Unlike most legal documents, wills are com- monly written by/for lay people. Given the im- portance and ubiquity of wills, formalizing and developing legal procedures for understading, eval- uating and executing wills is crucial. Extracting key information from legal wills is prerequisite for understanding, evaluating and executing wills. Information extraction (IE) from legal wills serves as foundation for many downstream tasks such as automatic will review and electronic will creation and execution.
We evaluate GPT-4 with in-context learn- ing on our dataset, in both in-domain (i.c.. examples from the same state) and out-of- domain (OOD) (i.c., examples from another state) settings. The evaluation results demon- strate that GPT-4 is capable of handling the legal information extraction task in this in- context-learnngg setting. However, GPT-4 is not perfect: we observed inconsistent out- puts (given prompt) as well as prompt over- generalization.
'Court documents are in the public domain in the U.S., hut wills are usually hidden behind pay-walled websites.

---

2 Related Work
Take the Direction event in our work for instance. If what is being directed by testator is another event (e.g., excuse a duty of the executor) in a will, the Direction event can have an Excuse event as its argument. Extracting nested events adds additional layer of complexity to the IE task. This higher level of complexity distinguishes our task (i.e., IE from legal wills) from IE in other legal domains.
2.1 IE from Legal Documents
There exist several IE datasets and models in lc- gal domain. II al. (2018) presents LexNLP, which is an open source Python package that can be used for natural language processing and infor- mation extraction from legal and regulatory texts. Chalkidis et al. (2017) introduces a manually anno- tated dataset for extracting key elements from con- tracts. Similarly, Tuggener et al. (2020) presents LEDGAR, a multi-label corpus for text classifica- tion of legal provisions in contracts. Chen al. (2020) introduces manually annotated dataset for named entity recognition and relation extrac- tion in the Chinese legal domain, based on drug- related criminal judgment documents. Kirsch et al. (2020) proposes a probabilistic rule model for ex- tracting information from court process documents. Martín-Chozas and Revenko (2021) describes model extracting Hohfeld's deontic relations from legal texts. Hendryeks et al. (2021) introduces CUAD, an expert-annotated dataset for identify- ing key clauses from contracts. Cao et al. (2022) presents CAILIE 1.0, a Chinese legal information extraction dataset for larceny.
There has been no prior endeavor to create an IE dataset from legal wills, despite their importance. There is a legal will dataset provided by Kwak et al. (2022), but its focus is on natural language inference rather than IE. Our work addresses this gap.
2.2 Evaluating LLMs on IE Tasks
Several studies have evaluated LLMs' performance on IE tasks in various domains. Agrawal et al. (2022) evaluated GPT-3 on diverse clinical extrac- tion tasks in both a zero- and a few-shot learning settings and demonstrated that the model can han- dle the tasks well without training in the clinical domain. Stammbach et al. (2022) evaluated GPT- 3 on the task of extracting character roles from narrative texts in zero-shot question answering setting, and reported that the model is capable of handling the character role extraction task. Hanafi et al. (2022) evaluated GPT-3 on a text-pattern ex- traction task in a zero-shot setting and compared its performance with the rule-based model. He found that the rule-based model outperforms GPT-3 when GPT-3 is not given any prompts. Jimenez Gutierrez et al. (2022) evaluated GPT-3 on two biomedical IE tasks and compared its few-shot performance with the performance of smaller pre-trained lan- guage models (PLM) fine-tuned on the tasks. The result demonstrates that GPT-3 significantly under- performs a smaller PLM fined-tuned on the tasks. The previous studies suggest that LLMs are capable of handling IE tasks in a zero-shot or few-shot set- ting with in-context learning. Nevertheless, there is no consensus the performance of LLMs. Some reported the TE performance of LLMs in a positive way (c.g., Agrawal et al., 2022; Stammbach et al., 2022), while others reported it in a somewhat nega- tive way (c.g., Hanafi et al., 2022; Jimenez Gutier- rez et al., 2022). Further investigation is needed to verify LLMs' capability on IE tasks.
While IE from other legal documents tends to focus entities and flat relations, IE from le- gal wills can involve information with more com- plexity, such as nested events.? Chalkidis et al. (2017) extracts various contract elements such as contract title, dates, relevant law articles, but it does not extract any relations among such elements nor extract events from the document. Chen et al. (2020), Kirsch et al. (2020), and Martín-Chozas and Revenko (2021) do extract relations, but the ex- tracted relations are all flat ones. Chen et al. (2020) extracts four relations relevant to key criminal be- havior, and these relations all have entities as their arguments. Likewise, the relations extracted from Kirsch et al. (2020) are all between entities (e.g., fine amount and fine units, etc.). The relations ex- tracted from Martín-Chozas and Revenko (2021) all have entities (e.g.. LegalEntityy LegalDocument, LegalConcept, and etc.) as their arguments.
In contrast, the target events in our work can have both entities and events as their arguments.
What we mean by "nested" versus "flat" is whether an event can have another event as its argument. If an event has another event as its argument, it is nested event. On the other hand, if an event only has entities as its arguments, it is flat event.
Recently, few attempts have been made to eval- uate LLMs on the legal IE task or closely related tasks. Barale et al. (2023) evaluates different ar-

---

chitectures, including two BERT-based LLMs, on the legal IE task (i.e., extracting key information from refugee cases in Canada). Savelka (2023)
hereby give, devise bequeath all my property. real, personal or mixed, to [Person-2], livivin the time my death
Entities:
evaluates GPT-4 on a semantic annotation of legal texts. However, the domain and the task of these previous works are distinct from those of our work. Barale et al. (2023) tests BERT-based LLMs on the IE from refugee cases, while our work tests GPT-4 the IE from legal wills. Savelka (2023) evalu- ates GPT-4 as we do, but the task is different from ours (i.e., semantic annotation vs. IE). Hence, we evaluate GPT-4 on legal IE tasks with our dataset to investigate its capability of handling IE in this challenging domain.
testator: 1, my. my
trggger: hereby give, devise and bequeath, death
assel: my property. real, personal mixed
beneficiary: [Person-2) 2]
condition: living
of my death
time: at the time of my death
Relations:
testatar-benefictary
lestator: my
benefiiiary: [Person-2]
testator-asset
testator: my
ssset: my property. real, personal mixed
beneficiary-assed
testator: [Person-2]
asset: all of my property, real, personal or mixed
Events:
bequest:
testator:
3 Dataset
trigger: hereby give, devise and bequeath
assel: all of property, real, personal or mixed
3.1 Data Collection
beneficiary: [Person-2]
death:
We collected 45 wills from Ancestry, which con- tains documents in the public domain?. The wills are from two US states (Tennessee and Idaho) and their execution dates range from 1978 to 2001. We performed an OCR on the collected wills to extract texts. More details on the OCR process can be found in Appendix A The will texts were anonymized with special tokens prior to the anno- tation process. Anonymization was done manually by our annotators.
trigger: death
lestator:
Figure 1: An example of anonymized will statement, and corresponding annotations.
that includes entities, binary relations, and n-ary events. When deciding the annotation taxonomy, we consulted with a law professor and a law stu- dent. Appendix B contains the complete taxonomy of entities, relations, and events annotated.
3.2 Data Annotation
Evaluation
We annotated the data with the Label Studio En- terprise edition'. Annotation was performed by two annotators. They established a detailed anno- tation guideline to keep the consistency of their annotations (See Appendix for the annotation guideline). Moreover, the two annotators had reg- ular meetings to maintain the quality of the anno- tations. The inter-annotator agreement score was 74%, which was automatically calculated by the Label Studio with the basic matching functionS. We annotated 26 types of entities, 18 types of binary relations, and 20 types of n-ary events, re- sulting in 10,108 annotations in total. With binary relation we extract relations between two entities (e.g., testator-beneficcan beneeicrrbb..,,,ary)) while with n-ary events we extract events involving several arguments (e.g., bequest). Figure shows annotation example
Our dataset can be used to evaluate LLMs' per- formance on IE task. To this end, we used GPT- 46 with in-context learning. When evaluating the model, we restricted the range of the task to extracting the four most-common entities (testa- tor, beneficiary, asset, will) and the four most- common relations (estator-beneffcian) testator- asset, beneficiary testator- will) to reduce the complexity. Our dataset was randomly split into four sets: a training set (30 Tennessee wills), a de- velopment set (5 Tennessee wills), a test set (5 Ten- nessee wills), and an out-of-domain (OOD) set (5 Idaho wills). The training set served as example pool for developing prompt, and the development set was used to optimize the prompt. The evalu- ation was conducted on the test set and the OOD set.
*Court documents are in the public domain in the US. "https: //ti://llbbb /labelstud.d 'The Label Studio provides not only agreement score per token, but also the inter-annotator agreement. For further information, see httpp::/ddcs humansignal. stats.htm
We treated the wills from Idaho as the OOD set given the differences between the wills from Ten- nessee and the wills from Idaho. The most obvious
iiiiailnncoooo con/ guide/
hhttes///opeaa..om//resrrctt

---

Datasets
Precision
Recall
F1
Test Entities
0.65
0.69 (0.64-0.7)
0.67 (0.62 0.711
Test Relations
0.75 0.77 (0.70 - 0.83)
0.76 (0.60
0.75 (0.69 -0.90)
OOD Entities
0.74 (0.69 0.800
0.76 (0.71 0.80)
OOD Relations
0.81 (0.70 - 0.89)
0.85 (0.78 0.92)
0.83 06766 0.88)
Table I: The result of evaluating GPT-4 on the legal IE task with in-context learning. We report precision, recall, and Fl scores with confidence intervals (calculated using bootstrap resampling with 10,000 samples) for entity and relation extraction. The "OOD" rows contain out-of-domain results, where we used examples from Tennessee on wills from Idaho.
difference between the two is that they are writ- ten to comply with different legal requirements, as Idaho and Tennessee have different probate codes, While Idaho is one of the states that adopted the entire Uniform Probate Code (UPC), which is model law promulgated by the Uniform Law Com- mission, Tennessee only adopted a few provisions of the UPC. Tennessee's statutory framework is based upon state statutes, largely derived from the state's common law; it is, in other words, an exam- ple of the probate code heterogeneity that the UPC attempted to remediate. As they have different pro- bate codes, the wills from Idaho and the wills from Tennessee must conform to different requirements. This can dictate the way the wills are formulated both substantively and stylistically.
many cases, GPT-4 extracted multiple instances of single relation, creating a large set of duplicates. These duplicates were ignored and removed before the evaluation.
Results
The evaluation results are summarized in Table The table suggests that GPT-4 handles the IE task reasonably well. The F1 scores range from 0.67 to 0.83, which are decent given the strictness of the evaluation metric and the difficulty of the tasks. The scores from the entity extraction tasks are slightly lower than the ones from the relation ex- traction tasks (test: 0.67 vs. 0.75, OOD: 0.76 vs. 0.83). This difference is attributable the rigor- ousness of the evaluation metric used for entity extraction task (i.e., exact text match for all en- tity mentions), rather than GPT-4's performance difference between the two tasks.
We evaluated the model in an in-context learning setting where only prompt with a few examples is given without fine-tuning the LLM. The prompt uti- lized in the evaluation was manually crafted using the training set and development set. The prompt consists of two parts: instruction and example. In the instruction part, a detailed instruction regarding the IE task is provided. In the example part, exam- ples of human annotations are given to offer further guidance. After the instruction and the examples are provided, a will statement on which the model should perform the IE task is given with brief instruction. For the relation extraction task, a set of entities extracted from the given will statement (by human annotators) are also provided. See Ap- pendix D for the actual prompts used in our study. The LLM outputs were compared with the hu- man annotator's 's annotations. For the entity extrac- tion task, we used exact text match as an evaluation metric, i.c., if the entire span of the extracted text does not exactly match the human annotation, it is considered incorrect. For the relation extraction task, we considered an extracted relation correct when any two mentions of the entity arguments match exactly and the correct type is produced. In
It is also noticeable that GPT-4 performed better on the OOD data than on the in-domain test data. This result suggests that the prompts created from our training partition can be transferred and used to extract information from OOD data (e.g.. legal wills from the US states other than Tennessee).
6 Discussion
The evaluation result shows that GPT-4 is capable of handling the legal IE tasks without given train- ing nor fine-tuning. However, the error analysis revealed two major issues of GPT-4's performance on IE tasks few-shot setting, which are 1) incon- sistency and 2) overgeneralization of the prompt.
6.1 Inconsistency
GPT-4 is not capable of setting consistent criteria from the information given from the prompt. When not given training or fine-tuning, failed to show consistent performance in various situations (c.g.. whether to include possessive pronouns/quantifers

---

Error type
Test
OOD
document-related entities (e.g.. "Tennessee Code Annotated, Section 35-50-110") as wills.
Span difference
143 (46%)
66 (29%)
False positive
91 (30%)
75 (33%)
Fulse negative
71 (23%)
77 (34%)
We witnessed the behavior of creating false pos- itives in relation extraction as well. As explained in the previous section, we provided a gold set of entities as part of the prompt for the relation ex- traction task, and thus did not expect to see many false positives. Nevertheless, GPT-4 did not limit itself to use only the provided set of entities but ex- tracted additional entities from the given will text. These additionally extracted entities were mostly false positives (similar to the ones explained above), and thus the relations extracted based on these false positive entities were also false positives.
Human mistake
(1%)
(4%)
Total
307
226
Frroor analysis the entity extraction task. For test data, span difference is the most common error type. For out-of-domain (OOD) data, all three types eror (except for human's 's mistake) are evenly present.
Table 2:
Error type
Test
OOD
False positive
16 (20%)
22 (41%)
False negative
53 (66%)
34 (42%)
Human mistake
11 (14%)
9 (17%)
Total
Table 3: Error analysis on the relation extraction task. For both test data and out-of-domain (OOD) data, the most common emror type false negatives. In OOD data, false positives are another major error type.
Given that this behavior does not occur when GPT-4 is asked identify key elements from wills without given a prompt, it is very likely that this issue occurs due GPT-4's overgeneralization of the prompt. Including examples for negative cases in the prompt might have prevented this issue from occurring.
part of the entity or not, whether to divide list of entities into multiple small entities or capture the list as a single entity, etc.). This inconsistency resulted a high number of span difference errors in the entity extraction task (46% for test data and 29% for OOD data).
7 Conclusion
We present an IE dataset consisting of 16,018 annotations from 45 legal wills. This dataset is valuable to both legal domain and AI domain, as it can serve stream tasks concerning legal wills. The dataset can also serve as an important evaluation platform for LLMs. We evaluated GPT-4 with in-context learning on our dataset, and demonstrated that the model can handle legal IE tasks reasonably well. However, the error analysis revealed two major issues of the GPT-4 in in-context learning setting, which are inconsistency and prompt over- generalization. Further investigation is needed to mitigate these issues and enhance the perfor- mance of GPT-4 in in-context learning setting. Our open-access datasct is publicly available at: https://git thub. com/ m1 4ai //i//w/wwll///////
GPT-4 shows inconsistency in the relation ex- traction task as well. Unlike human annotators, who always extract single instance of the relation between the same entities given a context, GPT-4 extracts a random number of instances of the re- lation between the same entities. It might extract all or none of them, or it can be somewhere in between. This arbitrary behavior of GPT-4 led to considerable number of redundant extractions. These redundant extractions are not "wrong", so they were ignored when evaluating the model's 's per- formance. Nevertheless, this behavior can be prob- lematic when somcone plans to utilize the model for relation extraction tasks. As its way of extract- ing relations is arbitrary, one cannot anticipate its behavior based on its prior outputs nor process the output in consistent way.
foundation for several down-
Limitations
6.2 Overgeneralization of the Prompt
The domain of our dataset is rather limited. only contains English data, and the data came from two US states (i.e., Tennessee and Idaho) only. We plan to expand our dataset to other domains (c.g., adding more wills from other US states) in the near future. Also, the evaluation on LLM was only conducted on subset of our dataset (i.e., four most common entities and four most common relations). It would be desirable to conduct an evaluation on the entire
We witnessed many false positives (Test: 30% and OOD: 33% for the entity extraction task; Test: 20% and OOD: 41% for the relation extraction task) from the outputs generated by GPT-4. In the entity extraction task, we encountered cases where the model falsely extracted human entities (e.g., "Co- Executors") as beneficiaries, moncy-related entities (e.g., "tax", "expenses", "debts") as assets, and

---

data as a follow-up study to verify the findings from this study. Even though our dataset provides foundation for developing automatic information extraction model, this study does not present such a model. Further endeavor is needed to develop such a model, but it is beyond the scope of our current study.
ference on Articial Intelligence and Law, ICAIL '17, page 19-28, New York, NY, USA. Association for Computing Machinery.
Yanguang Chen, Yuanyuan Sun, Zhihao Yang, and Hongfei Lin. 2020. Joint entity and relation extrac- tion for legal documents with legal feature enhance- ment. In Proceedings of the 28th International Con- ference on Computational Linguisties, pages 1561- 1571, Barcelona, Spain (Online). International Com- mittee on Computational Linguistics.
Ethics Statement
Our dataset contains legal wills collected from An- cestry. These legal wills are in public domain in the US, and we did not violate Ancestry's 's Terms and Conditions during the data collection process. All the collected wills were manually anonymized prior to annotation process.
Maeda Hanafi, Yannis Katsis, Ishan Jindal, and Lu- cian Popa. 2022. comparative analysis between systems and large language mod- els for pattern extraction tasks. In Proceedings of the Fourth Workshop on Data Science with Human- the-Loop (Language Advances), pages 43-50, Abu Dhabi, United Arab Emirates (Hybrid), Association for Computational Linguistics.
Acknowledgements
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball, 2021. CUAD: an expert-annotated NLPP dataset for legal contract review. CoRR, 0/s/2103.0611
We thank the reviewers for their thoughtful com- ments and suggestions. We also appreciate Label Studio for providing access to Label Studio En- terprise Cloud Platform through their Academic Program. This work was partially supported by the National Science Foundation (NSF) under grant #2217215, and by University of Arizona's Provost Investment Fund. Mihai Surdeanu and Clayton Morrison declare a financial interest lum.ai. This interest has been properly disclosed to the Univer- sity of Arizona Institutional Review Committee and is managed in accordance with its conflict of interest policies.
Michael J. Bommarito II, Daniel Martin Katz, and Eric M. Detterman. 2018. Lexnlp: Natural language processing and information extraction for legal and regulatory texts. CoRR, abs/1806.0366 800.03388.
Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context learn- ing for biomedical IE? think again. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 4497-4512, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Birgit Kirsch, Sven Giesselbach, Timothée Schmude, Malte Völkening, Frauke Rostalski, and Stefan Rüping. 2020. Using probabilistic soft logic to im- prove information extraction in the legal domain.
References
Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models are few-shot clinical information extractors. Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Procesingg pages 1998 20882202,2222 Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Alice Kwak, Jacob Israelsen, Clayton Morrison, Derek Bambauer, and Mihai Surdeanu. 2022. Validity as- sessment of legal will statements as natural language inference. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 6047- 6056, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Claire Barale, Michael Rovatsos, and Nehal Bhuta. 2023. Automated refugee case analysis: A NLP pipeline for supporting legal practitioners. In Find- ings of the Association for Computational Linguis- tics: ACL. 2023, pages 2992-3005, Toronto, Canada. Association for Computational Linguistics.
Patricia Martín Chirfn--hhozss and Artem Revenko. 2021. The- saurus enhanced extraction of hohfeld's relations from spanish labour law. In Joint Proceedings of the 2nd International Workshop Deep Learning meets Ontologies and Natural Language Processing (Deep- OntoNLP 2021) & 6th International Workshop on Ex- plainable Sentiment Mining and Emotion Detection (X- X-SEENTMENT 2021) co-located with co-located with 18th Extended Semantic Web Conference 2021, Hersonissos, Greece, June 6th 7th, 2021 (moved on- line), volume 2918 of CEUR Workshop Proceedings. pages 30-38. CEUR-WS.00
Yu Cao, Yuanyuan Sun, Ce Xu, Chunnan Li, Jinming Du, and Hongfei Lin. 2022. Cailie 1.0: A dataset for challenge of ai in law - information extraction v..0O. Al Open, 3:208 212.
Thas Chalkidis, Ion Androutsopoulos, and Achilleas Michos. 2017. Extracting contract elements. Pro- ceedings of the 16th Edition of the International Con-

---

National Science Foundation. 2022. Dass: frame- work for accountable smart contract wills. https: htwp:rrhhhhohwwwwdd/W/a
//www. nsf..nfff. wtt.:///wwwaraaarr./ss/////eaee/ hatp21222288sssttrrrcaa/errrrrr accessed 19 October 2023.
Online:
Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, and Σneko Agirre. 2023. Did chat- gpt cheat on your test? https://hitz-zenee github. io/Im- cont contamination/blog// Accessed: 2023-06-23.
Jaromir Savelka. 2023. Unlocking practical applications legal domain. Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law. ACM.
Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022. Heroes, villains, and victims, and GPT-3: tomated extraction of character roles without train- ing data. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 47-56. Seattle, United States. Association for Computational Linguistics.
Don Tuggener, Pius von Däniken, Thomas Peetz, and Mark Cieliebak. 2020. LEDGAR: large-scale multi-label corpus for text classification of legal pro- visions in contracts. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1235-1241, Marseille, France. European Lan- guage Resources Association.

---

A Legal Will Data Engineering Process with OCR
ex) My personal representative shall be given credit on the final settlement for all disburse- ments made to my beneficiaries under the provisions of this power.
The legal will data we collected are saved in PDF format. In order to extract texts from the PDF files, we utilized two Python libraries: 2df2image and pytesseract.
Named beneficiary: a person or entity named in a will as beneficiary ex) will, devise and bequeath my house and lot to [Person-16] and [Person-17].
We first converted the PDF files into PNG image files. After the conversion, texts were extracted from the images utilizing image_to_ss sts__tt_ssrinnng func- tion. Extracted texts were saved in a CSV file. The CSV file was manually reviewed by our an- notators. Any errors were fixed during this re- view process. The code used for this process can be found from the following link: https: //github. www.tthb.cooommtt.e cwm/ml 4ai/ie4wills
5. Will: legal document containing a person's wishes regarding the disposal of one's asset after death
ex) I, [Person-1]] 11, domiciled Memphis, Ten- nessee, do make, publish and declare this to be my Last Will and Testament, hereby re- voking all wills and codicils heretofore made by me.
B Taxonomy of entities, relations, and events
Asset: any money, personal property, or real estate owned by a testator
Our dataset contains 26 types of entities, 18 types binary relations, and 20 types of n-ary events. See Tables below for the full taxonomy and its statistics.
ex) I hereby give, devise and bequeath all of my property, real, personal or mixed, to [Person-2], if living at the time of my death.
7. Executor: a person who executes a will (= personal representative)
B.1 Entities
This section is formatted in the following way: Entity name: short description ex) An example sentence showing all the relevant entities in bold.
ex) I name, nominate and appoint my daugh- ter, [Person-3], Executor of this my will and estate, and direct that she be allowed to serve without bond.
1. Testator: a person who makes a will ex) I, [Person-1], domiciled in Memphis, Ten- nessee, do make, publish and declare this to be my Last Will and Testament, hereby revok- ing all wills and codicils heretofore made by me.
Wimess: a person witnessing a will ex) We, [Person-5] and [Person-6], the wit- nesses, sign our names to this instrument, consisting of four pages, and being first duly sworn do hereby declare to the undersigned authority that the testatrix signs and executes this instrument as her last Will and that she signs it willingly, and that each of us, the presence and hearing of the testatrix, hereby signs this Will as witness to the testatrix 's signing, and that to the best of our knowledge the testatrix is eighteen (18) years of age or older, of sound mind, and under constraint of undue influence.
2. Trigger: event trigger words. (extracted to be used in the n-ary event extractions)
ex) A trigger for the Nomination event: I name, nominate and appoint my niece, [Person-2], Executor of this my will and es- tate, and direct that she be allowed to serve without bond.
3. Condition: condition under which an event (e.g., will execution, bequest, etc.) occurs ex) I hereby give, devise and bequeath all of my property, real, personal or mixed, to [Person-2], 2], if living at the time of my death.
Time: any expression denoting a particular point in time
cx) hereby give, devise and bequeath all of my property, real, personal or mixed, to [Person-2], if living at the time of my death.
4. Beneficiary: a person or entity (e.g., organiza- tion) that receives something from a will
10. Duty: any duty directed by a testator to fidu- ciaries (e.g., executors, trustees, guardians, or

---

conservators)
17. Bond: any bonds (usually probate bonds, which is a type of bond ordered and required by a court before they will appoint a person or entity as the personal representative an estate)
ex) direct my Executor pay all my just debts and all fumeral expenses, which shall be probated, registered and allowed against my estate, as soon after my death as can con- veniently be done.
cx) I name, nominate and appoint my daugh- ter, [Person-3], Executor of this my will and estate, and direct that she be allowed to serve without bond.
11. State: any US state names
ex) [Person-l], domiciled in Memphis, Ten- nessee, do make, publish and declare this to be my Last Will and Testament, hereby revok- ing all wills and codicils heretofore made by me.
18. Codicil: testamentary or supplementary doc- ument that modifies or revokes a will or part of a will
ex) I, [Person- 11, domiciled in Memphis, Ten- nessee, do make, publish and declare this to be my Last Will and Testament, hereby revok- ing all wills and codicils heretofore made by me.
12. Date: any dates
ex) IN WITNESS WHEREOF, I have here- unto signed, published and declared this in- strument as my Last Will and Testament, in Lauderdale County, Tennessee, on this 12th day of June, 1989
19. Trustee: a person who manages a trust ex) My entire estate, after payment of debts, taxes and expenses, shall be distributed to the Trustee of the Living Trust Agreement of [Person-]], entered into as of this same date of this my Last Will and Testament, to be administered and distributed under the terms of the trust created in the said Living Trust Agreement of [Person-1].
13. County: any US county names ex) IN WIT- NESS WHEREOF, I have hereunto signed, published and declared this instrument as my Last Will and Testament, in Lauderdale County, Tennessce, on this 12th day of June, 1989
14. Right: any rights authorized by a testator to fiduciaries (c.g., executors, trustees, guardians, or conservators)
20. Non-beneficiary: person who is excluded from being beneficiary
ex) My personal representative shall have the authority and discretion to buy to sell or lease real property or any interest in real property which may have and use and apply the proceeds from sale or lease to the payment of debts, taxes, and expenses of administration of my estate and may gen- erally treat real property the same as per- sonalty.
ex) In the event that any other person persons other than those herein named my heirs should seek to inherit from me and establish a right to so inherit by final Decree of the Court of competent jurisdiction, then, such event, I give and bequeath unto such person or persons, nothing.
21. Tax: any taxes
15. Expense: any expenses
ex) authorize my personal representative pay from my general estate any interest which may accrue on debts or taxes due from my estate.
ex) I direct my Executor to pay all my just debts and all funeral expenses, which shall be probated, registered and allowed against my estate, soon after my death can con- veniently be done.
22. Affidavit: a legal statement sworn and signed by a testator and witnesses to confirm the va- lidity of will. It is usually attached to will. ex) IN WITNESS WHEREOF, have exe- cuted this my Last Will at Ripley, Tennessee this 13th day of September, 2001 and request the attesting witnesses to make the A Affidavit set out below.
16. Debt: any debts
ex) direct my Executor pay all my just debts and all funeral expenses, which shall be probated, registered and allowed against my estate, as soon after my death as can conve- niently be done.

---

23. Notary public: person who is authorized by state government witness the signing of important documents and administer oaths ex) Sworn to and subscribed before me on this, the 7th day of April, 1990.
Entity types
Number
Testator
1904
Trigger
1293
Condition
727
Beneficiary
56
Named Beneficiary
588
545
Asset
525
24. Trust: a fiduciary arrangement that allows a trustee to hold assets on behalf of benefi- ciary
Executor
465
Witness
315
Time
198
Duty
139
ex) I give, devise and bequeath the sum of five thousand dollars ($5,000.00) to my great- grandson [Person-1] to be held trust for his future use and benefit until he reaches the age twenty-five (25).
State
114
Date
108
County
85
Right
79
Expense
70
Debt
64
Bond
63
Codicil
34
25. Guardian: a person who has a legal right and responsibility of taking care of someone who cannot takes care of themselves (usually a mi- nor or an legally incompetent person)
Trustee
27
Non-beneficiary
25
Tax
20
Affidavit
18
Notary public
17
ex) If my personal representative determines that income principal is payable to minor or to a person under mental or physical dis- ability, whether or not adjudicated, then my fiduciary shall have the discretion to either make payments directly to the beneficiary, to the legally appointed guardian or conser- vator, or to distribute and pay such amounts directly for the benefit of such beneficiary.
Trust
10
Guardian
Conservator
Total
7497
Table 4: The entitie types contained in the dataset and their statisties.
1. Coreference resolution: the relation between entities referring to the same real-world entity ex) The relation between / and my in "I name, nominate and appoint my niece, [Person-2], Executor of this my will and estate, and direct that she be allowed to serve without bond."
26. Conservator: a person who handles the finan- cial and personal affairs who cannot handles such affairs by themselves (usually minor legally incompetent person)
2. Beneficiary-Asset: the relation between a ben- eficiary and asset
ex) If my personal representative determines that income or principal is payable to a minor or to a person under mental or physical dis- ability, whether or not adjudicated, then my fiduciary shall have the discretion to either make payments directly to the beneficiary, to the legally appointed guardian conservator, or distribute and pay such amounts directly for the benefit of such beneficiary.
ex) The relation between [Person-2] (Benefi- ciary) and all of my property, real, personal, or mixed (Asset) in "I hereby give, devise and bequeath all of my property, real, personal or mixed, to [Person-2], living at the time of my death."
3. Testator-Asset: the relation between testator
and asset
ex) The relation between (Testator) and all of my property, real, personal, or mixed (Asset) "I hereby give, devise and bequeath all my property, real, personal or mixed, to [Person-2], if living at the time of my death."
B.2 Relations
This section is formatted in the following way: Relation name: a short description
ex) An example of the relation being explained fol- lowed by example sentence.
(e.g.. The relation between entity I and entity 2 in "an example sentence showing the entities partici- pating in the relation being exemplified (i.e., entity and entity 2) in bold.")
Testator- Bestator-Beneficiarr: the relation between a testator and a beneficiary
ex) The relation between / (Testator) and [Person-2] (Asset) in hereby give, devise

---

and bequeath all of my property, real, personal or mixed, to [Person-2], if living the time of my death." "
in "We, the undersigned and subscribing wit- nesses, do hereby certify that we witnessed the foregoing Last Will and Testament of [Person- 1]. at her request, in her presence, and in the presence of each other, and that she signed the same in our presence, and in the presence of each of us, declaring the same to be her Last Will and Testament."
5. Testator Will: the relation between a testator and a will
ex) The relation between (Testator) and all former wills in hereby expressly revoke all former wills and codicils by me at any time heretofore made."
11. Witeess Will: the relation between a witness and will
6. Testator-Executor the relation between a tes- tator and an executor
ex) The relation between We and the foregoing Last Will and Testament in "We, the under- signed and subscribing witnesses, do hereby certify that we witnessed the foregoing Last Will and Testament of [Person-1] 1], at her quest, in her presence, and in the presence of each other, and that she signed the same in our presence, and in the presence of each of us, declaring the same to be her Last Will and Testament."
ex) The relation between / and my niece in "I name, nominate and appoint my niece, [Pe Executor of this my will and es- tate, and direct that she be allowed to serve without bond."
7. Inclusion: the relation between entity and another entity in which the former in- cludes the latter one
ex) The relation between All the rest of my property and any furnishings inside the house, any money have, and any other property of any kind in "All the rest of my property. including any furnishings inside the house, any money I have, and any other property of any kind, will and bequeath to [Person-6] of Maury City, Tennessee."
12. Spouse-Spouse: spousal relations
ex) The relation between / and my husband in "I appoint my husband, [Person-3], be Executor of this my Last Will and Testament and direct that no security be required of him as such."
13. Beneficiary- Executor: the relation between a beneficiary and an executor
8. Competence: the relation between a testator and a condition that shows the competence of the testator
ex) The relation between my personal repre- sentative and my beneficiaries in "Beginning immediately after my death, my personal rep- resentative may begin making payments my beneficiaries during the administration of my estate and need not wait for the final settlement of the estate."
ex) The relation between and being of sound and disposing mind and memory in "I, [Person-1]] 1]. of Gates, Lauderdale County, Ten- nessee, being of sound and disposing mind and memory, do hereby make, publish, and declare this instrument my LAST WILL AND TESTAMENT, hereby revoking all wills and codicils wills heretofore made by me."
14. Beneficiary-Wil Will: the relation between bene- ficiary and will
ex) The relation between this Will and any re- cipient in "All inheritance, estate, and similar taxes assessed by reason of my death shall be paid in respect of all property and interest required to be included in my estate for tax purposes whether the same passes under this Will or otherwise and without contribution from any recipient of any property required be included for tax purposes."
9. Parent-Child: the relation between parent and a child
ex) The relation between / and my son in "I direct that my household furnishings and contents be divided in kind equally between my son, [Person-2], and my daughter-in-law, [Person-3]."
10.
the relation between a wit-
ness and a testator
15. Eestator-Codicia the relation between a testa- tor and a codicil
ex) The relation between We and [Person-1]

---

ex) The relation between and codicils in "I, [Person-1]. domiciled in Memphis, Tennessee, do make, publish and declare this be my Last Will and Testament, hereby revoking all wills and codicils heretofore made by me."
Relation types
Number
Coreference resolution
2560
Beneficiary-Ase
376
Testator-Asset
Testator- Reneficaar
275
Testator- Will
245
Testator-Fxecutor
189
Inclusion
165
16. Testator-Truste:: the relation between testa- tor and trustee
Competence
156
Parent-Child
87
Sitness-Testatrr
57
ex) The relation between My and the Trustee in "My entire estate, after payment of debts, taxes and expenses, shall be distributed to the Trustee of the Living Trust Agreement of [Person-1], 11, entered into as of this same date of this my Last Will and Testament, to be administered and distributed under the terms of the trust created in the said Living Trust Agreement of [Person-
Witness-Will
49
Spouse-Spouse
37
Beneficiary-Fxccume
27
Beneficiary-Wo -Willll
18
Testator- Codicii
10
Testator-
Testator
Testator-Conservatoe Conservator
Total
4623
Table 5: The relation types contained in the dataset and their statistics.
17. Testator-Guardian: the relation between tator and a guardian
Bequest: an event in which testator bequeath asset to beneficiary
ex) The relation between my and the legally appointed guardian in "If my personal repre- sentative determines that income or principal is payable to minor or to person under mental or physical disability, whether or not adjudicated, then my fiduciary shall have the discretion either make payments directly to the beneficiary, the legally appointed guardian or conservator, or to distribute and pay such amounts directly for the benefit of such beneficiary."
ex) hereby give, devise and bequeath all of my property, real, personal or mixed, to [Person-2], if living at the time of my death.
Trigger: "hereby give, devise and be- queath"
Testator: "I"
Asset:
"all of my property, real, personal or mixed"
Beneficiary: "[Person-2]"
Condition:
"if living the time of my
death"
18 Testator-Conservaate Conservator: the relation between testator and conservator
2. Sign will: event in which testator or a witness signs will
ex) The relation between my and conservator in "If my personal representative determines that income or principal is payable a minor or to a person under mental or physical dis- ability, whether or not adjudicated, then my fiduciary shall have the discretion to either make payments directly to the beneficiary, to the legally appointed guardian or conservator, or to distribute and pay such amounts directly for the benefit of such beneficiary."
cx) IN WITNESS WHEREOF, have here- unto signed, published and declared this in- strument as my Last Will and Testament, in Lauderdale County, Tennessee, on this 11 day of April, 1994.
Trigger: "have hereunto signed"
Testator: "T"
Will: "this instrument"
Date: "on this 11 day of April, 1994"
B.3 Events
Condition: "IN WITNESS WHEREOF"
This section is formatted in the following way: Event name: short description
3. Direction: event in which testator gives direction to a fiduciary ex) I direct that my Executrix not be required to make an accounting to the Court.
ex) An example sentence
Argument name: "argument example ex- cerpted from the example sentence above"
Trigger: "direct"

---

Testator: "I"
Trigger:
"without"
Directed event:
Testator: "T"
Excuse: "my Executrix not be re- quired to make an accounting to the Court"
Exccutor: "she"
Bond: "bond"
7. Authorization: an event in which a testator authorizes fiduciary to right ex) I further give to my Personal Representa- tive all of the powers of Personal Representa- tive under the laws of the state of Idaho now effect and as may hereafter be amended.
4. Will creation: an event in which testator creates will
ex) I, [Person-]], an adult resident citizen of [Address-I], I, Lauderdale County, Tennessee, being of sound and disposing mind, memory and understanding, do hereby make, declare and publish this instrument my Last Will and Testament, expressly revoking any and all testamentary dispositions heretofore made by me.
Trigger: "further give"
Testator: "T"
Executor: "my Personal Representative" Right: "all of the powers of Personal Representative"
Trigger: "do hereby make, declare and publish"
Condition: "under the laws of the state of Idaho as now in effect and as may hereafter be amended""
Testator: "I"
Will: "this instrument"
8. Nomination: event in which testator nom- inates fiduciary
5. Attestation: an event in which a witness at- tests the validity of a will
ex) I name, nominate and appoint my daugh- ter, [Person-3], -3,, Executor of this my will and estate, and direct that she be allowed to serve without bond.
ex) We, the undersigned subscribing wit- nesses, do hereby certify that we witnessed the foregoing Last Will and Testament of [Person- 11, her request, in her presence and in the presence of each other, and that she signed the same in our presence, and in the presence of each of us, declaring the same to be her Last Will and Testament. This 11 day of April, 1994.
Trigger: "name, nominate and appoint"
Testator: "I"
Executor: "my daughter"
9. Death: an event in which any entity (e.g., tes- tator, beneficiary, executor, etc.) dies ex) In the event that my said granddaughter does not survive me, then I hereby give, de- vise and bequeath said property my great grandson, [Person-3]
Trigger: "do hereby certify"
Witness: "We"
Attested events:
- Atttestation: "we witnessed the fore- going Last Will and Testament of [Person- at her request, in her pres- ence and in the presence of each other"
Trigger: "does not survive"
Beneficiary: "my said granddaughter"
10. Revocation: an event which a testator re- vokes a will or a codicil
Sign will: "she signed the same in our presence, and in the presence of each of us"
ex) I. [Person-1], of Gates, Lauderdale County, Tennessee, being of sound and disposing mind and memory, do hereby make, publish, and de- clare this instrument my LAST WILL AND TESTAMENT, hereby revoking all wills and codicils wills heretofore made by me.
Date: "This 1l day of April, 1994"
6. Excuse: event in which testator excuses fiduciary from duty
ex) name, nominate and appoint my daugh- ter, [Person-3], Executor of this my will and estate, and direct that she allowed to serve without bond.
Trigger: "hereby revoking"
Testator: "my"
Will: "all wills"

---

Codicil: "codicils to wills"
Trigger: "shall receive nothing"
Beneficiary: "they"
11. Probate: an event in which will any part of the will is probated
Condition: "if any beneficiary under this Will should contest the terms of this Will"
ex) hereby direct my Executor to pay all of my just debts, funeral expenses, taxes and other expenses, which shall be probated, reg- istered and allowed against my estate as soon after my death as can be conveniently done.
15. Give: an event in which a testator gives a compensation to fiduciary
ex) I direct that my Executor shall receive a fee for his services of five (5) percent of my net estate after the above specific bequests are made.
Trigger: "shall be probated"
Debt: "all of my just debts"
Expense: "fumeral expenses"
Tax: "taxes"
Trigger: "shall receive"
Expense: "other expenses"
Executor: "my Executor"
Condition: "against my estate"
Asset: "five (5) percent of my net estate"
Time: "as soon after my death as can be conveniently done"
Time: "after the above specific bequests are made"
12. Codicil: an event in which codicil is made ex) [, 11, of Gates, Lauderdale County, Tennessee, being of sound and disposing mind and memory, do hereby make, publish, and declare this instrument my LAST WILL AND TESTAMENT, hereby revoking all wills and codicils to wills heretofore made by me.
16. Non probate instrument creation: an event in which probate instrument (c.g., trust) is created
ex) give, devise and bequeath the sum of five thousand dollars ($5,000.00) to my great- grandson [Person-1 11] to be held trust for his future use and benefit until he reaches the age of twenty-five (25).
Trigger: "made"
Testator: "me"
Trigger: "be held"
Codicil: "codicils"
Asset: "the sum of five thousand dollars ($5,000.00)
Time:
"heretofore"
Trust:
"trust"
13. Disqualification: an event in which a benefi- ciary fiduciary is disqualified
17. Renunciation: event in which fiduciary renounces
ex) If, for any reason, [Person-3] is unwill- ing or unable serve in this capacity, then I nominate and appoint his daughter, [Person-5], to serve in his place as Co-Exccutor without bond.
ex) If, for any reason, [Person-3] unwill- ing or unable to serve this capacity, then nominate and appoint his daughter, [Person-5]], to serve in his place as Co-Executor without bond.
Trigger: "is unwilling unable to serve"
Executor: "[Person-3]"
Trigger: "is unwilling"
Executor: "[Person-3]"
14. Removal: an event in which beneficiary is removed from the will
18. Notarization: an event in which an affidavit is notarized by notary public
ex) further direct that if any beneficiary un- der this Will should contest the terms of this Will they shall receive nothing by the terms of this Will and that share which they would have received shall be divided among the re- maining beneficiaries of my estate in the same manner as though they had predeceased me without issue.
ex) SWORN TO before me this September 14, 2001. NOTARY PUBLIC My Commission expires: 6/19/06
Trigger: "SWORN"
Notary public: "me"
Date: "this September 14, 2001"

---

19. Birth: an event in which beneficiary born ex) I have two children: [Person-2], born
[Date-1], 1J, and [Person-3], born [Date-2].
Trigger: "born"
Beneficiary: "[Person-2]"
Date: "[Date-1]"
Trigger: "born"
Beneficiary: "[Person-3]"
Date: "[Date-21"
20. Residual: an event in which asset becomes residuary estate
ex) If the gift of any item of property under this Article fails or lapses, such property shall become part of my residuary estate and shall be distributed as provided in Article 2.
Trigger: "shall become part of my resid- uary estate"
Asset: "such property"
Condition: "If the gift of any item of property under this Article 1 fails or lapses"
Event types
Number
Bequest
756
Sign will
542
Direction
530
Will creation
472
Atlestation
400
Authorization
249
Excuse
247
Nomination
208
Death
156
Revocation
105
Probate
Codicil
44
Disqualification
Removal
20
Give
Non probate instrument creation
Renunciation
Notarization
Birth
Residual
Total
3898
Table 6: The event types contained in the dataset and their statistics.

---

C Annotation Guideline
When multiple trigger words listed for single event, annotate the whole trigger words as single trigger entity ex) "make, declare, and publish" as a single event trigger
If you have any questions that are not already ad- dressed in the guideline below, please keep them as separate notes so that we can discuss together and add them to the guideline later.
Any date should be annotated as single date entity, including any preposition (e.g., "on") and year. ex) "on this 12th day of June, 1989" single date entity
C.1 Entity
Check if all the entities are annotated with correct labels (see the taxonomy to check the full entity list)
For annotating "excuse" event, use negative word (e.g., "no", "without", etc.) as trigger
Use the most specific label when there's mul- tiple possible labels ex) use NAMED BENEFICIARY instead of BENEFICIARY when applicable.'
Include "to" when annotating verb phrases ex) to pay all my just debts
When there's 's a priority among multiple benefi- ciaries/executors, capture the priority by using numbered beneficiary labels
During the review phase, make sure to check if the entities that were introduced later in the annotation phase (e.g.. Condition and Non- beneficiary) were properly annotated
ex) designate [Person-1] as an executor. If [Person- cannot be an executor, then [Person-2] should be an executor.
All of the following should be annotated as conditions:
In the sentence above, [Person-1] has priority over [Person-2]. So we'll want to Executorl for annotating [person-1] and Executor2 for annotating [person-2).
Competence: age or mental capacity of testator
Conditionals: if-clauses, clauses starting with "In the event..."
In other cases, we'll just use the Executor label.
Witnessing requirements: "at his/her quest", "in his/her presence", "in the pres- ence of each other", etc.
This applies to annotating Beneficiary as well.
Any other phrases that might count conditions:
When annotating entities, include quantifiers (all, some, any, both, etc.) and articles (a, the, this, that)
"IN WITNESS WHEREOF"
"In fee simple"
Label only what's present in the sentence. Don't worry about the entities omitted from the sentence
"Equally in equal amount"
"For X period time"
"Without bond"
Any special tokens (e.g., [Person-1]] should be captured as single entity, including brack-
Use the label Time only when the entity is temporal expression. Expressions such as "at my/her/his request" should be annotated as conditions.
Event trigger words (verbs/nouns triggering events) should be annotated with the "TRIG- GER" tag
When annotating assets, annotate the whole phrase one chunk. In the example below, the whole phrase ("my peacock collection, my humming birds, my sewing machine my jew- ciry armoire and all personal items (clothing, shoes coats etc.) which she may like") should be annotated as one entity.
"When annotating dataset, "beneficiary" and "named beneficiary" was distinguished. However, during evalua- tion process, this distinction was ignored and both were treated as "beneficiary".
"When annotating the dataset, the priority among the same entities (c.g., executor or beneficiary) was marked with num- hers. However, during the evaluation process, priority num- hering was ignored.
ex) will, give, devise and bequeath unto my

---

very dear friend, [Person-7] my peacock col- lection, my humming birds, my sewing ma- chine, my jewelry armoire and all personal items (clothing, shoes coats etc.) which she may like.
Be mindful with the direction of the arrow connecting trigger and arguments. It should ALWAYS start from a trigger and then con- nected to other arguments.
When there's a need for using a specific event as an argument another event (c.g., excuse, direction, authorization, and any other nested events), use the event's trigger as the argu- ment.
C.2 Relations
Check if all the relations have been annotated and correctly labeled (see the taxonomy to check the full relation list)
ex) [...] direct that she be allowed to serve without bond.
Be mindful with the order of the entities being connected. The entities that are listed first in the relation label should be the starting point and those that are listed second the relation label should be the ending point of the arrow connecting two entities
Here, Direction event will have Excuse event as one of its argument. In this case, the trigger of the Excuse event (i.e., "without") will be connected to the trig- ger of the Direction event to show that the Excuse event is one of the arguments of the Direction event.
ex) If we're annotating "Testator- "essator-Bennficiarr" the entity labeled "Testator" should be the starting point and the entity labeled "Bene- ficiary" should be the ending point.
Don't forget to connect all the Time and Con- dition entities to the relevant events.
Coreference resolution should pair the most closely located entities referring to the same real-world entity within a sentence
We will keep Death as an event. Any nouns/verbs relevant to death (e.g.. "death", "die", "does not survive", etc.) should be an- notated as triggers and the person involved in that event (i.e., person who dies is dead) will be connected the trigger
When there are multiple instances of entities participating in a relation, connect the most closely located instances and disregard all the other ones. We only need to mark the relation between the same real-world entities once, not multiple times. Coreference resolution will take care of this.
ex) name, nominate and appoint my niece. [Person-2]. Executor of this my will and es- tate, and direct that she allowed to serve without bond.
If annotating the "Testator- Executor" relation in this example, we'll connect "my" (Testator) and "my niece" (Executor) as they are the most closely located ones to each other.
don't need to connect other instances, as we already annotated their relation once. (e.g., We don't 't need to connect "T" and "(Person-21")
C,3 Events
Check all the events have been annotated and correctly labeled (see the taxonomy to check the full event list)

---

D Evaluation Prompts
D.2 Relation Extraction
Figure 2 and 3 below show the actual prompts used to evaluate GPT-4 on legal information extraction tasks. The prompts consist of two parts: instruc- tions and examples. After the prompt, a will text to perform information extraction on is given.
%% Your task is identily relations belween TESTATOR and BENEFICIARY, TESTATOR and ASSET, BENEFICIARY and ASSET, and TESTATOR and WILL from the will statement provided. Refer the example, entities and the extracted relations below as guide to accurately complete the task. Fill in the relevant information for each relation calegory mentioned.
%% Exampee:::
D.1 Entity Extraction
Will Text: ARTICLE 3. All the rest. residue and remainder of property which I may own my death, real, personal mixed, tangible intangible, of whatsoever nature and wheresoever situated including all property which may acquire or become entilled after the execulion bequeath and devise to my husband, [Person-3], he be living at my death, and if he is not living at my death, bequesth and devise property to my son [Person-2].
%8%Instrrctions: Your task to identify the instances (including pronouns) of TESTATOR, BENEFICIARY, WILL, and ASSET from the will statement provided. Refer to the example and the extracted information below as a guide to accurately complete the task. Fill in the relevant information for each category mentioned.
% %% #xaxxappe:
Entily ID: jiz94GhpJ2
rest. residue and remainder of the property which| own at my death, real, personal or mixed, tangible or intangible, of whatsoever nature and wheresoever situated including acquire or become entilled to after the execution of this Will bequeath and devise to my husband, [Person-3]. if he be living not living at my death, bequeath and
Will Text: ARTICLE
Annotation Text:
Start Index: 69
End Index: 70
property which may
Annotation Labels: [TESTATOR']
(9 more entities are given)
at my death, and devise this property my son [Person-2].
Relation Labels: TTESTATTR-BENEFICAAA
To jd: p-ge7sWiXg
Annotation Text:
From_id: r1PFNvb4TP
Start Index:
End Index: 70
more relations given)
Annotation Labels: [TESTATOR]]
%%Extract the relevant information from the following will statement and complete the form provided below:
(18 more entilies are given examples)
Will Text: I, [Person-1] of the [Address-1], County of Cassia and State of Idaho, of the age of 72 years, being of sound mind and memory, and not acting under duress, menace, fraud undue influence any person whomsoever, do publish, and declare this my Last Will and Testament in manner and form following, to-wit:
% %XExrtac the relevant information from the following will statement complete the form provided below:
Will Text: [Person-1] of the [Address-1] 1), County of Cassia and State of Idaho, of the age of 72 years, being of sound memory, and not acting under duress, menace, fraud or undue influence of any person whomsoever, do make, publish, and declare this my Last Will and Testament in manner and form following, to-wit:
Entity ID: 17r52xPaxP Annotation Text:
Start Index:
End Index:
Annotation Labels: [TTESATTO']
Figure 2: A prompt used to evaluate the GPT-4 on legal entity extraction task.
more entilies given)
Figure 3: A prompt used to evaluate the GPT-4 on legal relation extraction task.


Reviews:
[{"review_id": "5kZRFJKPYg", "reviewer": "Reviewer_Mcw9", "summary": "", "questions": "- How does will processing differs essentially from legal IE in other areas, apart from specific relations, in a way that previous methods and their results could not be transferred to wills?\n- Why the authors use indexes of the texts and not the tokens themselves in the entity extraction prompts? This seems to increase the difficulty for the models to learn entity descriptions.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Please find our responses to your comments and questions below.\n\n> 1. Importance of the task (motivation for the study)\n\nWills are important legal documents that allow people to maintain control over their assets. Unlike most legal documents, wills are commonly written by/for lay people. Given the importance and ubiquity of wills, formalizing and developing legal procedures for understanding, evaluating and executing wills is crucial. Extracting key information from legal wills is a prerequisite for understanding, evaluating and executing wills. In this vein, our task (IE from legal wills) serves as a foundation for many downstream tasks such as automatic will review and electronic will creation and execution. Also, will execution is a great use case for smart contract platforms. Unlike business contracts, court documents, including probated wills, are in the public domain in the US.\n\n> 2. Difference between IE from legal wills and from other legal documents\n\nWhile IE from other legal documents tend to focus on entities and simple relations, IE from legal wills need more complicated information (e.g., relations among entities, events). In order to understand, validate, and execute wills, it is crucial to understand relations between entities (e.g., relations between testator and beneficiary, relations between testator and witnesses, relations between assets and beneficiary, etc.). It is also essential to extract all the key events occurring in the wills along with their arguments (e.g., in the bequest event, it is important to understand what assets (“asset” argument) are given to whom (“beneficiary” argument) in what condition (“condition” argument).) However, existing IE datasets concentrate on extracting key entities and simple relations only. For instance, Chalkidis et al. (2017) extracts various contract elements such as contract title, dates, relevant law articles, but it doesn’t extract any relations among such elements nor extract events from the document. Chen et al. (2020) does extract four relations relevant to key criminal behaviors, but the level of sophistication is different from our work. Due to this difference, it is difficult to transfer IE framework from other legal domains to handle IE from legal wills.\n\n> 3. Limited scope of the dataset\n\nIt is correct that the scope of our dataset is somewhat narrow. However, we’d like to point out that our work is a continued effort and thus the scope of our dataset will continue to expand. Also, our experiment with LLM suggests that the knowledge from our dataset can be transferred to handle OOD data. This shows that our dataset can be applied to broader domains than its direct scope.\n\n> 4. The conclusions drawn by the authors, e.g. that the model performance can be extrapolated to OOD data from the very preliminary empirical study, are unsubstantiated.\n\nThe conclusion that the model’s performance can be extrapolated to OOD data is based upon our findings from the experiment. The decent F1 scores from OOD data (79% for entities and and 83% for relations) suggests that the knowledge from our dataset can be successfully transferred to other domains (e.g., wills from other US States). While we believe that our study has been conducted in a proper setting, we understand that there can be a concern with the relatively small sample size of test and OOD datasets. In order to mitigate this issue, we will add statistical significance analysis and confidence intervals to control for the relatively small data size when making revisions.\n\n> 5. Why do the authors use indexes of the texts and not the tokens themselves in the entity extraction prompts?\n\nWe’d like to clarify that we used both the actual tokens and the indices in the entity extraction prompts. For example, when extracting the text “I” as an entity, we provided both the text “I” and start and end indices (69 and 70, respectively) to the model. The reason we utilized both the texts and their indices in the prompt is to distinguish between the tokens that have the same texts (e.g., The reference of pronouns such as “his”, “her”, or “their” can be confusing if there are many candidates in the text).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Presents a new high quality dataset for legal IE on wills\nPerforms preliminary empirical studies employing LLM to legal IE in the dataset", "reasons_to_reject": "While will processing does seem to present some interesting applications, the authors, in my opinion, fail to motivate how important the task actually is and how it differs conceptually and practically from legal IE in other areas. It is not clear to me whether the work presents a significant contribution, as the proposed resource seems very limited, with 40 wills from the state of Tennessee and five wills from the state of Idaho. I also do think that the conclusions drawn by the authors, e.g. that the model performance can be extrapolated to OOD data from the very preliminary empirical study, are unsubstantiated.\n\nTl;DR\nThe work is weakly motivated by the authors.\nThe empirical study is rather limited and the conclusions drawn by the authors seem unsubstantiated by their work;\nThe generated dataset is of very limited scope (realistically one state in the USA) and its significance is not very clear to the area", "questions_to_authors": "- How does will processing differs essentially from legal IE in other areas, apart from specific relations, in a way that previous methods and their results could not be transferred to wills?\n- Why the authors use indexes of the texts and not the tokens themselves in the entity extraction prompts? This seems to increase the difficulty for the models to learn entity descriptions.", "paper_topic_and_main_contributions": "The present paper proposes a dataset for information extraction on legal texts, specifically wills, from two states of the USA and some preliminary experiments using GPT-4 in-context learning for legal IE.\nThe authors report on the data collection and annotation process, which involved domain experts and obtained acceptable agreement scores. Further the authors evaluate GPT-4 in a in-context learning approach for identification of mentioned entities and relation extraction restricted to the four most frequent classes in the corpus for each task . The authors further evaluate the performance of the model in a out of distribution setting by processing 5 will of a different state (Idaho), while the training and test examples all consisted of will from the state of Tennessee. The authors conduct an analysis of the errors identifying that  the model overgeneralises from the prompts by identifying entities that do not belong to the desired classes - which indicates that the model does not acquire the functional aspects of n=the classes in the context of the legal texts.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oWoc5upEfN", "reviewer": "Reviewer_shKx", "summary": "", "questions": "Question A: Please explain the design policy of the corpus. For example, at what level of detail do you want to annotate the text?\nQuestion B: Please explain the reason why the evaluation is performed only for four most common entity and relation types, rather than reducing the complexity of the evaluation. If there is an application that uses only these common entities and relations, it may be reasonable to have such an evaluation for the application.\nQuestion C: Please explain the reason why you called the will from the different state is treated as out-of-domain data. I suppose the style of writing a legal will may not be that different from state to state. However, I would expect there to be a big difference for cases with different assets and different family situations. Are there significant differences in the distribution of terms or writing style in the wills from different states?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Please find our responses to your comments and questions below.\n\n> 1. They do not discuss what kind of information they would like to extract from legal wills. Without such information, it is difficult to judge the value of this corpus just by the names of entities and relations.\n\nThank you for the feedback. If accepted, we will accompany the paper with an annotation manual that formalizes the taxonomy of extracted information. We will also provide concrete examples for each entity/relation category to show what information is contained in our dataset.\n\n> 2. Please explain the design policy of the corpus. For example, at what level of detail do you want to annotate the text?\n\nOur long term goal is to generate executable smart wills, and thus we annotated the information necessary for this goal as decided by the legal domain expert in our team. In order to understand, validate, and execute wills, it is crucial to extract all the key entities (e.g., testator, beneficiary, executor, witnesses, assets, just to name a few) and understand relations between entities (e.g., relations between testator and beneficiary, relations between testator and witnesses, relations between assets and beneficiary, etc.). It is also essential to extract all the key events occurring in the wills along with their arguments (e.g., in the bequest event, it is important to understand what assets (“asset” argument) are given to whom (“beneficiary” argument) in what condition (“condition” argument)). Therefore, the annotations contained in our dataset are much more detailed than those in other IE datasets constructed from other legal documents. We will accompany the paper with an annotation manual and more concrete examples if accepted, as promised earlier.\n\n> 3. Please explain the reason why the evaluation is performed only for four most common entity and relation types, rather than reducing the complexity of the evaluation. If there is an application that uses only these common entities and relations, it may be reasonable to have such an evaluation for the application.\n\nWe focused on the most common entities and relations in order to mitigate the cost of generating custom prompts for each entity/relation/event type. If accepted, we will expand to include most structures that occur frequently enough in the data.\n\n> 4. Please explain the reason why you called the will from the different state is treated as out-of-domain data. I suppose the style of writing a legal will may not be that different from state to state. However, I would expect there to be a big difference for cases with different assets and different family situations. Are there significant differences in the distribution of terms or writing style in the wills from different states?\n\nThere are both formal and informal differences among the wills from different states. First, obviously, wills from the different states will differ in that they must meet different legal requirements. That dictates substance, but it can also dictate style, such as the terms used for various entities, assets, and requirements. Second, practitioners and norms vary from state to state. There may be regional similarities—wills in the American South are generally more alike in style than they are when compared to ones from New Hampshire or California—but Tennessee and Idaho are not part of the same region. Plus, different case law, from different states’ courts, shapes wills as well, and states are likely to have divergence in case law.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They propose a new corpus for information extraction from legal wills and confirm the possibility of building an IE system using LLM.\n", "reasons_to_reject": "They do not discuss what kind of information they would like to extract from legal wills. Without such information, it is difficult to judge the value of this corpus just by the names of entities and relations. This problem is also related to the evaluation. The evaluation is done only for the four most common entities and relations. It is also difficult to interpret the evaluation results shown in the paper from the point of view of the final application process.", "questions_to_authors": "Question A: Please explain the design policy of the corpus. For example, at what level of detail do you want to annotate the text?\nQuestion B: Please explain the reason why the evaluation is performed only for four most common entity and relation types, rather than reducing the complexity of the evaluation. If there is an application that uses only these common entities and relations, it may be reasonable to have such an evaluation for the application.\nQuestion C: Please explain the reason why you called the will from the different state is treated as out-of-domain data. I suppose the style of writing a legal will may not be that different from state to state. However, I would expect there to be a big difference for cases with different assets and different family situations. Are there significant differences in the distribution of terms or writing style in the wills from different states?", "paper_topic_and_main_contributions": "This paper presents a new corpus for information extraction (IE) from legal wills using public domain court documents from two US states. They also evaluate the performance of LLM on this task.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R7WJZM6Ka6", "reviewer": "Reviewer_ZHWR", "summary": "", "questions": "- How do you explain that results are significantly higher for OOD data in table 1?\n- Same question for table 2 span difference 46% vs. 29% for OOD data: how can this difference be explained? ", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Please find our responses to your comments and questions below.\n\n> 1. The paper performs OOD experiment within the US states, but more generally it would be interesting to see how well this work could transfer to other english-speaking jurisdictions.\n\nThank you for the interesting suggestion. While the wills from other English-speaking jurisdictions are out of the scope of this paper, we agree that it is a great future research topic.\n\n> 2. Line 126 states: \"there has been no prior attempt to evaluate LLMs on legal IE tasks\". There has been previous research work on legal IE based on LLMs, mostly on BERT-based architectures, for example: https://aclanthology.org/2023.findings-acl.187/. To the best of my knowledge there are no studies using auto-regressive LLMs such as GPT for entity extraction, there is however some work on closely related tasks such as QA or annotations, see for example: https://arxiv.org/abs/2305.04417.\n\nThank you for letting us know of the works. The first work (Barale et al., 2023) was not published yet when we submitted our paper. We will make sure to include the suggested works in our relevant works section and revise the sentence in question. Still, we’d like to emphasize that our work is novel and distinct from the suggested works. The domain and the task (Barale et al. (2023): refugee case analysis; Savelka (2023): zero-shot semantic annotation of legal texts; our work: IE from legal wills in in-context learning setting) of the works are highly distinct from each other.\n\n> 3. Paragraph 3.1: it would be interesting to know more about the data engineering step with OCR, since this is very common when gathering legal data and is usually a challenging task.\n\nWe processed our collected data (in the PDF format) utilizing Python libraries called pdf2image and pytesseract. We first converted the PDF files into image files and then extracted texts using pytesseract’s “image_to_string” function. After the OCR process, the annotators manually checked the extracted texts and fixed any errors. If accepted, we will include an additional appendix with the details of this pipeline.\n\n> 4. Table 1 and paragraph 5: overall results are better on OOD entities and relations than there are on in domain entities and relations. The given explanation in paragraph 5 is about good transferability of the claims. This may be a partial explanation but it would be relevant to think in other directions as well: could it be that the test set is too small to accurately reflect the performance, in one direction or another? This difference in the results calls for a more precise explanation. How do you explain that results are significantly higher for OOD data in table 1? Same question for table 2 span difference 46% vs. 29% for OOD data: how can this difference be explained?\n\nWe interpreted that the model’s good performance on OOD suggests our dataset’s high transferability. While we believe that our study has been conducted in a proper setting, we understand that there can be a concern with the relatively small sample size of test and OOD datasets. In order to mitigate this issue, we will add statistical significance analysis and confidence intervals to control for the relatively small data size when making revisions.\n\n> 5. Missing References:\nOverall no important reference is missing, but here could be a few related papers that are not cited as related work:\nTax Law Entailment, at NLLP workshop 2020: https://arxiv.org/abs/2005.05257\nLexNLP: https://arxiv.org/abs/1806.03688\nLegal NLP review: https://aclanthology.org/2020.acl-main.466/\n\nThank you! We will add the suggested works to the relevant works section to emphasize the large number of distinct domains in the legal NLP domain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- There is generally a lack of resources in legal NLP, and introducing new datasets that are ready to be used by the community is essential to develop the field.\n- To the best of my knowledge, this is the first dataset containing wills.\n- The authors gather about 16, 000 gold standard annotations, over 45 documents, which is a reasonable size for a legal dataset.\n- The authors focus on the task of information extraction and linking, which are tasks that are commonly performed in the legal workload and that are often cited by legal professionals as ones that could be sped up using AI capabilities. \n- Paragraph 2.2 compare the legal domain with other domains requiring specific expertise such a healthcare IE and few-shot learning. This parallel between domain is very interesting since there are a number of shared challenges such as specific vocabulary and contextual meaning. An interesting addition to this paragraph would be to point out those similarities. \n- By performing experiment on out-of-domain data (documents from another state), the authors show that it is transferable at least to some extent within the USA. \n- Paragraph 6.1 and 6.2 include a precise analysis of the models errors in the context of legal IE by exposing both inconsistencies and overgeneralization. This is highly valuable for future research. \n", "reasons_to_reject": "- The paper performs OOD experiment within the US states, but more generally it would be interesting to see how well this work could transfer to other english-speaking jurisdictions.\n- Line 126 states: \"there has been no prior attempt to evaluate LLMs on legal IE tasks\". There has been previous research work on legal IE based on LLMs, mostly on BERT-based architectures, for example: https://aclanthology.org/2023.findings-acl.187/. To the best of my knowledge there are no studies using auto-regressive LLMs such as GPT for entity extraction, there is however some work on closely related tasks such as QA or annotations, see for example: https://arxiv.org/abs/2305.04417. \n- Paragraph 3.1: it would be interesting to know more about the data engineering step with OCR, since this is very common when gathering legal data and is usually a challenging task. \n- Table 1 and paragraph 5: overall results are better on OOD entities and relations than there are on in domain entities and relations. The given explanation in paragraph 5 is about good transferability of the claims. This may be a partial explanation but it would be relevant to think in other directions as well: could it be that the test set is too small to accurately reflect the performance, in one direction or another? This difference in the results calls for a more precise explanation. \n", "questions_to_authors": "- How do you explain that results are significantly higher for OOD data in table 1?\n- Same question for table 2 span difference 46% vs. 29% for OOD data: how can this difference be explained? ", "paper_topic_and_main_contributions": "The short-paper present a new dataset for legal NLP (legal wills) and manually annotated data for the task of entity extraction and relation extraction. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Overall no important reference is missing, but here could be a few related papers that are not cited as related work: \n- Tax Law Entailment, at NLLP workshop 2020: https://arxiv.org/abs/2005.05257\n- LexNLP: https://arxiv.org/abs/1806.03688\n- Legal NLP review: https://aclanthology.org/2020.acl-main.466/ "}}]

Author rebuttal:

Meta-review: The paper contributes a new high-quality dataset for legal Information Extraction on legal wills derived from public domain court documents from two US states. It discusses preliminary empirical studies to investigate the capability of ChatGPT4 on the task.  Reviewers and authors exchanges during discussion helped clarify some of the critical issues highlighted in the reviews. Overall, the paper was found to be interesting and innovative. However, some revisions are needed to make it more convincing. For instance, discussing the differences between the wills from the two states, as mentioned in response to the reviewers, would yield valuable insights for other researchers.  Given that this is a short paper, it needs at least to better explain the importance and novelty of IE on legal wills and to better justify the design choices by providing additional details, as indicated by reviewers. 

Here below is a summary of the main strengths and weaknesses identified: 

**Pros**  

- a new corpus for information extraction from legal wills; 

- demonstration  of the adequacy of the Gpt4 model for IE tasks; 

- focuson information extraction and linking, of special interest in legal profession; 

- comparison with other domains, interesting for highlighting shared challenges; 

- portability to other US states legal wills; 

- presence of error analysis. 

**Cons** 

- the dataset is limited, as it covers few states and few cases. Increasing its coverage will certainly provide stronger support to the claims;  

- the importance and novelty, or difference from similar tasks in legal IE, is not evident. The paper should explain it more clearly; 

- details on the extracted and evaluated information categories seems to be missing, which leaves some of the choices unexplained.
Decision: Accept-Findings