Paper 2 Data — An example of how to extract information from legal contracts using Artificial Intelligence.
By: Phil Godley
Tags: Financial Services, Machine Learning, Legal, Artificial Intelligence, NLP

1. Dataset

Please see: https://github.com/JsyPhil/paper-2-data/blob/main/F1-P2D-NER-CUAD-v1-Pre-Annotation-Workflow.ipynb

To create a dataset for such an NLP project, we first needed to find a corpus of legal documents, convert them to text and then pre-process these appropriately to be compatible with the pre-trained model’s approach to word embeddings and tokenization.

Fortunately, the CUADv1 legal agreement dataset was published by the Atticus Project in 2021 at the time of the project and this contains 510 general commercial legal contracts obtained from publicly available SEC filings.

Whilst the Atticus Project has created a dataset corpus of 13,000+ labels across these agreements, given that our use case is different, we will need to build our own dataset of labels.

Document Length

A current limitation of most NLP models is that they have a restriction on the length of the text from which predictions are to be made. Often this is 512 or 768 tokens depending on the model and it’s structure. Increasing this value has an exponential effect on size of the neural network and therefore the memory requirements and so is often hard-capped on the model.

Legal contracts will often be more than 512 words, which when tokenized will create many more tokens than this (see below for the explanation under Label Alignment).

This is easily solved in this use-case as most of the data to be extracted will be near the top of the legal contract so only the beginning of the contract will be needed.

If a whole contract is to be used in a model then this will need to be overcome. There are techniques available, models without a hard-cap and also models being developed for the purpose of long documents such as Big Bird from Google.

Labeling scheme

For the purpose of this project we want to extract the following basic data points from the legal contracts:

a) Agreement Date

b) Document Name

c) Party

We will use the Inside-Outside-Beginning (IOB) tagging convention to create the following feature labels for annotating and labeling the text in each agreement in our dataset:

0 B-AGMT_DATE

1 B-DOC_NAME

2 B-PARTY

3 I-AGMT_DATE

4 I-DOC_NAME

5 I-PARTY

6 O

The IOB convention will help the model understand and learn which word is part of the data we are aiming to extract. If two separate pieces of data that we need to extract are adjacent in the text then the IOB convention handles this. For example the agreement name and the agreement date may be next to each other on the front page of the agreement. In this case the feature labeling would look and be translated as follows:

Facility Agreement

7 May 2021

B-DOC_NAME, I-DOC_NAME,

B-AGMT_DATE, I-AGMT_DATE, I-AGMT_DATE

1, 4,

0, 3, 3

Therefore, we are able to create a feature label for every single word in every document, most of which will be ‘6' and not useful data for us, but at the same time we are able to train a model to know which few words in each contract contain the data which we want to extract, being feature labels ‘0’ to ‘5’.

Annotating each contract

There are a number of different options when it comes to feature labeling text to create a dataset for NLP projects. Some of these are proprietary commercial software options and some are open-source and free to use under license. A good open-source option is Doccano.

Doccano — Text Classification — Labels and Annotations

To get the documents into a format to import into Doccano, we prepared an initial workflow to import the text, clean the text and automate some of the tagging from the information available in the CUAD dataset.

Only some of the documents in the CUAD dataset are applicable for our use-case as some documents are templates, undated or don’t include counter-party information.

We built a workflow in Python to automate the creation of the annotations where possible. Where the automation is not possible, we are still able to identify these contracts and review the text manually in Doccano.

This annotation process in Doccano was time very well spent. All evidence suggests that the more annotations that can be made available for the NLP models, the better the models will perform. Similarly it is important that the annotations are completed consistently in every agreement, so not to confuse the training of the model. Therefore, each annotation was carefully reviewed after completion.

Following this workflow, we finished with a dataset of 314 documents containing 1,697 labels.

Number of labels in the dataset

2. Pre-processing the text

Please see: https://github.com/JsyPhil/paper-2-data/blob/main/F2-P2D-NER-CUAD-v1-Dataset-Preparation.ipynb

The text and feature labels are exported from Doccano using JSONL which is a multi-line JSON file format. This data then requires some further pre-processing to create a further JSON file format ready for ingestion into the NLP transformer models and to utilize the handy Datasets library, from Hugging Face.

This was done with a further Python workflow creating a list of every word in each contract exported from Doccano and matching this with a corresponding list of tags from 0 to 6 (as described above) relating to the annotated tag also exported from Doccano. This is then exported as a JSON file format.

This JSON data can be utilised and manipulated by the Datasets library and dovetails well into the Transformers library, for example when creating Training and Evaluation sets of data.

3. Fine-Tune the Transformer Models

Please see: https://github.com/JsyPhil/paper-2-data/blob/main/F3-P2D-NER-CUAD-v1-Fine-Tune-Transformer.ipynb

We utilized the ubiquitous Hugging Face open source NLP libraries to obtain and fine-tune our models.

Hugging Face has become the de-facto place to go for current state-of-the-art NLP models, datasets and other resources including tutorials and well-supported libraries.

It has helped that the data science teams at Google AI, Facebook AI, Microsoft and many others including universities have contributed to the models and libraries available at Hugging Face.

Hugging Face hosts many implementations and advancements of the original BERT model by Google AI from 2018. Some of these implementations are ‘fine-tuned’ examples addressing specific use cases, whilst other are ‘pre-trained’ examples which can be further trained for discrete use cases.

For the purposes of this project, it is the latter that we are interested in. Pre-trained models are a fantastic innovation as they allow anyone to use models which have been generally trained to a point with vast amounts of data and large compute resources, typically which are only available to ‘Big-Tech’ and ‘Academia’.

These pre-trained models can then be trained just a little bit more with your own relatively small amount of data to achieve great results.

There are some limitations which you have to accept, including that there may be some bias in the training data used to train these pre-trained models. Such limitations are usually well explained by the developer of the pre-trained model.

The Models

We experimented with four pre-trained variants of BERT for the purpose of fine-tuning our models:

DistilBERT base model (uncased)

DistilBERT base model (cased)

RoBERTa base model

DeBERTa-v2 (xl)

The workflow that we used to fine-tune our models was based on the Token Classification foundation workflow provided by Hugging Face. For our project we are only interested in the ‘Named Entity Recognition’ implementation to classify the entities in our text, i.e. the document name, date and parties.

The foundation workflow was adapted for our use case including:

Configuration settings for multiple experiments Bespoke feature labels Helper functions specifically to align label predictions from the DeBERTa-v2 (xl) model (this model does not have a Hugging Face AutoTokenizerFast class). Bespoke training schedule with warm up and cosine calculated learning rate for optimal tuning. Implementation of Weights and Biases for convenient experimentation and hyper-parameter tracking. Sample checking of the predictions

Label Alignment

It is worth specifically noting the complications which can arise from each model’s tokenization process. Different models will utilise different bespoke tokenizers, dictionaries and pre-trained word-embeddings. These may be based on sentencepiece, wordpiece or a byte-level BPE (same as GPT2). This is a current area of innovation in all the models. For example DeBERTa, developed by Microsoft, switched its tokenizer from byte-level BPE to sentencepiece across v1 to v2 in a matter of a few months. Hugging Face provide an excellent summary of the various tokenizer methods.

Accommodating different models and each of these tokenizers in a workflow requires some effort.

The tokenizer will add tokens to denote the start and end of vectors, these, among others are called ‘Special Tokens’. Also, if a word is to be put through a model and that word is not in the model’s pre-trained dictionary, then the tokenizer will break up that word into parts of words.

Therefore the amount of tokens passed through a model will be larger than the amount of words.

When a word is split into two or more tokens then the label assigned to the word is duplicated across the tokens. How this is done for any given sentence can differ greatly across each tokenizer.

When predictions are made from the models, we have to realign and re-complete the words as well as removing, or ignoring, the special tokens.

Fortunately, Hugging Face does accommodate this in some of the pre-trained models which are hosted on the platform, but not on others. We implement code to handle both these scenarios including building a bespoke helper function specifically for DeBERTa-v2 (xl).

Learning Rate Schedule

We found that using a bespoke learning rate schedule had a good impact on the performance of the models. Hyper-parameters can be seen below for our optimal learning rate schedule.

Learning Rate Schedule — Hyper-parameters

This had a warm-up from 0 to a maximum depending on the size of the batches and with a cosine decay. We believe that the bespoke learning rate schedule optimises the performance by allowing the pre-trained weights to be well maintained whilst being appropriately nudged for our specific use case.

Results

Some of the experiments undertaken can be seen on our page hosted on the Weights and Biases website.

Performance of the models — Weights and Biases

We found that the DeBERTa-v2 (xl) model produced the best outcome with F1 scores of 0.95 being consistently achieved at around 5 epochs and a batch size of 1 or 2 but was very resource intensive, in so far as even high end GPU hardware from 2020 could not run this model due to memory constraints. We had to utilize NVIDIA TITAN RTX GPU with 24GB of GPU memory to run this model. Even then the batch size had to be reduced to 1 or 2 samples per batch depending on the number and type of GPU otherwise runs would run out of memory. This model has 900 million parameters to fine-tune.

We also achieved good results with RoBERTa base with F1 scores of around 0.92 also consistently being achieved after around 5 or 6 epochs and a batch size of up to 16 depending on the type and number of GPUs available.This model has 125 million parameters to fine-tune.

Accordingly RoBERTa base was a good compromise with much less resource requirements for a very small reduction in performance.

The two DistiBERT models are great models to get the code up and running effectively including streamlining and de-bugging. This is because they are lightweight and run quickly. However, for inference, the performance isn’t good enough to justify using them over RoBERTa base.

The uncased version of DistilBERT performed better than the cased version.

We believe that the the larger DeBERTa-v2 and RoBERTa models may outperform further if more training data is available, which is often the case.

4. Final model

Please see: https://github.com/JsyPhil/paper-2-data/blob/main/F4-P2D-NER-CUAD-v1-Final-Model-Workflow.ipynb

We trained a final model for both RoBERTa and DeBERTa-v2 xl using the most promising hyper-parameters from our experiments.

We used the largest batch-sizes possible given the GPU memory for each model. Which was 16 and 2 for RoBERTa and DeBERTa-v2 (xl) respectively.

The optimizer used is Adam with a learning rate as defined above.

We found optimal performance after 8 epochs for both models. Although, we discovered similar performance after around 6 epochs up to the maximum of 10 epochs.

It is both important and recommended that the same text pre-processing is used for any legal contracts used by the model as that used for training.

5. Inference

Please see: http://localhost:8888/notebooks/Dropbox/Python/Paper%20To%20Data/paper-2-data/F5-P2D-NER-CUAD-v1-Inference.ipynb

The workflow of this single notebook provides for the following practical process:

Searches for and converts from PDF to text any number of legal contracts in PDF format in a defined folder or location. Pre-processes this text for ingestion into the fine-tuned final model. Predictions are accurately produced from the fine-tuned model for the agreement name, commencement date and parties to the contract. These predictions are then transformed into a structured tabular format and exported as a CSV file, togethet with the full text of the document for further use.

Exported tabular data from the NLP model.

Conclusion

We successfully demonstrated how NLP can be applied to a specific and valuable use case in the legal, professional and financial sectors. We hope that this may inspire technologists, business executives and managers alike to continue their learning and utilization of this important technology for the benefit of their businesses.

Special thanks to Malcolm Mason for his support and knowledge throughout this project.