LexNLP — Library For Automated Text Extraction & NER | by Rohan Gupta | Towards Data Science

A few weeks ago, I had to extract certain types of data from a set of documents and wondered what was the best way to do it. The documents were all leasing forms with data such as entity names, addresses, dates, amounts, conditions, etc.
I started out the old fashioned way, using regex to identify certain fields with their respective synonyms in each document. I had to manually set up the rules for extracting each type of field.
However, I later discovered LexNLP and its capabilities. All of my required data was extracted with LexNLP without having to write any rules. I was quite amazed by the library, but did not find any tutorials apart from the documentation. So, here I am creating a gateway to LexNLP for other data professionals who could benefit from its features.
Below is an overview of LexNLP, which is made by ContraxSuite. I provide examples for extracting certain kinds of data such as dates, entity names, money, and addresses. The library is currently available for extraction in English, Spanish and German.
LexNLP can extract all the following information from textual data:
acts, e.g., “section 1 of the Advancing Hope Act, 1986”
amounts, e.g., “ten pounds” or “5.8 megawatts”
citations, e.g., “10 U.S. 100” or “1998 S. Ct. 1”
companies, e.g., “Lexpredict LLC”
conditions, e.g., “subject to ...” or “unless and until ...”
constraints, e.g., “no more than” or “
copyright, e.g., “© Copyright 2000 Acme”
courts, e.g., “Supreme Court of New York”
CUSIP, e.g., “392690QT3”
dates, e.g., “June 1, 2017” or “2018–01–01”
definitions, e.g., “Term shall mean ...”
distances, e.g., “fifteen miles”
durations, e.g., “ten years” or “thirty days”
geographic and geopolitical entities, e.g., “New York” or “Norway”
money and currency usages, e.g., “$5” or “10 Euro”
percents and rates, e.g., “10%” or “50 bps”
PII, e.g., “212–212–2121” or “999–999–9999”
ratios, e.g.,” 3:1” or “four to three”
regulations, e.g., “32 CFR 170”
trademarks, e.g., “MyApp (TM)”
URLs, e.g., “http://acme.com/”
LexNLP requires python 3.6! So you are advised to create a completely new virtual environment in python 3.6 and download all requirements for the LexNLP library from the github link below.
github.com
Once you have everything installed, try import LexNLP. If it works then you’re all set to continue, but if you run into an error, go through the requirements again and make sure you have the right versions. Comment below if you need any help.
We will start off by importing each of the following libraries. Make sure to install each of these libraries:
from tika import parserimport glob, osimport pandas as pdfrom bs4 import BeautifulSoupimport codecsimport reimport numpy as np
Since all my files were in PDF/Doc formats, I would just have to extract the text as a string from each document. I will use the library Tika for this. The function I have written below gives me a data-frame with information on each file including the text needed for extraction.
def multipdftotxt(path):    df = pd.DataFrame(columns = ['S.No', 'File_Name', 'Author', 'Creation_Date', 'Title','Content'])    pdfs = []    i=0    os.chdir(path)    types = ['*.pdf', '*.doc', '*.docx']    textfiles = []        for typ in types:        textfiles.append(glob.glob(typ))    flat_list = []    for sublist in textfiles:        for item in sublist:            flat_list.append(item)    textfiles = flat_list            for file in textfiles:        print(file)        raw = parser.from_file(file)                text = raw['content']        dict2 = raw['metadata']        Author = dict2.get('Author')        Creation_Date = dict2.get('Creation-Date')        title = dict2.get('title')        i = i+1        df1 = {'S.No': i,'File_Name': file,'Author': Author,'Creation_Date': Creation_Date, 'Title': title,'Content': text}        df = df.append(df1, ignore_index=True)    df = df.replace('\n', ' \n ', regex=True, )     df = df.replace('\t', ' ', regex=True)    df = df.dropna(subset=['Content'])      return df
The image above shows us what the data-frame would look like. All I need from this data-frame is the columns ‘Text’ and ‘File names’, therefore I will extract these two columns in a dictionary using another function.
def dftod(df):    l = []    for i in df['Content']:        l.append(i)    emailname = []    for i in df['File_Name']:        emailname.append(i)    d = dict(zip(emailname, l))    k = [v.strip() for k,v in d.items()]    k = [re.sub(' +', ' ', temp) for temp in k]    k = [re.sub('\n +', '\n', temp) for temp in k]    k = [re.sub('\n+', '\n', temp) for temp in k]         d = dict(zip(emailname, k))    return d
Now, I have extracted the text in a dictionary and I am ready to use LexNLP’s extraction features.
Importing the right functions from LexNLP is the key to using the library properly. Below, I will show you how to extract specific types of data: Entity Names, Addresses, Dates, and Money.
import lexnlp.extract.en.entities.nltk_re#Remember d is our dictionary containing filenames and text.#For entity names, use lexnlp.extract.en.entities.nltk_re.get_companies(text)for filename,text in d.items():print(list(lexnlp.extract.en.entities.nltk_re.get_entities.nltk_re.get_companies(text)))Output:['Target Inc', 'Hawthorne LLC', 'Willburne & Co.']
from lexnlp.extract.en.addresses import address_features#You want to use lexnlp.extract.en.addresses.address_features.get_word_features(text)for filename,text in d.items():print(list(lexnlp.extract.en.addresses.address_features.get_word_features(text)))#Check for DateTime/Zip-code/Email-Address/URL:lexnlp.extract.en.addresses.address_features.is_datetime(text)lexnlp.extract.en.addresses.address_features.is_zip_code(text)lexnlp.extract.en.addresses.address_features.is_email(text)lexnlp.extract.en.addresses.address_features.is_url(text)
Dates can be extracted in the following formats:
February 1, 1998
2017–06–01
1st day of June, 2017
31 October 2016
15th of March 2000
import lexnlp.extract.en.datesfor filename,text in d.items():print(list(lexnlp.extract.en.dates.get_dates(text)))Output:[[datetime.date(1998, 2, 1)],[datetime.date(2017, 6, 1)],[datetime.date(2016, 10, 31)],[datetime.date(2000, 3, 15)]]
Money can be extracted in the following formats:
five dollars
5 dollars
5 USD
$5
As of now, only the following currencies can be detected through LexNLP:
USD/$: US Dollars
EUR/€: Euros
GBP/£: Great British pounds
JPY/¥: Japanese Yen
CNY/RMB/元/¥: Chinese Yuan/Renminbi
INR/Rs/₹: Indian Rupee
import lexnlp.extract.en.moneyfor filename,text in d.items():print(list(lexnlp.extract.en.money.get_money(text)))Output:[(5000000.00, 'GBP'),(100000.00, 'INR')]
For more information and resources, please visit the official documentation:
https://lexpredict-lexnlp.readthedocs.io
While LexNLP does the digging, enjoy your coffee and wait for those results to unfold!
Keep following! I appreciate the love. I hadn’t been writing as much in the last few months, but I’m glad to say that I’m back in the game.