Transform Invoices Into Tabular Data Using Python | by Pranjal Saxena | Towards Data Science

99% of text data is available in unstructured form. And, when we talk about the unstructured data that is hard to interpret and manage — Invoices are one example of unstructured data.
When we work in the analytics and data science field, we usually need data in tabular form to analyse them. By analysing, I mean — we can plot the insights, monitoring, forecasting future insights, managing future demands, much more.
As per the information on quora, a single large business can generate 33000 invoices in a month and based on a report by bizjournals — there are 16000+ large companies, which mean millions of invoices are being developed in a single day — which is a lot more unstructured data.
The good thing with the invoices are — most of them follow the same patterns. So that says — if we can match the pattern of a single invoice, we can deal with all kinds of invoices in any industry.
As of today, some of the industries have adopted automated invoice processing methods — the majority of them are still processing them manually, and a few are working on the same process of automating them to save time and money.
Let’s have a look at how we can automate this process using python programming. Our end product will be a function that will take the list of pdf files as input and generate an excel sheet where a single row will denote an invoice file in a structured way.
We will require a few libraries to read pdf files and applying a regular expression to our text data.
pdfplumber — to read pdf files
re — to apply regular expression
pandas — to create and manipulate our dataset
import pdfplumberimport reimport itertoolsimport pandas as pd
Here, we have three sections. The first uppermost section gives us the details about date & time, username, program name, and the menu code. The second middle section tells us about the available menu options for this invoice. Finally, the third last bottom section tells us about the authorization code under each menu option.
Our first goal is to make a python script that can distinguish between each section.
with pdfplumber.open("Menu_Authorizations.pdf") as pdf:    next_page_continue=False    for pcount in range(len(pdf.pages)):        page=pdf.pages[pcount]        text=page.extract_text()
The above code is helping us to open the pdf file. Here, we only have a single pdf file with 300 menus. Some menus are taking two pages as they have more menu options. We need to manage that case also based on regular expression results.
We can use extract_text() function to extract data from the pdf. As we are using a loop so, at a time, an invoice will be entertained.
When we iterate over the extracted text, our loop iterates the text, line by line. That means — we have to define regular expression based on lines our loop is executing.
Let’s first define the regular expression for each condition.
Menu number is present in the line where the line starts with “MENU”. Here, we need a regular expression that can detect that time.
menu_re= re.compile(r'^MENU:')
This single line of python can do that task for us.
As per the menu format — just below the menu code, we can find the list of menu options. Here, we need to make sure our line has respective headers for the menu options table.
menu_func_re=re.compile(r'^\s*\d+\s{2}[A-Za-z\s\d\W]+$')
The above regular expression can help us if our line starts with space, number, and followed by two spaces. In this way, we can detect this line.
The is the final section that we need to extract. We need to check if the line starts with the number followed by the code “APM”.
APM_re=re.compile(r'\s*\d+\s+APM')APM_re_internal=re.compile(r"^AUTHORIZED:")
The above two lines can do the task for us.
Once the regular expression part is done — we can connect all the code sections with the necessary conditions.
By running the code — we can get the lists of lists that denotes the documents and their relevant information.
Now, we can use pandas to convert the lists of lists to a data frame and then export that data frame to an excel sheet.
That’s all for this informative article. We have covered the step by step approach to convert the PDF invoices to an excel sheet. We have also covered the importance of this use case this time.
I hope you liked the article. Stay tuned for the upcoming article.
Thanks for the reading!
Here are some of my best picks:
towardsdatascience.com
towardsdatascience.com
betterprogramming.pub
towardsdatascience.com
Before you go...
If you liked this article and want to stay tuned with more exciting articles on Python & Data Science — do consider becoming a medium member by clicking here https://pranjalai.medium.com/membership.
Please do consider signing up using my referral link. In this way, the portion of the membership fee goes to me, which motivates me to write more exciting stuff on Python and Data Science.
Also, feel free to subscribe to my free newsletter: Pranjal’s Newsletter.