How to extract tables from PDF using Python Pandas and tabula-py | by Angelica Lo Duca | Towards Data Science

This tutorial is an improvement of my previous post, where I extracted multiple tables without Python pandas. In this tutorial, I will use the same PDF file, as that used in my previous post, with the difference that I manipulate the extracted tables with Python pandas.
The code of this tutorial can be downloaded from my Github repository.
Almost all the pages of the analysed PDF file have the following structure:
In the top-right part of the page, there is the name of the Italian region, while in the bottom-right part of the page there is a table.
I want to extract both the region names and the tables for all the pages. I need to extract the bounding box for both the tables. The full procedure to measure margins is illustrated in my previous post, section Define margins.
This script implements the following steps:
define the bounding box, which is represented through a list with the following shape: [top,left,bottom,width]. Data within the bounding box are expressed in cm. They must be converted to PDF points, since tabula-py requires them in this format. We set the conversion factor fc = 28.28.
extract data using the read_pdf() function
save data to a pandas dataframe.
In this example, we scan the pdf twice: firstly to extract the regions names, secondly, to extract tables. Thus we need to define two bounding boxes.
Firstly, I define the bounding box to extract the regions:
box = [1.5, 22,3.8,26.741]fc = 28.28         for i in range(0, len(box)):    box[i] *= fc
Then, Iimport the tabula-py library and we define the list of pages from which we must extract information, as well as the file name.
import tabula as tbpages = [3,5,6,8,9,10,12,14,16,18,22,24,26,28,30,32,34,36,38,40]file = “source/Bolletino-sorveglianza-integrata-COVID-19_17-marzo-2020_appendix.pdf”
Now I can read the list of regions from the pdf. I use the read_pdf() function and we set the output format to json.
regions_raw = tb.read_pdf(file, pages=pages,area=[box],output_format="json")
I note that the produced output is very complex. However, the general structure contains the region name of the i-th region in the position regions_raw[i]['data'][0][0]['text']. I build a list with all the regions, by looping into the region_raw list.
regions = []for i in range(0,len(regions_raw)):    regions.append(regions_raw[i]['data'][0][0]['text'])
I define the bounding box and we multiply each value for the conversion factor fc. In order to understand how the mechanism works, firstly, I extract the table of the first page and then we generalise to all the pages. In this example, the first page corresponds to page 3.
box = [8,10,25,26]for i in range(0, len(box)):    box[i] *= fc
Now I can read the pdf. In this case I set the output_format to DataFrame. The result is stored in tl, which is a list. I can convert it to a dataframe, simply using tl[0].
page = 3tl = tb.read_pdf(file, pages=page,area=[box],output_format="dataframe", stream=True)df = tl[0]df.head()
I note that the columns names are wrong. In addition, the first three rows are wrong. For this reason, I can rename the columns names by using the dataframe function rename().
df.rename(columns={ df.columns[0]: "Fascia d'età" , df.columns[1]: "Casi"}, inplace = True)df.head()
Now I can drop the first two rows by using the dropna() function.
df = df.dropna()df.head()
I can drop the new first row by selecting all the rows which do not contain this value.
df = df[df["Fascia d'età"] != "Fascia d'età"]df.head(8)
Now I add a new column to df, called Regione which contains the region name. I scan the pages list to extract the index of the current region.
region_column = []for i in range(0, len(df)):    index = pages.index(page)    region_column.append(regions[index])df['Regione'] = region_columndf.head()
Now I can generalise the previous code to extract the tables of all the pages. Firstly, I build an empty DataFrame, which will contain the values for all the regions. I will use the pd.concat() function to concatenate all the tables of alle the pages. I scan all the pages contained in the pages list.
import pandas as pddf = pd.DataFrame()for page in pages:        index = pages.index(page)    region = regions[index]    print(region)        tl = tb.read_pdf(file, pages=page,area=[box],output_format="dataframe", stream=True)        dft = tl[0]    dft.rename(columns={ dft.columns[0]: "Fascia d'età", dft.columns[1]: "Casi"}, inplace = True)        region_column = []    for i in range(0, len(dft)):        region_column.append(region)    dft['Regione'] = region_column        df = pd.concat([df, dft])
Similarly to the previous case, I drop all wrong records.
df.dropna(inplace=True)df = df[df["Fascia d'età"] != "Fascia d'età"]
Now I can save the result as a csv file.
df.to_csv('output.csv')
In this tutorial I have illustrated how to convert multiple PDF table into a single pandas DataFrame and export it as a CSV file.
The procedure involves three steps: define the bounding box, extract the tables through the tabula-py library and export them to a CSV file.
If you want to be updated on my research and other activities, you can follow me on Twitter, Youtube and Github.