From PDF to Excel. Using tabula and xlsxwriter | by Ednalyn C. De Dios | Towards Data Science

In the real world, we’ll often encounter data in all sorts of formats. Today, we’ll tackle the task of extracting tabular data from a PDF and exporting it to Excel.
The only caveat is, the pdf file must be machine-generated. PDFs of scanned pages do not work. This is one limitation of tabula.
With that said, let’s roll!
First, let’s make sure that we have a good java environment.
java -version
You should see something like:
Next, let’s install tabula-py with the following:
pip install tabula-py
If successful,
And lastly, let’s install xlsxwriter:
pip install xlsxwriter
And we should see something like this:
Now, let’s open a jupyter notebook and get to coding!
Let us begin with importing the following:
import pandas as pdimport tabulaimport xlsxwriter
Then let’s get the data:
data = tabula.read_pdf('../data/in/titanic.pdf', pages='all', stream=True)
The code above results in a list of dataframes. Since there’s only one dataframe in the list, let’s go ahead and extract the dataframe by itself. We can do with df[0].
Next, we’ll need to convert the column headers to the first row by resetting the index and transposing the dataframe twice.
df = data[0].reset_index().T.reset_index().T
And rename the columns as such:
df.columns=['x', 'passenger', 'pc_class', 'gender', 'age', 'parch', 'fare', 'cabin', 'embarked']
Let’s drop the first two columns with:
clean_df = df.drop(columns=['x']).reset_index(drop=True)
Now we are ready for prime time:
Let’s begin by making a new Excel file:
workbook = xlsxwriter.Workbook('../data/out/titanic.xlsx', {'nan_inf_to_errors': True})
And add a worksheet (tab) named “test”:
worksheet = workbook.add_worksheet('test')
Before writing the data into Excel, we need to convert our dataframe into a list first.
data = list(clean_df.to_records(index=False))
Let’s initialize the first cell (A1) of the Excel file with:
# Rows and columns are zero indexedrow = 0col = 0
Let’s examine the first row:
We need to fix that by manually inserting the column headers. Let’s insert the column names into index 0 of the list.
data.insert(0, ['passenger', 'pc_class', 'gender', 'age', 'parch', 'fare', 'cabin', 'embarked'])
Now we can write row by row.
# Iterate over the datafor passenger, pc_class, gender, age, parch, fare, cabin, embarked in data:    worksheet.write(row, col, passenger)    worksheet.write(row, col + 1, pc_class)    worksheet.write(row, col + 2, gender)    worksheet.write(row, col + 3, age)    worksheet.write(row, col + 4, parch)    worksheet.write(row, col + 5, fare)    worksheet.write(row, col + 6, cabin)    worksheet.write(row, col + 7, embarked)    row += 1
And finally, let’s close the workbook.
workbook.close()
Voila!
And that’s it! We got ourselves an Excel file.
In this post, we learned how to use tabula and xlsxwriter. We took a pdf file, extracted it to a dataframe, and then wrote the contents into an Excel file. Combined with for loops, we could easily ingest many pdfs and have a flat file that could feed into a database like Redshift. And with a little bit of hackery, we’ve got ourselves a winning combination for automation.
You can check out the repo on my Github for further examination. Here’s the whole code:
Thanks for stopping by and reading my post.
Stay tuned!
If you want to learn more about my journey from slacker to data scientist, check out the article below:
towardsdatascience.com
And if you’re thinking about switching gears and venture into data science, start thinking about rebranding now:
towardsdatascience.com
You can reach me on Twitter or LinkedIn.