Pandas has superpowers in reading Excel files | by Carsten Sandtner | Towards Data Science

Most companies have financial departments which produce Excel files with key figures and tables paired with complex auxiliary calculations and diagrams. It seems like they love complex files with a bunch of sheets and a cover page showing a summary of the numbers from the following sheets. These files are relatively good interpretable by humans. But if you want to get into the numbers as a Data Analyst you are facing some hurdles as there is no logical structure at first sight. Reading a sheet as a data frame results in one cluttered table. The sheets commonly have many tables with different headlines and layouts at one page. Did you know Pandas has some superpower opening and working with this kind of Excel files?
Pandas Excel-Reader is powerful and you don’t have to declutter the whole data frame with complex Python scripts. Let’s take a look at some features using two (semi-designed1) cases.
In my GitHub repository I’ve added the example files and a Jupyter Notebook with some annotations.
Before we start using pandas read_excel function, we have to install the python package xlrd. Just use your favorite package manager and install it.
conda install xlrd# orpip install xlrd
Pandas is ready to open and read Excel files. I’ve prepared two files to explain some great features.
Let’s start with the first file a simple file with one sheet. The sheet contains two tables, some explanations, a graph and auxiliary calculations.
If you read the file with pd.read_excel('Filename.xlsx'), you’ll get some messy cluttered data frame.
It’s very time-consuming to work with. But fear not, Pandas will help you! Taking a closer look at the sheet we’ll see that we have to limit the boundaries of the tables to be read. Pandas Excel reader has some nice features to pass the columns, which row is the headline and how many rows it should read.
Let’s drill these parameters down for this simple example.
The first table is located between columns A to M. The header is in row 2, and we need to read seven rows. Notice: Don’t count the header row, it’s implicit included. Now take a look at the needed parameters.
For defining the header we use header. Header expects the row of your sheet starting at 0! Excel starts counting at one! In our sheet we need row 2, which means we pass 1 as header parameter.
We are interested in columns A-M. Pandas Excel reader supports the Excel-syntax for selecting columns. In this case we have "A:M", and we pass this as a string to the usecolsparameter. The last parameter we need is the number of rows Pandas should read. Corresponding parameter is called nrows, and we pass 7 as we need seven rows starting from row two!
Let’s read the table with pandas:
df = pd.read_excel("demosheet1.xlsx", header=1, usecols="A:M", nrows=7)
Our data frame now contains a clean representation of our desired table
Nice, isn’t it? Now let’s read the second table of this sheet.
df_employees = pd.read_excel("200910_Produktivitätsreport August 2020.xlsx", header=12, usecols="A:G", nrows=10)
A header is located at row 13 (remember, pandas starts at 0, so we pass 12), we read columns A-G, and we are reading the next ten rows. That’s it for the second table.
This was a simple file with just one sheet. As told at the beginning, most files are more complex. They may contain several sheets and a cover page. Let’s take a look how Pandas Excel reader can handle this kind of sheets.
Imagine the following example. We have a management report with a cover page. Within the file we have three more sheets with further information/numbers. They all have similar structures and beside the table with interesting numbers they contain descriptions and further (for us useless) calculations. The cover page is not from interest for us as we want the data at the following sheets to work with.
If Pandas reads this kind of files, it will interpret only the first sheet found in the file. In our example: the cover page.
Now let’s figure out how to read other sheets with Pandas Excel reader.
The reader supports a parameter called sheet_name for passing the number or name of a sheet we want to read. Let’s assume a sheet is named Sheet 1, located at position two right after the cover page, and we want to read this one. We have two options to read the sheet:
Pass the position start counting at 0, where 0 is the first, 1 the second etc.Pass the name of a sheet as a string. In this case "Sheet 1"
Pass the position start counting at 0, where 0 is the first, 1 the second etc.
Pass the name of a sheet as a string. In this case "Sheet 1"
# Pass the positiondf = pd.read_excel("complex_one.xlsx", sheet_name=1)# Pass the name as a stringdf = pd.read_excel("complex_one.xlsx", sheet_name='Sheet 1')
As we are only interested in data starting at row ten, we just pass the parameters as described at the above example to access the data.
df = pd.read_excel("complex_one.xlsx", 	sheet_name='Sheet 1', 	header=9, 	usecols="A,B,D:H", 	nrows=19, )
You can see I’ve been using a more complex string for selecting columns. I use column A, B and the columns from D to H. I’ve skipped C as it’s not being interesting. My header is in row ten (remember start counting from 0 in Pandas) and I want to read 19 rows in total. 19 because the author of this sheet decided to have 19 possible rows with data. Even when less are used.
To get rid of rows with NaN, just use
df_sheet1 = df_complex.dropna()
That was easy. But there are two more sheets within our Excel file. The reader supports reading more than one sheet. sheet_name could handle a list of sheet names or numbers, too. You could mix names and numbers.
If you use a list of sheets, pandas returns a dictionary with data frames. As all sheets in our example have the same structure, we could use
df = pd.read_excel("complex_one.xlsx", 	sheet_name=['Sheet 1', 2, 3], 	header=9, 	usecols="A,B,D:H", 	nrows=19)
and we get three data frames with the same structure. Now you can work further with the raw data:
df_sheet1 = df_complex['Sheet 1'].dropna()df_sheet2 = df_complex[2].dropna()df_sheet3 = df_complex[3].dropna()
One last useful parameter in Pandas Excel reader is setting a datatype for columns. The column named # is being interpreted as a float by Pandas. But it should be an object as it contains an employees number. Excel reader has a parameter named dtype which expects a dictionary with.
Name_of_col: dtype
For example:
types = {	'#': object}df = pd.read_excel("complex_one.xlsx", 	sheet_name=['Sheet 1', 2, 3], 	header=9, 	usecols="A,B,D:H", 	nrows=19,	dtypes=types)
Now # is being interpreted as an object. Nice!
Pandas Excel reader has more features. Read the documentation for more inspiration what’s able to do with an Excel sheet in pandas.
Now you can read raw data from complex sheets which were created for being read by humans. We can declutter our data frames while reading them. Next step is making awesome stuff with this data. Create dashboards, visualizations and models which are being fired by management reports on Excel basis. Or add the data to a database etc. As Excel is still number one tool for nearly everybody working with numbers in companies, it’s handy to know how to read and use them in Data Analyses with Pandas. I hope you could learn something new, and I am pleased about notes and comments.
[1] semi-designed as these files are based on a real world scenario I’m dealing with.