Read Messy Excel Files Like a Pro | by Tim Meehan | Towards Data Science

As data scientists, we often interface with non-technical colleagues. When they provide data in the form of a spreadsheet, they probably don’t think about how pd.read_excel() will perform on our end, it’s just not their job to format their data to favor our tech stack.
Luckily, there are ways we can remedy this using pandas. In this article, I will walk you through some code I wrote to handle such a spreadsheet. One thing to note is the pd.read_excel() file will usually return something, so debugging issues with spreadsheet structure rests solely on manually inspecting the resulting dataframe.
The Spreadsheet
Let’s start by taking a look at the spreadsheet. This was generated by software used internally, so it’s something I encounter regularly. You’ll see that the spreadsheet has the following hurdles:
The header row is not the first rowthe last row of the data is not the last row in the spreadsheetthe first column of the data is not the first column of the spreadsheetThere are cell group labels (Customer Status and Region) which only exist in one row, so most of the rows will return NaN
The header row is not the first row
the last row of the data is not the last row in the spreadsheet
the first column of the data is not the first column of the spreadsheet
There are cell group labels (Customer Status and Region) which only exist in one row, so most of the rows will return NaN
As you can see, if your single goal was to look at this spreadsheet to get some insight, it’s very user friendly. It has a title, some filter parameters used to create the data and some boxes for a visual/hierarchical grouping. It’s when you try to read this into a pandas dataframe where things become problematic. Let’s try it and see what we get.
The above picture shows what pd.read_excel() would return for this spreadsheet without any additional arguments, ouch. Let’s work on getting the right header row. Pandas provides a header parameter which we can use to use that index as the starting position. We could hard code this index, but do we really want to change this every time we update the spreadsheet? No. In my case, the column names will not change, but the index of the header row and footer row will. Since I know the column names won’t change, we can read the file in, then search for the column name to retrieve the index of the header. I’ll use the header=None param so we don’t get random column names. I know my Customer Status column will be in column position 1.
raw_data = pd.read_excel('medium_example.xlsx', header=None)header_idx = raw_data[    raw_data[1].eq('Customer Status')].index.values[0]
Let’s do the same thing to find the footer row. In this case we will see Count in row column 1. The row containing ‘Count’ is a summary row of the spreadsheet. It represents the total number of products sold, but is not part of the actual data. The index we want is one row above the row containing the count value. we will correct for this by subtracting 1 from the index we find.
footer_idx = raw_data[    raw_data[1].eq('Count')].index.values[0] - 1
okay, now we have the index of the header row and last row of our desired data. We also know our first column of the data starts in the column B of the spreadsheet. Based on this, we can use this information to give the function a range to read from. To specify the rows, we’ll use the header parameter to specify our starting index. We can then use the nrows parameter to specify how many rows we will read following the header. This can be calculated by subtracting our header index from our footer index. We can also pass a range of excel columns which are denoted in alphabetical order. Here, we are taking columns B through Z. The data in the spreadsheet falls under columns B:E, however read_excel() will not return columns if they are empty, so this will help future proof the code if any columns get added.
# specify number of rows to read after header rownum_rows = footer_idx - header_idxclean_data = pd.read_excel(    io='medium_example.xlsx',     header=header_idx,    nrows=num_rows,    usecols="B:Z")
We passed in the same file as last time, but this time we used the header and footer indexes that was parsed and told the function to only read that window. We used the usecols param to tell pandas to ignore the first column. Let’s see what clean_data looks like:
This looks much better, but we still nave NaN values in most of our Customer Status and Region columns. In this case, we know that the first non-null value can fill the NaN values in the series until the next non-null value. Lucky for us, pandas has a built in function for this! You’ve most likely used it before, but it has a handy keyword arg that is less known. We will iterate through the columns that need this transformation and use Series.fillna(method='ffill').
for col in ['Customer Status', 'Region']:    clean_data[col] = clean_data[col].fillna(method='ffill')
Let’s see how this looks:
Nice! our values filled in as expected. We now have our data in the proper format for analysis. While this is great, in my opinion there’s one last step. Make it reusable! As I said before, this is something I will be using a lot, so I wrote a simple module I can import, and my colleagues using Python can as well. Here’s the module for reference:
import pandas as pdHEADER_COL_NAME = "Customer Status"FIRST_COLUMN = 1FOOTER_VALUE = "Count"FOOTER_VALUE_COLUMN = 1FFILL_COLUMNS = ["Customer Status", "Region"]class MessyExcelReader:    def __init__(self, file_path):        self.file_path = file_path        self.raw_df = self.read_raw_file()        self.header_idx = None        self.footer_idx = None        self.filtered_df = None    def read_raw_file(self):        return pd.read_excel(self.file_path, header=None)    def parse_for_header_loc(self):        header_row = self.raw_df[            self.raw_df[FIRST_COLUMN].eq(HEADER_COL_NAME)        ]        if len(header_row):            self.header_idx = header_row.index.values[0]        else:            raise ValueError(                f"No such value {HEADER_COL_NAME} in column {FIRST_COLUMN}"            )    def parse_for_footer_loc(self):        footer_row = self.raw_df[            self.raw_df[FOOTER_VALUE_COLUMN].eq(FOOTER_VALUE)        ]        if len(footer_row):            # subtract 1 from idx since we want row above "total"            self.footer_idx = footer_row.index.values[0] - 1        else:            raise ValueError(                f"No such value {FOOTER_VALUE} in column {FOOTER_VALUE_COLUMN}"            )    def read_index_locs(self):        num_rows = self.footer_idx - self.header_idx        df = pd.read_excel(            io=self.file_path,            header=self.header_idx,            nrows=num_rows,            usecols='B:Z'        )        self.filtered_df = df    def forward_fill_columns(self, cols):        for col in cols:            self.filtered_df[col] = self.filtered_df[col].fillna(method="ffill")    def read_file_clean(self):        self.parse_for_header_loc()        self.parse_for_footer_loc()        self.read_index_locs()        self.forward_fill_columns(FFILL_COLUMNS)        return self.filtered_df
With these tools, go forth and conquer messy spreadsheets!