Loading Ridiculously Large Excel Files in Python | by Yaakov Bressler | Towards Data Science

Sometimes you need to work with .xls or .xlx files. In the event that they’re larger than 1M rows, excel will either load the first 1M rows or crash. This article will demonstrate how to load such files in python.
Before we dive in, here’s some background information on how spreadsheets work on the backend.
Spreadsheets store data as row-wise arrays consisting of column keys with cell values.1
The functionality of spreadsheets deride from their behavior as a directed acyclic graph (DAG) which allow cyclical (arbitrary) relational functionality.2 This is best demonstrated with the illustration below which shows how some cell values are derived from their relationships to others.
Current size limits for excel are 1,048,576 rows by 16,384 columns — owing to memory resources. To be clear, data can be stored in an excel file which breaks these rules — but will not function in the excel program.
If you were to open an excel file of 1.25M rows, the program will drop all data below row 1M.
The easiest answer is that the file itself is easily shareable and provides better encryption than csv, txt, or json structure. Also, excel’s UI is incredibly user friendly and makes inspecting data really easy.
Some specific situations where I’ve received massive excel files:
External access not-allowed to database — existing user exports data to encrypted excel format and shares.
Data is being collected/generated by non-technical team. (Ex. agile marketing project.)
Data needs to be inspected by legal / compliance team before being shared.
Programmatic analysis or data manipulation has the advantage of:
reproducibility
automation
efficiency
reduction of spreadsheet risks
To prove this challenge and solution, let’s first create a massive excel file.
Assuming you have python installed on your computer, run the following command on your terminal:
pip install pandas openpyxl namegenerator
I included several data types (string, float, datetime) to make loading non-trivial.
Note: This will crash your machine if you have less than 8GB system memory.
My file is 210,714,241 bites (210 MB). Inspect your file with the code below to get the actual size.
f_path = "my_ridiculous_excel_file.xlsx"os.path.getsize(f_path)
Interestingly, csv format takes up significantly larger space — 324,425,683 bites (324 MB). Larger data size — but is serialized. (If you have any clue why / how this is happening, please leave as a comment.)
Loading excel files is a memory intensive action. The entire file is loaded into memory >> then each row is loaded into memory >> row is structured into a numpy array of key value pairs>> row is converted to a pandas Series >> rows are concatenated to a dataframe object.
data_path = "data/my_excel_file.xls"df = pd.read_excel(data_path, index_col=0, engine="openpyxl")
The above command took my computer 14 minutes to load. That’s way too long...
Is supposed to be faster than openpxyl but doesn’t work for > 1M rows of data...
The file is loaded to memory but data is loaded through a generator which allows mapped-retrieval of values. Still slow but a tiny drop faster than Pandas.
Openpyxl Documentation: Memory use is fairly high in comparison with other libraries and applications and is approximately 50 times the original file size.
wb = openpyxl.load_workbook(filename=data_path, read_only=True)ws = wb.active# Convert to a dfdf = pd.DataFrame(ws)
The above command took my computer 11 minutes 44 seconds to load.
wb = openpyxl.load_workbook(filename=data_path, read_only=True)ws = wb.active# Load the rowsrows = ws.rowsfirst_row = [cell.value for cell in next(rows)]# Load the datadata = []for row in rows:    record = {}    for key, cell in zip(first_row, row):        record[key] = cell.value    data.append(record)# Convert to a dfdf = pd.DataFrame(data)
The above command took my computer 11 minutes 8 seconds to load. Slightly better than the read_excel method, but still slow.
Unfortunately, couldn’t load.
Received the following assertion error:
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/xlrd/xlsx.py in do_row(self, row_elem)     635             self.rowx = int(row_number) - 1    636             explicit_row_number = 1--> 637         assert 0 <= self.rowx < X12_MAX_ROWS    638         rowx = self.rowx    639         colx = -1
Of what I was able to surmise from StackOverflow, this error is likely arising from exceeding allowed dimensions of the data or a non-allowed character — the latter of which is not likely.
If you’re working with massive excel files, try not to. They’re very difficult data structures to process — especially when your data is large. Consider serialized formats such as parquet, csv, json, or pickle (python’s binary stream).
If the data you’re working with is regularly larger than 1M rows, consider using a map-reduce tool — such as Dask, Apache Spark, or Apache Hadoop.
Answer to What Are The Data Structures Behind A Spreadsheet?. Software Engineering Stack Exchange. [Accessed 10 June 2020].Ack! Cell, Tree, DAG. Cs.gmu.edu. [Accessed 10 June 2020].
Answer to What Are The Data Structures Behind A Spreadsheet?. Software Engineering Stack Exchange. [Accessed 10 June 2020].
Ack! Cell, Tree, DAG. Cs.gmu.edu. [Accessed 10 June 2020].
Special thanks to: Aaron Hall, Hervé Yav, Ephraim Klestzick, Martin Durant