Stop persisting pandas data frames in CSVs
By: Vaclav Dekanovsky
Tags: Pandas, Python, Dataframes, Plotly Express, Code Performance

Excel and to_excel()

Sometimes it’s handy to export your data into an excel. It adds the benefit of easy manipulation, at a cost of the slowest reads and writes. It also ignores many datatypes. Timezones cannot be written into excel at all.

# exporting a dataframe to excel

df.to_excel(excel_writer, sheet_name, many_other_parameters)

Useful Parameters:

excel_writer — pandas excel writer object or file path

— pandas excel writer object or file path sheet_name — name of the sheet where the data will be output

— name of the sheet where the data will be output float_format — excel’s native number formatting

— excel’s native number formatting columns — option to alias data frames’ columns

— option to alias data frames’ columns startrow — option to shift the starting cell downward

— option to shift the starting cell downward engine — openpyxl or xlsxwriter

— or freeze_panes — option to freeze rows and columns

Advantages of excel:

allow custom formating and cell freezing

human-readable and editable format

Disadvantages of excel:

very slow reads/writes (20 times/40 times slower)

limit to 1048576 rows

serialization of the datetimes with timezones fails

More information on Pandas IO page.

Performance tests results for excel. Only 54% of columns kept the original datatype, it took 90% size of the CSV, but it took 20 times more time to write and 42 times more time to read

HDF5 and to_hdf()

Compressed format using an internal file-like structure suitable for huge heterogeneous data. It’s also ideal if we need to randomly access various parts of the dataset. If the data are stored as table (PyTable) you can directly query the hdf store using store.select(key,where="A>0 or B<5")

# exporting a dataframe to hdf

df.to_hdf(path_or_buf, key, mode, complevel, complib, append ...)

Useful Parameters:

path_or_buf — file path or HDFStore object

— file path or HDFStore object key —Identified or the group in the store

—Identified or the group in the store mode — write, append or read-append

— write, append or read-append format — fixed for fast writing and reading while table allow selecting just subset of the data

Advantages of HDF5:

for some data structures, the size and access speed can be awesome

Disadvantages of HDF5:

dataframes can be very big in size (even 300 times bigger than csv)

HDFStore is not thread-safe for writing

fixed format cannot handle categorical values

SQL and to_sql()

Quite often it’s useful to persist your data into the database. Libraries like sqlalchemy are dedicated to this task.

# Set up sqlalchemy engine

engine = create_engine(

'mssql+pyodbc://user:pass@localhost/DB?driver=ODBC+Driver+13+for+SQL+server',

isolation_level="REPEATABLE READ"

) # connect to the DB

connection = engine.connect() # exporting dataframe to SQL

df.to_sql(name="test", con=connection)

Useful Parameters:

name — name of the SQL table

— name of the SQL table con — connection engine usually by sqlalchemy.engine

— connection engine usually by chunksize — optionally load data in batches of the chunksize

Advantages of SQL:

Slower than persisting on disk (read 10 times/write 5 times, but this can be optimized)

Databases are understandable by all programmers

Disadvantages of SQL:

Some data format are not kept — category, int, floats and timedeltas

depending on the database performance can be slow

you may struggle to set up a DB connection in some cases

If you would like to increase the write time of .to_sql() try Kiran Kumar Chilla’s method described in Speed up Bulk inserts article.

Feather and to_feather()

Feather is a lightweight format for storing data frames and Arrow tables. It’s another option how to store the data, which is relatively fast and results in a small file size. It did not include it in the measurement because the engine locks the files for quite a long time and it’s hard to do several repetitions of the performance test. If you plan to persist a data frame once, feather can be an ideal option.

Other methods

Pandas offer even more persistence and reading methods. I’ve omitted json and fix-width-file because they have similar characteristics like csv. You can try to write directly to Google Big Query with .to_gbq() or stata format. New formats will definitely appear to address the need to communicate with a variety of cloud providers.

Thanks to this article, I started to like .to_clipboard() when I copy one-liners to emails, excel, or google doc.

Performance Test

Many of the methods have benefits over the CSV, but is it worth using these unusual approaches when CSV is so readily understandable around the world. Let’s have a look at the performance.

During the performance test I focus on 4 key measures:

data type preservation — how many % of columns remained original type after reading

— how many % of columns remained original type after reading compression/size — how big is the file in % of csv

— how big is the file in % of csv write_time — how long does it take to write this format as % of csv writing time

— how long does it take to write this format as % of csv writing time read_time — how long does it take to read this format as % of csv reading time

For this reason, I have prepared a dataset with 50K random numbers, strings, categories, datetimes and bools. The ranges of the numerical values come from numpy data types overview.

data = []

for i in range(1000000):

data.append(

[random.randint(-127,127), # int8

random.randint(-32768,32767), # int16

...

Generating random samples is a skill used almost in every test.

You can check the support function generating random strings and dates in the GitHub notebook, I’ll only mention one here:

def get_random_string(length: int) -> str:

"""Generated random string up to the specific lenght"""



letters = string.ascii_letters

result_str = ''.join([random.choice(letters) for i in range(random.randint(3,length))])

return result_str

Full code to generate the data frame is described in this gist:

Generate random data and measure the read/write speed in 7 iterations

Once we have some data we want to process them over and over again by different algorithms. You can write each of the tests separately, but let’s squeeze the test into one line:

# performance test

performance_df = performance_test(exporting_types) # results

performance_df.style.format("{:.2%}")

The performance_test function accepts a dictionary with the test definition, which looks like:

d = { ... "parquet_fastparquet": {

"type": "Parquet via fastparquet",

"extension": ".parquet.gzip",

"write_function": pd.DataFrame.to_parquet,

"write_params": {"engine":"fastparquet","compression":"GZIP"},

"read_function": pd.read_parquet,

"read_params": {"engine":"fastparquet"}

}

... }

The dictionary contains the functions which should be run, e.g. pd.DataFrame.to_parquet and the parameters. We iterate the dict and run one function after another:

path = "output_file"

# df is our performance test sample dataframe # persist the df

d["write_function"](df, path, **d["write_params"]) # load the df

df_loaded = d["read_function"](path, **d["read_params"]

I store the result into a dataframe to leverage Plotly.Express power to display the results with few lines of code:

# display the graph with the results

fig = pe.bar(performance_df.T, barmode='group', text="value") # format the labels

fig.update_traces(texttemplate='%{text:.2%}', textposition='auto') # add a title

fig.update_layout(title=f"Statistics for {dataset_size} records")

fig.show()

Performance test results. Data Format Perservation measure % success and size and speed are compared to the csv.

Sanity Check

Testing things on random samples is useful to get the first impression how good your application or tool is, but once it will have to meet the reality. To avoid any surprise, you should try your code on the real data. I’ve picked my favorite dataset — US Securities and Exchange Commission quarterly data dump — and run it through the performance test. I have achieved very similar results which persuaded me that my assumption was not completely wrong.