Stop persisting pandas data frames in CSVs | by Vaclav Dekanovsky | Towards Data Science

CSV is a great format for data exchange. It’s understood all around the world and editable in a regular notepad. That doesn’t mean that it’s suitable for persisting all DataFrames. CSVs can be slow to read and write, they take more disk space, and most importantly CSVs don’t store information about data types.
Advantage of the CSV:
Commonly understandable
Parsing and creation supported by most programming languages
Disadvantages of the CSV:
Bigger disk usage
Slower reads and writes
Don’t store information about data types
Ilia Zaitsev did a great analysis of speed and memory usage of various pandas persistence methods. I’ll touch the performance at the end when we will program an automated performance test. First, I want to focus on a different thing — how do these file formats handle various data types.
Pandas support a rich set of data types and some of them have multiple subtypes to make work with big data frames more efficient. The fundamental data types are:
object — strings or mixed types
string — since pandas 1.0.0
int — integer number
float — floating-point numbers
bool — boolean True and False values
datetime— date and time values
timedelta —time difference between two datetimes
category — limited list of values stored a memory-efficient lookup
Since pandas is using numpy arrays as its backend structures, the ints and floats can be differentiated into more memory efficient types like int8 , int16 , int32 , int64 , unit8 , uint16 , uint32 and uint64 as well as float32 and float64.
CSV doesn’t store information about the data types and you have to specify it with each read_csv(). Without telling CSV reader, it will infer all integer columns as the least efficient int64, floats as float64, categories will be loaded as strings as well as datetimes.
# for each loaded dataset you have to specify the formats to make DataFrame efficientdf = pd.read_csv(new_file,                 dtype={"colA": "int8",                        "colB": "int16",                        "colC": "uint8",                        "colcat": "category"},                 parse_dates=["colD","colDt"])
TimeDeltas are stored as string in CSVs -5 days +18:59:39.000000 and you would have to write a special parser to turn these strings back to timedelta format of pandas.
Timezones looks like 2020-08-06 15:35:06-06:00 and also would require special handling in read_csv() .
Luckily, csv is not the only option to persist the data frames. Reading Pandas’s IO tools you see that a data frame can be written into many formats, databases, or even a clipboard.
You can run the code yourself using this GitHub notebook. In the end, I’ll describe in detail how the data were created and I’ll guide you through the performance tests and sanity check using a real dataframe.
Pickle is the python native format for object serialization. It allows the python code to implement any kind of enhancement, like the latest protocol 5 described in PEP574 pickling out-of-band data buffers.
It also means that pickling outside of the Python ecosystem is difficult. However, if you want to store some pre-processed data for later or you just don’t want to lose hours of analytical work while not having any immediate use for the data, just pickle them.
# Pandas's to_pickle methoddf.to_pickle(path)
Contrary to .to_csv() .to_pickle() method accepts only 3 parameters.
path — where the data will be stored
compression — allowing to choose various compression methods
protocol — higher protocol can process a wider range of data more efficiently
Advantages of pickle:
Faster than CSV (5–300% of CSV write and 15–200% of CSV read depending on the compression method)
The resulting file is smaller (~50% of the csv)
It keeps the information about data types (100%)
no need to specify a plethora of parameters
Disadvantages of pickle:
Native to python, so it lacks support in other programming languages
Not reliable even across different python version
An error you get when you read a pickle created in the newer version of python:
ValueError: unsupported pickle protocol: 5
Apache Parquet is a compressed binary columnar storage format used in Hadoop ecosystem. It allows serializing complex nested structures, supports column-wise compression and column-wise encoding, and offers fast reads because it’s not necessary to read the whole column is you need only part of the data.
# Pandas's to_parquet methoddf.to_parquet(path, engine, compression, index, partition_cols)
.to_parquet() method accepts only several parameters.
path — where the data will be stored
engine — pyarrow or fastparquet engine. pyarrow is usually faster, but it struggles with timedelta format. fastparquet can be significantly slower.
compression — allowing to choose various compression methods
index — whether to store data frame’s index
partition_cols — specify the order of the column partitioning
Advantages of parquet:
Faster than CSV (starting at 10 rows, pyarrow is about 5 times faster)
The resulting file is smaller (~50% of CSV)
It keeps the information about data types (pyarrow cannot handle timedelta which slower fastparquet can)
Wide support in the Hadoop ecosystem allowing fast filtering throughout many partitions
Disadvantages of parquet:
duplicated column names not supported
pyarrow engine doesn’t support some data types
More specifics on Pandas IO page.
Sometimes it’s handy to export your data into an excel. It adds the benefit of easy manipulation, at a cost of the slowest reads and writes. It also ignores many datatypes. Timezones cannot be written into excel at all.
# exporting a dataframe to exceldf.to_excel(excel_writer, sheet_name, many_other_parameters)
Useful Parameters:
excel_writer — pandas excel writer object or file path
sheet_name — name of the sheet where the data will be output
float_format — excel’s native number formatting
columns — option to alias data frames’ columns
startrow — option to shift the starting cell downward
engine — openpyxl or xlsxwriter
freeze_panes — option to freeze rows and columns
Advantages of excel:
allow custom formating and cell freezing
human-readable and editable format
Disadvantages of excel:
very slow reads/writes (20 times/40 times slower)
limit to 1048576 rows
serialization of the datetimes with timezones fails
More information on Pandas IO page.
Compressed format using an internal file-like structure suitable for huge heterogeneous data. It’s also ideal if we need to randomly access various parts of the dataset. If the data are stored as table (PyTable) you can directly query the hdf store using store.select(key,where="A>0 or B<5")
# exporting a dataframe to hdfdf.to_hdf(path_or_buf, key, mode, complevel, complib, append ...)
Useful Parameters:
path_or_buf— file path or HDFStore object
key—Identified or the group in the store
mode— write, append or read-append
format— fixed for fast writing and reading while table allow selecting just subset of the data
Advantages of HDF5:
for some data structures, the size and access speed can be awesome
Disadvantages of HDF5:
dataframes can be very big in size (even 300 times bigger than csv)
HDFStore is not thread-safe for writing
fixed format cannot handle categorical values
Quite often it’s useful to persist your data into the database. Libraries like sqlalchemy are dedicated to this task.
# Set up sqlalchemy engineengine = create_engine(    'mssql+pyodbc://user:pass@localhost/DB?driver=ODBC+Driver+13+for+SQL+server',    isolation_level="REPEATABLE READ")# connect to the DBconnection = engine.connect()# exporting dataframe to SQLdf.to_sql(name="test", con=connection)
Useful Parameters:
name — name of the SQL table
con — connection engine usually by sqlalchemy.engine
chunksize — optionally load data in batches of the chunksize
Advantages of SQL:
Slower than persisting on disk (read 10 times/write 5 times, but this can be optimized)
Databases are understandable by all programmers
Disadvantages of SQL:
Some data format are not kept — category, int, floats and timedeltas
depending on the database performance can be slow
you may struggle to set up a DB connection in some cases
If you would like to increase the write time of .to_sql() try Kiran Kumar Chilla’s method described in Speed up Bulk inserts article.
medium.com
Feather is a lightweight format for storing data frames and Arrow tables. It’s another option how to store the data, which is relatively fast and results in a small file size. It did not include it in the measurement because the engine locks the files for quite a long time and it’s hard to do several repetitions of the performance test. If you plan to persist a data frame once, feather can be an ideal option.
Pandas offer even more persistence and reading methods. I’ve omitted json and fix-width-file because they have similar characteristics like csv. You can try to write directly to Google Big Query with .to_gbq() or stata format. New formats will definitely appear to address the need to communicate with a variety of cloud providers.
Thanks to this article, I started to like .to_clipboard() when I copy one-liners to emails, excel, or google doc.
Many of the methods have benefits over the CSV, but is it worth using these unusual approaches when CSV is so readily understandable around the world. Let’s have a look at the performance.
During the performance test I focus on 4 key measures:
data type preservation — how many % of columns remained original type after reading
compression/size — how big is the file in % of csv
write_time — how long does it take to write this format as % of csv writing time
read_time — how long does it take to read this format as % of csv reading time
For this reason, I have prepared a dataset with 50K random numbers, strings, categories, datetimes and bools. The ranges of the numerical values come from numpy data types overview.
data = []for i in range(1000000):    data.append(        [random.randint(-127,127),  # int8         random.randint(-32768,32767),  # int16        ...
Generating random samples is a skill used almost in every test.
You can check the support function generating random strings and dates in the GitHub notebook, I’ll only mention one here:
def get_random_string(length: int) -> str:    """Generated random string up to the specific lenght"""        letters = string.ascii_letters    result_str = ''.join([random.choice(letters) for i in range(random.randint(3,length))])    return result_str
Full code to generate the data frame is described in this gist:
Once we have some data we want to process them over and over again by different algorithms. You can write each of the tests separately, but let’s squeeze the test into one line:
# performance testperformance_df = performance_test(exporting_types)# resultsperformance_df.style.format("{:.2%}")
The performance_test function accepts a dictionary with the test definition, which looks like:
d = { ..."parquet_fastparquet": {        "type": "Parquet via fastparquet",        "extension": ".parquet.gzip",        "write_function": pd.DataFrame.to_parquet,        "write_params": {"engine":"fastparquet","compression":"GZIP"},        "read_function": pd.read_parquet,        "read_params": {"engine":"fastparquet"}    }... }
The dictionary contains the functions which should be run, e.g. pd.DataFrame.to_parquet and the parameters. We iterate the dict and run one function after another:
path = "output_file"# df is our performance test sample dataframe# persist the dfd["write_function"](df, path, **d["write_params"])# load the df df_loaded = d["read_function"](path, **d["read_params"]
I store the result into a dataframe to leverage Plotly.Express power to display the results with few lines of code:
# display the graph with the resultsfig = pe.bar(performance_df.T, barmode='group', text="value")# format the labelsfig.update_traces(texttemplate='%{text:.2%}', textposition='auto')# add a titlefig.update_layout(title=f"Statistics for {dataset_size} records")fig.show()
towardsdatascience.com
Testing things on random samples is useful to get the first impression how good your application or tool is, but once it will have to meet the reality. To avoid any surprise, you should try your code on the real data. I’ve picked my favorite dataset — US Securities and Exchange Commission quarterly data dump — and run it through the performance test. I have achieved very similar results which persuaded me that my assumption was not completely wrong.
towardsdatascience.com
Even though pickle clearly won the performance competition there can be use-cases when you would prefer to pick another format. We also had quite a special dataset and on the real data, the performance may differ.
Feel free to play with the code (GitHub) and try the performance on your favorite dataset. For me personally, .to_pickle() is great when I store a pre-processed dataset because I don’t have to worry about data format, I simply read_pickle() and hours of work materialize back to my notebook.
towardsdatascience.com
towardsdatascience.com
If you want to create inspiring infographics like the one above, use canva.com (affiliate link, when you click on it and purchase a product, you won't pay more, but I can receive a small reward; you can always write canva.com to your browser to avoid this). You can do more than infographic and some features are available for free.