Stop Using CSVs for Storage — Here Are the Top 5 Alternatives | by Dario Radečić | Towards Data Science

Everyone and their grandmother know what a CSV file is. But is it the optimal way to store data? Heck no. It’s probably the worst storage format if you don’t plan to view or edit data on the fly.
If you’re storing large volumes of data, opting for CSVs will cost you both time and money.
Today you’ll learn about five CSV alternatives. Each provides an advantage, either in read/write time or in file size. Some are even better in all areas.
Let’s set up the environment before going over the file formats.
You’ll need a couple of libraries to follow along. The best practice is to install them inside a virtual environment, so that’s exactly what you’ll do. The following code snippet creates a new virtual environment through Anaconda and installs every required library:
conda create --name file_formats python=3.8conda activate file_formatsconda install -c conda forge numpy pandas fastavro pyarrow feather-format jupyter jupyterlab
Once the installation finishes, you can execute the following command to start a JupyterLab session:
jupyter lab
The next step is to import the libraries and create an arbitrary dataset. You’ll make one with 5 columns and 10M rows:
import numpy as npimport pandas as pdimport featherimport pickleimport pyarrow as paimport pyarrow.orc as orc from fastavro import writer, reader, parse_schemanp.random.seed = 42df_size = 10_000_000df = pd.DataFrame({    'a': np.random.rand(df_size),    'b': np.random.rand(df_size),    'c': np.random.rand(df_size),    'd': np.random.rand(df_size),    'e': np.random.rand(df_size)})df.head()
Here’s how it looks like:
You now have everything needed to start experimenting with different data formats. Let’s cover ORC first.
ORC stands for Optimized Row Columnar. It’s a data format optimized for reads and writes in Hive. As Hive is painfully slow, folks at Hortonworks decided to develop the ORC file format to speed it up.
In Python, you can use the read_orc() function from Pandas to read ORC files. Unfortunately, there’s no alternative function for writing ORC files, so you’ll have to use PyArrow.
Here’s an example of writing Pandas DataFrames:
table = pa.Table.from_pandas(df, preserve_index=False)orc.write_table(table, '10M.orc')
And here’s the command for reading ORC files:
df = pd.read_orc('10M.orc')
You can learn more about ORC here:
towardsdatascience.com
Avro is an open-source project which provides services of data serialization and exchange for Apache Hadoop. It stores a JSON-like schema with the data, so the correct data types are known in advance. That’s where the compression happens.
Avro has an API for every major programming language, but it doesn’t support Pandas by default.
Here’s the set of commands for saving a Pandas DataFrame to an Avro file:
# 1. Define the schemaschema = {    'doc': 'Float data',    'name': 'Data',    'namespace': 'data',    'type': 'record',    'fields': [        {'name': 'a', 'type': 'float'},        {'name': 'b', 'type': 'float'},        {'name': 'c', 'type': 'float'},        {'name': 'd', 'type': 'float'},        {'name': 'e', 'type': 'float'},    ]}parsed_schema = parse_schema(schema)# 2. Convert pd.DataFrame to records - list of dictionariesrecords = df.to_dict('records')# 3. Write to Avro filewith open('10M.avro', 'wb') as out:    writer(out, parsed_schema, records)
Reading Avro files is no picnic either:
# 1. List to store the recordsavro_records = []# 2. Read the Avro filewith open('10M.avro', 'rb') as fo:    avro_reader = reader(fo)    for record in avro_reader:        avro_records.append(record)        # 3. Convert to pd.DataFramedf = pd.DataFrame(avro_records)
You can learn more about Avro here:
towardsdatascience.com
Apache Parquet is a data storage format designed for efficiency. The reason behind this is the column storage architecture, as it allows you to skip data that isn’t relevant quickly. This way, both queries and aggregations are faster, resulting in hardware savings.
The best news is — Pandas has full support for Parquet files.
Here’s the command for writing a Pandas DataFrame to a Parquet file:
df.to_parquet('10M.parquet')
And here’s the equivalent for reading:
df = pd.read_parquet('10M.parquet')
You can learn more about Parquet here:
towardsdatascience.com
You can use the pickle module to serialize objects and save them to a file. Likewise, you can then deserialize the serialized file to load them back when needed. Pickle has one major advantage over other formats — you can use it to store any Python object. One of the most widely used functionalities is saving machine learning models after the training is complete.
The biggest downside is that Pickle is Python-specific, so cross-language support isn’t guaranteed. It could be a deal-breaker for any project requiring data communication between Python and R, for example.
Here’s how to write a Pandas DataFrame to a Pickle file:
with open('10M.pkl', 'wb') as f:    pickle.dump(df, f)
You’ll only have to change the file mode when reading a Pickle file:
with open('10M.pkl', 'rb') as f:    df = pickle.load(f)
You can learn more about Pickle here:
towardsdatascience.com
Feather is a data format for storing data frames. It’s designed around a simple premise — to push data frames in and out of memory as efficiently as possible. It was initially designed for fast communication between Python and R, but you’re not limited to this use case.
You can use the feather library to work with Feather files in Python. It’s the fastest available option currently.
Here’s the command for saving Pandas DataFrames to a Feather file:
feather.write_dataframe(df, '10M.feather')
And here’s the command for reading:
df = feather.read_dataframe('10M.feather')
You can learn more about Feather here:
towardsdatascience.com
Many highly optimized file formats are useless if you need to change or even view the data on the fly. If that’s not the case, you should generally avoid CSVs.
Below is a comparison between CSV and every other mentioned data format in write time. The goal was to save previously created 10Mx5 dataset locally:
The differences are astronomical. Feather is about 115 times faster than CSV for storing identical datasets. Even if you decide to go with something more compatible such as Parquet, there’s still a 17 times decrease in write time.
Let’s talk about the read times next. The goal is to compare how much time it takes to read identical datasets in different formats:
CSVs aren’t so terrible here. Apache Avro is the absolute worse due to required parsing. Pickle is the fastest one, so it looks like the most promising option if you’re working only in Python.
And finally, let’s compare the file sizes on the disk:
Things don’t look good for CSVs. The file size reduction goes from 2.4x to 4.8x, depending on the file format.
To summarize, if you store gigabytes of data daily, choosing the correct file format is crucial. If you’re working only in Python, you can’t go wrong with Pickle. If you need something a bit more versatile, go with any other mentioned format.
What are your thoughts on these CSV alternatives? Which one(s) do you use if viewing and editing data on the fly isn’t required? Let me know in the comments below.
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.
medium.com
Follow me on Medium for more stories like this
Sign up for my newsletter
Connect on LinkedIn