Saving Metadata with DataFrames. Saving metadata with DataFrames using... | by Darren Smith | Towards Data Science

Metadata is important — okay, perhaps not exactly a life & death issue for most Python data engineers, but its power and utility should not be overlooked. It enriches data with essential context, such as when, where and how it was created. Collection and storage of metadata should be considered a key feature to include in data processing applications.
But how best to do this? If using the popular data analysis toolkit Pandas, how can metadata be stored with the ubiquitous DataFrame?
Unfortunately Pandas doesn’t have great support for metadata. There’s no conventional way to attach metadata to a DataFrame, or for portable storage of combined data & metadata.
The essential challenge is information coupling; how to ensure data & metadata remain linked together, not only during a single Python session, but also as they get persisted and passed from one system to the next. This latter question is our focus here; how can data & metadata be stored together in a portable and durable format?
Naively we might try to save DataFrames as “pickle” files, perhaps adding metadata as a custom attribute or using the experimental attrs. This feels natural, straightforward and initially appears to work. But this approach has serious drawbacks. Pickles can be very specific to the particular version of Python and Pandas used to create them, rendering them unusable by different versions. They are not readily portable to non-Python programs, nor particularly optimised for performance.
Parquet & Arrow approach
We will take a step-by-step look at an alternative approach, based on Parquet and Arrow, which allows a DataFrame and metadata to be persisted together in a single, portable file. Before proceeding to the details, a brief overview of Parquet and Arrow is in order, because these are relatively less familiar technologies.
Apache Parquet is a file format. It has the aim of efficient storage of tabular data. Unlike CSV, in which data is written row by row, Parquet stores data column by column. This approach benefits compression and read/write/query performance. Pandas has supported Parquet since version 0.21, so the familiar DataFrame methods to_csv and to_pickle are now joined by to_parquet. Parquet files typically have extension “.parquet”. A feature relevant to the present discussion is that Parquet supports the inclusion of file-level metadata.
While Parquet describes how data is arranged within files, Apache Arrow is concerned with its layout in memory. Arrow is a cross-language specification that describes how to store columnar data in memory. It serves as the internals of data processing applications & libraries, allowing them to efficiently work with and share large tabular datasets. It has many features, but two of interest here are its ability to convert to and from various Python types (DataFrames, dicts, Numpy arrays etc.), and its support for reading & writing Parquet files.
In the following solution we will first use Arrow to convert a DataFrame to an Arrow table and then attach metadata. This enriched table will then be saved as a Parquet file. Additionally we will see how to restore the DataFrame and metadata from the saved file.
But hang on, why don’t we just call the DataFrame’s to_parquet method to write a Parquet file directly? Well as of Pandas 1.1.4, this method doesn’t support custom metadata, hence we employ a work-around of going via Arrow.
Step-by-step code solution
The code that follows is written using Python 3.6.3, Pandas 0.23.0 and Arrow 2.0.0, all running on Ubuntu 18.04. The complete code can be found on github.
Let’s begin with the imports. We require Pandas, PyArrow (Python bindings for Arrow), Parquet and also JSON (which is used to serialise the metadata).
import pandas as pdimport pyarrow as paimport pyarrow.parquet as pqimport json
We introduce some data we wish to persist. This example imagines a weather IoT device recording a single data item, which is end-of-day values for two time-series variables, max-temperature and rainfall.
df = pd.DataFrame(  { 'temp': [12.1, 11, 13, 10, 10],    'rain': [9.2, 10.0, 2.2, 0.2, 0.4] },    index=pd.DatetimeIndex(['2020-10-12',                            '2020-10-13',                            '2020-10-14',                            '2020-10-15',                            '2020-10-16'],                           name='date'))
We also introduce some custom metadata. There is no universal format for metadata; typically it’s just key-value pairs, stored as a dict, with optional nesting of further dicts and lists. One caveat is that the value types can serialise to JSON.
custom_meta_content = {    'user': 'Wáng Fān',    'coord': '55.9533° N, 3.1883° W',    'time': '2020-10-17T03:59:59+0000'  # ISO-8601}
We require a metadata namespace key. As we will shortly see, the custom metadata will be inserted into a global metadata dict owned by an Arrow table, so we require a unique key to separate our custom metadata from other metadata which may already exist or be inserted later. A good choice is application or organisation name.
custom_meta_key = 'weatherapp.iot'
As noted earlier, our approach is to convert the DataFrame to an Arrow table. We use PyArrow to do this:
table = pa.Table.from_pandas(df)
Out of interest we can inspect this table’s metadata property (a dict instance) to see what it holds by default:
print(table.schema.metadata)
This shows some initial metadata. This has been created by Pandas and inserted with the key b'pandas'. Pandas adds its own metadata so that it can convert from an Arrow table back to a DataFrame.
Now to the add custom metadata to this table. This is not immediately possible because Arrow tables are immutable. Instead we construct a new Arrow table which is a copy of the original but with the metadata replaced; the replacement metadata will be a combination of the existing metadata and the custom metadata.
First to construct the combined metadata. This is a simple merge of the existing table metadata and the custom metadata. There are a few caveats to get this right. The metadata content must be JSON serialisable and encoded as bytes; the metadata key must also be encoded as bytes:
custom_meta_json = json.dumps(custom_meta_content)existing_meta = table.schema.metadatacombined_meta = {    custom_meta_key.encode() : custom_meta_json.encode(),    **existing_meta}
Next the Arrow table is shallow copied. The copy has the original table metadata replaced by the combined metadata:
table = table.replace_schema_metadata(combined_meta)
The original table has been discarded; only the copy is retained, and is referenced by the reused variable table. This table now contains both the custom metadata and the Pandas metadata. This can be verified with:
print(table.schema.metadata)
Saving to Parquet is straightforward. In this example we request compression, which compresses columns internally:
pq.write_table(table, 'example.parquet', compression='GZIP')
Reverse process
The DataFrame & metadata are now coupled together in single Parquet file, providing portability across programming languages and different versions of Python & Pandas. But how do we load this file to recover both the DataFrame and the metadata?
This reverse task is a lot less fuss. First read the Parquet file into an Arrow table.
restored_table = pq.read_table('example.parquet')
The DataFrame is obtained via a call of the table’s to_pandas conversion method. This operation uses the Pandas metadata to reconstruct the DataFrame, but this is under the hood details that we don’t need to worry about:
restored_df = restored_table.to_pandas()
The custom metadata is accessible via the Arrow table’s metadata object, by providing the custom metadata key used earlier (taking care to once again encode the key as bytes):
restored_meta_json = restored_table.schema.metadata[custom_meta_key.encode()]
This returns the metadata as a JSON string. The final step is to deserialise:
restored_meta = json.loads(restored_meta_json)
The metadata has been restored (in addition to the DataFrame); it is just a plain dict of key-value pairs, exactly as we originally created.
Summary
Metadata should be an important consideration of any data collection and processing system, but how it can be captured and stored is often overlooked.
Pandas lacks a dedicated mechanism for saving metadata to a DataFrame. However we have seen that recent libraries, such as Arrow and Parquet, do provide direct support for persisting DataFrames and metadata together in highly portable and performant files
Having a simple & conventional approach to couple data & metadata is the first step to its wider integration into our applications.