Package: gamma-io
Gamma IOExtensible I/O, filesystems and dataset layer for data-science projects.Overview"Gamma IO" provides an opinionated way to implement a "I/O and datasets" layer to avoid
cluttering your "business layer" data pipelines with infra-structured concerns. One of
the main goals is to provide a simple way to adapt this layer to your particular needs.We take anopen trunkapproach to code: the underlying code should be clean, very
easy to understand and extend. You can use the provided codeas a library dependency(eg.pip install gamma-io) oryou can simply "vendor" the code baseby copying it
over to your project as a Python module and extend it yourself.While we provide a decent amount of functionality (see below), we do not expect this
project to be able to read, write and manage data from every single possible data
source. Our main goal is to provide aconsistent and nice way to write glue codefor
the data storage layer.FeaturesClean interface to read/write datasets, separating infra configuration from data
transformations.Nice integration withgamma-config. But you can easily bring your
own configuration provider!Support for Pandas, PyArrow and Polars dataframes.Support for reading files from multiple filesystems viafsspec.First party support for partitioned Parquet datasets.Getting startedUsing pip and assuming you're using the optionalgamma-configintegration:pipinstallgamma-iogamma-config[jinja2]You can "scaffold" an initial configuration. In your project folder:python-mgamma.config.scaffoldRemove the sample files, then create yourself aconfig/20-datasets.yamlfile
with the contents:datasets:source:customers_1k:location:https://github.com/cjalmeida/gamma-io/raw/main/samples/customers-1000.zipformat:csvcompression:zipcustomers_1k_plain:location:https://github.com/cjalmeida/gamma-io/raw/main/samples/customers-1000.csvformat:csvraw:customers:location:"file:///tmp/gamma-io/data/customers"format:parquetcompression:snappypartition_by:[cluster]The file above provide two "layers": asourcelayer containing HTTPS remotecustomers_1kandcustomers_1k_plaindatasets, and arawlayer, containing acustomersdataset partitioned by theclustercolumn.In your code (or Jupyter Notebook) you can read these datasets as Pandas dataframe
easily:fromgamma.ioimportread_pandasdf=read_pandas("source","customers_1k")All details about dataset format and storage infrastructure is not cluttering the
codebase. Now let's write the dataset as set of partitioned Parquet files:fromgamma.ioimportread_pandas,write_pandas# read it againdf=read_pandas("source","customers_1k")# some transformation: let's add the cluster column to the datasetdf["cluster"]=(df["Index"]%3).astype(str)# write to our datasetwrite_pandas(df,"raw","customers")You can see it generated the Parquet structured partitioned in the "Hive" format:$tree/tmp/gamma-io/data

/tmp/gamma-io/data
└──customers├──cluster=0│└──part-0.parquet├──cluster=1│└──part-0.parquet└──cluster=2└──part-0.parquetConfiguring the filesystemIn the example above, thelocationconfiguration key points to where we can find the
dataset. The underlying infrastructure is based onfsspec, so it supports
manyfilesystem-like implementations out of the box.