Package: blizz
blizz –be blizzfuleffortless Python table schema managementIf you are looking for…📋 …an lightweight option to capture data schemas and field annotations for either Pandas or PySpark,⭐ …which makes table definitions available not just at runtime, but already while developing code
– with IDE type checking and autocomplete!❄️ …which allows dynamic schema inheritance/reference, to express several levels of derived data assets –
as common in stable data pipelines – tracing the full data field lineage across your project and boosting
refactoring and debugging productivity,🚀 …which offers pre-built PySpark & Pandas decorators for most common data checks & transformations – all based
on metadata,🏃 …which supports your workflow with boostrapping tools to generate basic code and Sphinx documentationsthenblizzis for you!Contents:InstallationUsage Tutorialblizz primitives: Relation and FieldBasic usage with pandasUsing field definitionsDefining metadata propertiesRelation HierarchiesMaking use of metadata declarations: blizz.check & blizz.applyUsing blizz with PysparkAPI Reference on Relation and FieldAPI Reference on blizz.check & blizz.applyBoostrapping – generate schema definitions from DataFramesBeyond: blizz Feature StoreBeyond: Generate DocumentationInstallationYou can install the latest stable version ofblizzsimply using Pip:pip install blizzNote, that eitherpandasorpysparkneed to be installed separately, if you do
not have them already.If you have any issues, questions or suggestions – do not hesitate to open an GitHub issue.A test installation from the latest sources is possible as well:pip install git+https://github.com/joerg-schneider/blizz/#egg=blizzUsage TutorialThe following is an introduction and basic tutorial toblizz. You can find also some example scripts insrc/tutorialof this repository.blizz primitives:RelationandFieldblizz.Relationandblizz.Fieldare the main primitives implementing its behaviour.A relation – in the computer scientific sense – refers here to any table/dataframe/... one
likes to define, made up of rows and columns, the latter defined byblizz.Field.
DataFrames from PySpark and Pandas are the supported means of capturing data instances to
which a givenRelationapplies to.The following sections explain how to useblizz.Relationandblizz.Field.Basic usage with PandasThe following is a minimal example of creating a Relation withblizz:define yourRelationas a subclass fromblizz'sRelationsuperclassadd some field names as membersaddload()as a classmethod and implement it to return either a Spark or Pandas DataFramefromblizzimportRelation,FieldimportpandasaspdclassIris(Relation):SEPAL_LENGTH=Field("sepal_length")SEPAL_WIDTH=Field("sepal_width")PETAL_LENGTH=Field("petal_length")PETAL_WIDTH=Field("petal_width")SPECIES=Field("species")@classmethoddefload(cls)->pd.DataFrame:returnpd.read_csv("https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7""/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv")Callingload()on the Relation, we can retrieve a dataframe:iris_df=Iris.load()Most importantly, now using the Relation's Schema, we can access/modifyiris_dfreferencing fields we like:print(iris_df[[Iris.SEPAL_LENGTH,Iris.SEPAL_WIDTH]])A command like above now references the member variables on Iris to access fields – these will also be autocompleted
by your IDE, checked if they exist (avoiding typos) and allow easy refactoring (e.g. if the column name changes in
the source – just change it once inField(...)!).Using Field definitions inside of load()Sinceload()is a class method, it is easily possible to refer to definedFieldsalready in it:fromblizzimportRelation,FieldimportpandasaspdclassIris(Relation):SEPAL_LENGTH=Field("sepal_length")SEPAL_WIDTH=Field("sepal_width")PETAL_LENGTH=Field("petal_length")PETAL_WIDTH=Field("petal_width")SPECIES=Field("species")# a newly defined field:SEPAL_PETAL_WIDTH=Field("sepal_petal_width")@classmethoddefload(cls)->pd.DataFrame:iris=pd.read_csv("https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7""/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv")iris[cls.SEPAL_PETAL_WIDTH]=iris[cls.SEPAL_WIDTH]+iris[cls.PETAL_WIDTH]returnirisIn this example, one can see how to add another, derived field to theRelationwhich is just
built on loading, leveraging existingFieldreferences.Defining Metadata properties on FieldFieldis implemented in an elegant way: It cleanly folds down to a Python string when you use it as part
of an expression (e.g. inside of square brackets for Pandas), but actually,it is not a string.In the previous examples,Fieldshave been only constructed using its default first attribute,.name.
However, it does also support a big variety of useful declarations one might like to make (seeAPI Reference on Relation and Fieldbelow!), such as defining datatypes, adding a description or
theField'sdefault value:classIris(Relation):# setting some example additional Field attributes:# 1. of course most importantly, one can set a datatype to complete the#    Relation's schema definition in the classical sense:SEPAL_LENGTH=Field("sepal_length",datatype=float)# for use with Pandas, `datatype` accepts Python inbuilts (e.g. float, object, int),# quoted names of Pandas datatypes, and also instances of numpy types, such as# numpy.int, numpy.number, etc.# 2. to capture metadata, and for documentation purposes, a description can be set:SEPAL_WIDTH=Field("sepal_width",description="The Sepal length")# 3. `default` also to capture a default row value for a Field:PETAL_LENGTH=Field("petal_length",default=0.0)PETAL_WIDTH=Field("petal_width")# 4. the boolean flag `key` allows to specify key fields:SPECIES=Field("species",key=True)@classmethoddefload(cls)->pd.DataFrame:iris=pd.read_csv("https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7""/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv")# using all defined key fields, one could run a simply deduplication command:iris_dedup=iris.drop_duplicates(subset=cls.get_key_fields())returniris_dedupRelation hierarchiesSince Fields are Python variables (class members ofRelation), this makes building connected Relations very easy:# based on Iris, we can define a derived second Relation `IrisSepal`:classIrisSepal(Relation):""" All records of Iris but filtered on a subset of Fields, just for the Sepal."""# we can simply reference the existing definitions that were made:SEPAL_LENGTH=Iris.SEPAL_LENGTHSEPAL_WIDTH=Iris.SEPAL_WIDTHSPECIES=Iris.SPECIES@classmethoddefload(cls)->pd.DataFrame:# we call Iris's load() method but filter it down to `IrisSepal`'s fields:returnIris.load()[cls.get_field_names()]In the example above, theload()method additionally shows that eachRelationoffers helpful
methods to interact with metadata – in this caseget_field_names()can retrieve all defined
fields as strings. SeeAPI Reference on Relation and Fieldfor the full list of available
methods.Making use of metadata declarations: blizz.check & blizz.applyHaving defined structured metadata on a Relation's fields can be very powerful to carry out
basic operations.For this purpose,blizzdefines two modules with Python function decorators, meant to be
added on-top of theload()method:blizz.check: defines several utilities to check a loaded Relation against the definitionblizz.apply: defines several utilities for often occuring transformations to the Relation's
dataframeAll of these utlities work are implemented with equal functionality on Pandas and Pyspark.Consider this example, which takes care of schema checking (field names, field types, field keys),
deduplication and checking for duplicates:fromblizzimportRelation,Fieldimportblizz.check,blizz.applyimportpandasaspdclassIris(Relation):SEPAL_LENGTH=Field("sepal_length",datatype=float)SEPAL_WIDTH=Field("sepal_width",datatype=float)PETAL_LENGTH=Field("petal_length",default=0.0)PETAL_WIDTH=Field("petal_width",datatype=float)SPECIES=Field("species_renamed",datatype=object,key=True,source_name="species")@classmethod@blizz.check.types@blizz.check.fields@blizz.check.keys@blizz.apply.defaults(fill=PETAL_LENGTH)# you can use field as arguments, too!@blizz.apply.deduplication@blizz.apply.renames# renames should be applied firstdefload(cls)->pd.DataFrame:returnpd.read_csv("https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7""/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv")print(Iris.load())It is important to note, that these decorator utilitiesexecute after the load() function
has finished, and the order of writing them matters. As an example, it makes more sense
to executeblizz.apply.deduplicationfirst, and then check its result withblizz.check.keys(unless you already expect it deduplicated, then avoidblizz.apply.deduplicationentirely!).In this use case, your decorators have to be defined in this order (lowest one runs first):@classmethod@blizz.check.keys@blizz.apply.deduplicationdefload(cls)->pd.DataFrame:passIn general, when you rename source fields, run@blizz.apply.renamesfirst (e.g. place it
lowest), so that following checks/applies find the fields they are looking for.It is possible to pass user defined functions meant to run as checks/applies usingblizz.check.funcandblizz.apply.funcdecorators – use the keyword argumentfunctionto pass in any
callable with a signature ofType[Relation], DataFrame -> DataFrame. Refer tocustom_decoratorsas an example script.Using blizz with PySparkUsingblizzwith PySpark works very similarly to the examples shown above – the only difference
lies in using different datatypes forFielddefinitions (either instances ofpyspark.sql.types.DataTypeor the PySpark datatypessimpleStringrepresentation (e.g. "int", "string", ...)), and of course
having to implementload()on each Relation differently, to return apyspark.sql.DataFrame:frompysparkimportSparkFilesfromblizzimportRelation,Fieldfrompyspark.sqlimportDataFramefrompyspark.sqlimportSparkSessionfrompyspark.sql.typesimportDoubleTypeimportblizz.checkclassIris(Relation):SEPAL_LENGTH=Field("sepal_length",datatype=DoubleType)SEPAL_WIDTH=Field("sepal_width",datatype="double")PETAL_LENGTH=Field("petal_length",datatype=DoubleType)PETAL_WIDTH=Field("petal_width",datatype=DoubleType)SPECIES=Field("species",datatype="string")@classmethod@blizz.check.types@blizz.check.fieldsdefload(cls,spark_session)->DataFrame:spark_session.sparkContext.addFile("https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7""/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv")df=spark_session.read.csv(SparkFiles.get("iris.csv"),inferSchema=True,header=True)returndf# set up a simple spark session:spark=SparkSession.builder.getOrCreate()# calling load(), we can retrieve a dataframe for the Relation:iris_df=Iris.load(spark)print(iris_df)# using the Relation's Schema, we can access/modify iris_df referencing fields we like:iris_df.select(Iris.SEPAL_WIDTH,Iris.SEPAL_LENGTH).show()spark.stop()API Reference on Relation and FieldTheField()constructor accepts the following arguments (besidesname, all optional):name: the target field name, referencing a field in the Relation as retrieved byload(). If
you want to retrieve an originally differently name field, then capture its source name insource_namesource_name: the field name as in the source – useful forload()to retrieve the Relation,
before fields get renamed – such renaming can be done byblizz.apply.renamesdatatype: the Field's datatype – for Pandas, can be defined as Python built-in, numpy type, or
as a String value ("int"). For PySpark, usepyspark.sql.types.DataTypeor the PySpark datatypessimpleStringrepresentation (e.g. "int", "string", ...).default: allows one to define a literal as a default value on NULLdescription: allows one to provide a string as a description/comment on the Fieldkey: a boolean flag indicating if this field is part of the primary keymock: allows one to define a callable mocking data for this Field – useful for testing of Relation's
with synthetic data.EachRelationdefines the following useful methods on class level:Relation.get_fields(): returns allFieldsas definedRelation.get_field_names(): returns allFieldsnames as definedRelation.get_types(): returns allFieldsdatatypes, as a DictField -> Datatype. Note that
datatypes here are as heterogenous, as you defined them, e.g. they can either be strings or applicable
classes such asintRelation.get_key_fields(): returns allFields, defined as keyRelation.get_key_field_names(): returns all names ofFields, defined as keyRelation.get_defaults(): return a Dictionary mapping fromFieldto its default value, where definedRelation.get_default(<Field>): return the default value for aFieldRelation.mock(): to be overridden by eachRelation– allows one to define a mocked version ofload()to produce synthetic data for testing.Of course it is entirely possibly, to define a subclass ofRelationfor your own purposes, which
deals with common use-cases by own class functions it adds! For instance, you might want to create
aSQLRelationwhich already bundles useful methods or definitions to allow you quickly
retrieving these kind of Relations from a SQL RDBMS.API Reference on blizz.check & blizz.applyblizz.checkoffers the following decorators for checking:blizz.check.fields: validates on the DataFrame returned byload(), that it contains at least all fields
(by name) as defined on the Relationblizz.check.types: validates that all fields that have defined data types on the Relation match with their
data types as in the DataFrame returned byload()blizz.check.keys: validates that the subset of all Fields marked as keys is uniqueblizz.check.func: run the user defined function passed as thefunctionkeyword parameter and check if
it raises an exception. The function's signature should beType[Relation], DataFrame -> DataFrame.Allblizz.checkdecorators accept the argumenton_fail, which can be set to:blizz.check.WARN: raise a warning on failing the checkblizz.check.RAISE: raise an exception on failing the checkblizz.applyoffers the following decorators for transformations:blizz.apply.deduplication: deduplicate the DataFrame returned byload()either by the Fields already marked
as keys of the Relation, or the subset of Field given as argumentkey:List[Field]to the decorator (which
has precedence). Additionally the
decorator supports the argumentssort_on:List[Field]andsort_order(which can be eitherblizz.apply.ASCorblizz.apply.DESC) – this can ensure which non-key records are preferrably kept on duplication. E.g. when you
find a duplicate based on the key-column, then pick among those duplicates the most recently updated row
– here one might use something like:sort_on = [UPDATE_TIMESTAMP], sort_order = blizz.apply.DESC.blizz.apply.defaults: fill defined defaults for NULL values in the DataFrame returned byload(), either
using the defined value defaults from the Relation's Fields (if exist) or as given with as the argumentfill: List[Field]to the decorator (which has precedence).blizz.apply.renames: perform Field renames for columns in the DataFrame as returned byload(), either
using the defined pairs ofField.source_name->Field.name(where exists) or using the rename definitions
as given as argumentcolumns:Dict[str,str]to the decorator, which will be added to the renames to perform.blizz.apply.func: run the user defined function passed as thefunctionkeyword parameter and return
the transformed data. The function's signature should beType[Relation], DataFrame -> DataFrame.Bootstrapping – generate schema definitions from DataFramesAs you've seen above,blizzcan be a powerful tool in your Python data pipelines. However it needs properly
defined Relations, which can be tedious at the start.For this,blizzcan boostrap Relations from existing PySpark or Pandas DataFrames usingblizz.relation_from_dataframe().>>>blizz.relation_from_dataframe(df=departments_df,name="Departments")importblizz.check
fromblizzimportRelation,Field
frompyspark.sqlimportDataFrame
frompyspark.sql.typesimport*


classDepartments(Relation):"""todo: describe relation Departments in this docstring"""DEPARTMENT_ID=Field(name="Department_ID",datatype=StringType)DEPARTMENT_NAME=Field(name="Department_Name",datatype=StringType)DOE=Field(name="DOE",datatype=TimestampType)@classmethod@blizz.check.keys(on_fail=blizz.check.WARN)@blizz.check.types(on_fail=blizz.check.WARN)@blizz.check.fields(on_fail=blizz.check.WARN)defload(cls)->DataFrame:"""todo: describe relation Departments load process in this docstring"""# todo: implement data source loading herepassYou can then simply copy and paste and adjust this as needed.Beyond: blizz Feature LibraryEXPERIMENTAL – only supported with PySparkOn top ofblizz's primitives, the plan is to build a lightweight Feature Library component for
Machine Learning. There is an example of this experimental API included as part ofsrc/tutorial. Conceptually the API allows to express
features programmatically, but in a well structured way using lightweight classes. This makes it
possible then, to decouple the feature generation from the pure feature definition.blizzhas also a CLI – serving as a client to instruct a PySpark process which features to build,
all based on a defined blizz Feature Library.In order to do this, one:defines a Feature Library in code as done withsrc/tutorial/example_feature_librarydefines a feature list definition: this is a YAML file of all FeatureGroups/Features one wants to
have calculated, being also able to specify parameters for parameterizable features and to
specify the output storage format. An example is given heresrc/tutorial/ExampleFeatureList.yamlcalls theblizzCLI with both and the output path:blizz build src/tutorial/ExampleFeatureList.yaml src/tutorial/example_feature_library my_featuresThis brings together the code-based definition of all data sources (byblizzprimitives) and features (byblizzFeatureGroup & Feature) with the instruction in the form of the YAML file what and how (parameterized features)
to calculate.Why is this approach powerful?how to retrieve data sources, check them, and build features on-top is purely captured in code. This means it
isfully supported by IDE type checks, autocompletion and refactoring abilities.due to this code centric means of definition, thewhole end to end pipeline is easily managed and versioned–
as a Python package.expressing ML features in a structured way allows to leverage acommon, powerful Spark based factoryto build
them – whoever wants features, only needs to callblizz build ...with their YAML build instructions and
pointing to a blizz Feature Library codebase (which can be checked out at a version as desired). As internally,blizzpipelines are just nested function calls on top ofpyspark.sql, thisheavily benefits from both
lazy evaluation and execution plan optimizations. This happens completely transparently thanks to Spark.in this way, definitions to calculate features can be centralized and shared purely on the code level. Consumers
of features, e.g. specialized ML models, just need to installblizz, write YAML build instructions and point
to the FeatureLibrary –their model does not even need to be in Python.Beyond: Generate DocumentationThanks to its rich metadata about Relations, Fields and Features, the task of documentation becomes easier
for data practitioners. On top of structured information serving as documentation, Python docstrings will
be automatically used, e.g. on class level underneath aRelation, when describingRelation.load()and
similarly forFeatureGroupandFeature.The philosophy here is, not write documentation twice – e.g. in code and somewhere else – but rather to
make it code driven as well.A nice (experimental) feature is the following, which will create a Sphinx project from ablizz.FeatureGroup's
metadata and serve it:blizzdocssrc/tutorial/example_feature_library--serve