Package: marshmallow-pyspark
marshmallow-pysparkMarshmallowis a popular package used for data serialization and validation.
One defines data schemas in marshmallow containing rules on how input data should be marshalled. Similar to marshmallow,pysparkalso comes with its own schema definitions used to
process data frames. This package enables users to utilize marshmallow schemas and its powerful data validation capabilities
in pyspark applications. Such capabilities can be utilized in data-pipeline ETL jobs where data consistency and quality
is of importance.InstallThe package can be install usingpip:$pipinstallmarshmallow-pysparkUsageData schemas can can define the same way as you would using marshmallow. A quick example is shown below:frommarshmallow_pysparkimportSchemafrommarshmallowimportfields# Create data schema.classAlbumSchema(Schema):title=fields.Str()release_date=fields.Date()# Input data frame to validate.df=spark.createDataFrame([{"title":"valid_1","release_date":"2020-1-10"},{"title":"valid_2","release_date":"2020-1-11"},{"title":"invalid_1","release_date":"2020-31-11"},{"title":"invalid_2","release_date":"2020-1-51"},])# Get data frames with valid rows and error prone rows# from input data frame by validating using the schema.valid_df,errors_df=AlbumSchema().validate_df(df)# Output of valid data framevalid_df.show()#    +-------+------------+#    |  title|release_date|#    +-------+------------+#    |valid_1|  2020-01-10|#    |valid_2|  2020-01-11|#    +-------+------------+# Output of errors data frameerrors_df.show()#    +--------------------+#    |             _errors|#    +--------------------+#    |{"row": {"release...|#    |{"row": {"release...|#    +--------------------+More OptionsOn top of marshmallow supported options, theSchemaclass comes with two additional initialization arguments:error_column_name: name of the column to store validation errors. Default value is_errors.split_errors: split rows with validation errors as a separate data frame from valid rows. When set toFalsethe
rows with errors are returned together with valid rows as a single data frame. The field values of all error rows are
set tonull. For user convenience the original field values can be found in therowattribute of the error JSON.
Default value isTrue.An example is shown below:frommarshmallowimportEXCLUDEschema=AlbumSchema(error_column_name="custom_errors",# Use 'custom_errors' as name for errors columnsplit_errors=False,# Don't split the input data frame into valid and errorsunkown=EXCLUDE# Marshmallow option to exclude fields not present in schema)# Input data frame to validate.df=spark.createDataFrame([{"title":"valid_1","release_date":"2020-1-10","garbage":"wdacfa"},{"title":"valid_2","release_date":"2020-1-11","garbage":"5wacfa"},{"title":"invalid_1","release_date":"2020-31-11","garbage":"3aqf"},{"title":"invalid_2","release_date":"2020-1-51","garbage":"vda"},])valid_df,errors_df=schema.validate_df(df)# Output of valid data frame. Contains rows with errors as# the option 'split_errors' was set to False.valid_df.show()#    +-------+------------+--------------------+#    |  title|release_date|             _errors|#    +-------+------------+--------------------+#    |valid_1|  2020-01-10|                    |#    |valid_2|  2020-01-11|                    |#    |       |            |{"row": {"release...|#    |       |            |{"row": {"release...|#    +-------+------------+--------------------+# The errors data frame will be set to Noneasserterrors_dfisNone# TrueLastly, on top of passing marshmallow specific options in the schema, you can also pass them in thevalidate_dfmethod.
These are options are passed to the marshmallow'sloadmethod:schema=AlbumSchema(error_column_name="custom_errors",# Use 'custom_errors' as name for errors columnsplit_errors=False,# Don't split the input data frame into valid and errors)valid_df,errors_df=schema.validate_df(df,unkown=EXCLUDE)DuplicatesMarshmallow-pyspark comes with the ability to validate one or more schema fields for duplicate values. This is achieved
by adding the field names to theUNIQUEattribute of the schema as shown:classAlbumSchema(Schema):# Unique valued field "title" in the schemaUNIQUE=["title"]title=fields.Str()release_date=fields.Date()# Input data frame to validate.df=spark.createDataFrame([{"title":"title_1","release_date":"2020-1-10"},{"title":"title_2","release_date":"2020-1-11"},{"title":"title_2","release_date":"2020-3-11"},# duplicate title{"title":"title_3","release_date":"2020-1-51"},])# Validate data framevalid_df,errors_df=AlbumSchema().validate_df(df)# List of valid rowsvalid_rows=[row.asDict(recursive=True)forrowinvalid_df.collect()]##   [#        {'title': 'title_1', 'release_date': datetime.date(2020, 1, 10)},#        {'title': 'title_2', 'release_date': datetime.date(2020, 1, 11)}#   ]## Rows with errorserror_rows=[row.asDict(recursive=True)forrowinerrors_df.collect()]##   [#        {'_errors': '{"row": {"release_date": "2020-3-11", "title": "title_2", "__count__title": 2}, '#                    '"errors": ["duplicate row"]}'},#        {'_errors': '{"row": {"release_date": "2020-1-51", "title": "title_3", "__count__title": 1}, '#                    '"errors": {"release_date": ["Not a valid date."]}}'}#    ]#The technique to drop duplicates but keep first is discussed in thislink.
In case there are multiple unique fields in the schema just add them to theUNIQUE, e.g.UNIQUE=["title", "release_date"].
You can even specify uniqueness for combination of fields by grouping them in a list:classAlbumSchema(Schema):# Combined values of "title" and "release_date" should be uniqueUNIQUE=[["title","release_date"]]title=fields.Str()release_date=fields.Date()# Input data frame to validate.df=spark.createDataFrame([{"title":"title_1","release_date":"2020-1-10"},{"title":"title_2","release_date":"2020-1-11"},{"title":"title_2","release_date":"2020-3-11"},{"title":"title_3","release_date":"2020-1-21"},{"title":"title_3","release_date":"2020-1-21"},{"title":"title_4","release_date":"2020-1-51"},])# Validate data framevalid_df,errors_df=AlbumSchema().validate_df(df)# List of valid rowsvalid_rows=[row.asDict(recursive=True)forrowinvalid_df.collect()]##   [#        {'title': 'title_1', 'release_date': datetime.date(2020, 1, 10)},#        {'title': 'title_2', 'release_date': datetime.date(2020, 1, 11)},#        {'title': 'title_3', 'release_date': datetime.date(2020, 1, 21)}#   ]## Rows with errorserror_rows=[row.asDict(recursive=True)forrowinerrors_df.collect()]##   [#        {'_errors': '{"row": {"release_date": "2020-1-21", "title": "title_3", '#                    '"__count__title": 2, "__count__release_date": 2}, '#                    '"errors": ["duplicate row"]}'},#        {'_errors': '{"row": {"release_date": "2020-1-51", "title": "title_4", '#                    '"__count__title": 1, "__count__release_date": 1}, '#                    '"errors": {"release_date": ["Not a valid date."]}}'},#        {'_errors': '{"row": {"release_date": "2020-3-11", "title": "title_2", '#                    '"__count__title": 2, "__count__release_date": 1}, '#                    '"errors": ["duplicate row"]}'}#    ]#WARNING: Duplicate check requires data shuffle per unique field. Having large number of unique fields will effect
spark job performance. By defaultUNIQUEis set to an empty list preventing any duplicate checks.FieldsMarshmallow comes with a variety of different fields that can be used to define schemas. Internally marshmallow-pyspark
convert these fields into pyspark SQL data types. The following table lists the supported marshmallow fields and their
equivalent spark SQL data types:MarshmallowPySparkRawuser specifiedStringStringTypeDateTimeTimestampTypeDateDateTypeBooleanBooleanTypeIntegerIntegerTypeFloatFloatTypeNumberDoubleTypeListArrayTypeDictMapTypeNestedStructTypeBy default theStringTypedata type is used for marshmallow fields not in the above table. Thespark_schemaproperty
of your defined schema can be used to check the converted spark SQL schema:# Gets the spark schema for the Album schemaAlbumSchema().spark_schema# StructType(List(StructField(title,StringType,true),StructField(release_date,DateType,true),StructField(_errors,StringType,true)))Custom FieldsMarshmallow_pyspark comes with support for an additionalRawfield. TheRawfield does not perform any formatting
and requires the user to specify the spark data type associated with the field. See the following example:frommarshmallow_pysparkimportSchemafrommarshmallow_pyspark.fieldsimportRawfrommarshmallowimportfieldsfrompyspark.sql.typesimportDateTypefromdatetimeimportdateclassAlbumSchema(Schema):title=fields.Str()# Takes python datetime.date objects and treats them as pyspark DateTyperelease_date=Raw(spark_type=DateType())# Input data frame to validate.df=spark.createDataFrame([{"title":"title_1","release_date":date(2020,1,10)},{"title":"title_2","release_date":date(2020,1,11)},{"title":"title_3","release_date":date(2020,3,10)},])# Validate data framevalid_df,errors_df=AlbumSchema().validate_df(df)# List of valid rowsvalid_rows=[row.asDict(recursive=True)forrowinvalid_df.collect()]##   [#        {'title': 'title_1', 'release_date': datetime.date(2020, 1, 10)},#        {'title': 'title_2', 'release_date': datetime.date(2020, 1, 11)},#        {'title': 'title_3', 'release_date': datetime.date(2020, 3, 10)}#   ]## Rows with errorserror_rows=[row.asDict(recursive=True)forrowinerrors_df.collect()]##   []#It is also possible to add support for custom marshmallow fields, or those missing in the above table. In order to do so,
you would need to create a converter for the custom field. The converter can be built using theConverterABCinterface:frommarshmallow_pysparkimportConverterABCfrompyspark.sql.typesimportStringTypeclassEmailConverter(ConverterABC):"""Converter to convert marshmallow's Email field to a pysparkSQL data type."""defconvert(self,ma_field):returnStringType()Thema_fieldargument in theconvertmethod is provided to handle nested fields. For an example you can checkoutNestedConverter. Now the final step would be to add the converter to theCONVERTER_MAPattribute of your schema:frommarshmallow_pysparkimportSchemafrommarshmallowimportfieldsclassUser(Schema):name=fields.String(required=True)email=fields.Email(required=True)# Adding email converter to schema.User.CONVERTER_MAP[fields.Email]=EmailConverter# You can now use your schema to validate the input data frame.valid_df,errors_df=User().validate_df(input_df)DevelopmentTo hack marshmallow-pyspark locally run:$pipinstall-e.[dev]# to install all dependencies$pytest--cov-config.coveragerc--cov=./# to get coverage report$pylintmarshmallow_pyspark# to check code quality with PyLintOptionally you can usemaketo perform development tasks.LicenseThe source code is licensed under Apache License Version 2.ContributionsPull requests always welcomed! :)