Package: json2spark-mapper
json-data-type-to-spark-sql-type-mapperA python mapper that converts JSON data types to a Spark SQL typeIntroductionSpark has built in support for convertingAvro data typesinto Spark SQL types, but lacks similar functionality with regard to JSON data types. Instead Spark has logic that caninfer typesfrom sample JSON documents. At first glance this might appear sufficient, however at closer inspection some disadvantages surface. For instance:It is impossible to define a StructField as optional. Every subsequent JSON document that is processed needs to supply all the fields that were present in the initial JSON document.Numeric values, both JSON integer and number, should be converted to the largest Spark type because ranges are unknown. This could lead additional storage requirements, although minimal in modern systems. When using a JSON schema that specifies ranges the right Spark type can be selected.JSON arrays can be a pain. In most positive scenario they act as alistcontaining a single type, but they could also be used to definetuplestructures with mandatory types and additional elements of any type.This package provides a mapping function that can be used similar to how avro schemas are used whilst keeping all relevant details to create a StructType with optimal StructFields. See thesupported types.Note:Looking for feedback from users that take this for a spin to work with real lifeHow to useInstall packageFirst make sure you install the module into your environment. There are various options assuming you have a Python 3.10 (or higher) environment set up:Install from PyPIpipinstalljson2spark-mapperInstall from TestPyPIpipinstall--index-urlhttps://test.pypi.org/simple/--extra-index-urlhttps://pypi.org/simple/json2spark-mapperNote: because the required package pyspark is not available in TestPyPI theextra-index-urlis needed.From sourcecheckout the projectnavigate to the root directorysimply issuepip install .gitclonehttps://github.com/vdweij/json-data-type-to-spark-sql-type-mapper.gitcdjson-data-type-to-spark-sql-type-mapper
pipinstall.Import moduleIn order to make the mapper functionfrom_json_to_sparkavailable in your Python file, use the following import statement:fromjson2spark_mapper.schema_mapperimportfrom_json_to_sparkCall mapping functionSimplest formwithopen("path-to-your-schema.json")asschema_file:json_schema=json.load(schema_file)struct_type=from_json_to_spark(json_schema)Setting default json draft versionPer default the most recent json draft version is used in case a schema lacks a@schemadefinition. The function usesJSON_DRAFTS.get_latest() from module json_schema_drafts.draftsto specify the default. This can be overridden by explictily setting
it to a desired draft version, like so:json_schema=json.load(schema_file,default_json_draft=JSON_DRAFTS.draft_2019_09)Don't forget to importJSON_DRAFT!fromjson2spark_mapper.schema_mapperimportJSON_DRAFTSForcing a json draft versionThe default behaviour is to use the draft version specified in the schema or when it is abcent to use thedefault_json_draftversion.
This can be overridden by specifying theforce_json_draftproperty, like so:json_schema=json.load(schema_file,force_json_draft=JSON_DRAFTS.draft_2019_09)The version specified viaforce_json_drafttakes precedence regardless of what is specified in the schema or by the defaultdefault_json_draft.Don't forget to importJSON_DRAFT!fromjson2spark_mapper.schema_mapperimportJSON_DRAFTSDefining own ResolversIt is possible to create your own readers for various Json types. This can be done by creating an implementation of either the abstract RootTypeResolveror via one of its abstract type resolvers.This resolver should then be added to the appropiateResolverRegistrythat needs to be created for each Json draft version that is needed.This registry could then in turn be added to theResolverAwareReaderand needs to be assigned toREADERofschame_mapper.TroubleshootingERROR: No matching distribution found for json2spark-mapper==[some-version]This library is packaged with the explict instructionpython_requires=">=3.10"which means pip will not be able to find this
library when tried to be installed from older Python versions.In case ofDatabricksmake sure to use a runtime higher than or equal to 13.2 that comes with Python 3.10.2.TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'Python 3.10 introducedPEP 604 – Allow writing union types as X | Yand this form of type
hints are used throughout the code. Make sure your project used Python 3.10 or higher.In case ofDatabricksmake sure to use a runtime higher than or equal to 13.2 that comes with Python 3.10.2.IssuesPlease check existing issues before creating a new one.DevelopmentFor development, install the[dev]dependencies of the package.
This will also installpre-commit.Install pre-commit so that it automatically runs whenever you create a
new commit, to ensure code quality before pushing.pipinstall.[dev]pre-commitinstallIn order to run unittest locally also install the[test]dependencies.pipinstall.[test]Pre-commitPre-commit is configured to lint and autofix all files with standard.Python code linting is done withblackandruffwith a selection
of ruff plugins. Details are in.pre-commit-config.yaml.Release NotesSee forrelease notes.MoreSee for moredevelopment information.