Package: json2parquet
This library wrapspyarrowto provide some tools to easily convert
JSON data into Parquet format. It is mostly in Python. It iterates over
files. It copies the data several times in memory. It is not meant to be
the fastest thing available. However, it is convenient for smaller data
sets, or people who don’t have a huge issue with speed.InstallationWith pip:pipinstalljson2parquetWith conda:condainstall-cconda-forgejson2parquetUsageHere’s how to load a random JSON dataset.fromjson2parquetimportconvert_json# Infer Schema (requires reading dataset for column names)convert_json(input_filename,output_filename)# Given columnsconvert_json(input_filename,output_filename,["my_column","my_int"])# Given columns and custom field namesfield_aliases={'my_column':'my_updated_column_name',"my_int":"my_integer"}convert_json(input_filename,output_filename,["my_column","my_int"],field_aliases=field_aliases)# Given PyArrow schemaimportpyarrowaspaschema=pa.schema([pa.field('my_column',pa.string),pa.field('my_int',pa.int64),])convert_json(input_filename,output_filename,schema)You can also work with Python data structures directlyfromjson2parquetimportload_json,ingest_data,write_parquet,write_parquet_dataset# Loading JSON to a PyArrow RecordBatch (schema is optional as above)load_json(input_filename,schema)# Working with a list of dictionariesingest_data(input_data,schema)# Working with a list of dictionaries and custom field namesfield_aliases={'my_column':'my_updated_column_name',"my_int":"my_integer"}ingest_data(input_data,schema,field_aliases)# Writing Parquet Files from PyArrow Record Batcheswrite_parquet(data,destination)# You can also pass any keyword arguments that PyArrow acceptswrite_parquet(data,destination,compression='snappy')# You can also write partitioned datewrite_parquet_dataset(data,destination_dir,partition_cols=["foo","bar","baz"])If you know your schema, you can specify custom datetime formats (only one for now).  This formatting will be ignored if you don’t pass a PyArrow schema.fromjson2parquetimportconvert_json# Given PyArrow schemaimportpyarrowaspaschema=pa.schema([pa.field('my_column',pa.string),pa.field('my_int',pa.int64),])date_format="%Y-%m-%dT%H:%M:%S.%fZ"convert_json(input_filename,output_filename,schema,date_format=date_format)Althoughjson2parquetcan infer schemas, it has helpers to pull in external ones as wellfromjson2parquetimportload_jsonfromjson2parquet.helpersimportget_schema_from_redshift# Fetch the schema from Redshift (requires psycopg2)schema=get_schema_from_redshift(redshift_schema,redshift_table,redshift_uri)# Load JSON with the Redshift schemaload_json(input_filename,schema)Operational NotesIf you are using this library to convert JSON data to be read bySpark,Athena,SpectrumorPrestomake sure you useuse_deprecated_int96_timestampswhen writing your Parquet files, otherwise you will see some really screwy dates.ContributingCode ChangesClone a fork of the libraryRunmake setupRunmake testApply your changes (don’t bump version)Add tests if neededRunmake testto ensure nothing brokeSubmit PRDocumentation ChangesIt is always a struggle to keep documentation correct and up to date.  Any fixes are welcome.  If you don’t want to clone the repo to work locally, please feel free to edit using Github and to submit Pull Requests via Github’s built in features.