Package: json-tools2
IndexInfer SchemaSchema evolvementJSON Transformjson_remove_fieldjson_traverse_transformjson_convert_to_arrayInfer SchemaWith functioninfer_schema, you can get schema from json object. The output complys toJSON Schema Specification. Here is an example:#!/usr/bin/env python# -*- coding: UTF-8 -*-fromjson_tools2importinfer_schemaimportjsondefmain():schema=infer_schema({"foo":1})print(json.dumps(schema,indent=4))if__name__=='__main__':main()Output{"type":"object","properties":{"foo":{"type":"integer"}}}When you have an array of element from different types, or object with the same property but from different type, we need to merge schema.case 1:2.1and2are of different types and we need to merge schema[2.1,2]case 2:2.1and2are of different types and we need to merge schema[{"foo":2.1},{"foo":2}]Following rules applies when we do schema merging:Fornull, any type can merged to it and return anullable type, for example,stingmerge tonulland return anullable stringForstring, onlynullcan merged tostringand returnnullable string.Fornumber, followings are the only valid type that can merge to itintegercan merged to it without changing itnullcan merge to it and becomenullable numberForinteger, followings are the only valid type that can merge to itnumbercan merged to it and upgrade it tonumbernullcan merge to it and becomenullable numberForboolean, followings are the only valid type that can merge to itnullcan merge to it and becomenullable booleanForobject, followings are the only valid type that can merge to itnullcan merge to it and becomenullable objectForarray, followings are the only valid type that can merge to itnullcan merge to it and becomenullable arrayWe useanyOfto representnullable type, for example:
This represent anullable integertype.{"anyOf":[{"type":"integer"},{"type":"null"}]}For any schema inferred, you should be able to validate the payload with the schema inferred, here is an example:#!/usr/bin/env python# -*- coding: UTF-8 -*-# You need to do "pip install jsonschema"fromjson_tools2importinfer_schemafromjsonschemaimportvalidateimportjsondefmain():payload={"foo":1}schema=infer_schema(payload)validate(instance=payload,schema=schema)# it should not throw exceptionif__name__=='__main__':main()Schema evolvementThink about such case:You receive json object every day.Every day, you want to load schema inferred from the json object from the prior day, and merge it with the schema inferred from today's object, and then save the inferred schema.Here is an example:#!/usr/bin/env python# -*- coding: UTF-8 -*-fromjson_tools2importget_schema,ValueSchemaimportjsondefmain():schema=get_schema(1)print(json.dumps(schema.to_json(),indent=4))saved_schema=schema.dump_json()# You got a json object saved_schema which you can save the inferred schema# load schemaschema=ValueSchema.load_json(saved_schema)schema.merge_from(get_schema(None))print(json.dumps(schema.to_json(),indent=4))if__name__=='__main__':main()Output{
    "type": "integer"
}
{
    "anyOf": [
        {
            "type": "integer"
        },
        {
            "type": "null"
        }
    ]
}JSON Transformjson_remove_fieldjson_remove_field(obj:Any,field_name:str)->Any:It removes field specified infield_namerecursively.Example:json_remove_field({"foo":1,"bar":{"foo":1,"t":2}},"foo")it will remove field "foo" recursively, so it will return{"bar":{"t":2}}json_traverse_transformjson_traverse_transform(obj: Any, path:str, handler:Callable[[Any], None]) -> Any:It traverse the input json objectobj, for each object that match the path pattern specified inpath, it calls the functionhandler. Eventuallyobjis retuened however, it might be mutated byhandler.Path pattern:Given object, a path represent a list of object extracted from object.

For example, given object X, here are example pathes:

"foo"        represent X.foo
"foo.bar"    represent X.foo.bar
"foo[]"      represent [X.foo[0], X.foo[1], ...]
"foo[].x"    represent [X.foo[0].x, X.foo[1].x, ...]
"foo[].x[]"  represent [X.foo[0].x[0], X.foo[0].x[1], ..., X.foo[1].x[0], X.foo[1].x[1], ...]Example:defhandler(o):iftype(o)!=dict:returnox=o.get("x",0)y=o.get("y",0)iftype(x)in(int,float)andtype(y)in(int,float):o["sum"]=x+yjson_traverse_transform({"cords":[{"x":1,"y":2},{"x":2,"y":3},]},"cords[]",handler)returns{"cords":[{"x":1,"y":2,"sum":3},{"x":2,"y":3,"sum":5},]}json_convert_to_arrayjson_convert_to_array(obj:Any,field_name:str,new_field_name:str,path="")->Any:For each matching child object based on pattern path, it convert it's fieldfield_namefrom dictionary intolist, result is stored in fieldnew_field_name, the original field is either deleted or overritten.Example:json_convert_to_array([{"q1":{"score":{"math":1,"physics":2}}},{"q1":{"score":{"math":2,"physics":3}}}],"score","score2","[].q1")returns[{"q1":{"score2":[{"key":"math","value":1},{"key":"physics","value":2}]}},{"q1":{"score2":[{"key":"math","value":2},{"key":"physics","value":3}]}},]