Package: pysimdjson-schemaful
pysimdjson-schemafulSchema-awarepysimdjsonloader for
efficient parsing of large excessive JSON inputs.When working with external APIs you have zero influence on, you may face the
following unfortunate edge-case (as we did):Particular endpoint responds with a relatively massive JSON-body, say, ≥ 1 MB.The amount of data you really need is several magnitudes smaller, e.g., 1 KB.There is no server-side filtering available.In such a case it may be very excessive in terms of memory, cpu time and delay to
deserialize and, subsequently, validate the whole response, even when using
fast JSON-deseralization libraries, such asorjson.In our particular case we needed less than 0.1% of ~5 MB responses, which we
validated withpydantic.
First, we compared several combinations of deserializers and validators:json+pydantic v1(Model.parse_raw(json.loads(data)))orjson+pydantic v1(Model.parse_raw(orjson.loads(data)))pysimdjson+pydantic v1(Model.parse_raw(simdjson.loads(data)))pydantic v2(Model.model_validate_json(data))To our surprise internalpydantic v2parser appeared to be ~2-3 times slower
thanjson+pydantic v1. The fastest wasorjson+pydantic v1(~2-3 times faster thanjsonand a bit faster than fullsimdjsonparsing).
Such a speed-up, however, still comes with excessive memory spending
(as a complete python dict object is created and populated on deserialization).Thus, we ended up usingpysimdjsonwith its fast lazy parsing and manually
iterated over nested JSON objects/arrays and extracted only required keys. It is
ugly, tedious and hard to maintain of course. However, it showed to be several
times faster thanorjsonand decreased memory consumption.Table of ContentsThe cruxWhen to use?InstallationUsageBasicReusing parserPydantic v1Pydantic v2Benchmarks (TBD)The cruxThis package aims to automate the manual labour of lazy loading with pysimdjson.Simply feed the JSON-schema in and the input data will be traversed
and loaded with pysimdjson accordingly.Supportspydantic>=1,<3python>=3.8,<3.12simdjson>=2,<6(with caveats)Does not support complex schemas (it may be not very reasonable from the
practical standpoint anyway), e.g.,anyOf(Union[Model1, Model2])...In such cases it will fully (not lazily) load the underlying objects.When to use?Input JSON data is large relatively to what is needed in there, i.e.,
selectivity is small.Other deserialization methods appear to be slower and/or more memory
consuming.If you can check all the boxes, then, this package may prove useful to you.Neveruse it as a default deserialization method: run some benchmarks for
your particular case first, otherwise, it may and will disappoint you.Installationpipinstallpysimdjson-schemafulIf you need pydantic supportpipinstall"pysimdjson-schemaful[pydantic]"UsageBasicimportjsonfromsimdjson_schemafulimportloadsschema={"type":"array","items":{"$ref":"#/definitions/Model"},"definitions":{"Model":{"type":"object","properties":{"key":{"type":"integer"},}}}}data=json.dumps([{"key":0,"other":1},{"missing":2},])parsed=loads(data,schema=schema)assertparsed==[{"key":0},{},]Example withadditionalProperties:schema={"type":"object","additionalProperties":{"$ref":"#/definitions/Model",},"definitions":{"Model":{"type":"object","properties":{"key":{"type":"integer"},}}}}data=json.dumps({"some":{"key":0,"other":1},"other":{"missing":2},})parsed=loads(data,schema=schema)assertparsed=={"some":{"key":0},"other":{},}Reusing parserWith re-used simdjson parser(recommended when used in a single thread,
otherwise better consult pysimdjson project on thread-safety):fromsimdjsonimportParserparser=Parser()parsed=loads(data,schema=schema,parser=parser)assertparsed=={"some":{"key":0},"other":{},}Pydantic v1With model (callBaseModel.parse_raw_simdjson):importjsonfromsimdjson_schemaful.pydantic.v1importBaseModelclassModel(BaseModel):key:intdata=json.dumps({"key":0,"other":1})obj=Model.parse_raw_simdjson(data)With type (callparse_raw_as_simdjson):importjsonfromtypingimportListfromsimdjson_schemaful.pydantic.v1importBaseModel,parse_raw_simdjson_asclassModel(BaseModel):key:intType=List[Model]data=json.dumps([{"key":0,"other":1},{"key":1,"another":2},])obj1,obj2=parse_raw_simdjson_as(Type,data)Pydantic v2With model (callBaseModel.model_validate_simdjson):importjsonfromsimdjson_schemaful.pydantic.v2importBaseModelclassModel(BaseModel):key:intdata=json.dumps({"key":0,"other":1})obj=Model.model_validate_simdjson(data)With type adapter (callTypeAdapter.validate_simdjson)importjsonfromtypingimportListfromsimdjson_schemaful.pydantic.v2importBaseModel,TypeAdapterclassModel(BaseModel):key:intadapter=TypeAdapter(List[Model])data=json.dumps([{"key":0,"other":1},{"key":1,"another":2},])obj1,obj2=adapter.validate_simdjson(data)BenchmarksTBD