Package: json-stream
json-streamSimple streaming JSON parser and encoder.WhenreadingJSON data,json-streamcan decode JSON data in
a streaming manner, providing a pythonic dict/list-like interface, or avisitor-based interfeace. Can stream from files,URLsoriterators.WhenwritingJSON data,json-streamcan stream JSON objects
as you generate them.These techniques allow you toreduce memory consumption and
latency.Readingjson-streamis a JSON parser just like the standard library'sjson.load(). It
will read a JSON document and convert it into native python types.importjson_streamdata=json_stream.load(f)Features:stream all JSON data types (objects, lists and simple types)stream nested datasimple pythoniclist-like/dict-like interfacestream truncated or malformed JSON data (up to the first error)native code parsing speedupsfor most common platformspure python fallback if native extensions not availableUnlikejson.load(),json-streamcanstreamJSON data from any file-like oriterableobject. This has the following benefits:it does not require the whole json document to be read into memory up-frontit can start producing data before the entire document has finished loadingit only requires enough memory to hold the data currently being parsedThere are specific integrations for streaming JSON data fromURLsusing therequests,httpx, orurllib.The objects thatjson-streamproduces can bere-outputusingjson.dump()with a little work.Usagejson_stream.load()json_stream.load()has two modes of operation, controlled by
thepersistentargument (default false).It is also possible to "mix" the modes as you consume the data.Transient mode (default)This mode is appropriate if you can consume the data iteratively. You cannot
move backwards through the stream to read data that has already been skipped
over. It is the mode youmustuse if you want to process large amounts of
JSON data without consuming large amounts of memory.In transient mode, only the data currently being read is stored in memory. Any
data previously read from the stream is discarded (it's up to you what to do
with it) and attempting to access this data results in aTransientAccessException.importjson_stream# JSON: {"count": 3, "results": ["a", "b", "c"]}data=json_stream.load(f)# data is a transient dict-like object# stream has been read up to "{"# use data like a dictresults=data["results"]# results is a transient list-like object# stream has been read up to "[", we now cannot read "count"# iterate transient listforresultinresults:print(result)# prints a, b, c# stream has been read up to "]"# attempt to read "count" from earlier in streamcount=data["count"]# will raise exception# stream is now exhausted# attempt to read from list that has already been iteratedforresultinresults:# will raise exceptionpassPersistent modeIn persistent mode all previously read data is stored in memory as
it is parsed. The returneddict-like orlist-like objects
can be used just like normal data structures.If you request an index or key that has already been read from the stream
then it is retrieved from memory. If you request an index or key that has
not yet been read from the stream, then the request blocks until that item
is found in the stream.importjson_stream# JSON: {"count": 1, "results": ["a", "b", "c"]}data=json_stream.load(f,persistent=True)# data is a streaming  dict-like object# stream has been read up to "{"# use data like a dictresults=data["results"]# results is a streaming list-like object# stream has been read up to "["# count has been stored data# use results like a lista_result=results[1]# a_result = "b"# stream has been read up to the middle of list# "a" and "b" have been stored in results# read earlier data from memorycount=data["count"]# count = 1# consume rest of listresults.read_all()# stream has been read up to "}"# "c" is now stored in results too# results.is_streaming() == False# consume everythingdata.read_all()# stream is now exhausted# data.is_streaming() == FalsePersistent mode is not appropriate if you care about memory consumption, but
provides an identical experience compared tojson.load().Mixed modeIn some cases you will need to be able to randomly access some part of the
data, but still only have that specific data taking up memory resources.For example, you might have a very long list of objects, but you cannot always
access the keys of the objects in stream order. You want to be able to iterate
the list transiently, but access the result objects persistently.This can be achieved using thepersistent()method of all thelistordict-like objects json_stream produces. Callingpersistent()causes the existing
transient object to produce persistent child objects.Note that thepersistent()method makes the children of the object it
is called on persistent, not the object it is called on.importjson_stream# JSON: {"results": [{"x": 1, "y": 3}, {"y": 4, "x": 2}]}# note that the keys of the inner objects are not ordereddata=json_stream.load(f)# data is a transient dict-like object# iterate transient list, but produce persistent itemsforresultindata['results'].persistent():# result is a persistent dict-like objectprint(result['x'])# print xprint(result['y'])# print y (error on second result without .persistent())print(result['x'])# print x again (error without .persistent())The opposite is also possible, going from persistent mode to transient mode, though
the use cases for this are more esoteric.# JSON: {"a": 1, "x": ["long", "list", "I", "don't", "want", "in", "memory"], "b": 2}data=load(StringIO(json),persistent=True).transient()# data is a persistent dict-list object that produces transient childrenprint(data["a"])# prints 1x=data["x"]# x is a transient list, you can use it accordinglyprint(x[0])# prints long# access earlier data from memoryprint(data["a"])# this would have raised an exception if data was transientprint(data["b"])# prints 2# we have now moved past all the data in the transient listprint(x[0])# will raise exceptionvisitor patternYou can also parse using a visitor-style approach where a function you supply
is called for each data item as it is parsed (depth-first).This uses a transient parser under the hood, so does not consume memory for
the whole document.importjson_stream# JSON: {"x": 1, "y": {}, "xxxx": [1,2, {"yyyy": 1}, "z", 1, []]}defvisitor(item,path):print(f"{item}at path{path}")json_stream.visit(f,visitor)Output:1 at path ('x',)
{} at path ('y',)
1 at path ('xxxx', 0)
2 at path ('xxxx', 1)
1 at path ('xxxx', 2, 'yyyy')
z at path ('xxxx', 3)
1 at path ('xxxx', 4)
[] at path ('xxxx', 5)Stream a URLjson_streamknows how to stream directly from a URL using a variety of packages.
Supported packages include:Python's batteries-includedurllibpackageThe popularrequestslibraryThe newerhttpxlibraryurlliburllib's response objects are already
file-like objects, so we can just pass them directly tojson-stream.importurllib.requestimportjson_streamwithurllib.request.urlopen('http://example.com/data.json')asresponse:data=json_stream.load(response)requestsTo stream JSON data fromrequests, you must
passstream=Truewhen making a request, and calljson_stream.requests.load()passing the response.importrequestsimportjson_stream.requestswithrequests.get('http://example.com/data.json',stream=True)asresponse:data=json_stream.requests.load(response)Note: these functions useresponse.iter_content()under the
hood with achunk_sizeof 10k bytes. This default allows us to perform effective reads from the response stream and
lower CPU usage. The drawback to this is thatrequestswill buffer each read until up to 10k bytes have been read
before passing the data back tojson_stream. If you need to consume data more responsively the only option is to tunechunk_sizeback to 1 to disable buffering.httpxTo stream JSON data fromhttpx, you must callstream()when
making your request, and calljson_stream.httpx.load()passing the response.importhttpximportjson_stream.httpxwithhttpx.Client()asclient,client.stream('GET','http://example.com/data.json')asresponse:data=json_stream.httpx.load(response)Under the hood, this works similarly to therequestsversion above, including
the caveat aboutchunk_size.Stream a URL (with visitor)The visitor pattern also works with URL streams.urllibimporturllib.requestimportjson_streamdefvisitor(item,path):print(f"{item}at path{path}")withurllib.request.urlopen('http://example.com/data.json')asresponse:json_stream.visit(response,visitor)requestsimportrequestsimportjson_stream.requestsdefvisitor(item,path):print(f"{item}at path{path}")withrequests.get('http://example.com/data.json',stream=True)asresponse:json_stream.requests.visit(response,visitor)Thechunk_sizenote also applies tovisit().httpximporthttpximportjson_stream.httpxdefvisitor(item,path):print(f"{item}at path{path}")withhttpx.Client()asclient,client.stream('GET','http://example.com/data.json')asresponse:json_stream.httpx.visit(response,visitor)Stream an iterablejson-stream's parsing functions can take any iterable object that produces encoded JSON asbyteobjects.importjson_streamdefsome_iterator():yieldb'{"some":'yieldb' "JSON"}'data=json_stream.load(some_iterator())assertdata['some']=="JSON"This is actually how therequestsandhttpxextensions work, as
both libraries provide methods to iterate over the response content.Encoding json-stream objectsYou can re-output (encode)persistentjson-streamdict-like andlist-like object back to JSON using the built-injson.dump()orjson.dumps()functions, but with a little additional work:importjsonimportjson_streamfromjson_stream.dumpimportJSONStreamEncoder,defaultdata=json_stream.load(f,persistent=True)# Option 1: supply json_stream.encoding.default as the default argumentprint(json.dumps(data,default=default))# Option 2: supply json_stream.encoding.JSONStreamEncoder as the cls argument# This allows you to create your own subclass to further customise encodingprint(json.dumps(data,cls=JSONStreamEncoder))If you are using a library that internally takes data you pass it and encodes
it usingjson.dump(). You can also use JSONStreamEncoder() as a context manager.It works by monkey-patching the built-inJSONEncoder.defaultmethod during the
scope of thewithstatement.# library codedefsome_library_function_out_of_your_control(arg):json.dumps(arg)# your codewithJSONStreamEncoder():some_library_function_out_of_your_control(data)Converting to standard Python typesTo convert a json-streamdict-like orlist-like object and all its
descendants to a standardlistanddict, you can use thejson_stream.to_standard_typesutility:# JSON: {"round": 1, "results": [1, 2, 3]}data=json_stream.load(f)results=data["results"]print(results)# prints <TransientStreamingJSONList: TRANSIENT, STREAMING>converted=json_stream.to_standard_types(results)print(converted)# prints [1, 2, 3]Thread safety (experimental)There is also a thread-safe version of thejson.dumpcontext manager:fromjson_stream.dump.threadingimportThreadSafeJSONStreamEncoder# your codewithThreadSafeJSONStreamEncoder():some_library_function_out_of_your_control(data)The thread-safe implementation will ensure that concurrent uses of the
context manager will only apply the patch for the first thread entering
the patched section(s) and will only remove the patch when the last
thread exits the patched sections(s)Additionally, if the patch is somehow called by a thread that isnotcurrently in a patched section (i.e. some other thread callingjson.dump) then that thread will block until the patch has been
removed. While such an un-patched thread is active, any thread attempting
to apply the patch is blocked.Rust tokenizer speedupsBy defaultjson-streamuses thejson-stream-rs-tokenizernative extension.This is a 3rd party Rust-based tokenizer implementations that provides
significant parsing speedup compared to pure python implementation.json-streamwill fallback to its pure python tokenizer implementation
ifjson-stream-rs-tokenizeris not available.Custom tokenizerYou can supply an alternative JSON tokenizer implementation. Simply pass
a tokenizer to theload()orvisit()methods.json_stream.load(f,tokenizer=some_tokenizer)The requests methods also accept a customer tokenizer parameter.WritingThe standard library'sjson.dump()function can only accept standard
python types such asdict,list,str.json-streamallows you to write streaming JSON output based on python
generators instead.For actually encoding and writing to the stream,json-streamstill uses the standard library'sjson.dump()function, but provides
wrappers that adapt python generators intodict/listsubclasses
thatjson.dump()can use.The means that you do not have to generate all of your data upfront
before callingjson.dump().UsageTo usejson-streamto generate JSON data iteratively, you must first
write python generators (or use any other iterable).To output JSON objects, the iterable must yield key/value pairs.To output JSON lists, the iterable must yield individual items.The values yielded can be either be standard python types or another otherStreamableobject, allowing lists and object to be arbitrarily nested.streamable_list/streamable_dictcan be used to wrap an existing
iterable:importsysimportjsonfromjson_streamimportstreamable_list# wrap existing iterabledata=streamable_list(range(10))# consume iterable with standard json.dump()json.dump(data,sys.stdout)Or they can be used as decorators on generator functions:importjsonimportsysfromjson_streamimportstreamable_dict# declare a new streamable dict generator function@streamable_dictdefgenerate_dict_of_squares(n):foriinrange(n):# this could be some memory intensive operation# or just a really large value of nyieldi,i**2# data is will already be Streamable because# of the decoratordata=generate_dict_of_squares(10)json.dump(data,sys.stdout)ExampleThe following example generates a JSON object with a nested JSON list.
It usestime.sleep()to slow down the generation and show that the
output is indeed written as the data is created.importsysimportjsonimporttimefromjson_stream.writerimportstreamable_dict,streamable_list# define a list data generator that (slowly) yields# items that will be written as a JSON list@streamable_listdefgenerate_list(n):# output n numbers and their squaresforiinrange(n):# range is itself a generatoryielditime.sleep(1)# define a dictionary data generator that (slowly) yields# key/value pairs that will be written as a JSON dict@streamable_dictdefgenerate_dict(n):# output n numbers and their squaresforiinrange(n):# range is itself a generatoryieldi,i**2time.sleep(1)# yield another dictionary item key, with the value# being a streamed nested listyield"a list",generate_list(n)# get a streamable generatordata=generate_dict(5)# use json.dump() to write dict generator to stdoutjson.dump(data,sys.stdout,indent=2)# if you already have an iterable object, you can just# call streamable_* on it to make it writabledata=streamable_list(range(10))json.dump(data,sys.stdout)Output:{"0":0,"1":1,"2":4,"3":9,"4":16,"a list":[0,1,2,3,4]}What are the problems with the standardjsonpackage?Reading withjson.load()The problem with thejson.load()stem from the fact that it must read
the whole JSON document into memory before parsing it.Memory usagejson.load()first reads the whole document into memory as a string. It
then starts parsing that string and converting the whole document into python
types again stored in memory. For a very large document, this could be more
memory than you have available to your system.json_stream.load()does not read the whole document into memory, it only
buffers enough from the stream to produce the next item of data.Additionally, in the default transient mode (see below)json-streamdoesn't store
up all of the parsed data in memory.Latencyjson.load()produces all the data after parsing the whole document. If you
only care about the first 10 items in a list of 2 million items, then you
have wait until all 2 million items have been parsed first.json_stream.load()produces data as soon as it is available in the stream.WritingMemory usageWhilejson.dump()does iteratively write JSON data to the given
file-like object, you must first produce the entire document to be
written as standard python types (dict,list, etc). For a very
large document, this could be more memory than you have available
to your system.json-streamallows you iteratively generate your data one item at
a time, and thus consumes only the memory required to generate that
one item.Latencyjson.dump()can only start writing to the output file once all the
data has been generated up front at standard python types.The iterative generation of JSON items provided byjson-streamallows the data to be written as it is produced.Future improvementsAllow long strings in the JSON to be read as streams themselvesAllow transient mode on seekable streams to seek to data earlier in
the stream instead of raising aTransientAccessExceptionA more efficient tokenizer?AlternativesNAYANAYAis a pure python JSON parser for
parsing a simple JSON list as a stream.Why not NAYA?It can only stream JSON containing a top-level listIt does not provide a pythonicdict/list-like interfaceYajl-PyYajl-Pyis a wrapper around the C YAJL JSON library that can be used to
generate SAX style events while parsing JSON.Why not Yajl-Py?No pure python implementationIt does not provide a pythonicdict/list-like interfacejsonslicerjsonsliceris another wrapper around the YAJL C library with a
path lookup based interface.Why not jsonslicer?No pure python implementationIt does not provide a pythonicdict/list-like interfaceMust know all data paths lookup in advance (or make multiple passes)ContributingSee the projectcontribution guide.DonationsORAcknowledgementsThe JSON tokenizer used in the project was taken from theNAYAproject.