Package: ijson-bigint
ijsonIjson is an iterative JSON parser with standard Python iterator interfaces.InstallationUsageHigh-level interfacesLower-level interfacesCommand linebytes/strsupportasynciosupportIntercepting eventsPush interfacesOptionsEventsPrefixBackendsPerformance tipsFAQAcknowledgementsInstallationIjson is hosted in PyPI, so you should be able to install it viapip:pip install ijsonBinary wheels are provided
for major platforms
and python versions.
These are built and published automatically
usingcibuildwheelvia Travis CI.UsageAll usage example will be using a JSON document describing geographical
objects:{"earth":{"europe":[{"name":"Paris","type":"city","info":{...}},{"name":"Thames","type":"river","info":{...}},// ...],"america":[{"name":"Texas","type":"state","info":{...}},// ...]}}High-level interfacesMost common usage is having ijson yield native Python objects out of a JSON
stream located under a prefix.
This is done using theitemsfunction.
Here’s how to process all European cities:importijsonf=urlopen('http://.../')objects=ijson.items(f,'earth.europe.item')cities=(oforoinobjectsifo['type']=='city')forcityincities:do_something_with(city)For how to build a prefix see theprefixsection below.Other times it might be useful to iterate over object members
rather than objects themselves (e.g., when objects are too big).
In that case one can use thekvitemsfunction instead:importijsonf=urlopen('http://.../')european_places=ijson.kvitems(f,'earth.europe.item')names=(vfork,vineuropean_placesifk=='name')fornameinnames:do_something_with(name)Lower-level interfacesSometimes when dealing with a particularly large JSON payload it may worth to
not even construct individual Python objects and react on individual events
immediately producing some result.
This is achieved using theparsefunction:importijsonparser=ijson.parse(urlopen('http://.../'))stream.write('<geo>')forprefix,event,valueinparser:if(prefix,event)==('earth','map_key'):stream.write('<%s>'%value)continent=valueelifprefix.endswith('.name'):stream.write('<object name="%s"/>'%value)elif(prefix,event)==('earth.%s'%continent,'end_map'):stream.write('</%s>'%continent)stream.write('</geo>')Even more bare-bones is the ability to react on individual events
without even calculating a prefix
using thebasic_parsefunction:importijsonevents=ijson.basic_parse(urlopen('http://.../'))num_names=sum(1forevent,valueineventsifevent=='map_key'andvalue=='name')Command lineA command line utility is included with ijson
to help visualise the output of each of the routines above.
It reads JSON from the standard input,
and it prints the results of the parsing method chosen by the user
to the standard output.The tool is available by running theijson.dumpmodule.
For example:$> echo '{"A": 0, "B": [1, 2, 3, 4]}' | python -m ijson.dump -m parse
#: path, name, value
--------------------
0: , start_map, None
1: , map_key, A
2: A, number, 0
3: , map_key, B
4: B, start_array, None
5: B.item, number, 1
6: B.item, number, 2
7: B.item, number, 3
8: B.item, number, 4
9: B, end_array, None
10: , end_map, NoneUsing-h/--helpwill show all available options.bytes/strsupportAlthough not usually how they are meant to be run,
all the functions above also acceptbytesandstrobjects (andunicodein python 2.7)
directly as inputs.
These are then internally wrapped into a file object,
and further processed.
This is useful for testing and prototyping,
but probably not extremely useful in real-life scenarios.asynciosupportIn python 3.5+ all of the methods above
work also on file-like asynchronous objects,
so they can be iterated asynchronously.
In other words, something like this:importasyncioimportijsonasyncdefrun():f=awaitasync_urlopen('http://..../')asyncforobjectinijson.items(f,'earth.europe.item'):ifobject['type']=='city':do_something_with(city)asyncio.run(run())An explicit set of*_asyncfunctions also exists
offering the same functionality,
except they will fail if anything other
than a file-like asynchronous object is given to them.
(so the example above can also be written usingijson.items_async).
In fact in ijson version 3.0
this was the only way to access
theasynciosupport.Intercepting eventsThe four routines shown above
internally chain against each other:
tuples generated bybasic_parseare the input forparse,
whose results are the input tokvitemsanditems.Normally users don’t see this interaction,
as they only care about the final output
of the function they invoked,
but there are occasions when tapping
into this invocation chain this could be handy.
This is supported
by passing the output of one function
(i.e., an iterable of events, usually a generator)
as the input of another,
opening the door for user event filtering or injection.For instance if one wants to skip some content
before full item parsing:importioimportijsonparse_events=ijson.parse(io.BytesIO(b'["skip", {"a": 1}, {"b": 2}, {"c": 3}]'))whileTrue:prefix,event,value=next(parse_events)ifvalue=="skip":breakforobjinijson.items(parse_events,'item'):print(obj)Note that this interception
only makes sense for thebasic_parse->parse,parse->itemsandparse->kvitemsinteractions.Note also that event interception
is currently not supported
by theasyncfunctions.Push interfacesAll examples above use a file-like object as the data input
(both the normal case, and forasynciosupport),
and hence are “pull” interfaces,
with the library reading data as necessary.
If for whatever reason it’s not possible to use such method,
you can stillpushdata
through yet a different interface:coroutines(via generators, notasynciocoroutines).
Coroutines effectively allow users
to send data to them at any point in time,
with a finaltargetcoroutine-like object
receiving the results.In the following example
the user is doing the reading
instead of letting the library do it:importijson@ijson.coroutinedefprint_cities():whileTrue:obj=(yield)ifobj['type']!='city':continueprint(obj)coro=ijson.items_coro(print_cities(),'earth.europe.item')f=urlopen('http://.../')forchunkiniter(functools.partial(f.read,buf_size)):coro.send(chunk)coro.close()All four ijson iterators
have a*_corocounterpart
that work by pushing data into them.
Instead of receiving a file-like object
and option buffer size as arguments,
they receive a singletargetargument,
which should be a coroutine-like object
(anything implementing asendmethod)
through which results will be published.An alternative to providing a coroutine
is to useijson.sendable_listto accumulate results,
providing the list is cleared after each parsing iteration,
like this:importijsonevents=ijson.sendable_list()coro=ijson.items_coro(events,'earth.europe.item')f=urlopen('http://.../')forchunkiniter(functools.partial(f.read,buf_size)):coro.send(chunk)process_accumulated_events(events)delevents[:]coro.close()process_accumulated_events(events)OptionsAdditional options are supported byallijson functions
to give users more fine-grained control over certain operations:Theuse_floatoption (defaults toFalse)
controls how non-integer values are returned to the user.
If set toTrueusers receivefloat()values;
otherwiseDecimalvalues are constructed.
Note that buildingfloatvalues is usually faster,
but on the other hand there might be loss of precision
(which most applications will not care about)
and will raise an exception when overflow occurs
(e.g., if1e400is encountered).
This option also has the side-effect
that integer numbers bigger than2^64(butsometimes2^32, seebackends)
will also raise an overflow error,
due to similar reasons.
Future versions of ijson
might change the default value of this option
toTrue.Themultiple_valuesoption (defaults toFalse)
controls whether multiple top-level values are supported.
JSON content should contain a single top-level value
(seethe JSON Grammar).
However there are plenty of JSON files out in the wild
that contain multiple top-level values,
often separated by newlines.
By default ijson will fail to process these
with aparse error: trailing garbageerror
unlessmultiple_values=Trueis specified.Similarly theallow_commentsoption (defaults toFalse)
controls whether C-style comments (e.g.,/* a comment */),
which are not supported by the JSON standard,
are allowed in the content or not.For functions taking a file-like object,
an additionalbuf_sizeoption (defaults to65536or 64KB)
specifies the amount of bytes the library
should attempt to read each time.Theitemsandkvitemsfunctions, and all their variants,
have an optionalmap_typeargument (defaults todict)
used to construct objects from the JSON stream.
This should be a dict-like type supporting item assignment.EventsWhen using the lower-levelijson.parsefunction,
three-element tuples are generated
containing a prefix, an event name, and a value.
Events will be one of the following:start_mapandend_mapindicate
the beginning and end of a JSON object, respectively.
They carry aNoneas their value.start_arrayandend_arrayindicate
the beginning and end of a JSON array, respectively.
They also carry aNoneas their value.map_keyindicates the name of a field in a JSON object.
Its associated value is the name itself.null,boolean,integer,double,numberandstringall indicate actual content, which is stored in the associated value.PrefixA prefix represents the context within a JSON document
where an event originates at.
It works as follows:It starts as an empty string.A<name>part is appended when the parser starts parsing the contents
of a JSON object member calledname,
and removed once the content finishes.A literalitempart is appended when the parser is parsing
elements of a JSON array,
and removed when the array ends.Parts are separated by..When using theijson.itemsfunction,
the prefix works as the selection
for which objects should be automatically built and returned by ijson.BackendsIjson provides several implementations of the actual parsing in the form of
backends located in ijson/backends:yajl2_c: a C extension usingYAJL2.x.
This is the fastest, butmightrequire a compiler and the YAJL development files
to be present when installing this package.
Binary wheel distributions exist for major platforms/architectures to spare users
from having to compile the package.yajl2_cffi: wrapper aroundYAJL2.x
using CFFI.yajl2: wrapper around YAJL 2.x using ctypes, for when you can’t use CFFI
for some reason.yajl: deprecated YAJL 1.x + ctypes wrapper, for even older systems.python: pure Python parser, good to use with PyPyYou can import a specific backend and use it in the same way as the top level
library:importijson.backends.yajl2_cffiasijsonforiteminijson.items(...):# ...Importing the top level library asimport ijsonuses the first available backend in the same order of the list above,
and its name is recorded underijson.backend.
If theIJSON_BACKENDenvironment variable is set
its value takes precedence and is used to select the default backend.You can also use theijson.get_backendfunction
to get a specific backend based on a name:backend=ijson.get_backend('yajl2_c')foriteminbackend.items(...):# ...Performance tipsIn more-or-less decreasing order,
these are the most common actions you can take
to ensure you get most of the performance
out of ijson:Make sure you use the fastest backend available.
Seebackendsfor details.If you know your JSON data
contains only numbers that are “well behaved”
consider turning on theuse_floatoption.
Seeoptionsfor details.Make sure you feed ijson with binary data
instead of text data.
Seefaq#1 for details.Play with thebuf_sizeoption,
as depending on your data source and your system
a value different from the default
might show better performance.
Seeoptionsfor details.FAQQ: Does ijson work withbytesorstrvalues?A: In short: both are accepted as input, outputs are onlystr.All ijson functions expecting a file-like object
should ideally be given one
that is opened in binary mode
(i.e., itsreadfunction returnsbytesobjects, notstr).
However if a text-mode file object is given
then the library will automatically
encode the strings into UTF-8 bytes.
A warning is currently issued (but not visible by default)
alerting users about this automatic conversion.On the other hand ijson always returns text data
(JSON string values, object member names, event names, etc)
asstrobjects in python 3,
andunicodeobjects in python 2.7.
This mimics the behavior of the systemjsonmodule.Q: How are numbers dealt with?A: ijson returnsintvalues for integers
anddecimal.Decimalvalues for floating-point numbers.
This is mostly because of historical reasons.
Since 3.1 a newuse_floatoption (defaults toFalse)
is available to returnfloatvalues instead.
See theoptionssection for details.Q: I’m getting anUnicodeDecodeError, or anIncompleteJSONErrorwith no messageA: This error is caused by byte sequences that are not valid in UTF-8.
In other words, the data given to ijson is notreallyUTF-8 encoded,
or at least not properly.Depending on where the data comes from you have different options:If you have control over the source of the data, fix it.If you have a way to intercept the data flow,
do so and pass it through a “byte corrector”.
For instance, if you have a shell pipeline
feeding data throughstdininto your process
you can add something like... | iconv-futf8-tutf8-c| ...in between to correct invalid byte sequences.If you are working purely in python,
you can create a UTF-8 decoder
using codecs’incrementaldecoderto leniently decode your bytes into strings,
and feed those strings (using a file-like class) into ijson
(see ourstring_reader_async internal classfor some inspiration).In the future ijson might offer something out of the box
to deal with invalid UTF-8 byte sequences.Q: I’m gettingparse error: trailing garbageorAdditional data founderrorsA: This error signals that the input
contains more data than the top-level JSON value it’s meant to contain.
This isusuallycaused by JSON data sources
containing multiple values, and isusuallysolved
by passing themultiple_values=Trueto the ijson function in use.
See theoptionssection for details.Q: Are there any differences between the backends?A: Apart from their performance,
all backends are designed to support the same capabilities.
There are however some small known differences:Theyajlbackend doesn’t supportmultiple_values=True.
It also doesn’t complain about additional data
found after the end of the top-level JSON object.
When usinguse_float=Trueit also doesn’t properly support
values greater than 2^32 in 32-bit platforms or Windows.
Numbers with leading zeros are not reported as invalid
(although they are invalid JSON numbers).
Incomplete JSON tokens at the end of an incomplete document
(e.g.,{"a": fals) are not reported asIncompleteJSONError.Thepythonbackend doesn’t supportallow_comments=TrueIt also internally works withstrobjects, notbytes,
but this is an internal detail that users shouldn’t need to worry about,
and might change in the future.Acknowledgementsijson was originally developed and actively maintained until 2016
byIvan Sagalaev.
In 2019 hehanded overthe maintenance of the project and the PyPI ownership.Python parser in ijson is relatively simple thanks toDouglas Crockfordwho invented a strict, easy to parse syntax.TheYAJLlibrary byLloyd Hilaielis the most popular and efficient way to parse JSON in an
iterative fashion.Ijson was inspired byyajl-pywrapper byHatem Nassrat. Though ijson borrows almost nothing
from the actual yajl-py code it was used as an example of integration with yajl
using ctypes.