Package: jsonslicer
jsonslicer - stream JSON parserOverviewJsonSlicer performs astreamoriterative,pullJSON
parsing, which means itdoes not loadwhole JSON into memory
and is able to parsevery largeJSON files or streams.  The
module is written in C and usesYAJLJSON parsing library, so it's also quitefast.JsonSlicer takes apathof JSON map keys or array indexes, and
providesiterator interfacewhich yields JSON data matching
given path as complete Python objects.Example{"friends":[{"name":"John","age":31},{"name":"Ivan","age":26}],"colleagues":{"manager":{"name":"Jack","age":33},"subordinate":{"name":"Lucy","age":21}}}fromjsonslicerimportJsonSlicer# Extract specific elements:withopen('people.json')asdata:ivans_age=next(JsonSlicer(data,('friends',1,'age')))# 26withopen('people.json')asdata:managers_name=next(JsonSlicer(data,('colleagues','manager','name')))# 'Jack'# Iterate over collection(s) by using wildcards in the path:withopen('people.json')asdata:forpersoninJsonSlicer(data,('friends',None)):print(person)# {'name': 'John', 'age': 31}# {'name': 'Ivan', 'age': 26}# Iteration over both arrays and dicts is possible, even at the same timewithopen('people.json')asdata:forpersoninJsonSlicer(data,(None,None)):print(person)# {'name': 'John', 'age': 31}# {'name': 'Ivan', 'age': 26}# {'name': 'Jack', 'age': 33}# {'name': 'Lucy', 'age': 21}# Map key of returned objects is available on demand...withopen('people.json')asdata:forposition,personinJsonSlicer(data,('colleagues',None),path_mode='map_keys'):print(position,person)# 'manager' {'name': 'Jack', 'age': 33}# 'subordinate' {'name': 'Lucy', 'age': 21}# ...as well as complete path informationwithopen('people.json')asdata:for*path,personinJsonSlicer(data,(None,None),path_mode='full'):print(path,person)# ('friends', 0) {'name': 'John', 'age': 31})# ('friends', 1) {'name': 'Ivan', 'age': 26})# ('colleagues', 'manager') {'name': 'Jack', 'age': 33})# ('colleagues', 'subordinate') {'name': 'Lucy', 'age': 21})# Extract all instances of deep nested fieldwithopen('people.json')asdata:age_sum=sum(JsonSlicer(data,(None,None,'age')))# 111APIjsonslicer.JsonSlicer(
    file,
    path_prefix,
    read_size=1024,
    path_mode=None,
    yajl_allow_comments=False,
    yajl_dont_validate_strings=False,
    yajl_allow_trailing_garbage=False,
    yajl_allow_multiple_values=False,
    yajl_allow_partial_values=False,
    yajl_verbose_errors=True,
    encoding=None,
    errors=None,
    binary=False,
)Constructs iterative JSON parser. which reads JSON data fromfile(a.read()-supportingfile-like objectcontaining a JSON document).fileis a.read()-supportingfile-like
objectcontaining a JSON document. Both binary and text files are supported,
but binary ones are preferred, because the parser has to operate on
binary data internally anyway, and using text input would require an
unnecessary encoding/decoding which yields ~3% performance overhead.
Note that JsonSlicer supports both unicode and binary output regardless
of input format.path_prefixis an iterable (usually a list or a tuple) specifying
a path or a path pattern of objects which the parser should extract
from JSON.For instance, in the example above a path('friends', 0, 'name')will yield string'John', by descending from the root element
into the dictionary element by key'friends', then into the array
element by index0, then into the dictionary element by key'name'. Note that integers only match array indexes and strings
only match dictionary keys.The path can be turned into a pattern by specifyingNoneas a
placeholder in some path positions. For instance,(None, None, 'name')will yield all four names from the example above, because
it matches an item under 'name' key on the second nesting level of
any arrays or map structure.Both strings and byte objects are allowed in path, regardless of
input and output encodings.  are automatically converted
to the format used internally.read_sizeis a size of block read by the parser at a time.path_modeis a string which specifies how a parser should
return path information along with objects. The following modes are
supported:'ignore'(the default) - do not output any path information, just
objects as is ('friends').{'name':'John','age':31}{'name':'Ivan','age':26}{'name':'Jack','age':33}{'name':'Lucy','age':21}Common usage pattern for this mode isforobjectinJsonSlicer(...)'map_keys'- output objects as is when traversing arrays and tuples
consisting of map key and object when traversing maps.{'name':'John','age':31}{'name':'Ivan','age':26}('manager',{'name':'Jack','age':33})('subordinate',{'name':'Lucy','age':21})This format may seem inconsistent (and therefore it's not the default),
however in practice only collection of a single type is iterated at
a time and this type is known, so this format is likely the most useful
as in most cases you do need dictionary keys.Common usage pattern for this mode isforobjectinJsonSlicer(...)# when iterating arraysforkeyobjectinJsonSlicer(...)# when iterating maps'full_paths'- output tuples consisting of all path components
(both map keys and array indexes) and an object as the last element.('friends',0,{'name':'John','age':31})('friends',1,{'name':'Ivan','age':26})('colleagues','manager',{'name':'Jack','age':33})('colleagues','subordinate',{'name':'Lucy','age':21})Common usage pattern for this mode isfor*path,objectinJsonSlicer(...)yajl_allow_commentsenables corresponding YAJL flag, which is
documented as follows:Ignore javascript style comments present in JSON input.  Non-standard,
but rather funyajl_dont_validate_stringsenables corresponding YAJL flag, which
is documented as follows:When set the parser will verify that all strings in JSON input
are valid UTF8 and will emit a parse error if this is not so.  When
set, this option makes parsing slightly more expensive (~7% depending
on processor and compiler in use)yajl_allow_trailing_garbageenables corresponding YAJL flag, which
is documented as follows:By default, yajl will ensure the entire input text was consumed
and will raise an error otherwise.  Enabling this flag will cause
yajl to disable this check.  This can be useful when parsing json
out of a that contains more than a single JSON document.yajl_allow_multiple_valuesenables corresponding YAJL flag, which
is documented as follows:Allow multiple values to be parsed by a single handle.  The entire
text must be valid JSON, and values can be seperated by any kind
of whitespace.  This flag will change the behavior of the parser,
and cause it continue parsing after a value is parsed, rather than
transitioning into a complete state.  This option can be useful
when parsing multiple values from an input stream.yajl_allow_partial_valuesenables corresponding YAJL flag, which
is documented as follows:When yajl_complete_parse() is called the parser will check that the
top level value was completely consumed.  I.E., if called whilst
in the middle of parsing a value yajl will enter an error state
(premature EOF).  Setting this flag suppresses that check and the
corresponding error.yajl_verbose_errorsenables verbose YAJL errors, with exception
message including the JSON text where the error occured, along with
an arrow pointing to the specific char.encodingmay be used to override output encoding, which is derived
from the input file handle if possible, or otherwise set to the
default one as Python builtinopen()would use (usually'UTF-8').errorsis an optional string that specifies how encoding and
decoding errors are to be handled. Defaults to'strict'binaryforces the output to be in form ofbytesobjects instead
ofstrunicode strings.The constructed object is as iterator. You may callnext()to extract
single element from it, iterate it viaforloop, or use it in generator
comprehensions or in any place where iterator is accepted.Performance/competitorsThe closest competitor isijson,
and JsonSlicer was written to be better. Namely,It's up to 35x faster depending on ijson backend (starting with 3.0,
ijson supports comparable performance via yajl2_c backend), close in
performance to Python's nativejsonmodule.It supports more flexible paths/patterns specifying which objects
to iterate over in JSON hierarchy and provides consistent interface
for iteration over arrays and dictionariesThe results of bundled benchmark on Python 3.8.2 / clang 8.0.1 /-O2 -DNDEBUG/ FreeBSD 12.1 amd64 / Core i7-6600U CPU @ 2.60GHz.FacilityTypeObjects/secjson.loads()str1147.6Kjson.load(StringIO())str1139.3KJsonSlicer (no paths, binary input, binary output)bytes1149.7KJsonSlicer (no paths, unicode input, binary output)bytes1134.5KJsonSlicer (no paths, binary input, unicode output)str1012.3KJsonSlicer (no paths, unicode input, unicode output)str996.2KJsonSlicer (full paths, binary output)bytes763.1KJsonSlicer (full paths, unicode output)str567.2Kijson.yajl2_cbytes1062.0Kijson.yajl2_cffibytes71.6Kijson.yajl2bytes56.4Kijson.pythonstr32.0KStatus/TODOJsonSlicer is currently in beta stage, used in production inRepologyproject. Testing foci are:Edge cases with uncommon encoding (input/output) configurationsAbsence of memory leaksRequirementsPython 3.6+yajl2.0.3+ (older versions lack pkgconfig file)pkg-config (build-time)C++ compiler (build-time)LicenseMIT license, copyright (c) 2019 Dmitry Marakasovamdmi3@amdmi3.ru.