Package: sleepyjson
sleepyjsonIn some situations, particularly in big data scenarios, it is necessary to extract information from a JSON file without needing to read the full content into memory. For an example, see the "Example" section below.sleepyjsonprovides a mechanism to deal with this scenario, where the JSON file is only parsed until the necessary information is found, and only that data is kept in memory.Although the package provides ways to handle random access to the contents of the file, random access runs in linear time on the size of the file. In fact, the whole idea of the package is to support memory-lightweightsequential processingof the JSON file.ExampleImagine you have a 10GB JSON file, where the top value is an array and the various items follow a predictable structure, as illustrated in the snippet below (pretend that the top level array contains millions of items and thesnippetskey contains large arrays with potentially long strings). Imagine as well that you want to extract the identifiers associated items that are dated from January of any year.[{"identifier":"AX1999","url":"http://www.example.org/url-path-with-a-slug","date":{"year":2020,"month":1,"day":1},"snippets":["A happy snippet of text found in the URL","Another snippet of text, this time sad","Yet another","And potentially many more"]},// ...]Traditionally (with the standard libraryjsonpackage), you would need to read the full dataset into memory.importjsonwithopen('data.json')asf:data=json.load(f)identifiers=[item['identifier']foritemindataifitem['date']['month']==1]Because you're reading the full file contents into thedatavariable, the memory consumption for this snippet is quite high.Withsleepyjson, you can keep memory usage low and still achieve the same result:importsleepyjsonwithopen('data.json')asf:reader=sleepyjson.Reader(f)identifiers=[item['identifier'].value()foriteminreaderifitem['date']['month'].value()==1]Notice the while the memory consumption is low, time complexity is linear for most purposes. If you want to get to a value near the end of a JSON file, the file must be fully parsed until the position you need to access. Additionally, because this is a pure python implementation, parsing is slow (Imaychange the parsing mechanism in the future to a compiled process to accelerate this).Comparison withjsonAs you can spot in the previous snippet of code,sleepyjsonrequires you to keep the file opened while you are moving within the JSON contents. This is because no content is moved into memory unless the code does so explicitly.Additionally, the contents of a value must be explicitly requested with the.value()method.Note: I want to change this method to something more explicit, likematerialize, to convey the meaning that we are not simply getting the value, but actually parsing and building a JSON value, which might be costly if the value is big.Thesleepyjson.Readerclass takes a file-like object, but doesn't read its contents until requested to. You can move around the file using iteration and indexation.Also,sleepyjson.Readercan read "JSON streams" in addition to regular JSON files. JSON streams are files that contain JSON values in succession. The reader can navigate within these files using the.next()method.# Assuming file `data.json` contains# ["an", "array"] {"an": "object"}fromsleepyjsonimportReaderwithopen('data.json')asf:reader=Reader(f)print(reader.value())# ['an', 'array']reader.next()print(reader.value())# {'an': 'object'}Partially valid JSONIn case your information needs from the file do not require the file to be read until the end,sleepyjsonparses only the necessary contents from the file, which means that the file does not need to be completely valid.Comments, trailing commasEven though python'sjsonpackage does not accept comments nor trailing commas, some popular packages elsewhere do. To support reading this "non-standard" data format,sleepyjsonunderstands double-slash comments and ignores trailing commas. So the following would be a valid JSON file from the point of view of this package:{// Comment"powers of two":[1,2,4,8,16,32,],}Simple documentationThe API surface of this package is simple, providing three classes:NodeNodeTypeReaderWhile you can produce instances of theNodeclass, I recommend you only instantiateReaderdirectly.NodeTypeis an enumeration class that represents the possible JSON value types.TheNodeclassThis class represents a value in the JSON file. It deliberately does not contain a full representation of the value, particularly for strings, arrays and objects, because doing so would defeat the purpose of the package. It does, however, offer mechanisms to access those value, by allowing iteration over arrays and objects, and allowing (but not requiring) the construction, in memory, of its contents.In the following examples, we assumenodepoints to the JSON object represented below:{"a":[0,1.337e3],"b":"string","c":[true,false,null]}In general, some operations are only valid for some types (namely indexing, iterating, etc.). If the corresponding methods are called on a node of an incorrect type, aValueErroris raised.TheNode.typeattributeReturns an instance ofNodeTypethat represents the type of JSON value under this node. Possible values are:NodeType.OBJECTNodeType.ARRAYNodeType.NUMBERNodeType.STRINGNodeType.TRUENodeType.FALSENodeType.NULLTheNode.valuemethodBuilds and returns the value of this node.If the node is a JSON true, false or null literal, it returnsTrue,FalseorNonerespectively.If the node is a number, it parses and returns the number (returns anintif no decimal and no exponent is provided,floatotherwise).If the node is a string, it parses the string, unescaping any escaped charactersIf the node is an array, it returns a python listIf the node is an object, it returns a python dictThe inner values of arrays and objects are recursively built with the.valuemethod as well.TheNode.__getitem__method (node[i])For arrays and objects, returns aNodethat represents the requested item. For arrays, you can index with integers. Negative value are allowed, but this requires parsing the entire array to determine the length of the array. For objects, you can index with strings. Indexing parses the node only until the correct item is found (except for indexing arrays with a negative value). If the item is not found, anIndexErroris raised (for arrays) or aKeyErroris raised (for objects).node['a'].value()# [0, 1337.0]node['c'][0].value()# Truenode['c'][-1].value()# Nonenode['x']# raises KeyErrorTheNode.__len__method (len(node))For arrays and objects, returns the length of the value. Determining the length parses the result but doesn't construct the items, which means it is easy on memory.len(node)# 3len(node['a'])# 2len(node['b'])# raises ValueError; you cannot determine the length of a stringTheNode.__iter__method (for i in node)Iterates over the items in an array, or over the keys in an object. This iterates in the order the values appear in the file. Also seeNode.items, to iterate over the keysandvalues of a JSON object.list(node)# ['a', 'b', 'c'][i.value()>0foriinnode['a']]# [False, True]TheNode.itemsmethodIterates over the items of a JSON object. The iterator returned from this method sequentially produces pairs of type(str, Node), where the first item in the key and the second item is the node representing the value associated with that key. The iterator respects the order in the file.TheNode.__contains__method (x in node)This defines theinoperator.x in nodeisTrueif:noderepresents a JSON array and one of its inner values is equal toxnoderepresents a JSON object and one of its keys is equal tox'a'innode# True0innode['a']# TrueTheNode.is_*methodsThere are several of these methods, each testing the type of value the node points to:Node.is_object: Determines whether the node points to an objectNode.is_array: Determines whether the node points to an arrayNode.is_string: Determines whether the node points to a stringNode.is_number: Determines whether the node points to a numberNode.is_true: Determines whether the node points to thetrueliteralNode.is_false: Determines whether the node points to thefalseliteralNode.is_boolean: Determines whether the node points to a boolean (trueorfalse)Node.is_null: Determines whether the node points to thenullliteralA note on key uniquenesssleepyjsondoesnotmake an effort to validate that keys on objects are unique. This means that iterating over the keys of an object can produce the same key more than once; however, retrieving the actual value of a JSON object preserves only one of those key-value pairs (since the returned object is actually a python dictionary).Additionally, because retrieving an item from an object stops when the key isfirstfound in the file, and building the python dictionary likely preserves thelastvalue associated with the key.As such, when a key is repeated in a JSON object, the following can happen:len(node) > len(node.value())node[key].value() != node.value()[key]TheReaderclassIn the following examples, we assumereaderto be constructed from a file whose contents are:{"a":[1,2,3]}true[null,false,true]TheReader.__init__constructorThis class constructor takes a file-like whose contents are in the JSON format. The file should contain a JSON value or a sequence of JSON values (a-la JSON streams). It can also receive multiple files.TheReader.nodeattributeReturns the node that is currently being read in the JSON stream. As a convenience, you can access the fields and methods of this node by calling them directly on the reader:reader.node.value()# {'a': [1, 2, 3]}reader.value()# {'a': [1, 2, 3]}TheReader.__len__method  (len(node))Returns the length of the current node. Equivalent tolen(reader.node).TheReader.__iter__method (for i in node)Iterates over the current node. Equivalent toiter(reader.node)TheReader.__getattr__method (node.*)This method gets the requested attribute fromreader.node, thus ensuring that the reader behaves in most ways like the node it is currently reading. Read theNodedocumentation to know more about this.TheReader.__getitem__method (node[i])This method implements random access to the contents of the current node. Equivalent toreader.node[i]. See the documentation for theNodeclass.reader['a'].value()# [1, 2, 3]len(reader['a'])# 3TheReader.nextmethodJumps to the next value on the JSON stream. Notice that if multiple files have been given in the constructor, this is the way to access the next files. There is no way to jump back to a previous value on the stream.reader.value()# {'a': [1, 2, 3]}reader.next()reader.value()# truereader.next()reader.value()# [None, False, True]If there are no more nodes in the current file and no more files to process, this method raises aStopIterationexception.TheReader.jumpmethodPerforms the.next()method a non-negative number of times.reader.value()# {'a': [1, 2, 3]}reader.jump(2)reader.value()# [None, False, True]