Package: orjsonl
orjsonlorjsonlis a lightweight, high-performance Python library for parsing jsonl files. It supports a wide variety of compression formats, including gzip, bzip2, xz and Zstandard. It is powered byorjson, the fastest and most accurate json serializer for Python.Installationorjsonlmay be installed withpip:pipinstallorjsonlTo read or write Zstandard files, install eitherzstdor thezstandardPython package.UsageThe code snippet below demonstrates how jsonl files can be saved, loaded, streamed, appended and extended withorjsonl:>>>importorjsonl>>># Create an iterable of Python objects.>>>data=['hello world',['fizz','buzz'],]>>># Save the iterable to a jsonl file.>>>orjsonl.save('test.jsonl',data)>>># Append a Python object to the jsonl file.>>>orjsonl.append('test.jsonl',{42:3.14})>>># Extend the jsonl file with an iterable of Python objects.>>>orjsonl.extend('test.jsonl',[True,False])>>># Load the jsonl file.>>>orjsonl.load('test.jsonl')['hello world',['fizz','buzz'],{42:3.14},True,False]>>># Stream the jsonl file.>>>list(orjsonl.stream('test.jsonl'))['hello world',['fizz','buzz'],{42:3.14},True,False]orjsonlcan also be used to process jsonl files compressed with gzip, bzip2, xz and Zstandard:>>>orjsonl.save('test.jsonl.gz',data)>>>orjsonl.append('test.jsonl.gz',{42:3.14})>>>orjsonl.extend('test.jsonl.gz',[True,False])>>>orjsonl.load('test.jsonl.gz')['hello world',['fizz','buzz'],{42:3.14},True,False]>>>[objforobjinorjsonl.stream('test.jsonl.gz')]['hello world',['fizz','buzz'],{42:3.14},True,False]Loaddefload(path:str|bytes|int|os.PathLike,decompression_threads:Optional[int]=None,compression_format:Optional[str]=None)->list[dict|list|int|float|str|bool|None]load()deserializes a compressed or uncompressed UTF-8-encoded jsonl file to a list of Python objects.pathis a path-like object giving the pathname (absolute or relative to the current working directory) of the compressed or uncompressed UTF-8-encoded jsonl file to be deserialized.decompression_threadsis an optional integer passed toxopen.xopen()as thethreadsargument that specifies the number of threads that should be used for decompression.compression_formatis an optional string passed toxopen.xopen()as theformatargument that overrides the autodetection of the file’s compression format based on its extension or content. Possible values are ‘gz’, ‘xz’, ‘bz2’ and ‘zst’.This function returns alistobject comprised of deserializeddict,list,int,float,str,boolorNoneobjects.Streamdefstream(path:str|bytes|int|os.PathLike,decompression_threads:Optional[int]=None,compression_format:Optional[str]=None)->Generator[dict|list|int|float|str|bool|None,None,None]stream()creates ageneratorthat deserializes a compressed or uncompressed UTF-8-encoded jsonl file to Python objects.pathis a path-like object giving the pathname (absolute or relative to the current working directory) of the compressed or uncompressed UTF-8-encoded jsonl file to be deserialized by thegenerator.decompression_threadsis an optional integer passed toxopen.xopen()as thethreadsargument that specifies the number of threads that should be used for decompression.compression_formatis an optional string passed toxopen.xopen()as theformatargument that overrides the autodetection of the file’s compression format based on its extension or content. Possible values are ‘gz’, ‘xz’, ‘bz2’ and ‘zst’.This function returns ageneratorthat deserializes the file todict,list,int,float,str,boolorNoneobjects.Savedefsave(path:str|bytes|int|os.PathLike,data:Iterable,default:Optional[Callable]=None,option:int=0,compression_level:Optional[int]=None,compression_threads:Optional[int]=None,compression_format:Optional[str]=None)->Nonesave()serializes an iterable of Python objects to a compressed or uncompressed UTF-8-encoded jsonl file.pathis a path-like object giving the pathname (absolute or relative to the current working directory) of the compressed or uncompressed UTF-8-encoded jsonl file to be saved.datais an iterable of Python objects to be serialized to the file.defaultis an optional callable passed toorjson.dumps()as thedefaultargument that serializes subclasses or arbitrary types to supported types.optionis an optional integer passed toorjson.dumps()as theoptionargument that modifies how data is serialized.compression_levelis an optional integer passed toxopen.xopen()as thecompresslevelargument that determines the compression level for writing to gzip, xz and Zstandard files.compression_threadsis an optional integer passed toxopen.xopen()as thethreadsargument that specifies the number of threads that should be used for compression.compression_formatis an optional string passed toxopen.xopen()as theformatargument that overrides the autodetection of the file’s compression format based on its extension. Possible values are ‘gz’, ‘xz’, ‘bz2’ and ‘zst’.Appenddefappend(path:str|bytes|int|os.PathLike,data:Any,newline:bool=True,default:Optional[Callable]=None,option:int=0,compression_level:Optional[int]=None,compression_threads:Optional[int]=None,compression_format:Optional[str]=None)->Noneappend()serializes and appends a Python object to a compressed or uncompressed UTF-8-encoded jsonl file.pathis a path-like object giving the pathname (absolute or relative to the current working directory) of the compressed or uncompressed UTF-8-encoded jsonl file to be appended.datais a Python object to be serialized and appended to the file.newlineis an optional Boolean flag that, if set toFalse, indicates that the file does not end with a newline and should, therefore, have one added before data is appended.defaultis an optional callable passed toorjson.dumps()as thedefaultargument that serializes subclasses or arbitrary types to supported types.optionis an optional integer passed toorjson.dumps()as theoptionargument that modifies how data is serialized.compression_levelis an optional integer passed toxopen.xopen()as thecompresslevelargument that determines the compression level for writing to gzip, xz and Zstandard files.compression_threadsis an optional integer passed toxopen.xopen()as thethreadsargument that specifies the number of threads that should be used for compression.compression_formatis an optional string passed toxopen.xopen()as theformatargument that overrides the autodetection of the file’s compression format based on its extension or content. Possible values are ‘gz’, ‘xz’, ‘bz2’ and ‘zst’.Extenddefextend(path:str|bytes|int|os.PathLike,data:Iterable,newline:bool=True,default:Optional[Callable]=None,option:int=0,compression_level:Optional[int]=None,compression_threads:Optional[int]=None,compression_format:Optional[str]=None)->Noneextend()serializes and appends an iterable of Python objects to a compressed or uncompressed UTF-8-encoded jsonl file.pathis a path-like object giving the pathname (absolute or relative to the current working directory) of the compressed or uncompressed UTF-8-encoded jsonl file to be extended.datais an iterable of Python objects to be serialized and appended to the file.newlineis an optional Boolean flag that, if set toFalse, indicates that the file does not end with a newline and should, therefore, have one added before data is extended.defaultis an optional callable passed toorjson.dumps()as thedefaultargument that serializes subclasses or arbitrary types to supported types.optionis an optional integer passed toorjson.dumps()as theoptionargument that modifies how data is serialized.compression_levelis an optional integer passed toxopen.xopen()as thecompresslevelargument that determines the compression level for writing to gzip, xz and Zstandard files.compression_threadsis an optional integer passed toxopen.xopen()as thethreadsargument that specifies the number of threads that should be used for compression.compression_formatis an optional string passed toxopen.xopen()as theformatargument that overrides the autodetection of the file’s compression format based on its extension or content. Possible values are ‘gz’, ‘xz’, ‘bz2’ and ‘zst’.LicenseThis library is licensed under theMIT License.