Package: inpout
python-inpoutSimple library for input/output using MessagePack and LZ4 compression in Python.InstallationYou can usepip(or any PyPI-compatible package manager) for installation:pip install inpoutor, if you prefer a local user installation:pip install --user inpoutFor Microsoft Windows users, you might need to runpipthrough the Python interpreter:python -m pip install inpoutNote:Visual C++ 14.0 is required for Windows installation. Get it withMicrosoft Visual C++ Build Tools:https://visualstudio.microsoft.com/downloads/UsageTo use the functionality of this library, simply import it in your Python programs:import inpoutHigh-Level FunctionsFor saving/loading data using MessagePack and LZ4 compression, the following high-level convenience functions are provided in the root namespace:load_obj(path, **kwargs): return a single object loaded from a file on disk.Seedata_unpacker()for details on the keyword arguments.load_iter(path, **kwargs): return an iterator of objects loaded from a file on disk.Seedata_unpacker()for details on the keyword arguments.save_obj(obj, path, **kwargs): save a single objectobjto a file on disk.Seedata_pack()for details on the keyword arguments.save_iter(iterable, path, **kwargs): save an interable of objectsiterableto a file on disk.Seedata_pack()for details on the keyword arguments.Context Manager FunctionsFor more flexibility, the following context manager functions are provided in the root namespace:data_unpacker(path, compression=True, **kwargs): create a data unpacker (MessagePack) context manager with optional compression (LZ4) support to be used as an iterable unpacker.path: path to the file on disk containing the data to read.compression: boolean flag for using LZ4 compression.kwargs: keyword arguments passed directly to the MessagePack unpacker. See below.data_pack(path, compression=True, level=None, append=False, **kwargs): create a data pack (MessagePack) context manager with optional compression (LZ4) support and file appending to be used as a packing function.path: path to the file on disk that will contain the written data.compression: boolean flag for using LZ4 compression.level: the compression level for the LZ4 compressor. Seecompressor()for details.append: boolean flag for opening the file on disk in appending mode.kwargs: keyword arguments passed directly to the MessagePack packer. See below.Packing FunctionsFor packing/unpacking data with MessagePack directly without compression, the following functions are provided ininpout.packing:pack(obj, stream, **kwargs): pack a single object using MessagePack (with extended types support) to a stream of bytes.obj: the object to pack.stream: the bytes stream to use for writing data.kwargs: keyword arguments passed directly to the MessagePack packer. See below.packb(obj, **kwargs): pack a single object using MessagePack (with extended types support) and return packed bytes.obj: the object to pack.kwargs: keyword arguments passed directly to the MessagePack packer. See below.unpack(stream, **kwargs): unpack a stream of packed bytes using MessagePack (with extended types support) and return a single unpacked object.stream: the bytes stream to use for reading data.kwargs: keyword arguments passed directly to the MessagePack unpacker. See below.unpackb(packed, **kwargs): unpack packed bytes using MessagePack (with extended types support) and return a single unpacked object.packed: the packed bytes to unpack.kwargs: keyword arguments passed directly to the MessagePack unpacker. See below.Compressing FunctionsFor compressing/decompressing arbitrary data with LZ4 directly without packing, the following context manager functions are provided ininpout.compression:decompressor(path): create a data decompressing context manager to be used as reader.path: path to the file on disk containing the compressed data.compressor(path, level=None, append=False): create a data compressing context manager to be used as a writer.path: path to the file on disk that will contain the compressed data.level: compression level to use. Defaults toLZ4F_COMPRESSION_MAXifNone. Values lower than3(including negative ones) use fast compression. Recommended range for hc-type compression is between4and9. More information can befound here.append: boolean flag for opening the file on disk in appending mode.Keyword Arguments for MessagePackFunctions involving data packing with MessagePack support optional keyword argumentskwargsto be passed directly to MessagePack packer and unpacker. Useful options are described below:use_list: can beTrue(default) orFalse.List is the default sequence/array type for Python. But tuples are lighter than lists. You can useuse_list=Falsewhile unpacking when performance is important for your program. Python objects that require hashable elements such asdictorsetcan't use lists as key, thereforeuse_list=Falseis required for unpacking data containing tuples as keys.ExamplesBelow is example code of how to use the main convenience functions of this library.fromdatetimeimportdatetimeimportinpout# create some Python objects to test, set and datetime are supported out of the boxobj1=[1,2,3,4,5]obj2=("test",1234)obj3={"test":1234,"test2":5678}obj4={"a","b","c",5,6,7,8}obj5=datetime.now()obj6={(1,2):"tuple_key"}# save all the above objects as a single tuple to diskinpout.save_obj((obj1,obj2,obj3,obj4,obj5),"test1.mp.lz4")# save all the above objects in order to disk one by one (iterator)iterator=(oforoin(obj1,obj2,obj3,obj4,obj5))inpout.save_iter(iterator,"test2.mp.lz4")# append more data to the same test file (save_obj and save_iter can be mixed)inpout.save_obj(obj1,"test2.mp.lz4",append=True)inpout.save_iter((obj2,obj3),"test2.mp.lz4",append=True)# save an object with a tuple as key to demonstrate 'use_list=False'inpout.save_obj(obj6,"test3.mp.lz4")# load the first test filedata=inpout.load_obj("test1.mp.lz4")print("DATA=%r"%(data,))# load the second test file (iterator)forobjininpout.load_iter("test2.mp.lz4"):print("OBJ=%r"%(obj,))# load the third test file using tuple types, otherwise it failsdata=inpout.load_obj("test3.mp.lz4",use_list=False)print("DATA=%r"%(data,))# demonstrate the data pack functionwithinpout.data_pack("test4.mp.lz4")aspack:forobjin(obj1,obj2,obj3,obj4,obj5,obj6):pack(obj)# demonstrate the data unpacker functionwithinpout.data_unpacker("test4.mp.lz4",use_list=False)asunpacker:forobjinunpacker:print("OBJ=%r"%(obj,))# demonstrate the data pack function (no compression)withinpout.data_pack("test4.mp",compression=False)aspack:forobjin(obj1,obj2,obj3,obj4,obj5,obj6):pack(obj)# demonstrate the data unpacker function (no compression)withinpout.data_unpacker("test4.mp",compression=False,use_list=False)asunpacker:forobjinunpacker:print("OBJ=%r"%(obj,))MessagePack Extended TypesThis library supports MessagePack extended types and includes encoders/decoders for two standard Python objects:set(typecode127) anddatetime(typecode126). These are automatically registered upon importing the library.setobjects are serialised as tuples containing their elements and reconstructed from these stored tuples.datetimeobjects are serialised as a tuple of two integers(seconds, microseconds)representing the number of seconds and microseconds since the UNIX epoch (00:00:00 Thursday, 1 January 1970). Timezone information is used for the conversion but not stored, thereforedatetimeobjects are reconstructed as naive, i.e. without timezone.You can also easily create your own encoders/decoders for Python objects and register them for this library to be used during serialisation/deserialisation:importinpoutclassMyType(object):def__init__(self,data1,data2):self.data1=data1self.data2=data2# define a representation for your type (encoder)# we will assign '50' as the typecode for this typedefencode_mytype(obj,packb,ext_type):returnext_type(50,packb((obj.data1,obj.data2)))# define how to create your type from your representation (decoder)defdecode_mytype(data,unpackb):data1,data2=unpackb(data)returnMyType(data1,data2)# register custom encoder/decoders for your typeinpout.packing.register_ext_type_encoder(MyType,encode_mytype)inpout.packing.register_ext_type_decoder(50,decode_mytype)# test saving/loading your typeobj=MyType("test",1234)inpout.save_obj(obj,"test.mp.lz4")obj2=inpout.load_obj("test.mp.lz4")print(obj2.data1,obj2.data2)You can use any typecode for your own extended types, however it must be between0and125(inclusive).More information about MessagePack extended types can befound here.Command-line ToolsThe library includes the following command-line tools that are installed automatically bypip:inpout-pprint: iterate and pretty-print data files generated by this library.This tool is based on theload_iterfunction with theuse_list=Falsekeyword argument. Compression is activated if filenames end with.lz4(case insensitive). Optionally, theNUMBERof objects to process from each input file can be also provided. Usage:$ inpout-pprint [-n NUMBER] FILENAME [FILENAME ...]LicenseThis software is under theApache License 2.0.Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.