Package: lzip
lzipis a Python wrapper for lzlib (https://www.nongnu.org/lzip/lzlib.html) to encode and decode Lzip archives (https://www.nongnu.org/lzip/).This package is compatible with arbitrary byte sequences but provides features to facilitate interoperability with Numpy'sfrombufferandtobytesfunctions. Decoding and encoding can be performed in chunks, enabling the decompression, processing and compression of files that do not fit in RAM. URLs can be used as well to download, decompress and process the chunks of a remote Lzip archive in one go.pip3installlzipQuickstartCompressCompress an in-memory buffer and write it to a file:importlziplzip.compress_to_file("/path/to/output.lz",b"data to compress")Compress multiple chunks and write the result to a single file (useful to avoid large in-memory buffers):importlzipwithlzip.FileEncoder("/path/to/output.lz")asencoder:encoder.compress(b"data")encoder.compress(b" to")encoder.compress(b" compress")UseFileEncoderwithout context management (with):importlzipencoder=lzip.FileEncoder("/path/to/output.lz")encoder.compress(b"data")encoder.compress(b" to")encoder.compress(b" compress")encoder.close()Compress a Numpy array and write the result to a file:importlzipimportnumpyvalues=numpy.arange(100,dtype="<u4")lzip.compress_to_file("/path/to/output.lz",values.tobytes())lzipcan use different compression levels. See thedocumentationbelow for details.DecompressRead and decompress a file to an in-memory buffer:importlzipbuffer=lzip.decompress_file("/path/to/input.lz")Read and decompress a file one chunk at a time (useful for large files):importlzipforchunkinlzip.decompress_file_iter("/path/to/input.lz"):# chunk is a bytes objectRead and decompress a file one chunk at a time, and ensure that each chunk contains a number of bytes that is a multiple ofword_size(useful to parse numpy arrays with a known dtype):importlzipimportnumpyforchunkinlzip.decompress_file_iter("/path/to/input.lz",word_size=4):values=numpy.frombuffer(chunk,dtype="<u4")Download and decompress data from a URL:importlzip# option 1: store the whole decompressed file in a single bufferbuffer=lzip.decompress_url("http://download.savannah.gnu.org/releases/lzip/lzip-1.22.tar.lz")# option 2: iterate over the decompressed file in small chunksforchunkinlzip.decompress_url_iter("http://download.savannah.gnu.org/releases/lzip/lzip-1.22.tar.lz"):# chunk is a bytes objectlzipcan also decompress data from an in-memory buffer. See thedocumentationbelow for details.DocumentationThe present package contains two libraries.lzipdeals with high-level operations (open and close files, download remote data, change default arguments...) whereaslzip_extensionfocuses on efficiently compressing and decompressing in-memory byte buffers.lzipuseslzip_extensioninternally. The latter should only be used in advanced scenarios where fine buffer control is required.QuickstartCompressDecompressDocumentationlzipFileEncoderBufferEncoderRemainingBytesErrorcompress_to_buffercompress_to_filedecompress_bufferdecompress_buffer_iterdecompress_filedecompress_file_iterdecompress_file_likedecompress_file_like_iterdecompress_urldecompress_url_iterlzip_extensionDecoderEncoderCompare optionsWord size and remaining bytesDefault parametersPublishlzipFileEncoderclassFileEncoder:def__init__(self,path,level=6,member_size=(1<<51)):"""Encode sequential byte buffers and write the compressed bytes to a file- path is the output file name, it must be a path-like object such as a string or a pathlib path- level must be either an integer in [0, 9] or a tuple (directory_size, match_length)0 is the fastest compression level, 9 is the slowestsee https://www.nongnu.org/lzip/manual/lzip_manual.html for the mapping betweeninteger levels, directory sizes and match lengths- member_size can be used to change the compressed file's maximum member sizesee the Lzip manual for details on the tradeoffs incurred by this value"""defcompress(self,buffer):"""Encode a buffer and write the compressed bytes into the file- buffer must be a byte-like object, such as bytes or a bytearray"""defclose(self):"""Flush the encoder contents and close the filecompress must not be called after calling closeFailing to call close results in a corrupted encoded file"""FileEncodercan be used as a context manager (with FileEncoder(...) as encoder).closeis called automatically in this case.BufferEncoderclassBufferEncoder:def__init__(self,level=6,member_size=(1<<51)):"""Encode sequential byte buffers and return the compressed bytes as in-memory buffers- level: see FileEncoder- member_size: see FileEncoder"""defcompress(self,buffer):"""Encode a buffer and return the compressed bytes as an in-memory buffer- buffer must be a byte-like object, such as bytes or a bytearrayThis function returns a bytes objectThe compression algorithm may decide to buffer part or all of the data,hence the relationship between input (non-compressed) buffers andoutput (conpressed) buffers is not one-to-oneIn particular, the returned buffer can be empty (b"") even if the input buffer is not"""deffinish(self):"""Flush the encoder contentsThis function returns a bytes objectcompress must not be called after calling finishFailing to call finish results in corrupted encoded buffers"""RemainingBytesErrorclassRemainingBytesError(Exception):def__init__(self,word_size,buffer):"""Raised by decompress_* functions if the total number of bytes is not a multiple of word_sizeThe remaining bytes are stored in self.bufferSee "Word size and remaining bytes" for details"""compress_to_bufferdefcompress_to_buffer(buffer,level=6,member_size=(1<<51)):"""Encode a single buffer and return the compressed bytes as an in-memory buffer- buffer must be a byte-like object, such as bytes or a bytearray- level: see FileEncoder- member_size: see FileEncoderThis function returns a bytes object"""compress_to_filedefcompress_to_file(path,buffer,level=6,member_size=(1<<51)):"""Encode a single buffer and write the compressed bytes into a file- path is the output file name, it must be a path-like object such as a string or a pathlib path- buffer must be a byte-like object, such as bytes or a bytearray- level: see FileEncoder- member_size: see FileEncoder"""decompress_bufferdefdecompress_buffer(buffer,word_size=1):"""Decode a single buffer and return the decompressed bytes as an in-memory buffer- buffer must be a byte-like object, such as bytes or a bytearray- word_size: see "Word size and remaining bytes"This function returns a bytes object"""decompress_buffer_iterdefdecompress_buffer_iter(buffer,word_size=1):"""Decode a single buffer and return an in-memory buffer iterator- buffer must be a byte-like object, such as bytes or a bytearray- word_size: see "Word size and remaining bytes"This function returns a bytes object iterator"""decompress_filedefdecompress_file(path,word_size=1,chunk_size=(1<<16)):"""Read and decode a file and return the decompressed bytes as an in-memory buffer- path is the input file name, it must be a path-like object such as a string or a pathlib path- word_size: see "Word size and remaining bytes"- chunk_size: the number of bytes to read from the file at oncelarge values increase memory usage but very small values impede performanceThis function returns a bytes object"""decompress_file_iterdefdecompress_file_iter(path,word_size=1,chunk_size=(1<<16)):"""Read and decode a file and return an in-memory buffer iterator- path is the input file name, it must be a path-like object such as a string or a pathlib path- word_size: see "Word size and remaining bytes"- chunk_size: see decompress_fileThis function returns a bytes object iterator"""decompress_file_likedefdecompress_file_like(file_like,word_size=1,chunk_size=(1<<16)):"""Read and decode a file-like object and return the decompressed bytes as an in-memory buffer- file_like is a file-like object, such as a file or a HTTP response- word_size: see "Word size and remaining bytes"- chunk_size: see decompress_fileThis function returns a bytes object"""decompress_file_like_iterdefdecompress_file_like_iter(file_like,word_size=1,chunk_size=(1<<16)):"""Read and decode a file-like object and return an in-memory buffer iterator- file_like is a file-like object, such as a file or a HTTP response- word_size: see "Word size and remaining bytes"- chunk_size: see decompress_fileThis function returns a bytes object iterator"""decompress_urldefdecompress_url(url,data=None,timeout=socket._GLOBAL_DEFAULT_TIMEOUT,cafile=None,capath=None,context=None,word_size=1,chunk_size=(1<<16)):"""Download and decode data from a URL and return the decompressed bytes as an in-memory buffer- url must be a string or a urllib.Request object- data, timeout, cafile, capath and context are passed to urllib.request.urlopensee https://docs.python.org/3/library/urllib.request.html for details- word_size: see "Word size and remaining bytes"- chunk_size: see decompress_fileThis function returns a bytes object"""decompress_url_iterdefdecompress_url_iter(url,data=None,timeout=socket._GLOBAL_DEFAULT_TIMEOUT,cafile=None,capath=None,context=None,word_size=1,chunk_size=(1<<16)):"""Download and decode data from a URL and return an in-memory buffer iterator- url must be a string or a urllib.Request object- data, timeout, cafile, capath and context are passed to urllib.request.urlopensee https://docs.python.org/3/library/urllib.request.html for details- word_size: see "Word size and remaining bytes"- chunk_size: see decompress_fileThis function returns a bytes object iterator"""lzip_extensionEven thoughlzip_extensionbehaves like a conventional Python module, it is written in C++. To keep the implementation simple, only positional arguments are supported (keyword arguments do not work). The Python classes documented below are equivalent to the classes exported by this low-level implementation.You can uselzip_extensionby importing it like any other module.lzip.pyuses it extensively.DecoderclassDecoder:def__init__(self,word_size=1):"""Decode sequential byte buffers and return the decompressed bytes as in-memory buffers- word_size is a non-zero positive integerall the output buffers contain a number of bytes that is a multiple of word_size"""defdecompress(self,buffer):"""Decode a buffer and return the decompressed bytes as an in-memory buffer- buffer must be a byte-like object, such as bytes or a bytearrayThis function returns a bytes objectThe compression algorithm may decide to buffer part or all of the data,hence the relationship between input (compressed) buffers andoutput (decompressed) buffers is not one-to-oneIn particular, the returned buffer can be empty (b"") even if the input buffer is not"""deffinish(self):"""Flush the encoder contentsThis function returns a tuple (buffer, remaining_bytes)Both buffer and remaining_bytes and bytes objectsbuffer should be empty (b"") unless the file was truncatedremaining_bytes is empty (b"") unless the total number of bytes decodedis not a multiple of word_sizedecompress must not be called after calling finishFailing to call finish delays garbage collection which can be an issuewhen decoding many files in a row, and prevents the algorithm from detectingremaining bytes (if the size is not a multiple of word_size)"""EncoderclassEncoder:def__init__(self,dictionary_size=(1<<23),match_len_limit=36,member_size=(1<<51)):"""Encode sequential byte buffers and return the compressed bytes as in-memory buffers- dictionary_size is an integer in the range [(1 << 12), (1 << 29)]- match_len_limit is an integer in the range [5, 273]- member_size is an integer in the range [(1 << 12), (1 << 51)]"""defcompress(self,buffer):"""Encode a buffer and return the compressed bytes as an in-memory buffer- buffer must be a byte-like object, such as bytes or a bytearrayThis function returns a bytes objectThe compression algorithm may decide to buffer part or all of the data,hence the relationship between input (decompressed) buffers andoutput (compressed) buffers is not one-to-oneIn particular, the returned buffer can be empty (b"") even if the input buffer is not"""deffinish(self):"""Flush the encoder contentsThis function returns a bytes objectcompress must not be called after calling finishFailing to call finish results in corrupted encoded buffers"""Compare optionsThe scriptcompare_options.pyuses thelziplibrary to compare the compression ratio of different pairs (dictionary_size, match_len_limit). It runs multiple compressions in parallel and does not store the compressed bytes. About 3 GB of RAM are required to run the script. Processing time depends on the file size and the number of processors on the machine.The script requires matplotlib (pip3 install matplotlib) to display the results.python3compare_options/path/to/uncompressed/file[--chunk-size=65536]Word size and remaining bytesDecoding functions take an optional parameterword_sizethat defaults to1. Decoded buffers are guaranteed to contain a number of bytes that is a multiple ofword_sizeto facilitate fixed-sized words parsing (for examplenumpy.frombytes). If the total size of the uncompressed archive is not a multiple ofword_size,lzip.RemainingBytesErroris raised after iterating over the last chunk. The raised exception provides access to the remaining bytes.Non-iter decoding functions do not provide access to the decoded buffers if the total size is not a multiple ofword_size(only the remaining bytes).The following example decodes a file and converts the decoded bytes to 4-bytes unsigned integers:importlzipimportnumpytry:forchunkinlzip.decompress_file_iter("/path/to/archive.lz",4):values=numpy.frombuffer(chunk,dtype="<u4")exceptlzip.RemainingBytesErroraserror:# this block is executed only if the number of bytes in "/path/to/archive.lz"# is not a multiple of 4 (after decompression)print(error)# prints "The total number of bytes is not a multiple of 4 (k remaining)"# where k is in [1, 3]# error.buffer is a bytes object and contains the k remaining bytesDefault parametersThe default parameters inlzipfunctions are not constants, despite what is presented in the documentation. The actual implementation looks like this:defsome_function(some_parameter=None):ifsome_parameterisNone:some_paramter=some_paramter_default_valueThis approach makes it possible to change default values at the module level at any time. For example:importlziplzip.compress_to_file("/path/to/output0.lz",b"data to compress")# encoded at level 6 (default)lzip.default_level=9lzip.compress_to_file("/path/to/output1.lz",b"data to compress")# encoded at level 9lzip.compress_to_file("/path/to/output2.lz",b"data to compress")# encoded at level 9lzip_default_level=0lzip.compress_to_file("/path/to/output1.lz",b"data to compress")# encoded at level 0lzipexports the followingdefaultdefault values:default_level=6default_word_size=1default_chunk_size=1<<16default_member_size=1<<51PublishBump the version number inversion.py.Create a new release on GitHub.