Package: py-lz4framed_ph4
OverviewThis is anLZ4-frame compression library for Python
v3.2+ (and 2.7+), bound to Yann Collet’sLZ4 C
implementation.Installing / packaging# To get from PyPIpip3installpy-lz4framed_ph4# To only build extension modules inline (e.g. in repository)python3setup.pybuild_ext-i# To build & install globallypython3setup.pyinstall# To install locally with pippipinstall--upgrade--find-links=..NotesThe above as well as all other python3-using commands should also run
with v2.7+This fork is based onhttps://github.com/Iotic-Labs/py-lz4framedImprovementsThis fork has several improvements I needed for my other project.Streamed decompression continuation (on reconnect)Streamed decompression state clone - checkpointingStreamed decompression marshalling - failure recovery, checkpointingMore on improvementsThe scenario improvements address is downloading & decompressing large
LZ4 data stream on the fly (hundreds of GBs). If the download stream is
interrupted the original decompressor had no way to resume the
decompression where it stopped.ContinuationThe main motivation is to recover from aforementioned interruptions.
Decompressor object now supports changing of the file-like object that
is read from. If input socket stream went down we can re-connect and
continue from the position it stopped. More in testtest_decompressor_fp_continuation.State cloningIf the processing logic is more complex you can useclone_decompression_contextto clone decompressor context (the whole
decompression state) and revert to this checkpoint if something breaks.
More in testtest_decompressor_fp_clone.State marshallingIn order to recover also from program crashes you can marshal /
serialize the decompressor context to the (byte) string which can be
later unmarshalled / deserialized and continue from that point.
Marshalled state can be stored e.g., to a file. More in testtest_decompressor_fp_marshalling.Random access archiveSituation: 800 GB LZ4 encrypted file. You want random access the file so
it can be map/reduced or processed in parallel from different offsets.Marshalled decompressor state takes only the required amount of memory.
If the state dump is performed on the block boundaries (i.e., when the
size hint from the previous call was provided by the input stream) the
marhsalled size would be only 184 B, in the best case scenario, 66 kB in
the worse case - when LZ4 file is using linked mode.Anyway, when state marshalling returns this small state the application
can build a meta file, the mapping: position in the input stream ->
decompressor context. With this meta file a new decompressor can jump to
the particular checkpoint.UsageSingle-function operation:importlz4framedcompressed=lz4framed.compress(b'binary data')uncompressed=lz4framed.decompress(compressed)To iteratively compress (to a file or e.g. BytesIO instance):withopen('myFile','wb')asf:# Context automatically finalises frame on completion, unless an exception occurswithCompressor(f)asc:try:while(...):c.update(moreData)exceptLz4FramedNoDataError:passTo decompress from a file-like object:withopen('myFile','rb')asf:try:forchunkinDecompressor(f):decoded.append(chunk)exceptLz4FramedNoDataError:# Compress frame data incomplete - error case...See alsolz4framed/__main__.pyfor
example usage.Documentationimportlz4framedprint(lz4framed.__version__,lz4framed.LZ4_VERSION,lz4framed.LZ4F_VERSION)help(lz4framed)Command-line utilitypython3-mlz4framedUSAGE:lz4framed(compress|decompress)(INFILE|-)[OUTFILE](De)compressesanlz4frame.InputisreadfromINFILEunlesssetto'-',inwhichcasestdinisused.IfOUTFILEisnotspecified,outputgoestostdout.TestsStaticThis library has been checked usingflake8andpylint, using a modified configuration - seepylint.rcandflake8.cfg.Unitpython3-munittestdiscover-v.Why?The only existing lz4-frame interoperable implementation I was aware of
at the time of writing
(lz4tools) had the following
limitations:Incomplete implementation in terms of e.g. reference & memory leaks
on failureLack of unit testsNot thread safeDoes not release GIL during low level (de)compression operationsDid not address the requirements for an external projectFurther readinghttps://github.com/lz4/lz4/wiki/lz4_Frame_format.mdhttps://github.com/lz4/lz4/wiki/lz4_Block_format.mdhttps://ticki.github.io/blog/how-lz4-works/https://fastcompression.blogspot.cz/2011/05/lz4-explained.html