Package: pymecompress
PYMECompressCompression for photon-noise limited images which keeps losses within the Poisson noise envelopePYMECompress consists of four parts:a fork of the Basic Compression Library originally by Marcus Geelnard,
modified to include a heavily optimized huffman coder (BCL license is avalable under pymecompress/bcl/doc/manual.pdf and would appear to be BSD compatible)a fast, AVX optimized, quantizer to perform "within noise level" quantization of photon-limited imagesa python wrapper of the above. Note that at this point, only huffman coding and quantization are exposed to pythonnumcodecs codecs (experimental) to permit simple usage with other IO packages - e.g. ZarrTogether they offer a single core throughput of ~500 -600MB/sInstallationcondaPrebuilt binaries of PYMEcompress are available as a conda package (pymecompress) on thedavid_baddeleyconda channel for python 2.7, 3.6 & 3.7, and installable via:conda install -c david_baddeley pymecompresssourceIf you want to modify/contribute to the package you will have to build from source.Because we use gcc compiler extensions for avx opcodes, we must use gcc/clang for compilation, regardless of platform.On OSX / linux, a standardpython setup.py installorpython setup.py developshould work.On Windows, you need to install mingw and run the build step first so that you can pass the compiler flag topython setup.py build- i.e. :python setup.py build --compiler=mingw32
python setup.py installA suitable environment for building pymecompress can be created using the following conda commandconda create -n <name> python=x.x numpy cython libpython m2w64-toolchainpipInstallation via pip is also available:pip install pymecompressalthough binary wheels are not available for all platforms so you may need to set up a build environment (gcc/mingw, as described for source installation) first.Usagenumcodecs codecsimportnumpyasnpfrompymecompressimportcodecs# vanilla huffman coding (lossless). NB - input buffer must be bytes/uint8huff=codecs.Huffman()d=np.ones(1000)assertnp.allclose(huff.decode(huff.encode(d.view('uint8'))).view(d.dtype),d)# with quantisation NB: input data type MUST be uint16huffq=codecs.HuffmanQuant16(offset=0,scale=1.0)ds=np.linspace(1,2**15).astype('uint16')assertnp.all((huffq.decode(huffq.encode(ds))-ds.astype('f'))<np.sqrt(ds))As a Zarr compression filterVERY EXPERIMENTAL!This is not yet well tested, but should work as described inhttps://zarr.readthedocs.io/en/stable/api/codecs.html. In brief ...importzarrfrompymecompressimportcodecsz=zarr.zeros(1000000,dtype='uint16',compressor=codecs.HuffmanQuant16(offset=0,scale=1))NBTo be able to read/open files saved using the pymecompress codecs you will probably need to runfrom pymecompress import codecsto register the codecs withnumcodecsbefore trying to open the file.Directly calling functionsAs you need to supply the original size to the decompression function, these are most suitable when putting the compressed data in an external wrapper e.g. PYMEs' PZFFormat which keeps track of the original data dimensions and dtype (we save a couple of bytes and a seek over using the codec versions above).importnumpyasnpimportpymecompressdata=np.linspace(1,2**15).astype('uint16')nbytes=data.nbytesc=pymecompress.HuffmanCompressQuant(data,quantizationOffset,quantizationScale).to_string()decompressed=pymecompress.HuffmanDecompress(np.fromstring(c,'u1'),nbytes)dequantized=(quantisationScale*decompressed)**2+quantizationOffset