Package: bloscpack
Author:Valentin HänelContact:valentin@haenel.coList:http://groups.google.com/group/bloscGithub:https://github.com/Blosc/bloscpackPyPi:https://pypi.python.org/pypi/bloscpackAnaconda:https://anaconda.org/pypi/bloscpackOhloh:https://www.ohloh.net/p/bloscpackVersion:Travis CI:Coveralls:Python Versions:License:And…:Table of ContentsDescriptionCode of ConductDependenciesStability of File FormatInstallationUsageExamplesPython APITestingBenchmarkOptimizing Chunk SizeBloscpack FormatCoding ConventionsHow to Optimize LoggingComparison to HDF5/PyTablesPrior ArtResources and Related PublicationsMaintainers Notes on Cutting a ReleaseTODOChangelogThanksAuthor, Copyright and LicenseDescriptionCommand line interface to and serialization format forBlosc, a high performance, multi-threaded, blocking and
shuffling compressor. Usespython-bloscbindings to interface with Blosc.
Also comes with native support for efficiently serializing and deserializing
Numpy arrays.Code of ConductThe Blosc community has adopted aCode of Conductthat we expect project participants
to adhere to. Please read the full text so that you can understand what actions
will and will not be tolerated.DependenciesPython 2.7, 3.4, 3.5, 3.6 or 3.7python-blosc(provides Blosc) andNumpy(as listed inrequirements.txt) for
running the codeThe Python packages listed intest_requirements.txtfor testing and
releasingStability of File FormatThe tool is considered alpha-stage, experimental, research software. It is not
unlikely thatthe internal storage format for the compressed files will
change in future. Pleasedo not depend critically on the files generated
(unless you know what you are doing)by Bloscpack. See the warranty disclaimer
in the licence at the end of this file.InstallationDisclaimer: There are a myriad ways of installing Python packages (and their
dependencies) these days and it is a futile endeavour to explain the procedures
in great detail again and again. Below are three methods that are known to
work. Depending on the method you choose and the system your are using you may
require any or all of: super user privileges, a C++ compiler and/or a virtual
environment. If you do run into problems or are unsure, it’s best to send an
email to the aforementioned mailing list asking for help.The package is available on PyPi, so you may use pip to install the
dependencies and bloscpack itself:$pipinstallbloscpackIf you want to install straight from GitHub, use pip’s VCS support:$pipinstallgit+https://github.com/Blosc/bloscpackOr, of course, download the source code or clone the repository and then use
the standardsetup.py:$gitclonehttps://github.com/Blosc/bloscpack$cdbloscpack$pythonsetup.pyinstallUsageBloscpack is accessible from the command line using theblpkexecutable
this has a number of global options and four subcommands:[c | compress],[d | decompress],[a | append]and[i | info]most of which each
have their own options.Help for global options and subcommands:$blpk--help[...]Help for each one of the subcommands:$blpkcompress--help[...]$blpkdecompress--help[...]$blpkinfo--help[...]$blpkappend--help[...]ExamplesBasicsBasic compression:$blpkcompressdata.datOr:$blpkcdata.dat… will compress the filedata.dattodata.dat.blpBasic decompression:$blpkdecompressdata.dat.blpdata.dcmpOr:$blpkddata.dat.blpdata.dcmp… will decompress the filedata.dat.blpto the filedata.dcmp. If you
leave out the[<out_file>]argument, Bloscpack will complain that the filedata.datexists already and refuse to overwrite it:$blpkdecompressdata.dat.blpblpk: error: output file 'data.dat' exists!If you know what you are doing, you can use the global option[-f|--force]to override the overwrite checks:$blpk--forcedecompressdata.dat.blpIncidentally this works for compression too:$blpkcompressdata.datblpk: error: output file 'data.dat.blp' exists!$blpk--forcecompressdata.datLastly, if you want a different filename:$blpkcompressdata.datcustom.filename.blp… will compress the filedata.dattocustom.filename.blpSettingsBy default, the number of threads that Blosc uses during compression and
decompression is determined by the number of cores detected on your system.
You can change this using the[-n|--nthreads]option:$blpk--nthreads1compressdata.datCompression with Blosc is controlled with the following options:[-t|--typesize]Typesize used by Blosc (default: 8):$ blpk compress--typesize8 data.dat[-l|--level]Compression level (default: 7):$ blpk compress--level3 data.dat[-s|--no-shuffle]Deactivate shuffle:$ blpk compress--no-shuffledata.dat[-c|--codec]Use alternative codec:$ blpk compress--codeclz4 data.datIn addition, there are the following options that control the Bloscpack file:[-z|--chunk-size]Desired approximate size of the chunks, where you can use human readable
strings like8Mor128Kormaxto use the maximum chunk size of
apprx.2GB(default:1MB):$ blpk compress--chunk-size128K data.dator$ blpk c-zmax data.dat[-k|--checksum<checksum>]Chose which checksum to use. The following values are permissible:None,adler32,crc32,md5,sha1,sha224,sha256,sha384,sha512, (default:adler32). As described in the header format, each
compressed chunk can be stored with a checksum, which aids corruption
detection on decompression:$ blpk compress--checksumcrc32 data.dat[-o|--no-offsets]By default, offsets to the individual chunks are stored. These are included
to allow for partial decompression in the future. This option disables that
feature. Also, a certain number of offsets (default: 10 * ‘nchunks’) are
preallocated to allow for appending data to the file:$ blpk compress--no-offsetsdata.datInfo SubcommandIf you just need some info on how the file was compressed[i | info]:$blpkinfodata.dat.blpblpk: BloscpackHeader:
blpk:     format_version: 3
blpk:     offsets: True
blpk:     metadata: False
blpk:     checksum: 'adler32'
blpk:     typesize: 8
blpk:     chunk_size: 1.0M (1048576B)
blpk:     last_chunk: 900.0K (921600B)
blpk:     nchunks: 1526
blpk:     max_app_chunks: 15260
blpk: 'offsets':
blpk: [134320,459218,735869,986505,1237646,...]
blpk: First chunk blosc header:
blpk: OrderedDict([('version', 2), ('versionlz', 1), ('flags', 1), ('typesize', 8), ('nbytes', 1048576), ('blocksize', 131072), ('ctbytes', 324894)])
blpk: First chunk blosc flags:
blpk: OrderedDict([('byte_shuffle', True), ('pure_memcpy', False), ('bit_shuffle', False), ('split_blocks', False), ('codec', 'blosclz')])Importantly, the header and flag information are for the first chunk only.
Usually this isn’t a problem because bloscpack compressed files do tend to have
homogeneous settings like codec used, typesize etc… However, there is nothing
that will stop you from appending to an existing bloscpack file using different
settings. For example, half the file might be compressed using ‘blosclz’
whereas the other half of the file might be compressed with ‘lz4’. In any case,
just be aware that the output is to be seen as an indication that is likely to
be correct for all chunks but must not be so necessarily.Adding MetdataUsing the[-m|--metadata]option you can include JSON from a file:$catmeta.json{"dtype": "float64", "shape": [200000000], "container": "numpy"}$blpkcompress--chunk-size=512M--metadatameta.jsondata.dat$blpkinfodata.dat.blpblpk: BloscpackHeader:
blpk:     format_version: 3
blpk:     offsets: True
blpk:     metadata: True
blpk:     checksum: 'adler32'
blpk:     typesize: 8
blpk:     chunk_size: 512.0M (536870912B)
blpk:     last_chunk: 501.88M (526258176B)
blpk:     nchunks: 3
blpk:     max_app_chunks: 30
blpk: 'offsets':
blpk: [922,78074943,140783242,...]
blpk: 'metadata':
blpk: {   u'container': u'numpy', u'dtype': u'float64', u'shape': [200000000]}
blpk: MetadataHeader:
blpk:     magic_format: 'JSON'
blpk:     meta_options: '00000000'
blpk:     meta_checksum: 'adler32'
blpk:     meta_codec: 'zlib'
blpk:     meta_level: 6
blpk:     meta_size: 59.0B (59B)
blpk:     max_meta_size: 590.0B (590B)
blpk:     meta_comp_size: 58.0B (58B)
blpk:     user_codec: ''It will be printed when decompressing:$blpkdecompressdata.dat.blpblpk: Metadata is:
blpk: '{u'dtype': u'float64', u'shape': [200000000], u'container': u'numpy'}'AppendingYou can also append data to an existing bloscpack compressed file:$blpkappenddata.dat.blpdata.datHowever there are certain limitations on the amount of data can be appended.
For example, if there is an offsets section, there must be enough room to store
the offsets for the appended chunks. If no offsets exists, you may append as
much data as possible given the limitations governed by the maximum number of
chunks and the chunk-size. Additionally, there are limitations on the
compression options. For example, one cannot change the checksum used. It is
however possible to change the compression level, the typesize and the shuffle
option for the appended chunks.Also note that appending is still considered experimental as ofv0.5.0.Verbose and Debug modeLastly there are two mutually exclusive options to control how much output is
produced.The first causes basic info to be printed,[-v|--verbose]:$blpk--verbosecompress--chunk-size0.5Gdata.datblpk: using 4 threads
blpk: getting ready for compression
blpk: input file is: 'data.dat'
blpk: output file is: 'data.dat.blp'
blpk: input file size: 1.49G (1600000000B)
blpk: nchunks: 3
blpk: chunk_size: 512.0M (536870912B)
blpk: last_chunk_size: 501.88M (526258176B)
blpk: output file size: 198.39M (208028617B)
blpk: compression ratio: 7.691250
blpk: done… and[-d|--debug]prints a detailed account of what is going on:$blpk--debugcompress--chunk-size0.5Gdata.datblpk: command line argument parsing complete
blpk: command line arguments are:
blpk:     force: False
blpk:     verbose: False
blpk:     offsets: True
blpk:     checksum: adler32
blpk:     subcommand: compress
blpk:     out_file: None
blpk:     metadata: None
blpk:     cname: blosclz
blpk:     in_file: data.dat
blpk:     chunk_size: 536870912
blpk:     debug: True
blpk:     shuffle: True
blpk:     typesize: 8
blpk:     clevel: 7
blpk:     nthreads: 4
blpk: using 4 threads
blpk: getting ready for compression
blpk: input file is: 'data.dat'
blpk: output file is: 'data.dat.blp'
blpk: input file size: 1.49G (1600000000B)
blpk: nchunks: 3
blpk: chunk_size: 512.0M (536870912B)
blpk: last_chunk_size: 501.88M (526258176B)
blpk: BloscArgs:
blpk:     typesize: 8
blpk:     clevel: 7
blpk:     shuffle: True
blpk:     cname: 'blosclz'
blpk: BloscpackArgs:
blpk:     offsets: True
blpk:     checksum: 'adler32'
blpk:     max_app_chunks: <function <lambda> at 0x1182de8>
blpk: metadata_args will be silently ignored
blpk: max_app_chunks is a callable
blpk: max_app_chunks was set to: 30
blpk: BloscpackHeader:
blpk:     format_version: 3
blpk:     offsets: True
blpk:     metadata: False
blpk:     checksum: 'adler32'
blpk:     typesize: 8
blpk:     chunk_size: 512.0M (536870912B)
blpk:     last_chunk: 501.88M (526258176B)
blpk:     nchunks: 3
blpk:     max_app_chunks: 30
blpk: raw_bloscpack_header: 'blpk\x03\x01\x01\x08\x00\x00\x00 \x00\x10^\x1f\x03\x00\x00\x00\x00\x00\x00\x00\x1e\x00\x00\x00\x00\x00\x00\x00'
blpk: Handle chunk '0'
blpk: checksum (adler32): '\x1f\xed\x1e\xf4'
blpk: chunk handled, in: 512.0M (536870912B) out: 74.46M (78074017B)
blpk: Handle chunk '1'
blpk: checksum (adler32): ')\x1e\x08\x88'
blpk: chunk handled, in: 512.0M (536870912B) out: 59.8M (62708295B)
blpk: Handle chunk '2' (last)
blpk: checksum (adler32): '\xe8\x18\xa4\xac'
blpk: chunk handled, in: 501.88M (526258176B) out: 64.13M (67245997B)
blpk: Writing '3' offsets: '[296, 78074317, 140782616]'
blpk: Raw offsets: '(\x01\x00\x00\x00\x00\x00\x00\xcdQ\xa7\x04\x00\x00\x00\x00\x18,d\x08\x00\x00\x00\x00'
blpk: output file size: 198.39M (208028617B)
blpk: compression ratio: 7.691250
blpk: donePython APIBloscpack has a versatile yet simple API consisting of a series of ‘arguments’
objects and high-level functions that can be invoked dependding on your input
and output needs.Nomenclature wise, Python 3 has done a lot for Bloscpack, because we always
need to represent compressed data as bytes deliberatey. This makes it easier
and more natural to distinguish between text, such a filenames and binary and
bytes objects such as compressed data.ArgumentsThe three argument types are:BloscArgsBloscpackArgsMetadataArgsas defined inbloscpack/args.py.  Instantiating any of them will create an
object with the defaults setup. The defaults are defined inbloscpack/defaults.py. You can use these in the high-level functions listed
below.You can override any and all defaults by passing in the respective
keyword-arguments, for example:>>>b=BloscArgs()# will create a default args object>>>b=BloscArgs(clevel=4)# change compression level to 4>>>b=BloscArgs(typesize=4,# change the typesize to 4>>>...clevel=9,# change the compression level to 9>>>...shuffle=False,# deactivate the shuffle filter>>>...cname='lz4')# let lz4 be the internal codecclassBloscArgs(MutableMappingObject):""" Object to hold Blosc arguments.

    Parameters
    ----------
    typesize : int
        The typesize used
    clevel : int
        Compression level
    shuffle : boolean
        Whether or not to activate the shuffle filter
    cname: str
        Name of the internal code to use

    """classBloscpackArgs(MutableMappingObject):""" Object to hold BloscPack arguments.

    Parameters
    ----------
    offsets : boolean
        Whether to include space for offsets
    checksum : str
        Name of the checksum to use or None/'None'
    max_app_chunks : int or callable on number of chunks
        How much space to reserve in the offsets for chunks to be appended.

    """classMetadataArgs(MutableMappingObject):""" Object to hold the metadata arguments.

    Parameters
    ----------
    magic_format : 8 bytes
        Format identifier for the metadata
    meta_checksum : str
        Checksum to be used for the metadata
    meta_codec : str
        Codec to be used to compress the metadata
    meta_level : int
        Compression level for metadata
    max_meta_size : int or callable on metadata size
        How much space to reserve for additional metadata

    """File / BytesThe following high-level functions exist for compressing and decompressing to
and from files and byte objects:pack_file_to_fileunpack_file_from_filepack_bytes_to_fileunpack_bytes_from_filepack_bytes_to_bytesunpack_bytes_from_bytesBeyond the target arguments such as the files and the bytes, eachpack_*function takes the following arguments:chunk_size : int
    the desired chunk size in bytes
metadata : dict
    the metadata dict
blosc_args : BloscArgs
    blosc args
bloscpack_args : BloscpackArgs
    bloscpack args
metadata_args : MetadataArgs
    metadata argsBelow are their sigantures:defpack_file_to_file(in_file,out_file,chunk_size=DEFAULT_CHUNK_SIZE,metadata=None,blosc_args=None,bloscpack_args=None,metadata_args=None):defunpack_file_from_file(in_file,out_file):defpack_bytes_to_file(bytes_,out_file,chunk_size=DEFAULT_CHUNK_SIZE,metadata=None,blosc_args=None,bloscpack_args=None,metadata_args=None):defunpack_bytes_from_file(compressed_file):defpack_bytes_to_bytes(bytes_,chunk_size=DEFAULT_CHUNK_SIZE,metadata=None,blosc_args=None,bloscpack_args=None,metadata_args=None,):defunpack_bytes_from_bytes(bytes_):NumpyNumpy arrays can be serialized as Bloscpack files, here is a very brief example:>>>a=np.linspace(0,1,3e8)>>>printa.size,a.dtype300000000 float64>>>bp.pack_ndarray_to_file(a,'a.blp')>>>b=bp.unpack_ndarray_from_file('a.blp')>>>(a==b).all()TrueLooking at the generated file, we can see the Numpy metadata being saved:$lha.blp-rw------- 1 esc esc 266M Aug 13 23:21 a.blp$blpkinfoa.blpblpk: BloscpackHeader:
blpk:     format_version: 3
blpk:     offsets: True
blpk:     metadata: True
blpk:     checksum: 'adler32'
blpk:     typesize: 8
blpk:     chunk_size: 1.0M (1048576B)
blpk:     last_chunk: 838.0K (858112B)
blpk:     nchunks: 2289
blpk:     max_app_chunks: 22890
blpk: 'offsets':
blpk: [202170,408064,554912,690452,819679,...]
blpk: 'metadata':
blpk: {   u'container': u'numpy',
blpk:     u'dtype': u'<f8',
blpk:     u'order': u'C',
blpk:     u'shape': [300000000]}
blpk: MetadataHeader:
blpk:     magic_format: 'JSON'
blpk:     meta_options: '00000000'
blpk:     meta_checksum: 'adler32'
blpk:     meta_codec: 'zlib'
blpk:     meta_level: 6
blpk:     meta_size: 67.0B (67B)
blpk:     max_meta_size: 670.0B (670B)
blpk:     meta_comp_size: 62.0B (62B)
blpk:     user_codec: ''Alternatively, we can also use a string as storage:>>>a=np.linspace(0,1,3e8)>>>c=pack_ndarray_to_bytes(a)>>>b=unpack_ndarray_from_bytes(c)>>>(a==b).all()TrueOr use alternate compressors:>>>a=np.linspace(0,1,3e8)>>>c=pack_ndarray_to_bytes(a,blosc_args=BloscArgs(cname='lz4'))>>>b=unpack_ndarray_from_bytes(c)>>>(a==b).all()Truedefpack_ndarray_to_file(ndarray,filename,chunk_size=DEFAULT_CHUNK_SIZE,blosc_args=None,bloscpack_args=None,metadata_args=None):defpack_ndarray_to_bytes(ndarray,chunk_size=DEFAULT_CHUNK_SIZE,blosc_args=None,bloscpack_args=None,metadata_args=None):defunpack_ndarray_from_file(filename):defunpack_ndarray_from_bytes(str_):If you are interested in the performance of Bloscpack compared to other
serialization formats for Numpy arrays, please look at the benchmarks presented
inthe Bloscpack paper from the EuroScipy 2013 conference proceedings.TestingInstalling DependenciesTesting requires some additional libraries, which you can install from PyPi
with:$pipinstall-rtest_requirements.txt[...]Basic TestsBasic tests, runs quickly:$nosetests[...]Heavier TestsExtended tests using a larger file, may take some time, but will be nice to
memory:$noseteststest/test_file_io.py:pack_unpack_hard[...]Extended tests using a huge file. This one take forever and needs loads (5G-6G)
of memory and loads of disk-space (10G). Use-sto print progress:$nosetests-stest/test_file_io.py:pack_unpack_extreme[...]Note that, some compression/decompression tests create temporary files (on
UNIXoid systems this is under/tmp/blpk*) which are deleted upon completion
of the respective test, both successful and unsuccessful, or when the test is
aborted with e.g.ctrl-c(usingatexitmagic).Under rare circumstances, for example when aborting the deletion which is
triggered on abort you may be left with large files polluting your temporary
space.  Depending on your partitioning scheme etc.. doing this repeatedly, may
lead to you running out of space on the file-system.Command Line Interface TestsThe command line interface is tested withcram:$cram--verbosetest_cmdline/*.cram[...]CoverageTo determine coverage you can pool together the coverage from the cram tests and
the unit tests:$COVERAGE=1cram--verbosetest_cmdline/*.cram[...]$nosetests--with-coverage--cover-package=bloscpack[...]Test RunnerTo run the command line interface tests and the unit tests and analyse
coverage, use the conveniencetest.shrunner:$./test.sh[...]BenchmarkUsing the providedbench/blpk_vs_gzip.pyscript on aIntel(R) Core(TM)i7-3667UCPU @ 2.00GHzCPU with 2 cores and 4 threads (active
hyperthreading), cpu frequency scaling activated but set to theperformancegovernor (all cores scaled to2.0 GHz), 8GB of DDR3 memory and a Luks encrypted
SSD, we get:$PYTHONPATH=../bench/blpk_vs_gzip.pycreate the test data..........done

Input file size: 1.49G
Will now run bloscpack...
Time: 2.06 seconds
Output file size: 198.55M
Ratio: 7.69
Will now run gzip...
Time: 134.20 seconds
Output file size: 924.05M
Ratio: 1.65As was expected from previous benchmarks of Blosc using the python-blosc
bindings, Blosc is both much faster and has a better compression ratio for this
kind of structured data. One thing to note here, is that we are not dropping
the system file cache after every step, so the file to read will be cached in
memory. To get a more accurate picture we can use the--drop-cachesswitch
of the benchmark which requires you however, to run the benchmark as root,
since dropping the caches requires root privileges:$PYTHONPATH=../bench/blpk_vs_gzip.py--drop-cacheswill drop caches
create the test data..........done

Input file size: 1.49G
Will now run bloscpack...
Time: 13.49 seconds
Output file size: 198.55M
Ratio: 7.69
Will now run gzip...
Time: 137.49 seconds
Output file size: 924.05M
Ratio: 1.65Optimizing Chunk SizeYou can use the providedbench/compression_time_vs_chunk_size.pyfile
to optimize the chunk-size for a given machine. For example:$sudoenvPATH=$PATHPYTHONPATH=.bench/compression_time_vs_chunk_size.pycreate the test data..........done
chunk_size    comp-time       decomp-time      ratio
512.0K        8.106235        10.243908        7.679094
724.08K       4.424007        12.284307        7.092846
1.0M          6.243544        11.978932        7.685173
1.41M         4.715511        10.780901        7.596981
2.0M          4.548568        10.676304        7.688216
2.83M         4.851359        11.668394        7.572480
4.0M          4.557665        10.127647        7.689736
5.66M         4.589349        9.579627         7.667467
8.0M          5.290080        10.525652        7.690499Running the script requires super user privileges, since you need to
synchronize disk writes and drop the file system caches for less noisy results.
Also, you should probably run this script a couple of times and inspect the
variability of the results.Bloscpack FormatThe input is split into chunks since a) we wish to put less stress on main
memory and b) because Blosc has a buffer limit of2GB(Version1.0.0and
above). By default the chunk-size is a moderate1MBwhich should be fine,
even for less powerful machines.In addition to the chunks some additional information must be added to the file
for housekeeping:header:a 32 bit header containing various pieces of informationmeta:a variable length metadata section, may contain user dataoffsets:a variable length section containing chunk offsetschunk:the blosc chunk(s)checksum:a checksum following each chunk, if desiredThe layout of the file is then:|-header-|-meta-|-offsets-|-chunk-|-checksum-|-chunk-|-checksum-|...|Description of the headerThe following 32 bit header is used for Bloscpack as of version0.3.0.  The
design goals of the header format are to contain as much information as
possible to achieve interesting things in the future and to be as general as
possible such that the persistence layer ofBlaze/BLZcan be implemented without
modification of the header format.The following ASCII representation shows the layout of the header:|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
| b   l   p   k | ^ | ^ | ^ | ^ |   chunk-size  |  last-chunk   |
                  |   |   |   |
      version ----+   |   |   |
      options --------+   |   |
     checksum ------------+   |
     typesize ----------------+

|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
|            nchunks            |        max-app-chunks         |The first 4 bytes are the magic stringblpk. Then there are 4 bytes which
hold information about the activated features in this file.  This is followed
by 4 bytes for thechunk-size, another 4 bytes for thelast-chunk-size,
8 bytes for the number of chunks,nchunksand lastly 8 bytes for the total
number of chunks that can be appended to this file,max-app-chunks.Effectively, storing the number of chunks as a signed 8 byte integer, limits
the number of chunks to2**63-1= 9223372036854775807, but this should not
be relevant in practice, since, even with the moderate default value of1MBfor chunk-size, we can still store files as large as8ZB(!) Given that
in 2012 the maximum size of a single file in the Zettabye File System (zfs) is16EB, Bloscpack should be safe for a few more years.Description of the header entriesAll entries are little-endian.version:(uint8)
format version of the Bloscpack header, to ensure exceptions in case of
forward incompatibilities.options:(bitfield)
A bitfield which allows for setting certain options in this file.bit 0 (0x01):If the offsets to the chunks are present in this file.bit 1 (0x02):If metadata is present in this file.checksum:(uint8)
The checksum used. The following checksums, available in the python
standard library should be supported. The checksum is always computed on
the compressed data and placed after the chunk.0:no checksum1:zlib.adler322:zlib.crc323:hashlib.md54:hashlib.sha15:hashlib.sha2246:hashlib.sha2567:hashlib.sha3848:hashlib.sha512typesize:(uint8)
The typesize of the data in the chunks. Currently, assume that the typesize
is uniform. The space allocated is the same as in the Blosc header.chunk-size:(int32)
Denotes the chunk-size. Since the maximum buffer size of Blosc is 2GB
having a signed 32 bit int is enough (2GB =2**31bytes). The special
value of-1denotes that the chunk-size is unknown or possibly
non-uniform.last-chunk:(int32)
Denotes the size of the last chunk. As with thechunk-sizeanint32is enough. Again,-1denotes that this value is unknown.nchunks:(int64)
The total number of chunks used in the file. Given a chunk-size of one
byte, the total number of chunks is2**63. This amounts to a maximum
file-size of 8EB (8EB = 2*63 bytes) which should be enough for the next
couple of years. Again,-1denotes that the number of is unknown.max-app-chunks:(int64)
The maximum number of chunks that can be appended to this file, excludingnchunks. This is only useful if there is an offsets section and if
nchunks is known (not-1), if either of these conditions do not apply
this should be0.The overall file-size can be computed aschunk-size* (nchunks - 1) +last-chunk-size. In a streaming scenario-1can be used as a placeholder.
For example if the total number of chunks, or the size of the last chunk is not
known at the time the header is created.The following constraints exist on the header entries:last-chunkmust be less than or equal tochunk-size.nchunks + max_app_chunksmust be less than or equal to the maximum value
of anint64.Description of the metadata sectionThis section goes after the header. It consists of a metadata-section header
followed by a serialized and potentially compressed data section, followed by
preallocated space to resize the data section, possibly followed by a checksum.The layout of the section is thus:|-metadata-header-|-data-|-prealloc-|-checksum-|The header has the following layout:|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
|         magic-format          | ^ | ^ | ^ | ^ |   meta-size   |
                                  |   |   |   |
              meta-options -------+   |   |   |
              meta-checksum ----------+   |   |
              meta-codec -----------------+   |
              meta-level ---------------------+

|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
| max-meta-size |meta-comp-size |            user-codec         |magic-format:(8 byte ASCII string)
The data will usually be some kind of binary serialized string data, for
exampleJSON,BSON,YAMLor Protocol-Buffers. The format
identifier is to be placed in this field.meta-options:(bitfield)
A bitfield which allows for setting certain options in this metadata
section. Currently unusedmeta-checksum:The checksum used for the metadata. The same checksums as for the data are
available.meta-codec:(unit8)
The codec used for compressing the metadata. As of Bloscpack version0.3.0the following codecs are supported.0:no codec1:zlib(DEFLATE)meta-level:(unit8)
The compression level used for the codec. Ifcodecis0i.e. the
metadata is not compressed, this must be0too.meta-size:(uint32)
The size of the uncompressed metadata.max-meta-size:(uint32)
The total allocated space for the data section.meta-comp-size:(uint32)
If the metadata is compressed, this gives the total space the metadata
occupies. If the data is not compressed this is the same asmeta-size.
In a sense this is the true amount of space in the metadata section that is
used.user-codec:Space reserved for usage of additional codecs. E.g. 4 byte magic string for
codec identification and 4 bytes for encoding of codec parameters.The total space left for enlarging the metadata section is simply:max-meta-size-meta-comp-size.JSON Example of serialized metadata:'{"dtype": "float64", "shape": [1024], "others": []}'If compression is requested, but not beneficial, because the compressed size
would be larger than the uncompressed size, compression of the metadata is
automatically deactivated.As of Bloscpack version0.3.0only the JSON serializer is supported and
used the stringJSONfollowed by four whitespace bytes as identifier.
Since JSON and any other of the suggested serializers has limitations, only a
subset of Python structures can be stored, so probably some additional object
handling must be done prior to serialize certain kinds of metadata.Description of the offsets entriesFollowing the metadata section, comes a variable length section of chunk
offsets. Offsets of the chunks into the file are to be used for accelerated
seeking. The offsets (if activated) follow the header. Each offset is a 64 bit
signed little-endian integer (int64). A value of-1denotes an unknown
offset. Initially, all offsets should be initialized to-1and filled in
after writing all chunks. Thus, If the compression of the file fails
prematurely or is aborted, all offsets should have the value-1.  Also, any
unused offset entries preallocated to allow the file to grow should be set to-1. Each offset denotes the exact position of the chunk in the file such
that seeking to the offset, will position the file pointer such that, reading
the next 16 bytes gives the Blosc header, which is at the start of the desired
chunk.Description of the chunk formatAs mentioned previously, each chunk is just a Blosc compressed string including
header. The Blosc header (as ofv1.0.0) is 16 bytes as follows:|-0-|-1-|-2-|-3-|-4-|-5-|-6-|-7-|-8-|-9-|-A-|-B-|-C-|-D-|-E-|-F-|
  ^   ^   ^   ^ |     nbytes    |   blocksize   |    ctbytes    |
  |   |   |   |
  |   |   |   +--typesize
  |   |   +------flags
  |   +----------versionlz
  +--------------versionThe first four are simply bytes, the last three are are each unsigned ints
(uint32) each occupying 4 bytes. The header is always little-endian.ctbytesis the length of the buffer including header andnbytesis the
length of the data when uncompressed. A more detailed description of the Blosc
header can be found in theREADME_HEADER.rst of the Blosc repositoryOverheadDepending on which configuration for the file is used a constant, or linear
overhead may be added to the file. The Bloscpack header adds 32 bytes in any
case. If the data is non-compressible, Blosc will add 16 bytes of header to
each chunk. The metadata section obviously adds a constant overhead, and if
used, both the checksum and the offsets will add overhead to the file. The
offsets add 8 bytes per chunk and the checksum adds a fixed constant value
which depends on the checksum to each chunk. For example, 32 bytes for theadler32checksum.Coding ConventionsNumpy rst style docstringsREADME cli examples should use long optionstesting: expected before receivednt.assert_equal(expected, received)Debug messages: as close to where the data was generatedSingle quotes around ambiguities in messagesoverwriting existing file: 'testfile'Exceptions instead of exitnose test generators parameterized testsUse the Wikipedia definition of compression ratio:http://en.wikipedia.org/wiki/Data_compression_ratioHow to Optimize LoggingSome care must be taken when logging in the inner loop. For example consider the
following two commits:https://github.com/Blosc/bloscpack/commit/0854930514eebaf7dbc6c4dcf3589dbcb9f2fdc9https://github.com/Blosc/bloscpack/commit/355bf90a8c13a2a1f792d43228c2a68c61476621If there are a larger number of chunks, calls todouble_pretty_sizewill be
executed (and may be costly)evenif no logging is needed.Consider the following script,loop-bench.py:importnumpyasnpimportbloscpackasbpimportbloscshuffle=Trueclevel=9cname='lz4'a=np.arange(2.5e8)bargs=bp.args.BloscArgs(clevel=clevel,shuffle=shuffle,cname=cname)bpargs=bp.BloscpackArgs(offsets=False,checksum='None',max_app_chunks=0)Timing withv0.7.0:In [1]: %run loop-bench.py

In [2]: %timeit bpc = bp.pack_ndarray_str(a, blosc_args=bargs, bloscpack_args=bpargs)
1 loops, best of 3: 423 ms per loop

In [3]: %timeit bpc = bp.pack_ndarray_str(a, blosc_args=bargs, bloscpack_args=bpargs)
1 loops, best of 3: 421 ms per loop

In [4]: bpc = bp.pack_ndarray_str(a, blosc_args=bargs, bloscpack_args=bpargs)

In [5]: %timeit a3 = bp.unpack_ndarray_str(bpc)
1 loops, best of 3: 727 ms per loop

In [6]: %timeit a3 = bp.unpack_ndarray_str(bpc)
1 loops, best of 3: 725 ms per loopAnd then using a development version that contains the two optimization commits:In [1]: %run loop-bench.py

In [2]: %timeit bpc = bp.pack_ndarray_str(a, blosc_args=bargs, bloscpack_args=bpargs)
1 loops, best of 3: 357 ms per loop

In [3]: %timeit bpc = bp.pack_ndarray_str(a, blosc_args=bargs, bloscpack_args=bpargs)
1 loops, best of 3: 357 ms per loop

In [4]: bpc = bp.pack_ndarray_str(a, blosc_args=bargs, bloscpack_args=bpargs)

In [5]: %timeit a3 = bp.unpack_ndarray_str(bpc)
1 loops, best of 3: 658 ms per loop

In [6]: %timeit a3 = bp.unpack_ndarray_str(bpc)
1 loops, best of 3: 655 ms per loopComparison to HDF5/PyTablesSince Blosc has already been supported for use in HDF5 files from within
PyTables, one might be tempted to question why yet another file format has to
be invented. This section aims to differentiate between HDF5/PyTables and
effectively argues that they are not competitors.Lightweight vs. Heavyweight. Bloscpack is a lightweight format. The format
specification can easily be digested within a day and the dependencies are
minimal. PyTables is a complex piece of software and the HDF5 file format
specification is a large document.Persistence vs. Database. Bloscpack is designed to allow for fast
serialization and deserialization of in-memory data. PyTables is more of a
database which for example allows complex queries to be computed on the
data.Additionally there are two network uses cases which Bloscpack is suited for
(but does not have support for as of yet):Streaming: Since bloscpack without offsets can be written in a single
pass it is ideally suited for streaming over a network, where you can
compress send and decompress individual chunks in a streaming fashion.Expose a file over HTTP and do partial reads from it, for example when
storing a compressed file in S3. You can easily just store a file on a
web server and then use the header information to read and decompress
individual chunks.Prior ArtThe following is a  list of important resources that were read during the
conception and initial stages of Bloscpack.The6pack utility included with FastLZ(the codec that
BloscLZ was derived from) was the initial inspiration for writing a command
line interface to Blosc.TheWikipedia article on the PNG formatcontains some
interesting details about the PNG header and file headers in general.TheXZ File Format Specificationgave rise to some ideas and
techniques about writing file format specifications and using checksums for
data integrity. Although the format and the document itself was a bit to
heavyweight for my tastes.TheSnappy framing formatand thefile container format for LZ4were also consulted, but I can’t remember if and what inspiration they gave
rise to.The homepages ofzlibandgzipwere also consulted at some point. The command line
interface ofgzip/gunzipwas deemed to be from a different era and as a
result git-style subcommands are used in Bloscpack.Resources and Related PublicationsMain Blosc websiteFrancesc Alted. *The Data Access Problem* EuroScipy 2009 Keynote PresentationFrancesc Alted. *Why modern CPUs are starving and what can be done about it*, Computing in Science & Engineering, Vol. 12, No. 2. (March 2010), pp. 68-71Francesc Alted: Sending Data from Memory to CPU (and back) faster than memcpy(). PyData London 2014slides0video0The Blosc Github organizationValentin Haenel. *Introducing Bloscpack* EuroScipy 2013 PresentationValentin Haenel. *Bloscpack: a compressed lightweight serialization format for numerical data*. Proceedings of the 6th European Conference on Python in Science (EuroSciPy 2013).Valentin Haenel.Fast Serialization of Numpy Arrays with Bloscpack. PyData Berlin 2014slides1,video1Maintainers Notes on Cutting a ReleaseSet the version as environment variableVERSION=vX.X.XUpdate the changelog andANNOUNCE.rstCommit usinggit commit-m"$VERSION changelog and ANNOUNCE.rst"Set the version number inbloscpack/version.pyCommit withgit commit-m"$VERSION"Make the tag usinggit tag-s-m"Bloscpack $VERSION" $VERSIONPush commits to Blosc githubgit push blosc masterPush commits to own githubgit push esc masterPush the tag to Blosc githubgit push blosc $VERSIONPush the tag to own githubgit push esc $VERSIONMake a source distribution usingpython setup.py sdist bdist_wheelUpload to PyPi usingtwine uploaddist/bloscpack-$VERSION*Bump version number to next dev version and resetANNOUNCE.rstAnnounce release on the Blosc listAnnounce release via TwitterTODODocumentationRefactor monolithic readme into Sphinx and publishCleanup and double check the docstrings for the public API classesCommand Linequiet verbosity levelExpose the ability to set ‘max_app_chunks’ from the command lineAllow to save metadata to a file during decompressionsubcommand e or estimate to estimate the size of the uncompressed data.subcommand v or verify to verify the integrity of the dataadd –raw-input and –raw-output switches to allow stuff like:
cat file | blpk –raw-input –raw-output compress > file.blpEstablish and document proper exit codesDocument the metadata saved during Numpy serializationProfiling and OptimizationUse the faster version of struct where you have a single stringMemory profiler, might be able to reduce memory used by reusing the buffer
during compression and decompressionBenchmark different codecsUse line profiler to check codeSelect different defaults for Numpy arrays, no offsets? no pre-alloc?Library Featurespossibly provide a BloscPackFile abstraction, like GzipFileAllow to not-prealloc additional space for metadataRefactor certain collections of functions that operate on data into objectsOffsets (maybe)partial decompression?since we now have potentially small chunks, the progressbar becomes relevant
againconfiguration file to store commonly used options on a given machineprint the compression time, either as verbose or debugInvestigate if we can use a StringIO object that returns memoryviews on read.Implement a memoryview Compressed/PlainSourceUse a bytearray to read chunks from a file. Then re-use that bytearray
during every read to avoid allocating deallocating strings the whole time.The keyword arguments to many functions are global dicts, this is a bad idea,
Make the immutable with a forzendict.Check that the checksum is really being checked for all PlainSinksBunch of NetworkSource/SinksHTTPSource/SinkMiscellaneousAnnounce on scipy/numpy lists, comp.compression, freshmeat, ohloh …Packaging and InfrastructureDebian packages (for python-blosc and bloscpack)Conda recipes (for python-blosc and bloscpack)Use tox for testing multiple python versionsBuild on travis and drone.io using pre-compiledChangelogv0.16.0     - Thu 27 Dec 2018Update of Python API and docsvarious minor fixesv0.15.0     - Wed 31 Oct 2018Halloween Release!Adding the Blosc code of conduct (#79 by @esc)Two new high-level functions: ‘pack_bytes_to_bytes’ and
‘unpack_bytes_from_bytes’ (#83 by @esc)Fix incorrect check for typesize-chunksize mismatch (#81 by @esc)Fix test to append without shuffle (#82 by @esc)Fix tests to respect snappy not being available by default (#85 by @esc)Fix tests to account for new default blocksize (#86 by @esc)Enable testing on Python 3.7 via Travis (#84 by @esc)v0.14.0     - Thu Oct 18 2018Remove official support for Python 2.6 (#77 by @esc)v0.13.0     - Thu May 24 2018Add license file and include in sdist packages (#75 by @toddrme2178)Print codec on info (#73 by @esc)Decode Blosc flags (#72 by @esc)Fix an embarrassing typo (#71 by @esc)Test zstd (#70 by @esc)Document args object (#69 by @esc)Various pep8 fixes by @escSupport for uploading wheels and using twine by @escFix use of coverage by @escBetter support for Python 2.6 by @escv0.12.0     - Fri Mar 09 2018Allow Pythonic None as checksum (#60 by @esc)Fix failing tests to comply with latest Blosc (#63 and #64 by FrancescElies)Support testing with Python 3.6 via Travis (#65 by @esc)Unpinn Blosc in conda recipe (who uses this?) (#61 by @esc)Cleanup README (#66 by @esc)Fix Trove classifiers (#67 by @esc)Random pep8 fixes by @escv0.11.0     - Mon Aug 22 2016Unpinn python-blosc and fix unit-tests (#51 and #57 fixed by @oogali)Improve the computation of the chunksize when it is not divisible by
typesize (#52 by FrancescAlted)v0.10.0     - Thu Dec 10 2015Fix for compressing sliced arrays (#43 reported by @mistycheney)Fixun/pack_bytes_fileto be available from toplevelFix the badges to come (mostly) fromhttps://img.shields.ioFixes for travis-ci, test Python 3.5 tooPin Blosc version to 1.2.7 viarequirements.txtandsetup.pydue to
breakage with Blosc 1.2.8.v0.9.0     - Tue Aug 18 2015Useast.literal_evalinstead ofnp.safe_evalwhich is much faster (#39 @cpcloud)Support for packing/unpacking bytes to/from file (#41)v0.8.0     - Sun Jul 12 2015Python 3.x compatibility (#14)v0.7.3     - Sat Jul 11 2015Fix deserialization of numpy arrays with nested dtypes that were created
with versions v0.7.1 and before. (#37)v0.7.2     - Wed Mar 25 2015Fix support for zero length arrays (and input in general) (#17 reported by @dmbelov)Catch whentypesizedoesn’t dividechunk_size(#18 reported by @dmbelov)Fix serialization of object arrays (#16 reported by @dmbelov)Reject Object dtype arrays since they cannot be compressed with BloscpackProvide backwards compatibility for older Numpy serializationsFix win32 compatibility of tests (#27 fixed by @mindw)Fix using setuptools for scripts and dependencies (#28 fixed by @mindw)Various misc fixesv0.7.1     - Sun Jun 29 2014Fix a bug related to setting the correct typesize when compressing Numpy
arraysOptimization of debug statements in the inner loopsv0.7.0     - Wed May 28 2014Modularize cram tests, even has something akin to a harnessRefactored, tweaked and simplified Source/Sink code and semanticsVarious documentation improvements: listing prior art, comparison to HDF5Improve benchmarking scriptsIntroduce a BloscArgs object for saner handling of the BloscArgsIntroduce a BloscpackArgs object for saner handling of the BloscpackArgsIntroduce MetadataHeader and MetdataArgs objects tooFix all (hopefully) incorrect uses of the term ‘compression ratio’Various miscellaneous fixes and improvementsv0.6.0     - Fri Mar 28 2014Complete refactor of Bloscpack codebase to support modularizationSupport fordrone.ioCI serviceImproved dependency specification for Python 2.6Improved installation instructionsv0.5.2     - Fri Mar 07 2014Fix project url in setup.pyv0.5.1     - Sat Feb 22 2014Documentation fixes and improvementsv0.5.0     - Sun Feb 02 2014Moved project to theBlosc organization on Githubv0.5.0-rc1 - Thu Jan 30 2014Support for Blosc 1.3.x (alternative codecs)v0.4.1     - Fri Sep 27 2013Fixed thepack_unpack_hardtest suiteFixed handling Numpy record and nested record arraysv0.4.0     - Sun Sep 15 2013Fix a bug when serializing numpy arrays to stringsv0.4.0-rc2 - Tue Sep 03 2013Package available via PyPi (since 0.4.0-rc1)Support for packing/unpacking numpy arrays to/from stringCheck that string and record arrays workFix installation problems with PyPi package (Thanks to Olivier Grisel)v0.4.0-rc1 - Sun Aug 18 2013BloscpackHeader class introducedThe info subcommand shows human readable sizes when printing the headerNow using Travis-CI for testing and Coveralls for coverageFurther work on the Plain/Compressed-Source/Sink abstractionsStart using memoryview in placesLearned to serialize Numpy arraysv0.3.0     - Sun Aug 04 2013Minor readme fixesIncrease number of cram testsv0.3.0-rc1 - Thu Aug 01 2013Bloscpack format changes (format version 3)Variable length metadata section with it’s own headerAbility to preallocate offsets for appending data (max_app_chunks)Refactor compression and decompression to use file pointers instead of
file name strings, allows using StringIO/cStringIO.Sanitize calculation of nchunks and chunk-sizeSpecial keywordmaxfor use with chunk-size in the CLISupport appending to a file andappendsubcommand
(including the ability to preallocate offsets)Support rudimentaryinfosubcommandAdd tests of the command line interface usingcramMinor bugfixes and corrections as usualv0.2.1     - Mon Nov 26 2012Backport to Python 2.6Typo fixes in documentationv0.2.0     - Fri Sep 21 2012Useatexitmagic to remove test data on abortChange prefix of temp directory to/tmp/blpk*Merge header RFC into monolithic readmev0.2.0-rc2 - Tue Sep 18 2012Don’t bail out if the file is smaller than default chunkSet the defaulttypesizeto8bytesUpgrade dependencies to python-bloscv1.0.5and fix testsMake extreme test less resource intensiveMinor bugfixes and correctionsv0.2.0-rc1 - Thu Sep 13 2012Implement new header format as described in RFCImplement checksumming compressed chunks with various checksumsImplement offsets of the chunks into the fileEfforts to make the library re-entrant, better control of side-effectsREADME is now rst not md (flirting with sphinx)Tons of trivial fixes, typos, wording, refactoring, renaming, pep8 etc..v0.1.1     - Sun Jul 15 2012Fix the memory issue with the testsTwo new suites:hardandextremeMinor typo fixes and correctionsv0.1.0     - Thu Jun 14 2012Freeze the first 8 bytes of the header (hopefully for ever)Fail to decompress on non-matching format versionMinor typo fixes and correctionsv0.1.0-rc3 - Tue Jun 12 2012Limit the chunk-size benchmark to a narrower rangeAfter more careful experiments, a default chunk-size of1MBwas
deemed most appropriateFixed a terrible bug, where during testing and benchmarking, temporary
files were not removed, oups…Adapted the header to have space for more chunks, include special marker
for unknown chunk number (-1) and format version of the compressed
fileAdded a note in the README about instability of the file formatVarious minor fixes and enhancementsv0.1.0-rc2 - Sat Jun 09 2012Default chunk-size now4MBHuman readable chunk-size argumentLast chunk now contains remainderPure python benchmark to compare against gzipBenchmark to measure the effect of chunk-sizeVarious minor fixes and enhancementsv0.1.0-rc1 - Sun May 27 2012Initial versionCompression/decompressionCommand line argument parserREADME, setup.py, tests and benchmarkThanksFrancesc Alted for writing Blosc in the first place, for providing continual
code-review and feedback on Bloscpack and for co-authoring the Bloscpack
file-format specification.Author, Copyright and License© 2012-2018 Valentin Haenel <valentin@haenel.co>Bloscpack is licensed under the terms of the MIT License.Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the “Software”), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.