Package: indexed-bzip2
indexed_bzip2This module provides an IndexedBzip2File class, which can be used to seek inside bzip2 files without having to decompress them first.
Alternatively, you can use this simply as aparallelizedbzip2 decoder as a replacement for Python's builtinbz2module in order to fully utilize all your cores.On a 12-core processor, this can lead to a speedup of6over Python'sbz2module, seethisexample.
Note that without parallelization,indexed_bzip2is unfortunately slower than Python'sbz2module.
Therefore, it is not recommended when neither seeking nor parallelization is used!The internals are based on an improved version of the bzip2 decoderbzcatfromtoybox, which was refactored and extended to be able to export and import bzip2 block offsets, seek to block offsets, and to add support for threaded parallel decoding of blocks.Seeking inside a block is only emulated, so IndexedBzip2File will only speed up seeking when there are more than one block, which should almost always be the cause for archives larger than 1 MB.Since version 1.2.0, parallel decoding of blocks is supported!
However, per default, the older serial implementation is used.
To use the parallel implementation you need to specify aparallelizationargument other than 1 toIndexedBzip2File, see e.g.this example.Table of ContentsPerformance comparison with bz2 moduleInstallationUsagePython LibraryVia RatarmountCommand Line ToolC++ LibraryInternal ArchitectureTracing the DecoderPerformance comparison with bz2 moduleResults for an AMD Ryzen 3900X 12-core (24 virtual cores) processor and withbz2FilePath='CTU-13-Dataset.tar.bz2', which is a 2GB bz2 compressed archive.ModuleRuntime / sBandwidth / (MB/s)Speedupbz23865.21indexed_bzip2 with parallelization = 06033.26.4indexed_bzip2 with parallelization = 14724.20.8indexed_bzip2 with parallelization = 22657.61.5indexed_bzip2 with parallelization = 69920.23.9indexed_bzip2 with parallelization = 126431.46.1indexed_bzip2 with parallelization = 246331.86.1indexed_bzip2 with parallelization = 326331.76.1The speedup ofindexed_bzip2over thebz2module withparallelization = 0is6.
When using only one core,indexed_bzip2is unfortunately slower by (559-392)/392 = 42%.These are simple timing tests for reading all the contents of a bzip2 file sequentially.importbz2importosimporttimefileSize=os.stat(bz2FilePath).st_sizewithbz2.open(bz2FilePath)asfile:t0=time.time()whilefile.read(512*1024):passbz2Duration=time.time()-t0print(f"Decoded file in{bz2Duration:.0f}s, bandwidth:{fileSize/bz2Duration/1e6:.1f}MB/s")The usage of indexed_bzip2 is slightly different:importindexed_bzip2importtime# parallelization = 0 means that it is automatically using all available cores.forparallelizationin[0,1,2,6,12,24,32]:withindexed_bzip2.IndexedBzip2File(bz2FilePath,parallelization=parallelization)asfile:t0=time.time()# Unfortunately, the chunk size is very performance critical! It might depend on the cache size.whilefile.read(512*1024):passibz2Duration=time.time()-t0print(f"Decoded file in{ibz2Duration:.0f}s"f", bandwidth:{fileSize/ibz2Duration/1e6:.1f}MB/s"f", speedup:{bz2Duration/ibz2Duration:.1f}")InstallationYou can simply install it from PyPI:python3-mpipinstall--upgradepip# Recommended for newer manylinux wheelspython3-mpipinstallindexed_bzip2To install with conda:condainstall-cconda-forgeindexed_bzip2The latest unreleased development version can be tested out with:python3-mpipinstall--force-reinstall'git+https://github.com/mxmlnkn/indexed_bzip2.git@master#egginfo=indexed_bzip2&subdirectory=python/indexed_bzip2'And to build locally, you can usebuildand install the wheel:cdpython/indexed_bzip2
rm-rfdist
python3-mbuild.
python3-mpipinstall--force-reinstall--userdist/*.whlUsagePython LibrarySimple open, seek, read, and closefromindexed_bzip2importIndexedBzip2Filefile=IndexedBzip2File("example.bz2",parallelization=os.cpu_count())# You can now use it like a normal filefile.seek(123)data=file.read(100)file.close()The first call to seek will ensure that the block offset list is complete and therefore might create them first.
Because of this the first call to seek might take a while.Use with context managerimportosimportindexed_bzip2asibz2withibz2.open("example.bz2",parallelization=os.cpu_count())asfile:file.seek(123)data=file.read(100)Storing and loading the block offset mapThe creation of the list of bzip2 blocks can take a while because it has to decode the bzip2 file completely.
To avoid this setup when opening a bzip2 file, the block offset list can be exported and imported.importindexed_bzip2asibz2importpickle# Calculate and save bzip2 block offsetsfile=ibz2.open("example.bz2",parallelization=os.cpu_count())block_offsets=file.block_offsets()# can take a while# block_offsets is a simple dictionary where the keys are the bzip2 block offsets in bits(!)# and the values are the corresponding offsets in the decoded data in bytes. E.g.:# block_offsets = {32: 0, 14920: 4796}withopen("offsets.dat",'wb')asoffsets_file:pickle.dump(block_offsets,offsets_file)file.close()# Load bzip2 block offsets for fast seekingwithopen("offsets.dat",'rb')asoffsets_file:block_offsets=pickle.load(offsets_file)file2=ibz2.open("example.bz2",parallelization=os.cpu_count())file2.set_block_offsets(block_offsets)# should be fastfile2.seek(123)data=file2.read(100)file2.close()Open a pure Python file-like object for indexed readingimportioimportosimportindexed_bzip2asibz2withopen("example.bz2",'rb')asfile:in_memory_file=io.BytesIO(file.read())withibz2.open(in_memory_file,parallelization=os.cpu_count())asfile:file.seek(123)data=file.read(100)Via RatarmountBecauseindexed_bzip2is used by default as a backend in ratarmount, you can useratarmountto mount single bzip2 files easily.
Furthermore, since ratarmount 0.11.0, parallelization is the default and does not have to be specified explicitly with-P.base64/dev/urandom|head-c$((512*1024*1024))|bzip2>sample.bz2# Serial decoding: 23 stimebzip2-c-dsample.bz2|wc-c

python3-mpipinstall--userratarmount
ratarmountsample.bz2mounted# Parallel decoding: 2 stimecatmounted/sample|wc-c# Random seeking to the middle of the file and reading 1 MiB: 0.132 stimeddif=mounted/samplebs=$((1024*1024))\iflag=skip_bytes,count_bytesskip=$((256*1024*1024))count=$((1024*1024))|wc-cCommand Line ToolA rudimentary command line tool exists but is not yet shipped with the Python module and instead has to be built from source.gitclonehttps://github.com/mxmlnkn/indexed_bzip2.gitcdindexed_bzip2mkdirbuildcdbuildcmake..cmake--build.--ibzip2The finishedibzip2binary is created in thetoolssubfolder.
To install it, it can be copied, e.g., to/usr/local/binor anywhere else as long as it is available in yourPATHvariable.
The command line options are similar to those of the existingbzip2tool.src/tools/ibzip2--help# Parallel decoding: 1.7 stimesrc/tools/ibzip2-d-c-P0sample.bz2|wc-c# Serial decoding: 22 stimebzip2-d-csample.bz2|wc-cC++ libraryBecause it is written in C++, it can of course also be used as a C++ library.
In order to make heavy use of templates and to simplify compiling with Pythonsetuptools, it is completely header-only so that integration it into another project should be easy.
The license is also permissive enough for most use cases.I currently did not yet test integrating it into other projects other than simply manually copying the source incoreandindexed_bzip2.
If you have suggestions and wishes like support with CMake or Conan, please open an issue.Internal ArchitectureThe parallelization of the bzip2 decoder and adding support to read from Python file-like objects required a lot of work to design an architecture which works and can be reasoned about.
An earlier architecture was discarded because it became to monolithic.
That discarded one was able to even work with piped non-seekable input, with which the current parallel architecture does not work with yet.
The serialBZ2Readerstill exists but is not shown in the class diagram because it is deprecated and will be removed some time in the future after theParallelBZ2Readerhas proven itself.
Clickhereor the image to get a larger image andhereto see an SVG version.Tracing the DecoderPerformance profiling and tracing is done withScore-Pfor instrumentation andVampirfor visualization.
This is one way, you could install Score-P with most of the functionalities on Ubuntu 22.04.sudoapt-getinstalllibopenmpi-devopenmpi-bingcc-11-plugin-devllvm-devlibclang-devlibunwind-dev\libopen-trace-format-devotf-tracelibpapi-dev# Install Score-P (to /opt/scorep)SCOREP_VERSION=8.0
wget"https://perftools.pages.jsc.fz-juelich.de/cicd/scorep/tags/scorep-${SCOREP_VERSION}/scorep-${SCOREP_VERSION}.tar.gz"tar-xf"scorep-${SCOREP_VERSION}.tar.gz"cd"scorep-${SCOREP_VERSION}"./configure--with-mpi=openmpi--enable-shared--without-llvm--without-shmem--without-cubelib--prefix="/opt/scorep-${SCOREP_VERSION}"make-j$(nproc)makeinstall# Add /opt/scorep to your path variables on shell startcat<<EOF >> ~/.bashrcif test -d /opt/scorep; thenexport SCOREP_ROOT=/opt/scorepexport PATH=$SCOREP_ROOT/bin:$PATHexport LD_LIBRARY_PATH=$SCOREP_ROOT/lib:$LD_LIBRARY_PATHfiEOFecho-1|sudotee/proc/sys/kernel/perf_event_paranoid# Check whether it worksscorep--version
scorep-infoconfig-summary# Actually do the tracingcdtools
bashtrace.sh