Package: pragzip
Pragzip: Parallel Random Access Gzip(Alternative Name: Rapidgzip: Random Access Parallel (Indexed) Decompression for Gzip Files)This repository contains the command line toolpragzip, which can be used for parallel decompression of almost any gzip file.The Python module provides aPragzipFileclass, which can be used to seek inside gzip files without having to decompress them first.
Alternatively, you can use this simply as aparallelizedgzip decoder as a replacement for Python's builtingzipmodule in order to fully utilize all your cores.The random seeking support is the same as provided byindexed_gzipbut further speedups are realized at the cost of higher memory usage thanks to a least-recently-used cache in combination with a parallelized prefetcher.Thisrepository is a light-weight fork of theindexed_bzip2repository, in which the main development takes place.
This repository was created for visibility reasons and in order to keep indexed_bzip2 and pragzip releases separate.
It will be updated at least for each release.
Issues regarding pragzip should be openedhere.Table of ContentsPerformanceDecompression with Existing IndexDecompression from ScratchInstallationUsageCommand Line ToolPython LibraryVia RatarmountC++ LibraryCitationInternal ArchitectureTracing the DecoderPerformanceThese are simple timing tests for reading all the contents of a gzip file sequentially.Results are shown for an AMD Ryzen 3900X 12-core (24 virtual cores) processor and withgzipFilePath="4GiB-base64.gz", which is a 4 GiB gzip compressed file with base64 random data.Be aware that the chunk size requested from the Python code does influence the performance heavily.
This benchmarks use a chunk size of 512 KiB.Decompression with Existing Index4GiB-base644GiB-base6420x-silesia20x-silesiaUncompressed Size4 GiB3.95 GiBCompressed Size3.04 GiB1.27 GiBModuleBandwidth/ (MB/s)SpeedupBandwidth/ (MB/s)Speedupgzip25012931pragzip (0  threads)448017.9483016.5pragzip (1  threads)2941.23501.2pragzip (2  threads)5802.36782.3pragzip (6  threads)16806.719406.6pragzip (12 threads)311012.5346011.8pragzip (24 threads)451018.0507017.3pragzip (32 threads)433017.3472016.1Decompression from ScratchPython4GiB-base644GiB-base6420x-silesia20x-silesiaUncompressed Size4 GiB3.95 GiBCompressed Size3.04 GiB1.27 GiBModuleBandwidth/ (MB/s)SpeedupBandwidth/ (MB/s)Speedupgzip25012931pragzip (0  threads)328013.122807.8pragzip (1  threads)2220.92360.8pragzip (2  threads)4281.74111.4pragzip (6  threads)12505.010953.7pragzip (12 threads)22909.213904.8pragzip (24 threads)330013.222807.8pragzip (32 threads)318012.724808.5Note that pragzip is generally faster when given an index because it can delegate the decompression to zlib while it has to use its own gzip decompression engine when no index exists yet.Note that values deviate roughly by 10% and therefore are rounded.The code used for benchmarking can be foundhere.InstallationYou can simply install it from PyPI:python3 -m pip install --upgrade pip  # Recommended for newer manylinux wheels
python3 -m pip install pragzipThe latest unreleased development version can be tested out with:python3-mpipinstall--force-reinstall'git+https://github.com/mxmlnkn/indexed_bzip2.git@master#egginfo=pragzip&subdirectory=python/pragzip'And to build locally, you can usebuildand install the wheel:cdpython/pragzip
rm-rfdist
python3-mbuild.
python3-mpipinstall--force-reinstall--userdist/*.whlUsageCommand Line Toolpragzip--help# Parallel decoding: 1.7 stimepragzip-d-c-P0sample.gz|wc-c# Serial decoding: 22 stimegzip-d-csample.gz|wc-cPython LibrarySimple open, seek, read, and closefrompragzipimportPragzipFilefile=PragzipFile("example.gz",parallelization=os.cpu_count())# You can now use it like a normal filefile.seek(123)data=file.read(100)file.close()The first call to seek will ensure that the block offset list is complete and therefore might create them first.
Because of this the first call to seek might take a while.Use with context managerimportosimportpragzipwithpragzip.open("example.gz",parallelization=os.cpu_count())asfile:file.seek(123)data=file.read(100)Storing and loading the block offset mapThe creation of the list of gzip blocks can take a while because it has to decode the gzip file completely.
To avoid this setup when opening a gzip file, the block offset list can be exported and imported.Open a pure Python file-like object for indexed readingimportioimportosimportpragzipaspragzipwithopen("example.gz",'rb')asfile:in_memory_file=io.BytesIO(file.read())withpragzip.open(in_memory_file,parallelization=os.cpu_count())asfile:file.seek(123)data=file.read(100)Via Ratarmountpragzipisplannedto be used as a backend inside ratarmount with version 0.12.
Then, you can useratarmountto mount single gzip files easily.base64/dev/urandom|head-c$((4*1024*1024*1024))|gzip>sample.gz# Serial decoding: 23 stimegzip-c-dsample.gz|wc-c

python3-mpipinstall--userratarmount
ratarmountsample.gzmounted# Parallel decoding: 3.5 stimecatmounted/sample|wc-c# Random seeking to the middle of the file and reading 1 MiB: 0.287 stimeddif=mounted/samplebs=$((1024*1024))\iflag=skip_bytes,count_bytesskip=$((2*1024*1024*1024))count=$((1024*1024))|wc-cC++ libraryBecause it is written in C++, it can of course also be used as a C++ library.
In order to make heavy use of templates and to simplify compiling with Pythonsetuptools, it is mostly header-only so that integration it into another project should be easy.
The license is also permissive enough for most use cases.I currently did not yet test integrating it into other projects other than simply manually copying the source insrc/core,src/pragzip, and if integrated zlib is desired alsosrc/external/zlib.
If you have suggestions and wishes like support with CMake or Conan, please open an issue.CitationA paper describing the implementation details and showing the scaling behavior with up to 128 cores has been submitted to andacceptedinACM HPDC'23, The 32nd International Symposium on High-Performance Parallel and Distributed Computing. If you use this software for your scientific publication, please cite it as:This is preliminiary. The final citation will become available end of June 2023.@inproceedings{rapidgzip,author={Knespel, Maximilian and Brunst, Holger},title={Rapidgzip: Parallel Decompression and Seeking in Gzip Files Using Cache Prefetching},year={2023},%isbn={},  % To be released end of Junepublisher = {Association for Computing Machinery},address   = {New York, NY, USA},url       = {https://doi.org/10.1145/3588195.3592992},doi       = {10.1145/3588195.3592992},abstract  = {Gzip is a file compression format, which is ubiquitously used. Although a multitude of gzip implementations exist, only pugz can fully utilize current multi-core processor architectures for decompression. Yet, pugz cannot decompress arbitrary gzip files. It requires the decompressed stream to only contain byte values 9–126. In this work, we present a generalization of the parallelization scheme used by pugz that can be reliably applied to arbitrary gzip-compressed data without compromising performance. We show that the requirements on the file contents posed by pugz can be dropped by implementing an architecture based on a cache and a parallelized prefetcher. This architecture can safely handle faulty decompression results, which can appear when threads start decompressing in the middle of a gzip file by using trial and error. Using 128 cores, our implementation reaches 8.7 GB/s decompression bandwidth for gzip-compressed base64-encoded data, a speedup of 55 over the single-threaded GNU gzip, and 5.6 GB/s for the Silesia corpus, a speedup of 33 over GNU gzip.},booktitle = {Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing},% pages   = {16–29},  % To be released end of Junenumpages  = {13},keywords  = {Gzip, Decompression, Parallel Algorithm, Performance, Random Access},location  = {Orlando, FL, USA},series    = {HPDC '23},}Internal ArchitectureThe main part of theinternal architectureused for parallelizing is the same as used forindexed_bzip2.Tracing the DecoderPerformance profiling and tracing is done withScore-Pfor instrumentation andVampirfor visualization.
This is one way, you could install Score-P with most of the functionalities on Ubuntu 22.04.Installation of DependenciesInstallation steps for Score-Psudoapt-getinstalllibopenmpi-devopenmpi-bingcc-11-plugin-devllvm-devlibclang-devlibunwind-dev\libopen-trace-format-devotf-tracelibpapi-dev# Install Score-P (to /opt/scorep)SCOREP_VERSION=8.0
wget"https://perftools.pages.jsc.fz-juelich.de/cicd/scorep/tags/scorep-${SCOREP_VERSION}/scorep-${SCOREP_VERSION}.tar.gz"tar-xf"scorep-${SCOREP_VERSION}.tar.gz"cd"scorep-${SCOREP_VERSION}"./configure--with-mpi=openmpi--enable-shared--without-llvm--without-shmem--without-cubelib--prefix="/opt/scorep-${SCOREP_VERSION}"make-j$(nproc)makeinstall# Add /opt/scorep to your path variables on shell startcat<<EOF >> ~/.bashrcif test -d /opt/scorep; thenexport SCOREP_ROOT=/opt/scorepexport PATH=$SCOREP_ROOT/bin:$PATHexport LD_LIBRARY_PATH=$SCOREP_ROOT/lib:$LD_LIBRARY_PATHfiEOFecho-1|sudotee/proc/sys/kernel/perf_event_paranoid# Check whether it worksscorep--version
scorep-infoconfig-summaryTracingResults for a version from 2023-02-04Comparison without and with rpmalloc preloaded