Package: indexed-gzip
indexed_gzipFast random access of gzip files in PythonOverviewInstallationUsageUsing withnibabelIndex import/exportWrite supportPerformanceAcknowledgementsLicenseOverviewTheindexed_gzipproject is a Python extension which aims to provide a
drop-in replacement for the built-in Pythongzip.GzipFileclass, theIndexedGzipFile.indexed_gzipwas written to allow fast random access of compressedNIFTIimage files (for which GZIP is the
de-facto compression standard), but will work with any GZIP file.indexed_gzipis easy to use withnibabel(http://nipy.org/nibabel/).The standardgzip.GzipFileclass exposes a random access-like interface (via
itsseekandreadmethods), but every time you seek to a new point in the
uncompressed data stream, theGzipFileinstance has to start decompressing
from the beginning of the file, until it reaches the requested location.AnIndexedGzipFileinstance gets around this performance limitation by
building an index, which containsseek points, mappings between
corresponding locations in the compressed and uncompressed data streams. Each
seek point is accompanied by a chunk (32KB) of uncompressed data which is used
to initialise the decompression algorithm, allowing us to start reading from
any seek point. If the index is built with a seek point spacing of 1MB, we
only have to decompress (on average) 512KB of data to read from any location
in the file.Intended useYou may findindexed_gzipuseful if you need to read from large GZIP files.
A major advantage ofindexed_gzipis that it will work with any GZIP file.
However, if you have control over the creation of your GZIP files, you may
wish to consider some alternatives:mgzipprovides an accelerated
GZIP compression and decompression library.Compression formats other than GZIP, such asbzip2andxz, have better
support for random access.Installationindexed_gzipis available onPyPi- to
install, simply type:pipinstallindexed_gzipYou can also installindexed_gzipfrom conda-forge:condainstall-cconda-forgeindexed_gzipTo compileindexed_gzip, make sure you havecythoninstalled (andnumpyif you want to compile the tests), and then run:pythonsetup.pydevelopTo run the tests, type the following; you will neednumpy,nibabel,pytest,pytest-cov, andcoverageinstalled:python-mindexed_gzip.testsUsageYou can use theindexed_gzipmodule directly:importindexed_gzipasigzip# You can create an IndexedGzipFile instance# by specifying a file name, or an open file# handle. For the latter use, the file handle# must be opened in read-only binary mode.# Write support is currently non-existent.myfile=igzip.IndexedGzipFile('big_file.gz')some_offset_into_uncompressed_data=234195# The index will be automatically# built on-demand when seeking.myfile.seek(some_offset_into_uncompressed_data)data=myfile.read(1048576)Using with in-memory dataYou can useindexed_gzipwith any Python file-like object. For example:importioimportindexed_gzipasigzip# Load some gzip data from somewherewithopen('my_file.gz')asf:data=f.read()# Create an IndexedGzipFile based on the# in-memory data buffergzf=igzip.IndexedGzipFile(fileobj=io.BytesIO(data))uncompressed=gzf.read(1048576)Using withnibabelYou can useindexed_gzipwithnibabel.nibabel>= 2.3.0 will
automatically useindexed_gzipif it is present:importnibabelasnibimage=nib.load('big_image.nii.gz')If you are usingnibabel2.2.x, you need to explicitly set thekeep_file_openflag:importnibabelasnibimage=nib.load('big_image.nii.gz',keep_file_open='auto')To useindexed_gzipwithnibabel2.1.0 or older, you need to do a little
more work:importnibabelasnibimportindexed_gzipasigzip# Here we are using 4MB spacing between# seek points, and using a larger read# buffer (than the default size of 16KB).fobj=igzip.IndexedGzipFile(filename='big_image.nii.gz',spacing=4194304,readbuf_size=131072)# Create a nibabel image using# the existing file handle.fmap=nib.Nifti1Image.make_file_map()fmap['image'].fileobj=fobjimage=nib.Nifti1Image.from_file_map(fmap)# Use the image ArrayProxy to access the# data - the index will automatically be# built as data is accessed.vol3=image.dataobj[:,:,:,3]Index import/exportIf you have a large file, you may wish to pre-generate the index once, and
save it out to an index file:importindexed_gzipasigzip# Load the file, pre-generate the# index, and save it out to disk.fobj=igzip.IndexedGzipFile('big_file.gz')fobj.build_full_index()fobj.export_index('big_file.gzidx')The next time you open the same file, you can load in the index:importindexed_gipasigzipfobj=igzip.IndexedGzipFile('big_file.gz',index_file='big_file.gzidx')Write supportindexed_gzipdoes not currently have any support for writing. Currently if
you wish to write to a file, you will need to save the file by alternate means
(e.g.  viagzipornibabel), and then re-create a newIndexedGzipFileinstance.  For example:importnibabelasnib# Load the entire image into memoryimage=nib.load('big_image.nii.gz')data=image.get_data()# Make changes to the datadata[:,:,:,5]*=100# Save the image using nibabelnib.save(data,'big_image.nii.gz')# Re-load the imageimage=nib.load('big_image.nii.gz')PerformanceA smalltest scriptis included withindexed_gzip; this script compares the performance of theIndexedGzipFileclass with thegzip.GzipFileclass. This script does the following:Generates a test file.Generates a specified number of seek locations, uniformly spaced
throughout the test file.Randomly shuffles these locationsSeeks to each location, and reads a chunk of data from the file.This plot shows the results of this test for a few compresed files of varying
sizes, with 500 seeks:AcknowledgementsTheindexed_gzipproject is based upon thezran.cexample (written by Mark
Alder) which ships with thezlibsource code.indexed_gzipwas originally inspired by Zalan Rajna's (@zrajna)zindexproject:Z. Rajna, A. Keskinarkaus, V. Kiviniemi and T. Seppanen
"Speeding up the file access of large compressed NIfTI neuroimaging data"
Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual
International Conference of the IEEE, Milan, 2015, pp. 654-657.

https://sourceforge.net/projects/libznzwithzindex/Initial work onindexed_gziptook place atBrainhackParis, at the Institut Pasteur,
24th-26th February 2016, with the support of theFMRIB Centre, at the
University of Oxford, UK.Many thanks to the following contributors (listed chronologically):Zalan Rajna (@zrajna): Bug fixes (#2)Martin Craig (@mcraig-ibme): Portingindexed_gzipto Windows (#3)Chris Markiewicz (@effigies): Option to drop file handles (#6)Omer Ozarslan (@ozars): Index import/export (#8)@DarioDaF: Windows overflow bug (#30)Sławomir Zborowski (@szborows):seek_pointsmethod (#35), README fixes
(#34)Ashwin Ramaswami (@epicfaace): Support for in-memory file objects (#55),
bug fixes (#63, #64, #65).Michał Górny (@mgorny): Remove hard dependency onnibabelfrom test suite
(#78).Alexander Gorban (@alexgorban) Fix memory leak (#82, #83).Maximilian Knespel (@mxmlnkn) Change default read buffer size to
improve performance (#90).Ben Beasley (@musicinmybrain) Python 3.12 compatibility (#126).Licenseindexed_gzipinherits thezliblicense, available for
perusal in theLICENSEfile.