Package: indexedtar
indexedtarAn indexed Tar for big data archives featuring fast random access with an index bundled inside the tarfile.The use case is to retrieve members of a "many members" tar archive without seeking
from one member to the next.GoalsInstallationUsage of theitarcliUsage of theIndexedTarclassBenchmarkConceptGoalsWe constrained this code as follows:Produce archives fully compliant with the tar specification to
preserve compatibility with existing toolsNo additional index file, the archive should contain the index and be 'all inclusive'Use only the python standard libraryInstallationUsing pypi.pip install indexedtarFrom the sources after cloning this repo.python setup.py installNote: when using pyenv I needed to relaunch my shell and virtualenv post-install to have theitarcli available.Launching unit testsLinting and unit tests require additional dependencies.$pipinstall-rrequirements.txt
$flake8--max-line-length120indexedtar
$black--checkindexedtar
$exportPYTHONPATH=".";py.test--cov=indexedtartests
...[88%]tests/test_itar.py.[100%]----------coverage:platformlinux,python3.8.12-final-0-----------
NameStmtsMissCover
--------------------------------------------
indexedtar/__init__.py172697%
indexedtar/itar.py37489%
--------------------------------------------
TOTAL2091095%Usage of theitarcliitar--help
usage:itar[-h][--targetTARGET][--fnmatch_filterFNMATCH_FILTER][--output_dirOUTPUT_DIR]actionarchive

IndexedTarbuild/extractutility.

positionalarguments:actionactiontoperform:"x"forextract,"l"forlisting,"c"forcreate,"a"forappendarchivepathtoarchivefile

optionalarguments:-h,--helpshowthishelpmessageandexit--targetTARGETfileordirectorytoadd--fnmatch_filterFNMATCH_FILTERfnmatchfilterforlisting/extractingarchivemembers--output_dirOUTPUT_DIRoutputdirectoryforextractionCreate an archive  with the files in thetests/datadirectory.itarctest.tar--targettests/dataList archive members matching a fnmatch pattern.itarltest.tar--fnmatch_filter"*3h.grib2"Extract members matching a fnmatch pattern to output directory.itarxtest.tar--fnmatch_filter"*arome*.grib2"--output_diroutUsage of theIndexedTarclassSee theunit testsfor usage examples.Create an archive.fromindexedtarimportIndexedTarDATA_DIR=pathlib.Path("/home/frank/dev/mf-models-on-s3-scraping")withIndexedTar("test.tar",mode="x:")asit:it.add_dir(DATA_DIR)Get a tarmember by indexwithIndexedTar(pathlib.Path("fat.tar"),mode="r:")asit:tinfo=it.getmember_at_index(5)# get 5th member from the archiveprint(tinfo.name)Get and extract members matching a regex or a fnmatch patternwithIndexedTar("indexed.tar","r:")asit:# find and extract members using fnmatchit.extract_members(it.get_members_fnmatching("2021_01_26/*"))# find and extract members using regexit.extract_members(it.get_members_re("^2021_02_01"))# extract to specific outputdir 'out'it.extract_members(it.get_members_fnmatching("*.grib2"),path=Path("out"))BenchmarkHDD for a 2.1 GB tarfile with 6094 membersWe extract the last member of the archive. Seebenchmark.py.(indexenv) [frank@localhost pyindexedtar]$ python benchmark.py 

python IndexedTar average extraction time: 0.0156 seconds
python Tar average extraction time: 1.5477 seconds
GNU Tar average extraction time: 0.0476 secondsSSD NVMe with a 2.1 GB tarfile containing 6094 membersReading 10 random members by name.python IndexedTar average extraction time: 0.0033 seconds
python Tar average extraction time: 0.3216 seconds
GNU Tar average extraction time: 0.0188 secondsSSD NVMe with a 27 GB tarfile containing 76175 membersReading 10 random members by name.python IndexedTar average extraction time: 0.0442 seconds
python Tar average extraction time: 3.9926 seconds
GNU Tar average extraction time: 0.1675 secondsConceptThe trick here is to have a 'normal' binary file
added at the beginning of the tar that serves as a
pre-allocation of 3 unsigned long long to
store header and data offsets + the size of our index.When we close the archive we write the index
as the last file in the tar and seek back to the
location of the offset and size to write it.The index itself is a json_tar_index.jsonlisting
all the files in the tar including duplicates. For each file we
store its tar header offset, its tar data offset and
its tar data length.[["my_first_file",3072,4608,352392],["my_second_file",357376,358912,352392],["my_third_file",711680,713216,352392]]######
_tar_offset.bin tar header
-----
_tar_offset.bin payload
unsigned long long value1 => points to >>>>>------------------|
unsigned long long value2 => points to index data
unsigned long long  value3 => index len                       |
######                                                        |
FILE 1 - tar header                                           |
-----                                                         |
FILE 1 - data          <<<<<<oooooooooooooooooooooooo         |
                                                    o         |
....                                                o         |
                                                    o         |
######                                              o         |
FILE N tar header                                   o         |
-----                                               o         |
FILE N data                                         o         |
######                                              o         |
_tar_index.json - tar header <<<<<<<<<--------------o---------|
------                                              o
_tar_index.json data                                o
[[FILE_1_NAME, FILE_1_TINFO_OFFSET, FILE_1_DATA_OFFSET>, FILE_1_SIZE],
...
[FILE_N_NAME, FILE_N_TINFO_OFFSET, FILE_N_DATA_OFFSET, FILE_N_SIZE]]
######This gives us the following workflow to retrieve a member 'A':open Indexedtar >>> read first member ( = index offset) >>> seek at index offset >>> read index >>> lookup 'A''s offset in index >>> read 'A'.Compatiblity checksOur archive stills open with the standard GNU tar cli tool or GUI 7zip client.(indextarenv)$ tar -tvf fat.tar | most
-rw-r--r-- 0/0              24 2021-09-29 23:50 _tar_offset.bin
-rw-r--r-- frank/frank  352392 2021-09-29 23:48 0_arpege-world_20210827_18_DLWRF_surface_acc_0-3h.grib2
-rw-r--r-- frank/frank  352392 2021-09-29 23:48 1_arpege-world_20210827_18_DLWRF_surface_acc_0-3h.grib2
-rw-r--r-- frank/frank  352392 2021-09-29 23:48 2_arpege-world_20210827_18_DLWRF_surface_acc_0-3h.grib2
-rw-r--r-- frank/frank  352392 2021-09-29 23:48 3_arpege-world_20210827_18_DLWRF_surface_acc_0-3h.grib2
-rw-r--r-- frank/frank  352392 2021-09-29 23:48 4_arpege-world_20210827_18_DLWRF_surface_acc_0-3h.grib2
-rw-r--r-- frank/frank  352392 2021-09-29 23:48 5_arpege-world_20210827_18_DLWRF_surface_acc_0-3h.grib2
...Todo and ideasadd highwayhash (SIMD, should perform ! ) checksums for each file in the indexSee if we could handle 'tar.gz' compressed archive using"IndexedGzip"?