Package: dirhash
dirhashA lightweight python module and CLI for computing the hash of any
directory based on its files' structure and content.Supports all hashing algorithms of Python's built-inhashlibmodule.Glob/wildcard (".gitignore style") path matching for expressive filtering of files to include/exclude.Multiprocessing for up to6x speed-upThe hash is computed according to theDirhash Standard, which is designed to allow for consistent and collision resistant generation/verification of directory hashes across implementations.InstallationFrom PyPI:pip install dirhashOr directly from source:git clone git@github.com:andhus/dirhash-python.git
pip install dirhash/UsagePython module:fromdirhashimportdirhashdirpath="path/to/directory"dir_md5=dirhash(dirpath,"md5")pyfiles_md5=dirhash(dirpath,"md5",match=["*.py"])no_hidden_sha1=dirhash(dirpath,"sha1",ignore=[".*",".*/"])CLI:dirhash path/to/directory -a md5
dirhash path/to/directory -a md5 --match "*.py"
dirhash path/to/directory -a sha1 --ignore ".*"  ".*/"Why?If you (or your application) need to verify the integrity of a set of files as well
as their name and location, you might find this useful. Use-cases range from
verification of your image classification dataset (before spending GPU-$$$ on
training your fancy Deep Learning model) to validation of generated files in
regression-testing.There isn't really a standard way of doing this. There are plenty of recipes out
there (see e.g. these SO-questions forlinuxandpython)
but I couldn't find one that is properly tested (there are some gotcha:s to cover!)
and documented with a compelling user interface.dirhashwas created with this as
the goal.checksumdiris another python
module/tool with similar intent (that inspired this project) but it lacks much of the
functionality offered here (most notably including file names/structure in the hash)
and lacks tests.PerformanceThe pythonhashlibimplementation of common hashing algorithms are highly
optimised.dirhashmainly parses the file tree, pipes data tohashliband
combines the output. Reasonable measures have been taken to minimize the overhead
and for common use-cases, the majority of time is spent reading data from disk
and executinghashlibcode.The main effort to boost performance is support for multiprocessing, where the
reading and hashing is parallelized over individual files.As a reference, let's compare the performance of thedirhashCLIwith the shell command:find path/to/folder -type f -print0 | sort -z | xargs -0 md5 | md5which is the top answer for the SO-question:Linux: compute a single hash for a given folder & contents?Results for two test cases are shown below. Both have 1 GiB of random data: in
"flat_1k_1MB", split into 1k files (1 MiB each) in a flat structure, and in
"nested_32k_32kB", into 32k files (32 KiB each) spread over the 256 leaf directories
in a binary tree of depth 8.ImplementationTest CaseTime (s)Speed upshell referenceflat_1k_1MB2.29-> 1.0dirhashflat_1k_1MB1.671.36dirhash(8 workers)flat_1k_1MB0.484.73shell referencenested_32k_32kB6.82-> 1.0dirhashnested_32k_32kB3.432.00dirhash(8 workers)nested_32k_32kB1.146.00The benchmark was run a MacBook Pro (2018), further details and source codehere.DocumentationPlease refer todirhash -h, the pythonsource codeand theDirhash Standard.