Package: hashget
hashgetHashget is networkdeduplicationtool working together with usual compression utilities (such as tar/gz/xz) and can becombinedwith other backup tools such asrestic.While usual compression tools uses mathematical algorithms for compressing data, hashget finds which files could be
downloaded from public (e.g. WordPress or Debian servers) or private (e.g. your company internal website) resources and
excludes it from archive (leaving only very short meta-information about it).Upon decompressing, hashget downloads these files, verifies hashsum and places it on target system with same
permissions, ownership, atime and mtime.Hashget compression is lossless, based on cryptographically strong SHA256 hashsum. (No collisions know for now)Chance of SHA256 collisionBitcoin was computing 300 quadrillion SHA-256 hashes per second. That's 300×1015hashes per second. 
Let's say you were trying to perform a collision attack and would "only" need to calculate 2128hashes. 
At the rate Bitcoin is going, it would take them 2128/(300*1015*86400*365.25)≈3.6×1013years.In comparison, our universe is only about 13.7×109years old.https://crypto.stackexchange.com/questions/47809/why-havent-any-sha-256-collisions-been-found-yetEffectivenessData sampleunpacked size.tar.gzhashget .tar.gzWordpress-5.1.143 Mb11 Mb ( 26% )155 Kb (0.3%)Linux kernel  5.0.4934 Mb161 Mb ( 20% )4.7 Mb (0.5%)Debian 9 (LAMP) LXC VM724 Mb165 Mb ( 23% )4.1 Mb (0.5%)Unpacked size measured withdu -shcommand. Ratio calculated asdh -shb compressed.tar.gz/du -shb original-dirin percents. Debian filesystem was clean and packed without temporary files (see example below).InstallationPip (recommended):pip3installhashget[plugins]or clone from git:gitclonehttps://github.com/yaroslaff/hashget.gitQuickStartCompressing (manual indexing)# prepare test data$mkdir-p/tmp/test/wp
$cd/tmp
$wgethttps://wordpress.org/wordpress-5.1.1.zip$cd/tmp/test/wp
$unzip-q/tmp/wordpress-5.1.1.zip# index data$hashget--submithttps://wordpress.org/wordpress-5.1.1.zip-pmy--hashserver# pack$hashget-zf/tmp/wordpress-hashget.tar.gz--pack.--hashserver
STEP1/3Indexing...
Indexingdonein0.07s.0local+0pulled+0new=0totalpackages
STEP2/3prepareexcludelistforpacking...
saved:1373files,1pkgs,size:37.9M.Download:11.0M
STEP3/3tarring...
.(38.1M)packedinto/tmp/wordpress-hashget.tar.gz(154.7K)-fto specify filename,-zto gzip it,--pack .commands which directory to pack and--hashserverwithout value disables
remote hashservers.You can check local indexes HashDB withhashget-adminutility.DecompressingUnpack .tar.gz and thenhashget -uthat directory (which has hidden file .hashget-restore.json).$mkdir/tmp/test/wp-unpacked
$cd/tmp/test/wp-unpacked
$tar-xzf/tmp/wordpress-hashget.tar.gz
$hashget-u.--hashserver
Recovered1373/1373files37.9Mbytes(0downloaded,11.0Mcached)in6.13sYou can delete .hashget-restore.json file after this if you want. Now /tmp/test/wp-unpacked restored from tiny 150K
hashget archive is same as /tmp/test/wp.Manual indexing is easy way to optimize packing of individual large packages.AdvancedDebian VM compressing (built-in plugin)Compressingtest machine:(Since it requires access to VM filesystem, run as user root or use sudo)hashget--pack/var/lib/lxc/mydebvm/rootfs/-zf/tmp/mydebvm.tar.gz\--excludevar/cache/aptvar/lib/apt/lists
STEP1/3Indexingdebianpackages...
Total:222packages
Indexingdonein0.02s.222local+0pulled+0new=222total.
STEP2/3prepareexcludelistforpacking...
saved:8515files,216pkgs,size:445.8M.Download:98.7M
STEP3/3tarring...
/var/lib/lxc/mydebvm/rootfs/(687.2M)packedinto/tmp/mydebian.tar.gz(4.0M)--excludedirective tells hashget and tar to skip some directories which are not necessary in backup.
(You can omit it, backup will be larger)Built-in debian plugin automatically discovers installed debian packages, downloads index files from hashserver
(to speed-up indexing) or download packages and index itself (if not found on hashserver). HashPackages (indexes) are
stored in local hashdb, so next time you will pack this directory, it will not download and index same data again and
will work much faster.For each package hashget archive will store link tosnapshot.debian.orgin restore-file.
snapshot links are permanent and works since 2010. Unfortunately, other deb/apt based linux distributions do not have
such snapshots, so this method is very effective only for Debian.
You maydisable this heuristicwhen packing other OS.Hint filesIf our package is indexed (like we just did with wordpress) it will be very effectively deduplicated on packing.
But what if it's not indexed? For example, if you cleaned hashdb cache or if you will restored this backup on other
machine and pack it again. It will take it's full space again.We will delete index for this file:hashget-admin--purge--hpwordpress-5.1.1-ru_RU.zip(you can get index filename withhashget-admin --list -p PROJECTcommand)Now, if you will make hashget --pack , it will make huge 22M archive again, our magic is lost...Now, create special smallhintfile hashget-hint.json (or .hashget-hint.json ,
if you want it to be hidden) in /tmp/test with this content:{"project":"wordpress.org","url":"https://ru.wordpress.org/wordpress-5.1.1-ru_RU.zip"}And now try compress it again:$hashget--pack/tmp/test-zf/tmp/test.tar.gz
STEP1/3Indexing...
submittinghttps://ru.wordpress.org/wordpress-5.1.1-ru_RU.zip
STEP2/3prepareexcludelistforpacking...
saved:1396files,1pkgs,size:52.2M.Download:11.7M
STEP3/3tarring...
/tmp/test(52.3M)packedinto/tmp/test.tar.gz(157.9K)Great! Hashget used hint file and automatically indexed file, so we got our fantastic compression rate again.Directories with hint files are packed effectively even if not indexed before. If you are developer,
you can include hashget-hint file inside your package files to make it backup-friendly.
This is much more simple way then writing plugin.Heuristic pluginsHeuristics are small plugins (installed when you didpip3 install hashget[plugins], or can be installed separately)
which can auto-detect some non-indexed files which could be indexed. You already know build-in heuristics for Debian
and hint files, but hashget could be extended with third-party plugins.Lets try test with linux kernel sources package (100Mb+):$mkdir/tmp/lk$cd/tmp/lk
$wget-qhttps://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.0.4.tar.xz
$tar-xflinux-5.0.4.tar.xz$du-sh.1.1G.If we will pack this machine same way as before we will see this:$hashget--pack/tmp/lk/-zf/tmp/lk.tar.gz--hashserver
STEP1/3Indexing...
submittinghttps://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.0.4.tar.xz
Indexingdonein199.27s.1local+0pulled+1new=2totalpackages
STEP2/3prepareexcludelistforpacking...
saved:50580files,1pkgs,size:869.3M.Download:100.4M
STEP3/3tarring...
/tmp/lk/(875.3M)packedinto/tmp/lk.tar.gz(4.6M)One very interesting line here is:submitting https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.0.4.tar.xzHashget detected linux kernel sources package, downloaded and indexed it. And we got fantastic result again: almost 200 times!
Package was not indexed before and was indexed during packing.This happened because hashget has heuristical plugin which detects linux kernel sources and guesses URL to index it.
This plugin puts index files for kernel packages into 'kernel.org' hashget project.If you will makehashget-admin --statusyou will see kernel.org project.hashget-admin --list -p PROJECTwill
show content of project:hashget-admin--list-pkernel.org
linux-5.0.5.tar.xz(767/50579)Even when new kernel package will be released (and it's not indexed anywhere), hashget will detect it and
automatically index (at least while new linux kernels will match same 'template' as it matches now for kernels
1.0 to 5.0.6).Users and developers of large packages can write their own hashget plugins usingLinux kernel hashget pluginas example.What you should indexYou should index ONLY static and permanent files, which will be available on same URL with same content during backup
lifetime. SeeExpirationchapter in wiki documentation.Not all projects provides such files. Usual linux package repositories has only latest files so it's not good for this
purpose, but debian has greatsnapshot.debian.orgrepository, which makes Debian great
for hashget compression.Do not indexlatestfiles, because content will change later (it's notstatic). E.g. you may indexhttps://wordpress.org/wordpress-5.1.1.zipbut you should not indexhttps://wordpress.org/latest.zipUsing local file poolIt's possible to use local or remote (e.g. on Intranet HTTP server) file pool. This can help to unpack archives even
when original server is not available.Pool populated automatically if --pool given to certain operations (--submit,--index,--pack,--postunpack/-u):For example:$mkdir/tmp/pool
$hashget--submithttps://wordpress.org/wordpress-5.1.1.zip--pool/tmp/pool/-pmy
$ls/tmp/pool
wordpress-5.1.1.zipWhen doing--postunpack(-u) with--pool, hashget will read files from pool, and write it there (if file
was not found in pool and downloaded, it will be saved to pool).# Empty pool$hashget-u.--pool/tmp/pool
...
Recovered8534/8534files450.0Mbytes(0downloaded,0frompool,98.7Mcached)in155.92s# Pool populated now$LANG=Cls/tmp/pool
adduser_3.115_all.debliblz4-1_0.0~r131-2+b1_amd64.deb
apache2-bin_2.4.25-3+deb9u6_amd64.debliblzma5_5.2.2-1.2+b1_amd64.deb
...# Now packages are taken from pool$hashget-u.--pool/tmp/pool
...
Recovered8534/8534files450.0Mbytes(0downloaded,98.7Mfrompool,0cached)in146.92sMore DocumentationFor more detailed documentation seeWiki.For example, seeIncremental backupschapter in wiki doc
andGlacierto know how to use Incremental/Differential backups
with Amazon Glacier or other storage engine.Also, please check my other open-source projectOkerr. AndOkerr wiki-documentation.