Package: pyhardlinkbackup
pyhardlinkbackupHardlink/Deduplication Backups with Python.Backups should be saved as normal files in the filesystem:accessible without any extra software or extra meta filesnon-proprietary formatCreate backups with versioningevery backup run creates a complete filesystem snapshot treeevery snapshot tree can be deleted, without affecting the other snapshotsDeduplication with hardlinks:Store only changed files, all other via hardlinksfind duplicate files everywhere (even if renamed or moved files)useable under Windows and LinuxRequirement: Python 3.6 or newer.Please: try, fork and contribute! ;)github.com/jedie/pyhardlinkbackup/actionstravis-ci.org/jedie/pyhardlinkbackupci.appveyor.com/project/jedie/pyhardlinkbackupcoveralls.io/r/jedie/pyhardlinkbackuprequires.io/github/jedie/pyhardlinkbackup/requirements/Example$ phlb backup ~/my/important/documents
...start backup, some time later...
$ phlb backup ~/my/important/documents
...This will create deduplication backups like this:~/pyhardlinkbackups
  └── documents
      ├── 2016-01-07-085247
      │   ├── phlb_config.ini
      │   ├── spreadsheet.ods
      │   ├── brief.odt
      │   └── important_files.ext
      └── 2016-01-07-102310
          ├── phlb_config.ini
          ├── spreadsheet.ods
          ├── brief.odt
          └── important_files.extInstallationWindowsinstall Python 3:https://www.python.org/downloads/Download the fileboot_pyhardlinkbackup.cmdcallboot_pyhardlinkbackup.cmdas admin (Right-click and useRun as administrator)If everything works fine, you will get a venv here:%ProgramFiles%\PyHardLinkBackupAfter the venv is created, call these scripts to finalize the setup:%ProgramFiles%\PyHardLinkBackup\phlb_edit_config.cmd- create a config .ini file%ProgramFiles%\PyHardLinkBackup\phlb_migrate_database.cmd- create database tablesTo upgrade pyhardlinkbackup, call:%ProgramFiles%\PyHardLinkBackup\phlb_upgrade_pyhardlinkbackup.cmdTo start the Django webserver, call:%ProgramFiles%\PyHardLinkBackup\phlb_run_django_webserver.cmdLinuxDownload the fileboot_pyhardlinkbackup.shcallboot_pyhardlinkbackup.shIf everything works fine, you will get a venv here:~\pyhardlinkbackupAfter the venv is created, call these scripts to finalize the setup:~/PyHardLinkBackup/phlb_edit_config.sh- create a config .ini file~/PyHardLinkBackup/phlb_migrate_database.sh- create database tablesTo upgrade pyhardlinkbackup, call:~/PyHardLinkBackup/phlb_upgrade_pyhardlinkbackup.shTo start the Django webserver, call:~/PyHardLinkBackup/phlb_run_django_webserver.shStarting a backup runTo start a backup run, use this helper script:Windows batch:%ProgramFiles%\PyHardLinkBackup\pyhardlinkbackup_this_directory.cmdLinux shell script:~/PyHardLinkBackup/pyhardlinkbackup_this_directory.shCopy this file to a location that should be backed up and just call it to run a backup.Verifying an existing backup$ cd pyhardlinkbackup/
~/PyHardLinkBackup $ source bin/activate

(PyHardLinkBackup) ~/PyHardLinkBackup $ phlb verify --fast ~/PyHardLinkBackups/documents/2016-01-07-102310With–fastthe files’ contents will not be checked.
If not given: The hashes from the files’ contents will be calculated and compared. Thus, every file must be completely read from filesystem, so it will take some time.A verify run does:Do all files in the backup exist?Compare file sizesCompare hashes from hash-fileCompare files’ modification timestampsCalculate hashes from files’ contents and compare them (will be skipped if–fastused)Configurationphlb will use a configuration file named:PyHardLinkBackup.iniSearch order is:current directory down to rootuser directoryE.g. if the current working directoy is/foo/bar/my_files/then the search path will be:/foo/bar/my_files/PyHardLinkBackup.ini/foo/bar/PyHardLinkBackup.ini/foo/PyHardLinkBackup.ini/PyHardLinkBackup.ini~/PyHardLinkBackup.iniThe user home directory under Windows/LinuxCreate / edit default .iniYou can just open the editor with the user directory .ini file with:(PyHardLinkBackup) ~/PyHardLinkBackup $ phlb configThe defaults are stored here:/phlb/config_defaults.iniExcluding files/folders from backup:There are two ways to exclude files/folders from your backup.
Use the follow settings in yourPyHardLinkBackup.ini# Directory names that will be recursively excluded from backups (comma separated list!)
SKIP_DIRS= __pycache__, temp

# glob-style patterns to exclude files/folders from backups (used with Path.match(), Comma separated list!)
SKIP_PATTERNS= *.pyc, *.tmp, *.cacheThe filesystem scan is divided into two steps:
1. Just scan the filesystem tree
2. Filter and load meta data for every directory itemTheSKIP_DIRSis used in the first step.
TheSKIP_PATTERNSis used the the second step.Upgrading pyhardlinkbackupTo upgrade to a new version just start this helper script:Windows:phlb_upgrade_pyhardlinkbackup.cmdLinux:phlb_upgrade_pyhardlinkbackup.shSome notesWhat is ‘phlb’ and ‘manage’ ?!?phlbis a CLI.manageis similar to a normal Djangomanage.py, but it always
uses the pyhardlinkbackup settings.Why in hell do you use Django?!?Well, just because of the great database ORM and the Admin Site. ;)How to go into the Django admin?Just start:Windows:phlb_run_django_webserver.cmdLinux:phlb_run_django_webserver.shAnd then request ‘localhost’
(Note:–noreloadis needed for Windows with venv!)Running the unit testsJust start:phlb_run_tests.cmd/phlb_run_tests.shor do this:$ cd pyhardlinkbackup/
~/PyHardLinkBackup $ source bin/activate
(PyHardLinkBackup) ~/PyHardLinkBackup $ manage testUsing the CLI$ cd pyhardlinkbackup/
~/PyHardLinkBackup $ source bin/activate
(PyHardLinkBackup) ~/PyHardLinkBackup $ phlb --help
Usage: phlb [OPTIONS] COMMAND [ARGS]...

  pyhardlinkbackup

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  add     Scan all existing backup and add missing ones...
  backup  Start a Backup run
  config  Create/edit .ini config file
  helper  link helper files to given path
  verify  Verify a existing backupAdd missing backups to the databasephlb addcan be used in different scenarios:recreate the databaseadd a backup manuallyphlb adddoes this:scan the complete file tree underBACKUP_PATH(default:~/PyHardLinkBackups)recreate all hash filesadd all files to databasededuplicate with hardlinks, if possibleSo it’s possible to recreate the complete database:delete the current.sqlitefilerunphlb addto recreateAnother scenario is e.g.:DSLR images are stored on a network drive.You have already a copy of all files locally.You would like to add the local copy to pyhardlinkbackup.Do the following steps:move the local files to a subdirectory belowBACKUP_PATHe.g.:~/PyHardLinkBackups/pictures/2015-12-29-000015/Note: the date format in the subdirectory name must match theSUB_DIR_FORMATTERin your configrun:phlb addNow you can runphlb backupfrom your network drive to make a new, up-to-date backup.Windows DevelopmentSome notes about setting up a development environment on Windows:/dev/WindowsDevelopment.creoleAlternative solutionsAttic:https://attic-backup.org/(Not working on Windows, own backup archive format)BorgBackup:https://borgbackup.readthedocs.io/(Fork of Attic with lots of improvements)msbackup:https://pypi.python.org/pypi/msbackup/(Uses tar for backup archives)Duplicity:http://duplicity.nongnu.org/(No Windows support, tar archive format)Burp:http://burp.grke.org/(Client/Server solution)dirvish:http://www.dirvish.org/(Client/Server solution)restic:https://github.com/restic/restic/(Uses own backup archive format)See also:https://github.com/restic/others#list-of-backup-softwareHistorydev-compare v0.13.0…masterTBC18.03.2020 - v0.13.0 -compare v0.12.3…v0.13.0Dynamic chunk sizereplaceCHUNK_SIZEinPyHardLinkBackup.iniwithMIN_CHUNK_SIZEFix misleading error msg for dst OSError, bad exception handling#23Fix “run django server doesn’t work”#39Fix “add” commandDeactivate PyPy tests on Travis CI17.03.2020 - v0.12.3 -compare v0.12.2…v0.12.3Fix#44- wroing file size in process barusepytest-randomlyupdate requirements06.03.2020 - v0.12.2 -compare v0.12.1…v0.12.2Enhance log file contentUpdate requirementsFix too verbose output by decrease debug level05.03.2020 - v0.12.1 -compare v0.12.0…v0.12.1revert lowercasePyHardLinkBackupfor environment destination and default backup directory.05.03.2020 - v0.12.0 -compare v0.11.0…v0.12.0Refactor backup process: Usehttps://github.com/jedie/IterFilesystemfor less RAM usage and faster start on big source treesmodernized project/sources:Update to Django v2.2.x TLSuse poetryadd Makefilerun tests with pytest and toxrun tests only with python 3.8, 3.7, 3.6run tests on github, tooremove support for python 3.5 (os.scandirfallback removed)NOTE:Windows support is not tested, yet! (Help wanted)03.03.2019 - v0.11.0 -compare v0.10.1…v0.11.0Update boot filesUse django v1.11.x09.09.2016 - v0.10.1 -compare v0.10.0…v0.10.1BugfixCatch scan dir errors #2426.04.2016 - v0.10.0 -compare v0.9.1…v0.10.0move under Windows the install location from%APPDATA%\PyHardLinkBackupto%ProgramFiles%\PyHardLinkBackupto ‘migrate’: Just delete%APPDATA%\PyHardLinkBackupand install viaboot_pyhardlinkbackup.cmd26.04.2016 - v0.9.1 -compare v0.9.0…v0.9.1run migrate database in boot processAdd missing migrate scripts10.02.2016 - v0.9.0 -compare v0.8.0…v0.9.0Work-a-round for Windows MAX_PATH limit: Use\\?\path prefix internally.movePath2()to external lib:https://github.com/jedie/pathlib_revised04.02.2016 - v0.8.0 -compare v0.7.0…v0.8.0New: add all missing backups to database:phlb add(s.above)03.02.2016 - v0.7.0 -compare v0.6.4…v0.7.0New: verify a existing backupIMPORTANT:run database migration is needed!01.02.2016 - v0.6.4 -compare v0.6.2…v0.6.4Windows: Bugfix temp rename error, because of the Windows API limitation, see:#13Linux: Bugfix scanner if symlink is brokenDisplay local variables on low level errors29.01.2016 - v0.6.3 -compare v0.6.2…v0.6.3Less verbose and better information about SKIP_DIRS/SKIP_PATTERNS hits28.01.2016 - v0.6.2 -compare v0.6.1…v0.6.2Handle unexpected errors and continue backup with the next fileBetter handle interrupt key during execution28.01.2016 - v0.6.1 -compare v0.6.0…v0.6.1Bugfix #13 by using a better temp rename routine28.01.2016 - v0.6.0 -compare v0.5.1…v0.6.0New: faster backup by compare mtime/size only if old backup files exists27.01.2016 - v0.5.1 -compare v0.5.0…v0.5.1IMPORTANT:run database migration is needed!New.inisetting:LANGUAGE_CODEfor change translationmark if backup was finished compledDisplay information of last backup runAdd more information into summary file27.01.2016 - v0.5.0 -compare v0.4.2…v0.5.0refactory source tree scan. Split in two passed.CHANGESKIP_FILESin.iniconfig to:SKIP_PATTERNSBackup from newest files to oldest files.Fix#10:New–namecli option (optional) to force a backup name.Display error message if backup name can be found (e.g.: backup a root folder)22.01.2016 - v0.4.2 -compare v0.4.1…v0.4.2work-a-round for junction under windows, see also:https://www.python-forum.de/viewtopic.php?f=1&t=37725&p=290429#p290428(de)Bugfix in windows batches: go into work dir.print some more status information in between.22.01.2016 - v0.4.1 -compare v0.4.0…v0.4.1Skip files that can’t be read/write. (and try to backup the remaining files)21.01.2016 - v0.4.0 -compare v0.3.1…v0.4.0Search forPyHardLinkBackup.inifile in every parent directory from the current working dirincrease default chunk size to 20MBsave summary and log file for every backup run15.01.2016 - v0.3.1 -compare v0.3.0…v0.3.1fix unittest run under windows15.01.2016 - v0.3.0 -compare v0.2.0…v0.3.0database migration neededAdd ‘no_link_source’ to database (e.g. Skip source, if 1024 links created under windows)14.01.2016 - v0.2.0 -compare v0.1.8…v0.2.0good unittests coverage that covers the backup process08.01.2016 - v0.1.8 -compare v0.1.0alpha0…v0.1.8install and runable under Windows06.01.2016 - v0.1.0alpha0 -d42a5c5first Release on PyPi29.12.2015 -commit 2ce43commit ‘Proof of concept’Linkshttps://pypi.python.org/pypi/pyhardlinkbackup/https://www.python-forum.de/viewtopic.php?f=6&t=37723(de)https://github.com/jedie/pathlib_revisedDonatingpaypal.me/JensDiemerFlattr This!SendBitcoinsto1823RZ5Md1Q2X5aSXRC5LRPcYdveCiVX6FNote: this file is generated from README.creole2020-03-1810:02:44 with"python-creole"