Package: abgleich
ABGLEICHSYNOPSISabgleichis a simple ZFS sync tool. It displays source and target ZFS zpool, dataset and snapshot trees. It creates meaningful snapshots only if datasets have actually been changed. It compares a source zpool tree to a target, backup zpool tree. It pushes backups from a source to a target. It cleanes up older snapshots on the source side if they are present on the target side. It runs on a command line and produces nice, user-friendly, human-readable, colorized output. It also includes a GUI.CLI EXAMPLEGUI EXAMPLEsnapbackupcleanupINSTALLATIONThe base CLI tool can be installed as follows:pipinstall-vUabgleichAn installation also including a GUI can be triggered by running:pipinstall-vUabgleich[gui]RequiresCPython3.6 or later, aUnix shellandssh. GUI support requiresQt5in addition. Tested withOpenZFS0.8.x on Linux.abgleich, CPython and the Unix shell must only be installed on one of the involved systems. Any remote system will be contacted via ssh and provided with direct ZFS commands.INITIALIZATIONAll actions involving a remote host assume thatsshwith public key authentication instead of passwords is correctly configured and working.Let's assume that everything insource_tank/dataand below should be synced withtarget_tank/some_backup/data.source_tankandtarget_tankare zpools.datais the "prefix" for the source zpool,some_backup/datais the corresponding "prefix" for the target zpool. Forabgleichto work,source_tank/dataandtarget_tank/some_backupmust exist.target_tank/some_backup/datamust not exist. The latter will be created byabgleich. It is highly recommended to set the mountpoint oftarget_tank/some_backuptononebefore runningabgleichfor the first time.Rights to run the following commands are required:commandsourcetargetzfs listxxzfs getxxzfs snapshotxzfs sendxzfs receivexzfs destroyxconfig.yamlComplete example configuration file:source:zpool:tank_ssdprefix:host:localhostuser:port:target:zpool:tank_hddprefix:BACKUP_SOMEMACHINEhost:bigdatauser:zfsadminport:include_root:yeskeep_snapshots:2keep_backlog:Truealways_changed:nowritten_threshold:1048576check_diff:yessuffix:_backupdigits:2ignore:-home/user/CACHE-home/user/CCACHEssh:compression:nocipher:aes256-gcm@openssh.comcompatibility:target_samba_noshare:yestarget_autosnapshot_ignore:yeszpooldefines the name of the zpools on source and target sides. Theprefixvalue defines a "path" to a dataset underneath thezpool, so the name of the zpool itself is not part of theprefix. Theprefixcan be empty on either side. Prefixes can differ between source and target side.hostspecifies a value used byssh. It does not have to be an actual host name. It can also be an alias from ssh's configuration. If ahostis set tolocalhost,sshwont be used and theuserfield can be left empty or omitted. Both source and target can be remote hosts orlocalhostat the same time. Theportparameter specifies a customsshport. It can be left empty or omitted.sshwill then use its defaults or configuration to determine the correct port.include_rootindicates whether{zpool}{/{prefix}}should be included in all operations.keep_snapshotsis an integer and must be greater or equal to1. It specifies the number of snapshots that are kept per dataset on the source side when a cleanup operation is triggered.keep_backlogis either an integer or a boolean. It specifies if (or how many) snapshots are kept on the target side if the target side is cleaned. Snapshots that are part of the overlap with the source side are never considered for removal.suffixcontains the name suffix for new snapshots.Whether or not snapshots are generated is based on the following sequence of checks:Dataset is ignored: NODataset has no snapshot: YESIf thealways_changedconfiguration option is set toyes: YESIf thetaggingconfiguration option underneathcompatibilityis set to yes and the last snapshot of the dataset has not been tagged byabgleichas a backup: YESwrittenproperty of dataset equals0: NODataset is a volume: YESIf thewritten_thresholdconfiguration is set and thewrittenproperty of dataset is larger than the value ofwritten_threshold: YESIf thecheck_diffconfiguration option is set tono: YESIfzfs diffproduces any output relative to the last snapshot: YESOtherwise: NOSettingalways_changedtoyescausesabgleichto beliefe that all datasets have always changed since the last snapshot, completely ignoring what ZFS actually reports. No diff will be produced & checked for values ofwrittenlower thanwritten_threshold. Checking diffs can be completely deactivated by settingcheck_difftono.digitsspecifies how many digits are used for a decimal number describing the n-th snapshot per dataset per day as part of the name of new snapshots.ignorelists stuff underneath theprefixwhich will be ignored by this tool, i.e. no snapshots, backups or cleanups.sshallows to fine-tune the speed of backups. In fast local networks, it is best to setcompressiontonobecause the compression is usually slowing down the transfer. However, for low-bandwidth transmissions, it makes sense to set it toyes. For significantly better speed in fast local networks, make sure that both the source and the target system support a common cipher, which is accelerated byAES-NIon both ends. Thesshport can be specified per side via theportconfiguration option, i.e. for source and/or target.Custom pre- and post-processing can be applied aftersendand beforereceiveper side via shell commands specified in theprocessingconfiguration option (underneathsourceandtarget). This can be useful for a custom transfer compression based on e.g.lzmaorbzip2.compatibilityadds options for makingabgleichmore compatible with other tools. Iftarget_samba_noshareis active, thesharesmbproperty will - as part of backup operations - be set toofffor{zpool}{/{prefix}}on the target side, preventing sharing/exposing backup datasets by accident. Iftarget_autosnapshot_ignoreis active, thecom.sun:auto-snapshotproperty will - similarly as part of backup operations - be set tofalsefor{zpool}{/{prefix}}on the target side, tellingzfs-auto-snapshotto ignore the dataset.USAGEAll potentially changing or destructive actions are listed in detail before the user is asked to confirm them. None of the commands listed below create, change or destroy a zpool, dataset or snapshot on their own without the user's explicit consent.abgleich tree config.yaml [source|target]Show ZFS tree with snapshots, disk space and compression ratio. Appendsourceortarget(optional).abgleich snap config.yamlDetermine which datasets on the source side have been changed since last snapshot. Generate snapshots on the source side where applicable.abgleich compare config.yamlCompare source ZFS tree with target ZFS tree. See what is missing where.abgleich backup config.yamlSend (new) datasets and new snapshots from source to target.abgleich cleanup config.yaml [source|target]Cleanup older local snapshots on source side if they are present on both sides. Of those snapshots present on both sides, keep at leastkeep_snapshotsnumber of snapshots on source side. Or: Cleanup older snapshots on target side. Beyond the overlap with source, keep at leastkeep_backlogsnapshots. Ifkeep_backlogisFalse, all snapshots older than the overlap will be removed. Ifkeep_backlogisTrue, no snapshots will be removed. Ifabgleich cleanruns against the target side, an extra warning will be displayed and must be confirmed by the user before any dangerous actions are attempted.abgleich wizard config.yamlRuns a sequence ofsnap,backupandcleanupin a wizard GUI. This command is only available ifabgleichwas installed with GUI support.SPEEDabgleichuses Python'stype hintsand enforces them withtypeguardat runtime. It furthermore makes countless assertions.The enforcement of types and assertions can be controlled through thePYTHONOPTIMIZEenvironment variable. If set to0(the implicit default value), all checks are activated.abgleichwill run slow. For safety, this mode is highly recommended. For significantly higher speed, all type checks and most assertions can be deactivated by settingPYTHONOPTIMIZEto1or2, e.g.PYTHONOPTIMIZE=1 abgleich tree config.yaml. This is not recommended. You may want to check if another tool or configuration has altered this environment variable by runningecho $PYTHONOPTIMIZE.