Package: dup-composer
Dup-composerDup-composer is a front-end script for Duplicity, that lets you define your backups in a configuration file and execute them in a simple way.This is work in progress, hence it is not ready for production use yet.To get started:InstallDuplicityversion 0.7 or newer.InstallDup-composer.Create your YAML configuration file and define your backup groups configuration in this file.Executedupcomp, define the groups you wish to execute and whether you want to do abackuporrestore.You can find more detail for each step in its respective section.RequirementsDuplicity 0.7+ installed and theduplicityscript on your PATH.Python 3.5+Keyring 19.2+*SecretStorage 3.1.1+*PyYAML 5.1+** Will be installed bypipautomatically.InstallationInstall fromPyPI:pip3 install dup-composerConfigurationThe configuration should followYAML 1.1syntax.Let's start at the top, with the list of backup groups - the three dots (...) are placeholders for child and scalar nodes:backup_groups:my_first_backup_group:...my_second_backup_group:...The parent node of the groups is calledbackup_groups, which is currently the root of the configuration structure, but further configuration nodes might be added on the top level in the future.Backup groupFor each backup group, you have to have the following structure in place:my_first_backup_group:encryption:...backup_provider:...backup_file_prefixes:...volume_size:...sources:...Theencryptionnode (mandatory) is the parent of the encryption related configuration, the children ofbackup_provider(mandatory) specify all the provider related properties.backup_file_prefixesis optional and contains the child nodes for archive, signature and manifest file prefix configuration. Thevolumes_size(mandatory) node determines the size of the backup archive file chunks in MBs.EncryptionThere are primarily two ways to set up theencryptionnode at the moment.Encryption is turned off:encryption:enabled:noEncryption is turned on:encryption:enabled:yesgpg_key:123456789ABCDEFFgpg_passphrase:examplepassphrase123If theenablednode is set tono, encryption is disabled, there is no need to configure thegpg_keyandgpg_passphrasenodes. When encryption is enabled however, they are mandatory. This key will be used both for signing and encrypting the backup data.Keyring support:The value ofgpg_passphrasecan also be read from a keyring. See thekeyring support documentfor details on setting up the keyring to use withDup-composer. Once the keyring has been set up, you can specify the passphrase in the following format:gpg_passphrase: ['service_name_in_the_keyring', 'account_name_in_the_keyring'].Backup providerThebackup_providerconfiguration to be configured largely depends on the type of the provider, determined by the URL scheme:backup_provider:url:file://This configuration setsDup-composerup to save the backup files on thelocal filesystem. There is no need to specify a concrete path here, as that will be determined by thesourcessection of the configuration. The URL will just set thecontextfor those paths.For aremote SFTP or SCP backup, you need a slightly different configuration:backup_provider:url:sftp://myscpuser@host.example.com/password:examplepassword123In this case, you need to specify the username of the remoteSFTPhost in the first part of theSFTPURL, which is what you would do usingDuplicitydirectly as well. Use thepasswordnode to specify the password.You can useSCPthe same way, but use thescp://protocol in the URL instead ofsftp://.Keyring support:The value ofpasswordcan also be read from a keyring. See thekeyring support documentfor details on setting up the keyring to use withDup-composer. Once the keyring has been set up, you can specify the password in the following format:password: ['service_name_in_the_keyring', 'account_name_in_the_keyring'].Finally, you have to configureAWS S3like this:backup_provider:url:s3://s3.sa-east-1.amazonaws.com/my-backup-bucketaws_access_key:EXAMPLEACCESSKEYaws_secret_key:ExAmPlESeCrEtKeYThe S3 bucket URL is configured as theurlnode value, whileaws_access_keyandaws_secret_keyneed to contain yourAWSgenerated keys for the bucket. Like with the rest of the providers, the actual path, folder, within the bucket shouldn't be added to the URL.Keyring support:The value ofaws_secret_keycan also be read from a keyring. See thekeyring support documentfor details on setting up the keyring to use withDup-composer. Once the keyring has been set up, you can specify the secret key in the following format:aws_secret_key: ['service_name_in_the_keyring', 'account_name_in_the_keyring'].Backup file prefixesThe next feature comes handy if you want toprefix the generated backup file namesin a specific way. I use this to set upbucket rulesinS3, that move my archive files toGlacier. Here is an example of the configuration:backup_file_prefixes:manifest:manifest_archive:archive_signature:signature_The prefixes can be specifically set up for each file type generated at the backup location. Set these up as needed; you can leave thebackup_file_prefixesnode out altogether, if you don't need this feature.VolumesThevolume_sizenode is rather simple: a number should be given as its value; this determines thearchive size in megabytes.SourcesUnder thesourcesnode in the configuration hierarchy, you can specify alist of locations(paths) you want to back up, where to back them up and where the restored data should go. You can set up multiple sources within a single group. Here is an example set of two sources configured:sources:/var/www/html:backup_path:/root/backupsrestore_path:/root/restored/home/tommy:backup_path:/home/bkup/my-laptop-backupsrestore_path:/root/restored-from-backupThe source child nodes/var/www/htmland/home/tommydeterminethe directory you want to back up, andbackup_pathprescribesthe location the backup files will be saved to. In practice, the value ofbackup_pathwill be appended to the value of the providerurlnode discussed earlier; hence these two fragments give the true backup location.restore_pathis not used during the backup step, but specifying it is mandatory at the moment. I will remove this requirement very soon, as it doesn't make any sense, until an actual restore has to happen.There are a few limitations on the path data provided in this configuration:They can't begin with a hyphen "-".They can't contain any backslash characters.They can't contain any newline characters.Extra: Make sure to check quoting rules if you need to add any characters, that have a special meaning in theYAMLsyntax.Include and exclude filtersYou can configure a list of pathsincludedin orexcludedfrom the backup for each backup source. The configuration ends up being passed toDuplicitythrough the--includeand--excludecommand line options.For example, taking the following configuration filefiltered-backup.yml:backup_groups:my_filtered_backup:encryption:enabled:nobackup_provider:url:file://volume_size:400sources:/var/www/html:backup_path:/home/backups/web_server_docrootrestore_path:/var/www/htmlfilters:-type:excludepath:/var/www/html/no_bak-type:includepath:/var/www/thml/no_bak/importantand running Dup-composer like this:dupcomp-cfiltered-backup.ymlbackupwill result in the followingDuplicitycommand:duplicity--no-encryption--volsize400--exclude/var/www/html/no_bak--include/var/www/html/no_bak/important/var/www/htmlfile:///home/backups/web_server_docrootImportant:You can list as manyincludeandexcludefilters as needed.The order in which you list the filters matters, as it will be passed in the same order toDuplicity. Please consult theDuplicity man pagesto understand how the order impacts the processing.The filters are only supported for backups.Examplebackup-compose.ymlexample:backup_groups:my_local_backups:encryption:enabled:nobackup_provider:url:file://volume_size:200sources:/var/www/html:backup_path:/root/backups/var/www/htmlrestore_path:/root/restored/var/www/html/home/tommy:backup_path:/root/backups/home/tommyrestore_path:/root/restored/home/tommymy_s3_backups:encryption:enabled:yesgpg_key:xxxxxxgpg_passphrase:xxxxxxbackup_provider:url:s3://s3.sa-east-1.amazonaws.com/my-backup-bucketaws_access_key:xxxxxxaws_secret_key:xxxxxxbackup_file_prefixes:manifest:manifest_archive:archive_signature:signature_volume_size:50sources:/etc:backup_path:/etcrestore_path:/root/restored/etc/home/shared:backup_path:/home/sharedrestore_path:/root/restored/home/sharedmy_sftp_backups_with_keyring:encryption:enabled:nobackup_provider:url:sftp://myscpuser@backuphost1.example.com/password:['backuphost1','myscpuser']keyring:username:keyringuserbus_address:/run/user/1003/busvolume_size:200sources:/home/fun:backup_path:/home/funrestore_path:/root/restored/home/fun/home/katy:backup_path:/home/katyrestore_path:/root/restored/home/katyConfiguration change safeguardTo reduce the risk of unintentional changes, or changes that might have a negative effect on existing backup chains,Dup-composerwill print an informational message and abort the execution if any of the existing backup groups are changed. The user can verify, if the change is intentional and if so, rerunDup-composerwith the-soption to disable the safeguard.Usagedupcomp-h
-----
usage:dupcomp.py[-d][-s][-f][-c<configpath>]backup|restore[backup_group1backup_group2...]optionalarguments:-ddryrun(justprintthecommandstobeexecuted)-c<configpath>usetheconfigurationfileat<configpath>-sskiptheconfigurationchangesafeguardstep-fforcefullbackup-----