Package: borgctl
borgctl - borgbackup without bash scriptsborgbackupis cool and easy to use. But we all end up writing bash scripts for creating/listing/pruning/checking/... backups. borgctl is a borg wrapper that uses yaml configuration files to make life a bit easier.Motivation (my use cases)I don't want to write a bash script on every server to make backups. I don't want to have an extra script for listing/pruning backups. I want to backup to multiple storage backends. I backup different things to different remote backends (money constraint). I don't want to store my borg passphrase on disk. I also don't want to enter it for every backend. I want to make backups in append-only-mode. To prune/compact backups, Yubikey authentication is mandatory. I want to see when my last backup was done (monitoring). I want usability and easy deployment.Now, I make backups withborgctl-cbackend1-append.yml-cbackend2-append.yml-cbackend3-append.ymlcreateI prune/compact backups with (needs Yubikey)borgctl-cbackend1-full.yml-cbackend2-full.yml-cbackend3-full.yml--cronMyi3status bar shows a redB:7dif my last backup is 7 days old. On some servers, I like tocreate,pruneandcompactbackups by just usingborgctl --cron. There is alsoborgmagic, but it does not handle all my use cases (to admit: I first reinvented the wheel and then stumbled across borgmatic. I also thought it just takes a day to write a small python script ...)FeaturesYaml configuration files: Specify what to backup and where to backupUsability: Just callborgctl create,borgctl list,borgctl prune, ...Flexibility: Run borgctl with multiple configuration files to handle different use cases (e. g. backup to multiple remote backends or runborg createin append-only-mode andborg prune/compactwith a Yubikey)Ask the user for the borg passphrase (if you don't want to store it on disk, just specifyaskas passphrase in the config file). Re-use the entered password for other remote storage backends (you don't have to re-enter it again)For semi-automated setups: There are some helpers to generate ssh keys and print the authorized_keys entry (restrict access to repository)Run multiple borg commands by runningborgctl --cron(likecreate+prune+compact, commands can be specified in the config file)Monitoring: Write a state file (text file with timestamp) after successfully executing borg commandsLogging: Write everything to a log file that automatically rotatesEasy to deploy and useQuickstartUsagekmille@linbox:~borgctl--help
usage:borgctl[-h][-l[LIST]][-d][-s][-a][-cCONFIG][--cron][-p][--version]{break-lock,check,compact,config,create,delete,diff,extract,export-tar,import-tar,info,init,key,list,mount,prune,rename,umount,upgrade,with-lock}...

borgctlisasimpleborgbackupwrapper.Runningversion0.4.9.

Theworkingdirectoryis/etc/borgctl/forrootor$XDG_CONFIG_HOME/borgctlor~/.config/borgctlfornon-rootusers.
Thelogdirectoryis/var/log/borgctl/forrootor$XDG_STATE_HOME/borgctlor~/.local/state/borgctlfornon-rootusers.

positionalarguments:{break-lock,check,compact,config,create,delete,diff,extract,export-tar,import-tar,info,init,key,list,mount,prune,rename,umount,upgrade,with-lock}options:-h,--helpshowthishelpmessageandexit-l[LIST],--list[LIST]listexistingborgctlconfigfilesin/home/kmille/.config/borgctl-d,--generate-default-configwritedefaultconfigto/home/kmille/.config/borgctl/default.ymlorprintsittostdoutifthefilealreadyexists-s,--generate-ssh-keywriteanewed25519sshkeyto~/.ssh/borg_$configandupdatetheconfigfile-a,--generate-authorized-keysprinttheauthorized_keysentry.Italsoshowstheretrictedentry-cCONFIG,--configCONFIGspecifytheconfigfiletouse.Defaultstodefault.yml.Youcanspecifymultipleconfigfileswith-cdefault.yml-clocal-disk.yml.Iftheconfigfilecontainsa/,thenarelative/absolutepathisasumed.Ifnot,/home/kmille/.config/borgctl/$configwillbeused--cronrunmultipleborgcommandsinarow.Thecommandstorunarespecifiedintheconfigfile(cron_commands)-p,--generate-passphrasegenerateadicewarelikepassphrase--versionshowversionandexitInstallationYou can find the latest release on theGithub Releasepage. There are different ways to get borgctl up and running.Use thePyPipackage withpipinstallborgctlThe source and binary package can also be found on the Release page (*.whl and *.tar.gz)Use the Debian packageInstall the latest package with:cd/tmpexportLATEST_RELEASE=$(curl-shttps://api.github.com/repos/kmille/borgctl/tags|jq-r'.[0].name')wget"https://github.com/kmille/borgctl/releases/download/$LATEST_RELEASE/borgctl_$LATEST_RELEASE-1_amd64.deb"aptinstall./borgctl*.deb
rm./borgctl*.debThe deb package should work on all Debian based distros. It is created bycreate-deb-package.sh. It requiresborgbackupandpython3-ruamel.yamlas dependencies.Use theArch Linux/Manjaro packagewithyayborgctlYou can also download borgctl-*any.pkg.tar.zst and install it withpacman -U.Configurationborgctl uses config files. If you runborgctlwithout specifying a config file, it expects a default.yml in the config directory. You can specify one or more config files with-c/--config. If the config file contains a /, the config file is interpreted as relative/absolute path. There is also a logging configuration (logging.conf) stored, which is used by borg and borgctl itself.Default config location for root user:/etc/borgctlDefault config location for non-root users:$XDG_CONFIG_HOME/borgctlor~/.config/borgctlThe output of borg and borgctl will be written to borg.log. The file gets logrotated automatically.Default log directory for root user:/var/log/borgctl/Default log directory for non-root users:$XDG_STATE_HOME/borgctlor~/.local/state/borgctlThe state files are also written to the log directory. You can useborgctl --listto list all config files.For example:kmille@linbox:~sudoborgctl--list
backend1-append.yml
...

kmille@linbox:~sudoborgctllist2024-01-0910:37:17,967INFOUsingconfigfile/etc/borgctl/default.yml

kmille@linbox:~sudoborgctl-cbackend1-append.ymllist2024-01-0910:37:55,148INFOUsingconfigfile/etc/borgctl/backend1-append.yml

kmille@linbox:/etcsudoborgctl-cborgctl/backend1-append.ymllist2024-01-0910:38:47,498INFOUsingconfigfileborgctl/backend1-append.ymlDefault configuration file generated byborgclt --generate-default-config:---repository:""ssh_key:""prefix:""passphrase:""mount_point:"/mnt"borg_create_backup_dirs:-"/"borg_create_excludes:-".cache"borg_create_arguments:-"--one-file-system"-"--compression=lz4"borg_prune_arguments:-"--keep-last10"-"--dry-run"cron_commands:-"create"-"prune"-"compact"state_commands:-"create"-"prune"# https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variablesenvs:"BORG_RELOCATED_REPO_ACCESS_IS_OK":"yes"borg_binary:"/usr/bin/borg"Walkthrough/How borgctl behavesborgctl needs a configuration file. If you run it without specifying one, a default.yml in the default config location is expected to exist. If you run it and there is no default.yml, you can use--generate-default-configto create one:root@linbox:~borgctllist2023-12-2610:58:31,564INFOUsingconfigfile/etc/borgctl/default.yml2023-12-2610:58:31,564ERRORCouldnotloadconfigfile/etc/borgctl/default.yml
Pleaseuse--generate-default-configtocreateadefaultconfig

root@linbox:~borgctl--generate-default-config2023-12-2610:59:28,107INFOPleasemakeabackupofthepassphrase!2023-12-2610:59:28,112INFOSuccessfullywroteconfigfileto/etc/borgctl/default.yml/etc/borgctl/default.ymlis based on thedefault config fileand gets new diceware-like password for the borg passphrase . Also the hostname is set as borg prefix. It's crucial to make a backup of the passphrase!Generating ssh keysroot@linbox:~borgctl--generate-ssh-key2023-12-2611:06:12,784INFOUsingconfigfile/etc/borgctl/default.yml2023-12-2611:06:12,785INFORunning['ssh-keygen','-t','ed25519','-N','','-q','-f','/root/.ssh/borg_default','-C','kmille_borg_default@linbox']2023-12-2611:06:12,794INFOSuccessfullycreatedsshkey/root/.ssh/borg_default2023-12-2611:06:12,805INFOUpdatedssh_keyin/etc/borgctl/default.yml
root@linbox:~cat/root/.ssh/borg_default.pubssh-ed25519AAAAC3NzaC1lZDI1NTE5AAAAINpVZZjIAzeyGq0oLKVeLzEECoe7RXg6YpAcPIsakboFkmille_borg_default@linboxIf you specify a ssh key in the config file, it will be used as output file (if it does not exist). If not, the default name for the ssh key is$currentUsername_borg_$configPrefix.Generating authorized_keys entryroot@linbox:~borgctl--generate-authorized_keys2023-12-2611:07:47,549INFOUsingconfigfile/etc/borgctl/default.yml2023-12-2611:07:47,551INFOUsingsshkey/root/.ssh/borg_default.pubfromconfigfile2023-12-2611:07:47,551INFOAddthislinetoauthorized_keys:
ssh-ed25519AAAAC3NzaC1lZDI1NTE5AAAAINpVZZjIAzeyGq0oLKVeLzEECoe7RXg6YpAcPIsakboFkmille_borg_default@linbox2023-12-2611:07:47,551INFOUsethislineforrestrictedaccess:command="borg serve --restrict-to-path /opt/test-backup",restrictssh-ed25519AAAAC3NzaC1lZDI1NTE5AAAAINpVZZjIAzeyGq0oLKVeLzEECoe7RXg6YpAcPIsakboFkmille_borg_default@linbox2023-12-2611:07:47,553INFOOrthisall-in-onecommand:echo-e'command="borg serve --restrict-to-path /opt/test-backup",restrict ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINpVZZjIAzeyGq0oLKVeLzEECoe7RXg6YpAcPIsakboF kmille_borg_default@linbox\n'|sshbackup-host1'cat >> /home/backuper/.ssh/authorized_keys'List all config filesroot@linbox:~borgctl--list
backend1-append.yml
backend2-append.yml
backend1-full.yml
backend2-full.yml

root@linbox:~borgctl--listfull
backend1-full.yml
backend2-full.ymlRunning borg commands with borgctlPlease update the config file (what to backup (borg_create_backup_dirsandborg_create_excludesand where to backup (repository)). Then you can create your first backup:root@linbox:~borgctlcreate2023-12-2611:11:35,197INFOUsingconfigfile/etc/borgctl/default.yml2023-12-2611:11:35,199INFOExecuting:BORG_REPO="/root/borg-repo"BORG_LOGGING_CONF="/etc/borgctl/logging.conf"BORG_RSH="ssh -i /root/.ssh/borg_default"BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"/usr/bin/borg--verbosecreate--progress--stats--one-file-system--compression=lz4--exclude=.cache::linbox_2023-12-26_11:11:35/usr/bin2023-12-2611:11:35,718INFOCreatingarchiveat"/root/borg-repo::linbox_2023-12-26_11:11:35"2023-12-2611:12:07,934INFO------------------------------------------------------------------------------2023-12-2611:12:07,935INFORepository:/root/borg-repo2023-12-2611:12:07,936INFOArchivename:linbox_2023-12-26_11:11:352023-12-2611:12:07,937INFOArchivefingerprint:079e9c226455ff1b816be986bf49029c3cdf28ec04e9f6bf2d387ea9a4f24d872023-12-2611:12:07,937INFOTime(start):Tue,2023-12-2611:11:352023-12-2611:12:07,938INFOTime(end):Tue,2023-12-2611:12:062023-12-2611:12:07,938INFODuration:31.04seconds2023-12-2611:12:07,939INFONumberoffiles:38692023-12-2611:12:07,940INFOUtilizationofmax.archivesize:0%2023-12-2611:12:07,940INFO------------------------------------------------------------------------------2023-12-2611:12:07,941INFOOriginalsizeCompressedsizeDeduplicatedsize2023-12-2611:12:07,942INFOThisarchive:1.55GB777.29MB777.13MB2023-12-2611:12:07,942INFOAllarchives:1.55GB777.29MB777.43MB2023-12-2611:12:07,943INFO2023-12-2611:12:07,943INFOUniquechunksTotalchunks2023-12-2611:12:07,944INFOChunkindex:431043222023-12-2611:12:07,945INFO------------------------------------------------------------------------------2023-12-2611:12:08,101INFOUpdatedstatefile/var/log/borgctl/borg_state_default_create.txt
root@linbox:~root@linbox:~cat/var/log/borgctl/borg_state_default_create.txt2023-12-26_11:12:08#root@linbox:~root@linbox:~ls/var/log/borgctl/borg.log-rw-r--r--1rootroot11KDec2611:12/var/log/borgctl/borg.logThe executed borg command is also printed to stderr. You can adjust the parameters borgctl uses. For example you can add--json /usr/local/bin/, which will be added to borg:root@linbox:~borgctlcreate--json/usr/local/bin2023-12-2611:15:42,954INFOUsingconfigfile/etc/borgctl/default.yml2023-12-2611:15:42,956INFOExecuting:BORG_REPO="/root/borg-repo"BORG_LOGGING_CONF="/etc/borgctl/logging.conf"BORG_RSH="ssh -i /root/.ssh/borg_default"BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"/usr/bin/borg--verbosecreate--progress--stats--one-file-system--compression=lz4--exclude=.cache--json::linbox_2023-12-26_11:15:42/usr/bin--json/usr/local/bin2023-12-2611:15:43,518INFOCreatingarchiveat"/root/borg-repo::linbox_2023-12-26_11:15:42"{"archive":{"command_line":["/usr/bin/borg",
...Adjusting default arguments for single borg commandsFor every borg command, you can add command line parameter/arguments in the config file by specifyingborg_$borgcommand_arguments. They are used automatically. For example, here are default arguments for the create command:borg_create_arguments:-"--one-file-system"-"--compression=lz4"For convenient mounting, I like to use:borg_mount_arguments:-"--strip-components=2"-"--foreground"-"--last1"Specifying an archiveIf you want to specify an archive, you have to prepend ::root@linbox:~borgctllist2023-12-2611:18:38,323INFOUsingconfigfile/etc/borgctl/default.yml2023-12-2611:18:38,325INFOExecuting:BORG_REPO="/root/borg-repo"BORG_LOGGING_CONF="/etc/borgctl/logging.conf"BORG_RSH="ssh -i /root/.ssh/borg_default"BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"/usr/bin/borg--verboselist
linbox_2023-12-26_11:11:35Tue,2023-12-2611:11:35[079e9c226455ff1b816be986bf49029c3cdf28ec04e9f6bf2d387ea9a4f24d87]linbox_2023-12-26_11:15:42Tue,2023-12-2611:15:43[4599ac310b99703407600bc8bfa58ddb80da7be3d613b49fdf75691b99d293e0]root@linbox:~root@linbox:~borgctllistlinbox_2023-12-26_11:11:352023-12-2611:19:06,975INFOUsingconfigfile/etc/borgctl/default.yml2023-12-2611:19:06,977INFOExecuting:BORG_REPO="/root/borg-repo"BORG_LOGGING_CONF="/etc/borgctl/logging.conf"BORG_RSH="ssh -i /root/.ssh/borg_default"BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"/usr/bin/borg--verboselistlinbox_2023-12-26_11:11:35
Remote:ssh:Couldnotresolvehostnamelinbox_2023-12-26_11:Nameorservicenotknown2023-12-2611:19:07,436ERRORConnectionclosedbyremotehost.Isborgworkingontheserver?2023-12-2611:19:07,518ERRORborgfailedwithexitcode:2root@linbox:~root@linbox:~borgctllist::linbox_2023-12-26_11:11:352023-12-2611:18:33,787INFOUsingconfigfile/etc/borgctl/default.yml2023-12-2611:18:33,789INFOExecuting:BORG_REPO="/root/borg-repo"BORG_LOGGING_CONF="/etc/borgctl/logging.conf"BORG_RSH="ssh -i /root/.ssh/borg_default"BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"/usr/bin/borg--verboselist::linbox_2023-12-26_11:11:35
drwxr-xr-xrootroot0Tue,2023-12-2610:56:58usr/bin-rwxr-xr-xrootroot21905Fri,2023-12-2220:59:55usr/bin/tzselect
-rwxr-xr-xrootroot51216Fri,2023-12-2220:59:55usr/bin/zdump
-rwxr-xr-xrootroot59448Fri,2023-12-2220:59:55usr/bin/zic-rwxr-xr-xrootroot27176Sat,2023-03-1817:07:07usr/bin/gdbm_dump
...Running multiple borg commands with --cron--cronis nice if you want to have a cronjob that creates, prunes and compacts backups. In the config file, you can specify which borg commands should be run if--cronis supplied:cron_commands:-"create"-"prune"-"compact"Example outupt:root@linbox:~borgctl--cron2023-12-2611:22:06,988INFOUsingconfigfile/etc/borgctl/default.yml2023-12-2611:22:06,990INFORunning'borg create'in--cronmode2023-12-2611:22:06,990INFOExecuting:BORG_REPO="/root/borg-repo"BORG_LOGGING_CONF="/etc/borgctl/logging.conf"BORG_RSH="ssh -i /root/.ssh/borg_default"BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"/usr/bin/borg--verbosecreate--progress--stats--one-file-system--compression=lz4--exclude=.cache::linbox_2023-12-26_11:22:06/usr/bin2023-12-2611:22:07,532INFOCreatingarchiveat"/root/borg-repo::linbox_2023-12-26_11:22:06"...2023-12-2611:22:08,338INFOUpdatedstatefile/var/log/borgctl/borg_state_default_create.txt2023-12-2611:22:08,338INFORunning'borg prune'in--cronmode2023-12-2611:22:08,338INFOExecuting:BORG_REPO="/root/borg-repo"BORG_LOGGING_CONF="/etc/borgctl/logging.conf"BORG_RSH="ssh -i /root/.ssh/borg_default"BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"/usr/bin/borg--verboseprune--list--keep-last102023-12-2611:22:08,898INFOKeepingarchive(rule:secondly#1):     linbox_2023-12-26_11:22:06           Tue, 2023-12-26 11:22:07 [9be95c7822b2e195fbcdfa63787d20ceb13ba04a7d560d627f2f92941e1dc23e]2023-12-2611:22:08,898INFOKeepingarchive(rule:secondly#2):     linbox_2023-12-26_11:21:35           Tue, 2023-12-262023-12-2611:22:08,969INFOUpdatedstatefile/var/log/borgctl/borg_state_default_prune.txt2023-12-2611:22:08,970INFORunning'borg compact'in--cronmode2023-12-2611:22:08,970INFOExecuting:BORG_REPO="/root/borg-repo"BORG_LOGGING_CONF="/etc/borgctl/logging.conf"BORG_RSH="ssh -i /root/.ssh/borg_default"BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"/usr/bin/borg--verbosecompact2023-12-2611:22:09,459INFOcompactionfreedabout1.48kBrepositoryspace.
root@linbox:~Monitoring borg: state filesborgctl also writes state files, if a borg command runs successfully. It contains the current date. You can use it for monitoring. State files are written to the log directory. The format isborg_state__$config_file_prefix_$borg_command.txt. In the config file you can specify a list of commands for which a state file should be created.state_commands:-"create"-"prune"root@linbox:~cat/var/log/borgctl/borg_state_default_create.txt2023-12-26_11:22:08borg passphrase: ask and ask-alwaysIf you don't want to keep your borg passphrase on your disk, you can useaskorask-always. If you specifyaskin your config file as passphrase, you will be ask during runtime for the passphrase. The difference betweenaskandask-always: That's interesting if you run borgctl with multiple config files and different configurations. You always get asked for the password if you specifyask-always. If you run borgctl with three config files and every config file hasaskspecified, it only asks you for the first run. Then, it uses the previously entered password.MiscIn the config file, you can specify the borg binary (borg_binary) used for invocation. You can also add environment variables. If you need help for a borg command, you can just addhelp(likeborgctl list help). You can change default arguments for specific borg commands by adding/modifyingborg_$command_argumentsin the config file (likeborg_prune_arguments).Using the Yubikey for authenticationIf you want to authenticate with a Yubikey without touching~/.ssh/config, you can keepssh_keyempty and add"BORG_RSH":"ssh-I/usr/lib64/pkcs11/opensc-pkcs11.so-oForwardAgent=no-oIdentityAgent=no"as environment variable in the config file. But you can also stay with your.ssh/configif you like.LoggingIn the config directory you will find a file called logging.conf. It is used by borgctl and borg itself for logging. The borg output (stdout and stderr) is printed to stdout, the borgctl output is printed to stderr. Everything gets logged to borg.log, which can be found in the log directory. The log file gets rotated automatically after reaching 1 megabyte. Docs can be foundhereandhere.Hereare the docs from borgbackup about logging.Monitoring with py3statusI usemonitoring_last_backup.pywithpy3statuson my laptop. It showsB:3dwhen my last backup was 3 days ago. The color changes from green to red after 7 days. You can runpython3 monitoring_last_backup.pyto test it.DevelopmentYou needpoetry. Get get started, usepoetry install. Then you can runpoetry run borgctl --help. The code can be found in the borgctl directory. Deployment/monitoring stuff is in the contrib directory. Tests can be run withpoetry run pytest tests -s -v -x.  To run the mypy checks, runpoetry run mypy. Mypy is configured inpyproject.toml.