Package: rsync-system-backup
The rsync-system-backup program usesrsyncto create full system backups ofLinuxsystems. Supported backup destinations include local disks (possibly
encrypted usingLUKS) and remote systems that are running anSSHserver orrsync daemon. Each backup produces a timestamped snapshot and these
snapshots are rotated according to a rotation scheme that you can configure.
The package is currently tested on cPython 2.7, 3.4, 3.5, 3.6, 3.7 and PyPy
(2.7).StatusInstallationUsageCommand lineHow it worksThe (lack of) backup formatSnapshots and hard linksRotation of snapshotsBackup destinationsEncrypted USB disksLocal server (rsync daemon)Remote server (rsync daemon over SSH tunnel)Alternative connection methodsUsing rsync daemonEnabling rsync daemonTunneling rsync daemon connectionsContactLicenseStatusWhile this project brings together more than ten years of experience in
creating (system) backups usingrsync, all of the actual Python code was
written in the first few months of 2017 and has seen limited real world use.
The project does however have an automated test suite with more than 90% test
coverage and my intention is to extend the test coverage further.In May 2018 I changed the status from alpha to beta as part of release 1.0. The
bump in major version number was triggered by a backwards incompatible code
change, however at that point I had been usingrsync-system-backupto make
local backups of several of my Linux systems for the majority of a year. Also
several colleagues of mine have used the how-to on setting up unattended
backups to an encrypted USB disk.WarningPlease use the--dry-runoption when you’re getting familiar
with howrsync-system-backupworks and don’t remove the option
until you’re confident that you have the right command line,
because usingrsync-system-backupin the wrong way can cause
data loss (for example by accidentally swapping theSOURCEandDESTINATIONarguments).InstallationThersync-system-backuppackage is available onPyPIwhich means
installation should be as simple as:$pipinstallrsync-system-backupThere’s actually a multitude of ways to install Python packages (e.g. theper
user site-packages directory,virtual environmentsor just installing
system wide) and I have no intention of getting into that discussion here, so
if this intimidates you then read up on your options before returning to these
instructions ;-).UsageThere are two ways to use thersync-system-backuppackage: As the command
line programrsync-system-backupand as a Python API. For details about the
Python API please refer to the API documentation available onRead the Docs.
The command line interface is described below.Command lineUsage:rsync-system-backup [OPTIONS] [SOURCE] DESTINATIONUse rsync to create full system backups.The required DESTINATION argument specifies the (possibly remote) location
where the backup is stored, in the syntax of rsync’s command line interface.
The optional SOURCE argument defaults to ‘/’ which means the complete root
filesystem will be included in the backup (other filesystems are excluded).Please use the--dry-runoption when getting familiar with this program and
don’t remove it until you’re confident that you have the right command line,
because using this program in the wrong way can cause data loss (for example
by accidentally swapping the SOURCE and DESTINATION arguments).Supported locations include:Local disks (possibly encrypted using LUKS).Remote systems that allow SSH connections.Remote systems that are running an rsync daemon.Connections to rsync daemons tunneled over SSH.The backup process consists of several steps:First rsync is used to transfer all (relevant) files to a destination
directory (whether on the local system or a remote system). Every time
a backup is made, this same destination directory is updated.After the files have been transferred a ‘snapshot’ of the destination
directory is taken and stored in a directory with a timestamp in its
name. These snapshots are created using ‘cp--archive--link’.Finally the existing snapshots are rotated to purge old backups
according to a rotation scheme that you can customize.Supported options:OptionDescription-b,--backupCreate a backup using rsync but don’t create a snapshot and don’t rotate
old snapshots unless the--snapshotand/or--rotateoptions are also given.-s,--snapshotCreate a snapshot of the destination directory but don’t create a backup
and don’t rotate old snapshots unless the--backupand/or--rotateoptions
are also given.This option can be used to create snapshots of an rsync daemon module using
a ‘post-xfer exec’ command. If DESTINATION isn’t given it defaults to the
value of the environment variable$RSYNC_MODULE_PATH.-r,--rotateRotate old snapshots but don’t create a backup and snapshot unless the--backupand/or--snapshotoptions are also given.This option can be used to rotate old snapshots of an rsync daemon module
using a ‘post-xfer exec’ command. If DESTINATION isn’t given it defaults to
the value of the environment variable$RSYNC_MODULE_PATH.-m,--mount=DIRECTORYAutomatically mount the filesystem to which backups are written.When this option is given andDIRECTORYisn’t already mounted, the
‘mount’ command is used to mount the filesystem to which backups are
written before the backup starts. When ‘mount’ was called before the
backup started, ‘umount’ will be called when the backup finishes.An entry for the mount point needs to be
defined in /etc/fstab for this to work.-c,--crypto=NAMEAutomatically unlock the encrypted filesystem to which backups are written.When this option is given and theNAMEdevice isn’t already unlocked, the
cryptdisks_start command is used to unlock the encrypted filesystem to
which backups are written before the backup starts. When cryptdisks_start
was called before the backup started, cryptdisks_stop will be called
when the backup finishes.An entry for the encrypted filesystem needs to be defined in /etc/crypttab
for this to work. If the device of the encrypted filesystem is missing and
rsync-system-backup is being run non-interactively, it will exit gracefully
and not show any desktop notifications.If you want the backup process to run fully unattended you can configure a
key file in /etc/crypttab, otherwise you will be asked for the password
each time the encrypted filesystem is unlocked.-t,--tunnel=TUNNEL_SPECConnect to an rsync daemon through an SSH tunnel. This provides encryption
for rsync client to daemon connections that are not otherwise encrypted.
The value ofTUNNEL_SPECis expected to be an SSH alias, host name or IP
address. Optionally a username can be prefixed (followed by ‘@’) and/or a
port number can be suffixed (preceded by ‘:’).-i,--ionice=CLASSUse the ‘ionice’ program to set the I/O scheduling class and priority of
the ‘rm’ invocations used to remove backups.CLASSis expected to be one of
the values ‘idle’, ‘best-effort’ or ‘realtime’. Refer to the man page of
the ‘ionice’ program for details about these values.-u,--no-sudoBy default backup and snapshot creation is performed with superuser
privileges, to ensure that all files are readable and filesystem
metadata is preserved. The-u,--no-sudooption disables
the use of ‘sudo’ during these operations.-n,--dry-runDon’t make any changes, just report what would be done. This doesn’t
create a backup or snapshot but it does run rsync with the--dry-runoption.--multi-fsAllow rsync to cross filesystem boundaries. This option has the opposite
effect of the rsync option--one-file-systembecause rsync-system-backup
defaults to running rsync with--one-file-systemand must be instructed
not to using--multi-fs.-x,--exclude=PATTERNSelectively exclude certain files from being included in the backup.
Refer to the rsync documentation for allowedPATTERNsyntax. Note that
rsync-system-backup always uses the ‘rsync--one-file-system’ option.-f,--forceBy default rsync-system-backup refuses to run on non-Linux systems because
it was designed specifically for use on Linux. The use of the-f,--forceoption sidesteps this sanity check. Please note that you are on your own if
things break!--disable-notificationsBy default a desktop notification is shown (using notify-send) before the
system backup starts and after the backup finishes. The use of this option
disables the notifications (notify-send will not be called at all).-v,--verboseMake more noise (increase logging verbosity). Can be repeated.-q,--quietMake less noise (decrease logging verbosity). Can be repeated.-h,--helpShow this message and exit.How it worksI’ve been finetuning my approach to Linux system backups for years now and
during that timersynchas become my swiss army knife of choice :-). I also
believe that comprehensive documentation can be half the value of an open
source project. The following sections attempt to provide a high level
overview of my system backup strategy:The (lack of) backup formatSnapshots and hard linksRotation of snapshotsBackup destinationsThe (lack of) backup formatEach backup is a full copy of the filesystem tree, stored in the form of
individual files and directories on the destination. This “backup format” makes
it really easy to navigate through and recover from backups because you can use
whatever method you are comfortable with, whether that is a file browser,
terminal,Pythonscript or evenchroot:-).NoteYou may want to configureupdatedbto exclude the directory
containing your system backups, otherwise thelocatedatabase
will grow enormously.Snapshots and hard linksEvery time a backup is made the same destination directory is updated with
additions, updates and deletions since the last backup. After the backup is
done a snapshot of the destination directory is created using the commandcp--archive--linkwith the current date and time encoded in the name.Due to the use ofhard linkseach “version” of a file is only stored once.
Becausersyncby default doesn’t modify files inplace it breakshard linksand thereby avoids modifying existinginodes. This ensures that the contents
of snapshots don’t change when a new backup updates existing files. The
combination of hard links and the avoidance of inplace modifications
effectively provides a limited form ofdeduplication. Each snapshot requires a
couple of megabytes to store the directory names and hard links but the
contents of files aren’t duplicated.The articleEasy Automated Snapshot-Style Backups with Linux and Rsynccontains more details about this technique.Rotation of snapshotsSnapshots can be rotated according to a flexible rotation scheme, for example
I’ve configured my laptop backup rotation to preserve the most recent 24 hourly
backups, 30 daily backups and endless monthly backups.Backup destinationsWhile developing, maintaining and evolving backup scripts for various Linux
laptops and servers I’ve learned that backups for different systems require
different backup destinations and connection methods:Encrypted USB disksAutomatic mountingLocal server (rsync daemon)Remote server (rsync daemon over SSH tunnel)Encrypted USB disksThere’s aLUKSencrypted USB disk on my desk at work that I use to keep
hourly, daily and monthly backups of my work laptop. The disk is connected
through the same USB hub that also connects my keyboard and mouse so I can’t
really forget about it :-).Automatic mountingBefore the backup starts, the encrypted disk is automatically unlocked and
mounted. The use of a key file enables this process to run unattended in the
background. Once the backup is done the disk will be unmounted and locked
again, so that it can be unplugged at any time (as long as a backup isn’t
running of course).Local server (rsync daemon)My personal laptop transfers hourly backups to thersync daemonrunning on
the server in my home network using a direct TCP connection without SSH. Most
of the time the laptop has an USB Ethernet adapter connected but the backup
runs fine over a wireless connection as well.Remote server (rsync daemon over SSH tunnel)My VPS (virtual private server) transfers nightly backups to thersync
daemonrunning on the server in my home network over anSSH tunnelin
order to encrypt the traffic and restrict access. The SSH account is configured
to allow tunneling but disallow command execution. This setup enables the rsync
client and server to run with root privileges without allowing the client to
run arbitrary commands on the server.Alternative connection methodsBacking up to a local disk limits the effectiveness of backups but using SSH
access between systems gives you more than you bargained for, because you’re
allowing arbitrary command execution. Thersync daemonprovides an
alternative that does not allow arbitrary command execution. The following
sections discuss this option in more detail.Using rsync daemonTo be able to write files as root and preserve all filesystem metadata, rsync
must be running with root privileges. However most of my backups are stored on
remote systems and opening up remote root access over SSH just to transfer
backups feels like a very blunt way to solve the problem :-).Fortunately another solution is available: Configure an rsync daemon on the
destination and instruct your rsync client to connect to the rsync daemon
instead of connecting to the remote system over SSH. The rsync daemon
configuration can restrict the access of the rsync client so that it can only
write to the directory that contains the backup tree.In this setup no SSH connections are used and the traffic between the rsync
client and server is not encrypted. If this is a problem for you then continue
reading the next section.Enabling rsync daemonOn Debian and derivatives like Ubuntu you can enable and configure anrsync
daemonquite easily:Make sure that rsync is installed:$sudoapt-getinstallrsyncEnable the rsync daemon by editing/etc/default/rsyncand changing the
lineRSYNC_ENABLE=falsetoRSYNC_ENABLE=true. Here’s a one liner
that accomplishes the task:$sudosed-i's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/'/etc/default/rsyncCreate the configuration file/etc/rsyncd.confand define at least
one module. Here’s an example based on my rsync daemon configuration:# Global settings.max connections=4log file=/var/log/rsyncd.log# Defaults for modules.read only=nouid=0gid=0# Daily backups of my VPS.[vps_backups]path=/mnt/backups/vps/latestpost-xfer exec=/usr/sbin/process-vps-backups# Hourly backups of my personal laptop.[laptop_backups]path=/mnt/backups/laptop/latestpost-xfer exec=/usr/sbin/process-laptop-backupsThepost-xferexecdirectives configure the rsync daemon to create a
snapshot once the backup is done and rotate old snapshots afterwards.Once you’ve created/etc/rsyncd.confyou can start the rsync daemon:$sudoservicersyncstartIf you’re using a firewall you should make sure that the rsync daemon port
is whitelisted to allow incoming connections. The rsync daemon port number
defaults to 873. Here’s an iptables command to accomplish this:$sudoiptables-AINPUT-ptcp-mtcp--dport873-mcomment--comment"rsync daemon"-jACCEPTTunneling rsync daemon connectionsWhen your backups are transferred over the public internet you should
definitely use SSH to encrypt the traffic, but if you’re at all security
conscious then you probably won’t like having to open up remote root access
over SSH just to transfer backups :-).The alternative is to use a non privileged SSH account to set up anSSH
tunnelthat redirects network traffic to the rsync daemon. The login shell of
the SSH account can be set to/usr/sbin/nologin(or something similar like/bin/false) todisable command execution, in this case you need to pass-Nto the SSH client.ContactThe latest version ofrsync-system-backupis available onPyPIandGitHub.
The documentation is hosted onRead the Docsand includes achangelog. For
bug reports please create an issue onGitHub. If you have questions,
suggestions, etc. feel free to send me an e-mail atpeter@peterodding.com.LicenseThis software is licensed under theMIT license.© 2018 Peter Odding.