Package: rotate-backups
Backups are good for you. Most people learn this the hard way (including me).
Nowadays my Linux laptop automatically creates a full system snapshot every
four hours by pushing changed files to anrsyncdaemon running on the server
in my home network and creating a snapshot afterwards using thecp-alcommand (the articleEasy Automated Snapshot-Style Backups with Linux and
Rsyncexplains the basic technique). The server has a second disk attached
which asynchronously copies from the main disk so that a single disk failure
doesn’t wipe all of my backups (the “time delayed replication” aspect has also
proven to be very useful).Okay, cool, now I have backups of everything, up to date and going back in
time! But I’m running through disk space like crazy… A proper deduplicating
filesystem would be awesome but I’m running crappy consumer grade hardware and
e.g. ZFS has not been a good experience in the past. So I’m going to have to
delete backups…Deleting backups is never nice, but an easy and proper rotation scheme can help
a lot. I wanted to keep things manageable so I wrote a Python script to do it
for me. Over the years I actually wrote several variants. Because I kept
copy/pasting these scripts around I decided to bring the main features together
in a properly documented Python package and upload it to thePython Package
Index.Therotate-backupspackage is currently tested on cPython 2.7, 3.5+ and PyPy
(2.7). It’s tested on Linux and Mac OS X and may work on other unixes but
definitely won’t work on Windows right now.FeaturesInstallationUsageCommand lineConfiguration filesCustomizing the rotation algorithmSupported configuration optionsHow it worksContactLicenseFeaturesDry run modeUse it.I’m serious. If you don’t androtate-backupseats more backups
than intended you have no right to complain ;-)Flexible rotationRotation with any combination of hourly, daily, weekly, monthly and yearly
retention periods.Fuzzy timestamp matching in filenamesThe modification times of the files and/or directories are not relevant. If
you speak Python regular expressions, here is how the fuzzy matching
works:# Required components.
(?P<year>\d{4}) \D?
(?P<month>\d{2}) \D?
(?P<day>\d{2}) \D?
(
   # Optional components.
   (?P<hour>\d{2}) \D?
   (?P<minute>\d{2}) \D?
   (?P<second>\d{2})?
)?All actions are loggedLog messages are saved to the system log (e.g./var/log/syslog) so you
can retrace what happened when something seems to have gone wrong.InstallationTherotate-backupspackage is available onPyPIwhich means installation
should be as simple as:$pipinstallrotate-backupsThere’s actually a multitude of ways to install Python packages (e.g. theper
user site-packages directory,virtual environmentsor just installing
system wide) and I have no intention of getting into that discussion here, so
if this intimidates you then read up on your options before returning to these
instructions ;-).UsageThere are two ways to use therotate-backupspackage: As the command line
programrotate-backupsand as a Python API. For details about the Python
API please refer to the API documentation available onRead the Docs. The
command line interface is described below.Command lineUsage:rotate-backups [OPTIONS] [DIRECTORY, ..]Easy rotation of backups based on the Python package by the same name.To use this program you specify a rotation scheme via (a combination of) the--hourly,--daily,--weekly,--monthlyand/or--yearlyoptions and the
directory (or directories) containing backups to rotate as one or more
positional arguments.You can rotate backups on a remote system over SSH by prefixing a DIRECTORY
with an SSH alias and separating the two with a colon (similar to how rsync
accepts remote locations).Instead of specifying directories and a rotation scheme on the command line you
can also add them to a configuration file. For more details refer to the online
documentation (see also the--configoption).Please use the--dry-runoption to test the effect of the specified rotation
scheme before letting this program loose on your precious backups! If you don’t
test the results using the dry run mode and this program eats more backups than
intended you have no right to complain ;-).Supported options:OptionDescription-M,--minutely=COUNTIn a literal sense this option sets the number of “backups per minute” to
preserve during rotation. For most use cases that doesn’t make a lot of
sense :-) but you can combine the--minutelyand--relaxedoptions to
preserve more than one backup per hour.  Refer to the usage of the-H,--hourlyoption for details aboutCOUNT.-H,--hourly=COUNTSet the number of hourly backups to preserve during rotation:IfCOUNTis a number it gives the number of hourly backups to preserve,
starting from the most recent hourly backup and counting back in time.Alternatively you can provide an expression that will be evaluated to get
a number (e.g. ifCOUNTis “7 * 2” the result would be 14).You can also pass “always” forCOUNT, in this case all hourly backups are
preserved.By default no hourly backups are preserved.-d,--daily=COUNTSet the number of daily backups to preserve during rotation. Refer to the
usage of the-H,--hourlyoption for details aboutCOUNT.-w,--weekly=COUNTSet the number of weekly backups to preserve during rotation. Refer to the
usage of the-H,--hourlyoption for details aboutCOUNT.-m,--monthly=COUNTSet the number of monthly backups to preserve during rotation. Refer to the
usage of the-H,--hourlyoption for details aboutCOUNT.-y,--yearly=COUNTSet the number of yearly backups to preserve during rotation. Refer to the
usage of the-H,--hourlyoption for details aboutCOUNT.-t,--timestamp-pattern=PATTERNCustomize the regular expression pattern that is used to match and extract
timestamps from filenames.PATTERNis expected to be a Python compatible
regular expression that must define the named capture groups ‘year’,
‘month’ and ‘day’ and may define ‘hour’, ‘minute’ and ‘second’.-I,--include=PATTERNOnly process backups that match the shell pattern given byPATTERN. This
argument can be repeated. Make sure to quotePATTERNso the shell doesn’t
expand the pattern before it’s received by rotate-backups.-x,--exclude=PATTERNDon’t process backups that match the shell pattern given byPATTERN. This
argument can be repeated. Make sure to quotePATTERNso the shell doesn’t
expand the pattern before it’s received by rotate-backups.-j,--parallelRemove backups in parallel, one backup per mount point at a time. The idea
behind this approach is that parallel rotation is most useful when the
files to be removed are on different disks and so multiple devices can be
utilized at the same time.Because mount points are per system the-j,--paralleloption will also
parallelize over backups located on multiple remote systems.-p,--prefer-recentBy default the first (oldest) backup in each time slot is preserved. If
you’d prefer to keep the most recent backup in each time slot instead then
this option is for you.-r,--relaxedBy default the time window for each rotation scheme is enforced (this is
referred to as strict rotation) but the-r,--relaxedoption can be used
to alter this behavior. The easiest way to explain the difference between
strict and relaxed rotation is using an example:When using strict rotation and the number of hourly backups to preserve
is three, only backups created in the relevant time window (the hour of
the most recent backup and the two hours leading up to that) will match
the hourly frequency.When using relaxed rotation the three most recent backups will all match
the hourly frequency (and thus be preserved), regardless of the
calculated time window.If the explanation above is not clear enough, here’s a simple way to decide
whether you want to customize this behavior or not:If your backups are created at regular intervals and you never miss an
interval then strict rotation (the default) is probably the best choice.If your backups are created at irregular intervals then you may want to
use the-r,--relaxedoption in order to preserve more backups.-i,--ionice=CLASSUse the “ionice” program to set the I/O scheduling class and priority of
the “rm” invocations used to remove backups.CLASSis expected to be one of
the values “idle” (3), “best-effort” (2) or “realtime” (1). Refer to the
man page of the “ionice” program for details about these values. The
numeric values are required by the ‘busybox’ implementation of ‘ionice’.-c,--config=FILENAMELoad configuration fromFILENAME. If this option isn’t given the following
default locations are searched for configuration files:/etc/rotate-backups.ini and /etc/rotate-backups.d/*.ini~/.rotate-backups.ini and ~/.rotate-backups.d/*.ini~/.config/rotate-backups.ini and ~/.config/rotate-backups.d/*.iniAny available configuration files are loaded in the order given above, so
that sections in user-specific configuration files override sections by the
same name in system-wide configuration files. For more details refer to the
online documentation.-C,--removal-command=CMDChange the command used to remove backups. The value ofCMDdefaults torm``-f``R. This choice was made because it works regardless of whether
“backups to be rotated” are files or directories or a mixture of both.As an example of why you might want to change this, CephFS snapshots are
represented as regular directory trees that can be deleted at once with a
single ‘rmdir’ command (even though according to POSIX semantics this
command should refuse to remove nonempty directories, but I digress).-u,--use-sudoEnable the use of “sudo” to rotate backups in directories that are not
readable and/or writable for the current user (or the user logged in to a
remote system over SSH).-S,--syslog=CHOICEExplicitly enable or disable system logging instead of letting the program
figure out what to do. The values ‘1’, ‘yes’, ‘true’ and ‘on’ enable system
logging whereas the values ‘0’, ‘no’, ‘false’ and ‘off’ disable it.-f,--forceIf a sanity check fails an error is reported and the program aborts. You
can use--forceto continue with backup rotation instead. Sanity checks
are done to ensure that the given DIRECTORY exists, is readable and is
writable. If the--removal-commandoption is given then the last sanity
check (that the given location is writable) is skipped (because custom
removal commands imply custom semantics).-n,--dry-runDon’t make any changes, just print what would be done. This makes it easy
to evaluate the impact of a rotation scheme without losing any backups.-v,--verboseIncrease logging verbosity (can be repeated).-q,--quietDecrease logging verbosity (can be repeated).-h,--helpShow this message and exit.Configuration filesInstead of specifying directories and rotation schemes on the command line you
can also add them to a configuration file.Configuration files are text files in the subset ofini syntaxsupported by
Python’sconfigparsermodule. They can be located in the following places:DirectoryMain configuration fileModular configuration files/etc/etc/rotate-backups.ini/etc/rotate-backups.d/*.ini~~/.rotate-backups.ini~/.rotate-backups.d/*.ini~/.config~/.config/rotate-backups.ini~/.config/rotate-backups.d/*.iniThe available configuration files are loaded in the order given above, so that
user specific configuration files override system wide configuration files.You can load a configuration file in a nonstandard location using the command
line option--config, in this case the default locations mentioned above
are ignored.Each section in the configuration defines a directory that contains backups to
be rotated. The options in each section define the rotation scheme and other
options. Here’s an example based on how I userotate-backupsto rotate the
backups of the Linux installations that I make regular backups of:# /etc/rotate-backups.ini:# Configuration file for the rotate-backups program that specifies# directories containing backups to be rotated according to specific# rotation schemes.[/backups/laptop]hourly=24daily=7weekly=4monthly=12yearly=alwaysionice=idle[/backups/server]daily=7 * 2weekly=4 * 2monthly=12 * 4yearly=alwaysionice=idle[/backups/mopidy]daily=7weekly=4monthly=2ionice=idle[/backups/xbmc]daily=7weekly=4monthly=2ionice=idleAs you can see in the retention periods of the directory/backups/serverin
the example above you are allowed to use expressions that evaluate to a number
(instead of having to write out the literal number).Here’s an example of a configuration for two remote directories:# SSH as a regular user and use `sudo' to elevate privileges.[server:/backups/laptop]use-sudo=yeshourly=24daily=7weekly=4monthly=12yearly=alwaysionice=idle# SSH as the root user (avoids sudo passwords).[server:/backups/server]ssh-user=roothourly=24daily=7weekly=4monthly=12yearly=alwaysionice=idleAs this example shows you have the option to connect as the root user or to
connect as a regular user and usesudoto elevate privileges.Customizing the rotation algorithmSince publishingrotate-backupsI’ve found that the default rotation
algorithm is not to everyone’s satisfaction and because the suggested
alternatives were just as valid as the choices that I initially made,
options were added to expose the alternative behaviors:DefaultAlternativeStrict rotation (the time window
for each rotation frequency is
enforced).Relaxed rotation (time windows are
not enforced). Enabled by the-r,--relaxedoption.The oldest backup in each time slot
is preserved and newer backups in
the time slot are removed.The newest backup in each time slot
is preserved and older backups in
the time slot are removed. Enabled
by the-p,--prefer-recentoption.Supported configuration optionsRotation schemes are defined using theminutely,hourly,daily,weekly,monthlyandyearlyoptions, these options support the
same values as documented for the command line interface.Thetimestamp-patternoption can be used to customize the regular
expression that’s used to extract timestamps from filenames. The value is
expected to be a Python compatible regular expression that must contain the
named capture groups ‘year’, ‘month’ and ‘day’ and may contain the groups
‘hour’, ‘minute’ and ‘second’. As an example here is the default regular
expression:# Required components.
(?P<year>\d{4} ) \D?
(?P<month>\d{2}) \D?
(?P<day>\d{2}  ) \D?
(?:
    # Optional components.
    (?P<hour>\d{2}  ) \D?
    (?P<minute>\d{2}) \D?
    (?P<second>\d{2})?
)?Note how this pattern spans multiple lines: Regular expressions are compiled
using there.VERBOSEflag which means whitespace (including newlines) is
ignored.Theinclude-listandexclude-listoptions define a comma separated
list of filename patterns to include or exclude, respectively:Make surenotto quote the patterns in the configuration file,
just provide them literally.If an include or exclude list is defined in the configuration file it
overrides the include or exclude list given on the command line.Theprefer-recent,strictanduse-sudooptions expect a boolean
value (yes,no,true,false,1or0).Theremoval-commandoption can be used to customize the command that is
used to remove backups.Theioniceoption expects one of the I/O scheduling class namesidle,best-effortorrealtime(or the corresponding numbers).Thessh-useroption can be used to override the name of the remote SSH
account that’s used to connect to a remote system.How it worksThe basic premise ofrotate-backupsis fairly simple:You pointrotate-backupsat a directory containing timestamped backups.It will scan the directory for entries (it doesn’t matter whether they are
files or directories) with a recognizable timestamp in the name.NoteAll of the matched directory entries are considered to be backups
of the same data source, i.e. there’s no filename similarity logic
to distinguish unrelated backups that are located in the same
directory. If this presents a problem consider using the--includeand/or--excludeoptions.The user defined rotation scheme is applied to the entries. If this doesn’t
do what you’d expect it to you can try the--relaxedand/or--prefer-recentoptions.The entries to rotate are removed (or printed in dry run).ContactThe latest version ofrotate-backupsis available onPyPIandGitHub. The
documentation is hosted onRead the Docsand includes achangelog. For bug
reports please create an issue onGitHub. If you have questions, suggestions,
etc. feel free to send me an e-mail atpeter@peterodding.com.LicenseThis software is licensed under theMIT license.© 2020 Peter Odding.