Package: backup-warden
Backup WardenIn today's digital landscape, every business is taking backups of their data (hopefully). These backups can be challenging as they often require substantial disk space and/or cloud storage that can lead to significant financial expenses. If you're in search of a solution to handle your backups and retain only the necessary ones according to your retention policies, look no further! With Backup Warden, it will supervise and maintain your backups simplifying your overall data life cycle and enabling you to have smarter resource utilization.Thanks to xolox for his work onrotate-backupsthat gave me a lot of inspiration for this project!UsageOptionDescriptionDefault Value--hourlyNumber of hourly backups to preserve72--dailyNumber of daily backups to preserve7--weeklyNumber of weekly backups to preserve6--monthlyNumber of monthly backups to preserve12--yearlyNumber of yearly backups to preservealways-c,--configLocation of the config file/etc/backup_warden.ini-s,--sourceSource of where the backups are stored. Select from:local,ssh,s3local-b,--bucketName of the AWS S3 bucket-p,--pathSpecify a path to traverse all directories it contains for granular retention policies-e,--environmentEnvironment the backups are rotated in (used for Slack alert only)-t,--timestamp-patternThe timestamp pattern using a regex expression to parse out of filenames-l,--log-fileEnable logging to this file path-I,--includeInclude backups based on their directory path and/or filename (separated by comma)-E,--excludeExclude backups based on their directory path and/or filename (separated by comma)-H,--ssh-hostSSH host/alias to use--ssh-sudoWrap SSH commands with sudo for escalated privilegesFalse--filestatUse the file's last modified date instead of parsing timestamp from filenameFalse--prefer-recentKeep the most recent backup in each time slot instead of oldestFalse--relaxedTime windows are not enforcedFalse--syslogUse syslogFalse--debugLog debug messages that can help troubleshootFalse--deleteCommit to deleting backups (DANGER ZONE)False-V,--versionDisplay version and exit-h,--helpShow this help message and exitNote: Boolean options such as--filestatcan be specified asyes/no,true/false, or1/0in the configAdditional Information for OptionsOption:hourly,daily,weekly,monthly,yearlyThese options determine the number of backups to retain for each respective frequencyYou have the flexibility to provide an expression that will be evaluated to calculate a value. For example, using--hourly=5+2would result in 7Alternatively, you can specify "always" as the value to preserve all backups for that particular frequencyOption:sourceThere are currently three available sources, each functioning differently when scanning directories to find backups.localThis option is straightforward and doesn't really require any additional explanation. It is a simple method for locating backupssshTo use this source, you need to configure the SSH config file (~/.ssh/config) with the relevant host information. It also supports aliases defined in the SSH config, as well as jump hostss3One thing to note is thes3_endpoint_urloption. This lets you specify an endpoint other than the default to be able to use an alternative like DigitalOcean Spaces (i.e.https://nyc3.digitaloceanspaces.com)Option:pathUsing Config FileWhenpathis used under the[main]section in config, it significantly alters Backup Warden's functionality. In this case, Backup Warden will traverse through every directory and file under the given path until it locates a backup. Once a backup is found, it associates the backup with a config section usingfnmatchfor pattern matching that defines its retention policy. If there isn't a config section that matches all possibilities of a path found, it's ignored. Ifpathis not specified, Backup Warden will only scan the path defined in each config section.Using thepathoption provides granular control over retention policies and allows for flexible path name conventions. It enables you to define custom retention rules based on very specific paths.Using Parameters--pathis the same as usingpathunder[main]section in config.Note: Specifying apathmay result in a performance impact if there are a lot of non-backup directories/files within the specified path. This shouldn't be an issue though unless your setup is very abnormal. You can useexclude_listto help out if this is a scenario.Option:timestamp-patternThetimestamp-patternoption provides the flexibility to customize the regular expression used for extracting timestamps from filenames. The value for this option should be a Python-compatible regular expression that includes the named capture groups 'year', 'month', and 'day'. Additionally, it can optionally include the groups 'hour', 'minute', and 'second'. 'unixtime' is also supported (see below for how to use it)Here is an example of the default regular expression:# Required components(?P<year>\d{4})\D?(?P<month>\d{2})\D?(?P<day>\d{2})\D?(?:# Optional components(?P<hour>\d{2})\D?(?P<minute>\d{2})\D?(?P<second>\d{2})?)?Regular expressions are compiled using there.VERBOSEflag which ignores whitespace, including newlines.If your backups utilize Unix timestamps instead of standard timestamps, you can specify a pattern like:(?P<unixtime>\d+)Option:filestatIn cases where your backup files do not contain a timestamp, you have the option to use the last modified time of the backup instead. However, it is important to note that when utilizing this parameter, you will also need to modify thetimestamp-patternto accurately identify which directories/files are considered backups. For example, if all of your backups have filenames starting with "backup-", you would change thetimestamp-patterntobackup-\S+.If your backup file names are not standardized and do not follow a specific pattern, this feature is currently not supported.Option:relaxedBackup Warden offers the--relaxedoption to modify its default rotation behavior. By default, Backup Warden enforces strict time windows for each rotation scheme. However, with the--relaxedoption, you can relax this enforcement. Here's a clear explanation of the difference between strict and relaxed rotation:Strict Rotation: When the number of hourly backups to preserve is set to three, only backups created within the relevant time window (the hour of the most recent backup and the two hours leading up to it) will match the hourly frequency. Choose this option if your backups are created at regular intervals without any missed intervalsRelaxed Rotation: With the--relaxedoption enabled, the three most recent backups will all match the hourly frequency and be preserved, regardless of the calculated time window. Choose this option if your backups are created at irregular intervals, as it allows for the preservation of more backupsOption:include-list/exclude-listThese options utilizefnmatch, allowing the use of asterisks as wildcards. This enables precise definition of which backups should be excluded from deletion. Include and exclude can be used together for fine-grained control.For example, to exclude thecluster1from Backup Warden's operations, you can use the--exclude-list="*cluster1*"argument. This ensures that any directories/files containingcluster1in their names will be excluded.To further expand the exclusion criteria, you can exclude backups from the year 2022 by using--exclude-list="*cluster1*, *2022*".The same concept applies to theexclude-listoption under each section in the config file:[/path/backups/*/logical]hourly=72daily=7weekly=6monthly=12yearly=alwaysinclude_list=exclude_list=*cluster1*, *2022*Include functions in the opposite manner. If you want to only include specific backups, you can utilize this feature. It can be used as a command-line argument using--include-list, or as theinclude_listoption in a config path section.Installation & ExecutionMust be using Python 3.8+Using PyPi:pipinstallbackup-warden

backup-warden--configconfig/example.iniUsing Poetry:curl-sSLhttps://install.python-poetry.org|python3-

poetryinstall

poetryrunbackup-warden--configconfig/example.iniConfigurationBackup Warden offers two methods for setting it up: parameters and a config file. The recommended approach is to use a config file, which allows customization of directory paths and their respective retention policies. You can find examples of the config filehereWith a config file, each section represents a specific path containing backups to be rotated. Within each section, you can define the rotation scheme and other options. Please refer to the above information for detailed instructions on how to utilize pattern matching effectively when using thepathoption.Note: If you specify a config file along with config path(s), command-line parameters will have no effect. The methods are not interchangeable.Under the[main]section in the config file, you can set the following options:bucketpathsourceenvironmentssh_hostssh_sudosysloglog_files3_endpoint_urls3_access_key_ids3_secret_access_keyslack_webhookFor each config[path]section, you can set the following options:hourly,daily,weekly,monthly,yearlytimestamp_patterninclude_list/exclude_listfilestatrelaxedprefer_recentYou can also set the following options using environment variables, which will override the corresponding config values:S3_ENDPOINT_URLAWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYSLACK_WEBHOOKNot usingpathWhen thepathparameter is omitted under the[main]section, Backup Warden does not accept wildcarding for config sections. In this scenario, Backup Warden will solely scan the specified config sections without traversing through additional directories.This means that instead of scanning the entire file system or applying wildcards to match multiple paths, Backup Warden will focus solely on the directories specified within the config sections. It will not explore subdirectories or perform any recursive scanning.By adhering to this behavior, Backup Warden provides a targeted approach, limiting the scope of its backup scanning and rotation operations to the explicitly defined directories within the config sections.Directory structure/path/backups/logical/$backups/path/backups/physical/$backupsConfig[/path/backups/logical]hourly=15daily=3weekly=5monthly=12yearly=always[/path/backups/physical]hourly=10daily=7weekly=4monthly=12yearly=alwaysThis is a basic setup and works as you expect.UsingpathWhen thepathoption is used under the[main]section, Backup Warden allows wildcarding to be applied flexibly within the config sections.By specifying thepathoption, you gain the ability to use wildcards to define the directories or files that Backup Warden should scan for backups. This enables a more dynamic and versatile approach. Backup Warden will effectively traverse through the specified paths, including subdirectories if necessary, to locate the backups based on the wildcard patterns provided.Directory structure/path/backups/cluster1/logical/$backups/path/backups/cluster2/logical/$backups/path/backups/cluster1/physical/$backups/path/backups/cluster2/physical/$backupsConfig[main]path=/path/backups[/path/backups/*/logical]hourly=15daily=3weekly=5monthly=12yearly=always[/path/backups/*/physical]hourly=10daily=7weekly=4monthly=12yearly=alwaysBackup Warden's design incorporates hierarchical directory structure awareness, allowing for precise configuration and retention policies.By defining/path/backups/*/logicalas a config section, Backup Warden acknowledges the wildcard (*) as a placeholder that matches any subdirectory under/path/backups/and assigns it to thelogicalconfig section.When a retention policy is set for a broader path, such aspath/backups, it will not override or take precedence over a more specific path like/path/backups/cluster1/logical. Backup Warden's scanning and rotation operations respect the defined hierarchy, ensuring that retention policies are accurately applied to the corresponding backup directories without unintentionally affecting others.AlertingBackup Warden offers a convenient Slack integration feature that allows you to stay informed about your backups if you specify a Slack Webhook URL. Benefit from the following alerts:Non-Backup Alert: Get notified if a path doesn't have a backup in the past 24 hoursSuccess Alert: Receive notification after a successful execution, along with detailed statistics about what it didFailure Alert: In case of a failed execution, be promptly notified to address any potential issuesFunctionality OverviewBackup Warden employs the following steps to carry out the backup rotation process:Specify Paths: You need to provide apathand/or use config sections for paths to inform Backup Warden about the locations where the backups are storedScan for Backups: Backup Warden scans each specified path to locate backups. These backups can be in the form of either directories or files. Backup Warden identifies backups by searching for timestamps in their names. If you're usingfilestat, it will not look for a timestamp, but what you specify in place of itApply Rotation Scheme: Backup Warden applies the defined rotation scheme to the identified backups. If the outcome doesn't align with your expectations, you can experiment with therelaxedand/orprefer-recentoptions to achieve the desired behaviorBackup Deletion: Backups that are determined to be rotated based on the rotation scheme will be deleted if thedeleteoption is passed. If it isn't, Backup Warden will skip the deletion step and preserve the rotated backupsOutput Example