Package: s3-sysbackup
s3-sysbackup PackageA very flexible, Unix-style facility for backing content up toAWS S3with time/space folding, using the S3Object Locksystem forwrite-once-read-many(WORM) storage, which can help provide additional security or compliance around backup requirements.S3 charges not only for storage, but also for management operations on data in S3.s3-sysbackupworks to minimize both kinds of charges, including a limited ability to amortize backup content storage over multiple machines.The Command LineThis package provides a console entry point nameds3-sysbackup, a program which takes a subcommand as its first argument.  Help is available from the program by using the-hor--helpoption, either with or without a subcommand.Thecreate-resourcesSubcommandThis subcommand creates the resources in AWS necessary for storage and retrieval of backup data.  These resources include not only an S3 bucket, but also IAM roles, groups, and policies to best manage the S3 bucket it creates.The created resources are organized into a "package," which by default has the names3-sysbackup.  Multiple instance can be created with different package names.  Package names must be globally unique within your account.  Each package exports the bucket name from CloudFormation within the region used for the CloudFormation stack, meaning that either a non-default name is needed when a second package is created in the same region, or the package stacks need to be created in separate regions.Thewrite-configSubcommandThis writes a configuration file (and possibly a credentials file) for use bys3-sysbackup archive.  It also writes unit files forsystemdso the archiver will run on a daily basis.  This command can create an IAM user for the backup system to authenticate to AWS as, using the IAM resources created bys3-sysbackup create-resourcesto strictly limit the capabilities of the account.When this subcommand is used to create an IAM account for uploading backup data, it can save those credentials in an encrypted file for transfer to and use on another system.  Creating this encrypted file requires the presence of theccryptprogram on the system.  Since this encryption only supports a single password, if you need to share the encrypted credentials with another user,do notuse your own account password (either system password or AWS password).ThearchiveSubcommandThearchivesubcommand is the workhorse of this backup system: it evaluates which data needs to be sent up to S3, manages object retention periods, and builds the backup manifests for each run.  This subcommand is typically invoked by asystemd"timer" unit on a daily basis.Configurations3-sysbackupconfiguration lives in a directory structure on disk, with the default location being/etc/backup.  Within the directory there must be aconffile, which is in Pythonconfigparserformat.  Backup strategies, as detailed below, each have their own subdirectory of the configuration directory.The sections of theconffile are detailed in the subsections below.confFile:[aws]SectionThis section gives configuration information for access to the AWS API.  The available settings are:SettingValue DescriptionprofileSets the profile to use -- may be from the standard profile system or from a custom credentials file specified with thecredentials_filesettingcredentials_filePath to an AWS credentials file to userole_arnUsed only in conjunction with thecredentials_filesetting, gives the ARN of an IAM Role to assume for authorizing AWS actionsPlease note that, ifcredentials_fileisnotused,profilecan be used to identify a standardly configured AWS profile that uses an assumed IAM Role, which is why therole_arnsetting only works if acredentials_fileis given.confFile:[s3]SectionThis section configures usage of S3.  The available settings are:SettingValue DescriptionbucketExplicitly sets the name of the bucket into which the backup should be made; if not given (or explicitly set on theSaverbeforeload_configis called), CloudFormation is used to identify the bucketconfFile:[cloudformation]SectionWhen an S3 bucket is not explicitly configured either on theSaverobject or through thebucketsetting in the[s3]section, theSaverwill attempt to query CloudFormation for an export – namedbackup-bucket, unless otherwise specified as indicated below – giving the name of the destination bucket.SettingValue DescriptionregionAWS region in which the CloudFormation stack to be queried resides; if the profile in use does not configure a default region, this is necessary for use of CloudFormationexportThe name of the export to read for the bucket nameconfFile:[content]SectionSettingValue Descriptionlock_timeoutThe number of seconds to wait for an advisory read lock on a "picked" filedays_retainedThe number of days for which content should be retained via S3's Object LockBackup Strategiess3-sysbackupemploys two different strategies for backup: a file-based strategy calledquiltingand a dynamic data strategy calledsnapshotting.  Information from both strategies is included in themanifest fileuploaded to S3 for each backup run on each machine.QuiltingQuilting, like the Git version control system, is content based: it keys the objects in which it stores content on S3 by the content found within the file.  The keys for these objects are then "patched together" (i.e.quilted) by the information this strategy incorporates into the manifest file uploaded for each backup.  When a file's content hasn't changed since the last backup, that file content need not be uploaded again.  And when multiple machines are backing up copies of the same file, the file content only needs to be uploaded by the first machine to back it up.The manifest, like a recursive version of Git'strees, has entries referencing a file's content, path on the system, and permissions, but also its ownership.  It also contains information about symbolic links and directories, so as much as possible of the backed-up system can be restored.  Where possible, ownership uses symbolic names and permissions are captured as ACLs, but fallbacks to numeric representations can occur.Selection of files for the quilting strategy is done by executables placed in (or symlinked into) thepickers.dsubdirectory of the configuration directory.  Each executable file in that directory is run and the lines it prints to STDOUT are interpreted as file paths; one typical usage is to put one or more executable shell scripts that callfindwith the desired parameters.To save on transfer and storage costs with S3, each file selected by apickerwill be evaluated as to whether gzipping will improve storage efficiency.  If the test indicates that the file compresses well, the GZip algorithm will be applied before the file is transferred to S3; the application of GZip will be noted on the S3 object's metadata.SnapshottingNot all data to be backed up resides conveniently in a file that can be assumed unchanging or reasonably readable at a given moment, with databases as a good example.  For these data, it is typically necessary to invoke a tool both to retrieve the backup data and also to restore it.s3-sysbackupaccommodates this need through itssnapshottingstrategy.Executable files in thesnapshots.dsubdirectory of the configuration directory provide both the snapshot content and commentary about the snapshot.  Each of these executables will be invoked with a single argument: a decimal string giving the file descriptor to which it should write the snapshot data.  Text sent to STDOUT is captured as a description of the snapshot included in the S3 object's metadata.  A line sent to STDOUT starting with::restore-through::is interpreted as giving a (shell formatted) command for restoring the snapshot data (piped to STDIN) onto a system, which is additionally saved in a separate metadata entry.With each run of the backup system, each snapshot will be taken and uploaded to S3.  It is the responsibility of the snapshotter executables to implement any beneficial compression.  The object keys of all uploaded snapshot content are collected within the uploaded manifest file for the backup run.As a note, if writing a snapshotter executable as abashscript, the construct>&${1:-1}can be useful: it redirects output to the file descriptor passed as the first argument, defaulting to STDOUT (file descriptor 1) if not specified.  This will not, of course, work within a function, though it could be adapted by copying the target descriptor number into a less general variable name:# At top level of scriptCONTENT_FD=${1:-1}# Defining a functionfunctionsome_func(){# ...some-pipeline-producing-content>&$CONTENT_FD# ...}