Package: collective.recipe.backup
Easy Zope backup/restore recipe for buildoutContentsEasy Zope backup/restore recipe for buildoutIntroductionCompatibilityDevelopmentExample usageBacked up dataData that isnotbacked upIs your backup backed up?BackupSnapshotsZipbackupsRestoreNames of created scriptsSupported optionsCron job integrationBlob storagersyncAlternative restore sourceContributorsSponsorshipChange history5.0.0 (2023-06-12)5.0.0a3 (2023-03-15)5.0.0a2 (2023-02-01)5.0.0a1 (2022-11-08)4.2.0 (2021-08-30)4.1.1 (2019-10-07)4.1.0 (2019-04-10)4.0.1 (2018-04-19)4.0 (2017-12-22)4.0b5 (2017-11-17)4.0b4 (2017-08-18)4.0b3 (2017-07-05)4.0b2 (2017-06-26)4.0b1 (2017-05-31)3.1 (2017-02-24)3.0.0 (2015-12-31)2.22 (2015-12-30)2.21 (2015-10-06)2.20 (2014-11-11)2.19 (2014-06-16)2.18 (2014-04-29)2.17 (2014-02-07)2.16 (2014-01-14)2.15 (2013-09-16)2.14 (2013-09-09)2.13 (2013-07-15)2.12 (2013-06-28)2.11 (2013-05-06)2.10 (2013-03-30)2.9 (2013-03-06)2.8 (2012-11-13)2.7 (2012-09-27)2.6 (2012-08-29)2.5 (2012-08-08)2.4 (2011-12-20)2.3 (2011-10-05)2.2 (2011-09-14)2.1 (2011-09-01)2.0 (2011-08-26)1.7 (2010-12-10)1.6 (2010-09-21)1.5 (2010-09-08)1.4 (2010-08-06)1.3 (2009-12-08)1.2 (2009-10-26)1.1 (2009-08-21)1.0 (2009-02-06)0.9 (2008-12-05)0.8 (2008-09-23)0.7 (2008-09-19)0.6 (2008-09-19)0.5 (2008-09-18)0.4 (2008-08-19)0.3.1 (2008-07-04)0.2 (2008-07-03)0.1 (2008-07-03)IntroductionThis recipe is mostly a wrapper around thebin/repozoscript in
your Zope buildout.  It requires that this script is already made
available.  If this is not the case, you will get an error like this
when you run one of the scripts:bin/repozo: No such file or directory.  This should be there when you
are usingplone.recipe.zeoserver.  If this is
not the case, the easiest way of getting abin/repozoscript is to
add a new section in yourbuildout.cfg(do not forget to add it in thepartsdirective):[repozo]
recipe = zc.recipe.egg
eggs = ZODB
# or this for an older version:
# eggs = ZODB3
scripts = repozo
dependent-scripts = truebin/repozois a Zope script to make backups of yourData.fs.
Looking up the settings can be a chore. And you have to pick a
directory where to put the backups. This recipe providessensible
defaultsfor your common backup tasks. Making backups a piece of
cake is important!bin/backupmakes an incremental backup.bin/restorerestores the latest backup created by the backup script.bin/snapshotbackupmakes a full snapshot backup, separate from the
regular backups. Handy right before a big change in the site.bin/snapshotrestorerestores the latest full snapshot backup.bin/zipbackupmakes a zip backup.  This zips the Data.fs and the
blobstorage, handy for copying production data to your local
machine, especially the blobstorage with its many files.
Actually, zipping the Data.fs is standard,
and we do not zip the blobstorage,
because most files in there are already compressed.
But we do combine the blobs in one tar archive.Note: the Data.fs and blobstorage (or other storages) arenotcombined
in one file; you need to download multiple files.  Enable this
script by using theenable_zipbackupoption.bin/ziprestorerestores the latest zipbackup.CompatibilityThe recipe is tested with Python 2.7, 3.6, 3.7, 3.8, 3.9.
In Plone terms it works fine on Plone 4, 5 and 6.Note that the integration withplone.recipe.zope2instanceis not tested on Python 3.
It would pulls in too many dependencies, like Zope and ZODB.DevelopmentCode repository:https://github.com/collective/collective.recipe.backupIssue tracker:https://github.com/collective/collective.recipe.backup/issuesObvious fixes, like fixing typos, are fine on master.
For larger changes or if you are unsure, please create a branch or a pull request.The code comes with abuildout.cfg.  Please bootstrap the
buildout and run the createdbin/testto see if the tests still
pass.  Please try to add tests if you add code.
To run the tests for all supported Python versions, runtox.The long description of this package (as shown on PyPI), used to
contain a big file with lots of test code that showed how to use the
recipe.  This grew too large, so we left it out.  It is probably
still good reading if you are wondering about the effect some
options have.  Seesrc/collective/recipe/backup/tests/*.rst.We are tested on GitHub Actions:https://github.com/collective/collective.recipe.backup/actionsQuestions and comments tohttps://community.plone.orgor toMaurits van Rees.Example usageThe simplest way to use this recipe is to add a part inbuildout.cfglike this:[buildout]
parts = backup

[backup]
recipe = collective.recipe.backupYou can set lots of extra options, but the recipe authors like to
think they have created sane defaults, so this single line stating the
recipe name should be enough in most cases.Running the buildout adds thebackup,snapshotbackup,zipbackup,restore,snapshotrestoreandziprestorescripts to thebin/directory of the buildout.
Some are not added by default, others can be switched off.Backed up dataWhich data does this recipe backup?The Zope Object DataBase (ZODB) filestorage, by default located atvar/filestorage/Data.fs.The blobstorage (since version 2.0) if your buildout uses it, by
default located atvar/blobstorage.Data that isnotbacked upWhich data does this recipenotbackup?  Everything else of course,
but specifically:Data stored inRelStoragewillnotbe backed up.  (You could
still use this recipe to back up the filesystem blobstorage,
possibly with theonly_blobsoption.)Other data stored in SQL, perhaps via SQLAlchemy, willnotbe
backed up.It doesnotcreate a backup of your entire buildout directory.Is your backup backed up?Note that the backups are by default created in thevardirectory
of the buildout, so if you accidentally remove the entire buildout,
you also lose your backups.  It should be standard practice to use thelocationoption to specify a backup location in for example the
home directory of the user.  You should also arrange to copy that
backup to a different machine/country/continent/planet.BackupCallingbin/backupresults in a normal incremental repozo backup
that creates a backup of theData.fsinvar/backups.  When you
have a blob storage it is by default backed up tovar/blobstoragebackups.SnapshotsA quick backup just before updating the production server is a good
idea.  But you may not want to interfere with the regular backup
regime.  For that, thebin/snapshotbackupis great. It places a
full backup in, by default,var/snapshotbackups.ZipbackupsFor quickly grabbing the current state of a production database so you
can download it to your development laptop, you want a full and zipped
backup.  The zipped part is important for the blobstorage, because you
do not want to usescpto recursively copy over all those blob
files: downloading one tarball is faster.You can use thebin/zipbackupscript for this.  This script
overrides a few settings, ignoring whatever is set in the buildout
config section:archive_blobis turned on.keepis set to 1 to avoid keeping lots of needless backups.keep_blob_daysis ignored because it is a full backup.The script places a full backup in, by default,var/zipbackupsand
it puts a tarball of the blobstorage invar/blobstoragezips.This script is not created by default.
You can enable it by setting theenable_zipbackupoption to true.
Also, ifbackup_blobsis false, the scripts are useless, so we do not create them, even when you have enabled them explicitly.RestoreCallingbin/restorerestores the very latest normal incrementalrepozobackup and restores the blobstorage if you have that.You can restore the very latest snapshotbackup withbin/snapshotrestore.You can restore the zipbackup withbin/ziprestore.You can also restore the backup as of a certain date. Just pass a date argument.
According torepozo: specify UTC (not local) time.
The format isyyyy-mm-dd[-hh[-mm[-ss]]].
So as a simple example, restore to 25 december 1972:bin/restore 1972-12-25or to that same date, at 2,03 seconds past 1:bin/restore 1972-12-25-01-02-03Since version 2.3 this also works for restoring blobs.
We restore the directory from the first backup at or before the specified date.
(Note that before version 4.0 we restored the directory from the first backup after the specified date,
which should be fine as long as you did not do a database pack in between.)Since version 2.0, the restore scripts ask for confirmation before
starting the restore, as this is a potentially dangerous command.
(“Oops, I have restored the live site but I meant to restore the test
site.”)  You need to explicitly type ‘yes’:This will replace the filestorage (Data.fs).
This will replace the blobstorage.
Are you sure? (yes/No)?Note that for large filestorages and blobstoragesit may take long to restore.
You should do a test restore and check how long it takes.
Seconds?  Minutes?  Hours?
Is that time acceptable or should you take other measures?Names of created scriptsA backup part will normally be called[backup], leading to abin/backupandbin/snapshotbackup.  Should you name your part
something else,  the script names will also be different, as will the createdvar/directories (since version 1.2):[buildout]
parts = plonebackup

[plonebackup]
recipe = collective.recipe.backup
enable_zipbackup = trueThat buildout snippet will create these scripts:bin/plonebackup
bin/plonebackup-full
bin/plonebackup-zip
bin/plonebackup-snapshot
bin/plonebackup-restore
bin/plonebackup-ziprestore
bin/plonebackup-snapshotrestoreSupported optionsThe recipe supports the following options, none of which are needed by
default. The most common ones to change arelocationandblobbackuplocation, as those allow you to place your backups in
some system-wide directory like/var/zopebackups/instancename/and/var/zopebackups/instancename-blobs/.archive_blobUsetararchiving functionality.falseby default. Set it totrueand backup/restore will be done withtarcommand. Note thattarcommand must be available on machine if this option is set totrue.
This option also works with snapshot backup/restore commands. As this
counts as a full backupkeep_blob_daysis ignored.
See thecompress_bloboption if you want to compress the archive.alternative_restore_sourceYou can restore from an alternative source.  Use case: first make
a backup of your production site, then go to the testing or
staging server and restore the production data there.  SeeAlternative restore sourcealternative_restore_sourcesBackwards compatibility spelling foralternative_restore_source.
This will no longer work in version 7.backup_blobsBackup the blob storage.  Default isTrueon Python 2.6 (Plone
4) and higher, andFalseotherwise.  This requires theblob_storagelocation to be set.  If noblob_storagelocation has been set and we cannot find one by looking in the
other buildout parts, we quit with an error (since version 2.22).
Ifbackup_blobsis false,enable_zipbackupcannot be true,
because thezipbackupscript is not useful then.blob_storageLocation of the directory where the blobs (binary large objects)
are stored.  This is used in Plone 4 and higher, or on Plone 3 if
you useplone.app.blob.  This option is ignored if backup_blobs isfalse.  The location is not set by default.  When there is a part
usingplone.recipe.zeoserver,plone.recipe.zope2instanceorplone.recipe.zope2zeoserver, we check if that has a
blob-storage option and use that as default.  Note that we pick
the first one that has this option and we do not care about
shared-blob settings, so there are probably corner cases where we
do not make the best decision here.  Use this option to override
it in that case.blob-storageAlternative spelling for the preferredblob_storage, asplone.recipe.zope2instancespells it asblob-storageand we are
using underscores in all the other options.  Pick one.blob_timestampsNew in version 4.0.  Default is true (this was false before version 4.2).
If false, we createblobstorage.0.
The next time, we rotate this toblobstorage.1and create a newblobstorage.0.
Withblob_timestamps = true, we create stable directories that we do not rotate.
They get a timestamp, the same timestamp that the ZODB filestorage backup gets.
For example:blobstorage.1972-12-25-01-02-03.
Or witharchive_blob = true:blobstorage.1972-12-25-01-02-03.tar.
Because the filename is unpredictable, since version 4.1 we create alatestsymlink
to the most recent backup.
Blob timestamps are not used with zipbackup, because this only keeps 1 backup,
which means there is no confusion about which filestorage backup it belongs to.blobbackuplocationDirectory where the blob storage will be backed up to.  Defaults
tovar/blobstoragebackupsinside the buildout directory.blobsnapshotlocationDirectory where the blob storage snapshots will be created.
Defaults tovar/blobstoragesnapshotsinside the buildout
directory.blobziplocationDirectory where the blob storage zipbackups will be created.
Defaults tovar/blobstoragezipsinside the buildout
directory.compress_blobNew in version 4.0.  Default is false.
This is only used when thearchive_bloboption is true.
When switched on, it will compress the archive,
resulting in a.tar.gzinstead of atarfile.
When restoring, we always look for both compressed and normal archives.
We used to always compress them, but in most cases it hardly decreases the size
and it takes a long time anyway.  I have seen archiving take 15 seconds,
and compressing take an additional 45 seconds.
The result was an archive of 5.0 GB instead of 5.1 GB.datafsIn case theData.fsisn’t in the defaultvar/filestorage/Data.fslocation, this option can overwrite it.debugIn rare cases when you want to know exactly what’s going on, set debug totrueto get debug level logging of the recipe itself.repozois also run
with--verboseif this option is enabled.enable_snapshotrestoreHaving asnapshotrestorescript is very useful in development
environments, but can be harmful in a production buildout. The
script restores the latest snapshot directly to your filestorage
and it used to do this without asking any questions whatsoever
(this has been changed to require an explicityesas answer).
If you don’t want asnapshotrestorescript, set this option to false.enable_zipbackupCreatezipbackupandziprestorescripts.  Default: false.
Ifbackup_blobsis not on, these scripts are always disabled,
because they are not useful then.fullBy default, incremental backups are made. If this option is set totrue,bin/backupwill always make a full backup.incremental_blobsNew in version 4.0.  Default is false.
When switched on, it will use the--listed-incrementaloption oftar.
Note: this only works with the GNU version oftar.
On Mac you may need to install this withbrew installgnu-tarand change yourPATHaccording to the instructions.
It will create a metadata orsnapshot fileso that a second call to the backup script will create a second tarball with only the differences.
For some reason, all directories always end up in the second tarball,
even when there are no changes; this may depend on the used file system.
This option is ignored when thearchive_bloboption is false.
This optionrequirestheblob_timestampsoption to be true,
because it needs the tarball names to be stable, instead of getting rotated.
If you have explicitly setblob_timestampsto false, buildout will exit with an error.
Note that thelatestsymlink to the most recent backup is not created withincremental_blobstrue.
For large blobstorages it may take long to restore, so do test it out.
But that is wise in all cases.
Essentially, this feature seems to trade off storage space reduction with restore time.keepNumber of full backups to keep. Defaults to2, which means that the
current and the previous full backup are kept. Older backups are removed,
including their incremental backups. Set it to0to keep all backups.keep_blob_daysNumber ofdaysof blob backups to keep.  Defaults to14, so
two weeks.  This isonlyused for partial (full=False)
backups, so this is what gets used normally when you do abin/backup.  This option has been added in 2.2.  For full
backups (snapshots) we just use thekeepoption.  Recommended
is to keep these values in sync with how often you do azeopackon
theData.fs, according to the formulakeep * days_between_zeopacks = keep_blob_days.  The default matches one
zeopack per seven days (2*7=14).
Since version 4.0, this option is ignored unlessonly_blobsis true.
Instead, we remove the blob backups that have no matching filestorage backup.locationLocation where backups are stored. Defaults tovar/backupsinside the
buildout directory.locationprefixLocation of the folder where all other backup and snapshot folders will
be created. Defaults tovar/.
Note that this does not influence where we look for a source filestorage or blobstorage.only_blobsOnly backup the blobstorage, not theData.fsfilestorage.  False
by default.  May be a useful option if for example you want to
create onebin/filestoragebackupscript and onebin/blobstoragebackupscript, usingonly_blobsin one andbackup_blobsin the other.post_commandCommand to execute after the backup has finished.  One use case
would be to unmount the remote file system that you mounted
earlier using thepre_command.  See thatpre_commandabove for
more info.pre_commandCommand to execute before starting the backup.  One use case would
be to mount a remote file system using NFS or sshfs and put the
backup there.  Any output will be printed.  If you do not like
that, you can always redirect output somewhere else (mycommand > /dev/nullon Unix).  Refer to your local Unix guru for more
information.  If the command fails, the backup script quits with
an error.  You can specify multiple commands.quickCallrepozowith the--quickoption.  This option was
introduced tocollective.recipe.backupin version 2.19, withdefault value true.  Due to all the checksums that the repozo
default non-quick behavior does, an amount of data is read that is
three to four times as much as is in the actual filestorage.  With
the quick option it could easily be just a few kilobytes.
Theoretically the quick option is less safe, but it looks like it
can only go wrong when someone edits the.datfile in the
repository or removes a.deltafsfile.Thequickoption only influences the createdbin/backupscript.  It has no effect on the snapshot or restore scripts.The repozo help says about this option: “Verify via md5 checksum
only the last incremental written.  This significantly reduces the
disk i/o at the (theoretical) cost of inconsistency.  This is a
probabilistic way of determining whether a full backup is
necessary.”rsync_optionsAdd extra options to the defaultrsync-acommand. Default is no
extra parameters. This can be useful for example when you want to restore
a backup from a symlinked directory, in which casersync_options =--no-l-kdoes the trick.rsync_hard_links_on_first_copyWhen usingrsync, the blob files for the first backup are copied
and then subsequent backups make use of hard links from this initial
copy, to save time and disk space.
Enable this option to also use hard links for the initial copy to further reduce
disk usage.
This is safe for ZODB blobs, since they are not modified in place.
Theblob_storageand the backup folderblobbackuplocationhave to be in the same partition for hard links to be possible.snapshotlocationLocation where snapshot backups of the filestorage are stored. Defaults tovar/snapshotbackupsinside the buildout directory.use_rsyncUsersyncwith hard links for backing up the blobs.  Default is
true.rsyncis probably not available on all machines though, and
I guess hard links will not work on Windows.  When you set this to
false, we fall back to a simple copy (shutil.copytreefrom
Python in fact).ziplocationLocation where zip backups of the filestorage are stored. Defaults tovar/zipbackupsinside the buildout directory.An example buildout snippet using various options, would look like this:[backup]
recipe = collective.recipe.backup
location = ${buildout:directory}/myproject
keep = 2
datafs = subfolder/myproject.fs
full = true
debug = true
snapshotlocation = snap/my
enable_snapshotrestore = true
pre_command = echo 'Can I have a backup?'
post_command =
    echo 'Thanks a lot for the backup.'
    echo 'We are done.'Paths in directories or files can use relative (../) paths, and~(home dir) and$BACKUP-style environment variables are
expanded.Cron job integrationbin/backupis of course ideal to put in your cronjob instead of a wholebin/repozo....line. But you don’t want the “INFO” level logging that you
get, as you’ll get that in your mailbox. In your cronjob, just add-qor--quiet, andbin/backupwill shut up unless there’s a problem.
This option ignores the debug variable, if set to true in buildout.Speaking of cron jobs?  Take a look atzc.recipe.usercrontabif you want to handle
cronjobs from within your buildout.  For example:[backupcronjob]
recipe = z3c.recipe.usercrontab
times = 0 12 * * *
command = ${buildout:directory}/bin/backupBlob storageAdded in version 2.0.We can backup the blob storage.  Plone 4 uses a blob storage to store
files (Binary Large OBjects) on the file system.  In Plone 3 this is
optional.  When this is used, it should be backed up of course.  You
must specify the source blob_storage directory where Plone (or Zope)
stores its blobs.  As indicated earlier, when we do not set it
specifically, we try to get the location from other parts, for example
theplone.recipe.zope2instancerecipe:[buildout]
parts = instance backup

[instance]
recipe = plone.recipe.zope2instance
user = admin:admin
blob-storage = ${buildout:directory}/var/somewhere

[backup]
recipe = collective.recipe.backupIf needed, we can tell buildout that weonlywant to backup blobs or
specifically donotwant to backup the blobs.  Specifying this using
thebackup_blobsandonly_blobsoptions might be useful in
case you want to separate this into several scripts:[buildout]
newest = false
parts = filebackup blobbackup

[filebackup]
recipe = collective.recipe.backup
backup_blobs = false

[blobbackup]
recipe = collective.recipe.backup
blob_storage = ${buildout:directory}/var/blobstorage
only_blobs = trueWith this setupbin/filebackupnow only backs up the filestorage
andbin/blobbackuponly backs up the blobstorage.New in version 4.0: you may want to specifyblob_timestamps = true.
Then we create stable directories that we do not rotate.
For example:blobstorage.1972-12-25-01-02-03instead ofblobstorage.0.rsyncBy default we usersyncto create backups.  We create hard links
with this tool, to save disk space and still have incremental backups.
This probably requires a unixy (Linux, Mac OS X) operating system.
It is based on this article by Mike Rubel:http://www.mikerubel.org/computers/rsync_snapshots/We have not tried this on Windows.  Reports are welcome, but best is
probably to set theuse_rsync = falseoption in the backup part.
Then we simply copy the blobstorage directory.Alternative restore sourceAdded in version 2.17.
Changed in version 5: only one source is supported.You can restore from an alternative source.  Use case: first make a
backup of your production site, then go to the testing or staging
server and restore the production data there.In thealternative_restore_sourceoption you can define different
filestorage and blobstorage backup source directories using this
syntax:alternative_restore_source =
    storagename datafs1_backup [blobdir1_backup]The storagename must beData(or1) for the standardData.fsand optionally its blobstorage.The result is abin/altrestorescript.This will work for a standard buildout with a single filestorage and
blobstorage:[backup]
recipe = collective.recipe.backup
alternative_restore_source =
    Data /path/to/production/var/backups /path/to/production/var/blobstoragebackupsThe above configuration usesrepozoto restore the Data.fs from
the/path/to/production/var/backupsrepository to the standardvar/filestorage/Data.fslocation.  It copies the most recent
blobstorage backup from/path/to/production/var/blobstoragebackups/to the standardvar/blobstoragelocation.Calling the script with a specific date is supported just like the
normal restore script:bin/altrestore 2000-12-31-23-59The recipe will fail if the alternative source does not match the
standard filestorage and blobstorage.  For
example, you get an error when thealternative_restore_sourceis
missing theDatakey, when it has an extra key, when a
key has no paths, when a key has an extra or missing blobstorage.During install of the recipe, so during thebin/buildoutrun, it
does not check if the sources exist: you might have the production
backups on a different server and need to setup a remote shared
directory, or you copy the data over manually.Note that the script takes thearchive_blobanduse_rsyncoptions
into account.  So if the alternative restore source contains a blob
backup that was made witharchive_blob = true, you need analtrestorescript that also uses this setting.Contributorscollective.recipe.backup is basically a port of ye oldeinstancemanager’s
backup functionality. That backup functionality was coded mostly by Reinout
van Rees and Maurits van Rees, both fromZest softwareCreating the buildout recipe was done byReinoutwith some fixes byMaurits, who is now the main developer and maintainer.The snapshotrestore script was added by Nejc Zupan (niteoweb).The fullbackup script was added byTom ‘Spanky’ Kapanka.Archive blob backups feature added byMatej Cotman(niteoweb).SponsorshipWork on collective.recipe.backup has been made possible by Ghent University, or UGent.
Seehttps://www.ugent.be.
Ghent University is a top 100 university and one of the major universities in Belgium.Change history5.0.0 (2023-06-12)Bug fixes:Final release, no changes since previous alpha.  [maurits]5.0.0a3 (2023-03-15)Breaking changes:Removedadditional_filestoragesoption.
Removed support inalternative_restore_sourcesfor multiple storages: only the standard single filestorage plus blobstorage is supported.
Renamedalternative_restore_sourcestoalternative_restore_source, keeping the old alias for now.
[maurits] (Issue #64)Removedgzipoption.
From now on, we always call repozo with the gzip option, you can no longer turn this off.
[maurits] (Issue #67)Removed optiongzip_blob.
This was a backwards compatibility alias forarchive_blob.
You should use that now it you want to create a tar archive.
Additionally use thecompress_bloboption if you are really sure you want to compress the archive.
[maurits] (Issue #68)Removed optionquick.
From now on, we always call repozo with the quick option for normal backups, you can no longer turn this off.
For full backups (snapshotbackups and zipbackups), the quick option is not used.
[maurits] (Issue #69)Bug fixes:Deprecate setting blob_timestamps to false.
Seeissue 65.
[maurits] (Issue #65)5.0.0a2 (2023-02-01)Bug fixes:Add missing changelog entries for release 5.0.0a1.  [maurits]5.0.0a1 (2022-11-08)Breaking changes:Drop support for Python 2.7-3.7, add support for 3.10.
[maurits] (Issue #62)No longer support theenable_fullbackupoption and thefullbackupscript.
A full backup is automatically made by the first call ofbin/backupafter you have done a zeopack.
If you still want such a script, look atbin/snapshotbackupscript or use thefulloption, optionally in a second buildout section.
[maurits] (Issue #66)New features:Added an option to use hard links for the first backup copy
[ale-rt] (Issue #57)Add support for Python 3.11.  [maurits] (Issue #311)Bug fixes:Drop code-analysis, add other QA checks: isort, black, flake8.
[maurits] (Issue #63)4.2.0 (2021-08-30)New features:Changed default ofblob_timestampsto true.
We might remove the previous behavior (creatingblobstorage.0,blobstorage.1etcetera) completely in version 5.
[maurits] (Issue #61)Bug fixes:Switch the test infrastructure to GitHub Actions and tox.
Also test Python 3.8 and 3.9. [ale-rt, maurits] (Issue #58)Fixedlatestsymlink when using optionsblob_timestampsandno_rsync.
Previously it pointed to for exampleblobstorage.2021-01-02-03-04-05/blobstorageinstead of the directory above it.
[maurits] (Issue #61)When creating a zipbackup, do not create a blob timestamp.
We always only keep one zipbackup, so there is no need for timestamps to know which filestorage backup it belongs to.
[maurits] (Issue #61)4.1.1 (2019-10-07)Bug fixes:When using timestamps, use correct filestorage backup location for blob timestamp of snapshot and zip backup.
[maurits] (Issue #55)4.1.0 (2019-04-10)New features:Create symlink to latest timestamped blobstorage backup.
Whenblob_timestampsis false,blobstorage.0is a stable filename,
but withblob_timestampstrue, such a stable name was missing.
[maurits] (Issue #48)Dropped official Python 2.6 support, added Python 3.7 support.
No code change, but we are no longer testing 2.6.
[maurits] (Issue #49)Bug fixes:Updated package versions in test buildout.
Improved code quality according to flake8 and with black.
[maurits] (Issue #49)Removed already disabled support for sitepackage_safe_scripts.  [maurits] (Issue #51)4.0.1 (2018-04-19)Bug fixes:With archive_blob true, still allow restoring non archives. We prefer
archives then, but if no proper archive is found, we look for non archives.
[maurits] (Issue #44)Sort os.listdir, for improved functioning on Mac. [maurits] (Issue #45)4.0 (2017-12-22)Updated readme and added sponsorship note.  [maurits]4.0b5 (2017-11-17)Addedincremental_blobsoption.
This creates tarballs with only the changes compared to the previous blob backups.
This option is ignored when thearchive_bloboption is false.
[maurits]Improved code quality, reducing complexity.  [maurits]Refactored the main functions to not have so much code duplication.
The normal, full, snapshot and zip backups had almost the same code.
This made it hard to add new options.
[maurits]4.0b4 (2017-08-18)Test Python 3.6 (and 2.6 and 2.7) on Travis from now on.  [maurits]Ignore the zope2instance recipe integration tests on Python 3.
They would need a compatiblemailingloggerpackage.
Seeissue #31. [maurits]Tests: use cleaner way to check the mock repozo output.
Share this setup between tests.
This makes the output order the same on Python 2 and 3.
Seeissue #31. [maurits]4.0b3 (2017-07-05)Added basic Python 3 support.  We do not test with it yet,
but you should not get NameErrors anymore.
Seeissue #31. [maurits]4.0b2 (2017-06-26)No longer create thefullbackupscript by default.
You can still enable it by settingenable_fullbackuptotrue.
[maurits]Without explicitblob-storageoption, default tovar/blobstorage.
Take thevaroption from zeoserver/client recipes into account.
Fixesissue #27.
[maurits]Do not create hidden backup.0when blob_storage ends with a slash.
Fixesissue #26.
[maurits]4.0b1 (2017-05-31)Make custom backup locations relative to thelocationprefixoption or thevardirectory.
Until now, thelocationprefixoption was only used if you did not set custom locations.
Custom location would be relative to the buildout directory.
Now they are relative to thelocationprefixoption, with thevardirectory as default.
So if you used a relative path, your backups may end up in a different path.
Absolute paths are not affected: they ignore the locationprefix.
[maurits]When log level is DEBUG, show time stamps in the log.  [maurits]Addedcompress_bloboption.  Default is false.
This is only used when thearchive_bloboption is true.
When switched on, it will compress the archive,
resulting in a.tar.gzinstead of atarfile.
When restoring, we always look for both compressed and normal archives.
We used to always compress them, but in most cases it hardly decreases the size
and it takes a long time anyway.  I have seen archiving take 15 seconds,
and compressing take an additional 45 seconds.
The result was an archive of 5.0 GB instead of 5.1 GB.
[maurits]Renamedgzip_bloboption toarchive_blob.
Kept the old name as alias for backwards compatibility.
This makes room for letting this create an archive without zipping it.
[maurits]Automatically remove old blobs backups that have no corresponding filestorage backup.
We compare the timestamp of the oldest filestorage backup with the timestamps of the
blob backups.  This can be the name, if you useblob_timestamps = true,
or the modification date of the blob backup.
This means that thekeep_blob_daysoption is ignored, unless you useonly_blobs = true.
[maurits]When backing up a blobstorage, use the timestamp of the latest filestorage backup.
If a blob backup with that name is already there, then there were no database changes,
so we do not make a backup.
This is only done when you use the newblob_timestamps = trueoption.
[maurits]When restoring to a specific date, find the first blob backup at or before
the specified date.  Otherwise fail.  The repozo script does the same.
We used to pick the first blob backupafterthe specified date,
because we assumed that the user would specify the exact date that is
in the filestorage backup.
Note that the timestamp of the filestorage and blobstorage backups may be
a few seconds apart, unless you use theblob_timestamps == trueoption.
In the new situation, the user should pick the date of the blob backup
or slightly later.
[maurits]Addedblob_timestampsoption.  Default is false.
By default we createblobstorage.0.
The next time, we rotate this toblobstorage.1and create a newblobstorage.0.
Withblob_timestamps = true, we create stable directories that we do not rotate.
They get a timestamp, the same timestamp that the ZODB filestorage backup gets.
For example:blobstorage.1972-12-25-01-02-03.
[maurits]When restoring, first run checks for all filestorages and blobstorages.
When one of the backups is missing, we quit with an error.
This avoids restoring a filestorage and then getting into trouble
due to a missing blobstorage backup.  [maurits]3.1 (2017-02-24)Add alocationprefixoption to configure a folder where all other
backup and snapshot folders will be created [erral]Only claim compatibility with Python 2.6 and 2.7.  [maurits]Updated test buildout to use most recent versions.  [maurits]3.0.0 (2015-12-31)Refactored the init and install methods of this recipe.  During the
init phase we were reading the buildout configuration, but during
this phase the configuration is still being build.  So differences
could occur, especially in the order of execution of parts.  This
was not good.  Most code is now moved from the init to the install
(and update) method.  This has less possible problems.  Downside:
some configuration errors are caught later.
[maurits]Readzeo-var,var,file-storagefrom buildout sections.
Update default backup and Data.fs locations based on this.
[maurits]2.22 (2015-12-30)Do not acceptbackup_blobsfalse andenable_zipbackuptrue.
The zipbackup script is useless without blobs.
[maurits]Set defaultbackup_blobsto true on Python 2.6 (Plone 4) and
higher.  Otherwise false.  If noblob_storagecan be found, we
quit with an error.
[maurits]Accepttrue,yes,on,1, in lower, upper or mixed
case as true value.  Treat all other values in the buildout options
as false.
[maurits]Find plone.recipe.zope2instance recipes also when they are not
completely lower case.  The zope2instance recipe itself works fine
when it has mixed case, so we should accept this too.
[maurits]2.21 (2015-10-06)When restoring, createvar/filestorageif needed.
Fixes #23.
[maurits]2.20 (2014-11-11)Addenable_fullbackupoption.  Default: true, so no change
compared to previous version.
[maurits]Create backup/snapshot/zipbackup directories only when needed.
Running the backup script should not create the snapshot
directories.
[maurits]Add zipbackup and ziprestore scripts whenenable_zipbackup = true.
[maurits]2.19 (2014-06-16)Call repozo with--quickwhen making an incremental backup.
This is a lot faster.  Theoretically it lead to inconsistency if
someone is messing in your backup directory.  You can return to the
previous behavior by specifyingquick = falsein the backup
recipe part in your buildout config.
[maurits]check and create folders now happens after pre_commands is run
[@djay]2.18 (2014-04-29)Addrsync_optionsoption.  These are added to the defaultrsync-acommand. Default is no extra parameters. This can be
useful for example when you want to restore a backup from a
symlinked directory, in which casersync_options =--no-l-kdoes the trick.
[fiterbek]2.17 (2014-02-07)Addalternative_restore_sourcesoption.  This creates abin/altrestorescript that restores from an alternative backup
location, specified by that option.  You can use this to restore a
backup of the production data to your testing or staging server.
[maurits]When checking if the backup script will be able to create a path,
remove all created directories.  Until now, only the final directory
was removed, and not any created parent directories.
[maurits]Testing: split the single big doctest file into multiple files, to
make the automated tests less dependent on one another, making it
easier to change them and add new ones.
[maurits]No longer test with Python 2.4, because Travis does not support it
out of the box.  Should still work fine.
[maurits]2.16 (2014-01-14)Do not create blob backup dirs when not backing up blobs.
Do not create filestorage backup dirs when not backing up filestorage.
Fixeshttps://github.com/collective/collective.recipe.backup/issues/17[maurits]2.15 (2013-09-16)Restore compatibility with Python 2.4 (Plone 3).
[maurits]2.14 (2013-09-09)Archive blob backups with buildout optiongzip_blob.
[matejc]2.13 (2013-07-15)When printing that we halt the execution due to an error running
repozo, actually halt the execution.
[maurits]2.12 (2013-06-28)Backup directories are now created when we launchbackuporfullbackuporsnapshotbackupscripts, no more during
initialization.
[bsuttor]2.11 (2013-05-06)Print the names of filestorages and blobstorages that will be
restored.  Issue #8.
[maurits]Added a new command-line argument :--no-promptdisables user
input when restoring a backup or snapshot. Useful for shell scripts.
[bouchardsyl]Fixed command-line behavior with many arguments and not only a date.
[bouchardsyl]2.10 (2013-03-30)Addedfullbackupscript that defaults tofull=true.  This
could have been handled by making a new part, but it seemed like
overkill to have to generate a complete new set of backup scripts,
just to get one for full.
[spanky]2.9 (2013-03-06)Fixed possible KeyError:blob_snapshot_location.
[gforcada]2.8 (2012-11-13)Fixed possible KeyError:blob_backup_location.https://github.com/collective/collective.recipe.backup/issues/3[maurits]2.7 (2012-09-27)additional_filestorages improved: blob support and custom location.
[mamico]2.6 (2012-08-29)Added pre_command and post_command options.  See the documentation.
[maurits]2.5 (2012-08-08)Moved code to github:https://github.com/collective/collective.recipe.backup[maurits]2.4 (2011-12-20)Fixed silly indentation error that prevented old blob backups from
being deleted when older thankeep_blob_daysdays.
[maurits]2.3 (2011-10-05)Quit the rest of the backup or restore when a repozo call gives an
error.  Main use case: when restoring to a specific date repozo will
quit with an error when no files can be found, so we should also not
try to restore blobs then.
[maurits]Allow restoring the blobs to the specified date as well.
[maurits]2.2 (2011-09-14)Refactored script generation to make a split between initialization
code and script arguments.  This restores compatibility with
zc.buildout 1.5 for system pythons.  Actually we no longer create so
called ‘site package safe scripts’ but just normal scripts that work
for all zc.buildout versions.
[maurits]Added optionkeep_blob_days, which by default specifies that
only for partial backups we keep 14 days of backups.  See the
documentation.
[maurits]Remove old blob backups when doing a snapshot backup.
[maurits]2.1 (2011-09-01)Raise an error when the four backup location options
(blobbackuplocation, blobsnapshotlocation, location and
snapshotlocation) are not four distinct locations (or empty
strings).
[maurits]Fixed possible TypeError: ‘Option values must be strings’.
Found by Alex Clark, thanks.
[maurits]2.0 (2011-08-26)Backup and restore blobs, using rsync.
[maurits]Ask if the user is sure before doing a restore.
[maurits]1.7 (2010-12-10)Fix generated repozo commands to work also
when recipe is configured to have a nonData.fsmain db plus additional filestorages.
e.g.:
datafs= var/filestorage/main.fs
additional = catalog
[hplocher]1.6 (2010-09-21)Added the option enable_snapshotrestore so that the creation of the
script can be removed. Backwards compatible, if you don’t specify it
the script will still be created. Rationale: you may not want this
script in a production buildout where mistakenly using
snapshotrestore instead of snapshotbackup could hurt.
[fredvd]1.5 (2010-09-08)Fix: when running buildout with a config in a separate directory
(likebin/buildout-cconf/prod.cfg) the default backup
directories are no longer created inside that separate directory.
If you previously manually specified one of the location,
snapshotlocation, or datafs parameters to work around this, you can
probably remove those lines.  So: slightly saner defaults.
[maurits]1.4 (2010-08-06)Added documentation about how to get the required bin/repozo script
in your buildout if for some reason you do not have it yet (like on
Plone 4 when you do not have a zeo setup).
Thanks to Vincent Fretin for the extra buildout lines.
[maurits]1.3 (2009-12-08)Added snapshotrestore script.  [Nejc Zupan]1.2 (2009-10-26)The part name is now reflected in the created scripts and var/ directories.
Originally bin/backup, bin/snapshotbackup, bin/restore and var/backups
plus var/snapshotbackups were hardcoded.  Those are still there when you
name your part[backup].  With a part named[NAME], you get
bin/NAME, bin/NAME-snapshot, bin/NAME-restore and var/NAMEs plus
var/NAME-snapshots.  Request by aclark for plone.org.  [reinout]1.1 (2009-08-21)Run the cleanup script (removing too old backups that we no longer
want to keep) for additional file storages as well.
Fixeshttps://bugs.launchpad.net/collective.buildout/+bug/408224[maurits]Moved everything into a src/ subdirectory to ease testing on buildbot (which
would grab all egss in the eggs/ dir that buildbot’s mechanism creates.
[reinout]1.0 (2009-02-06)Quote all paths and arguments so that it works on paths that contain
spaces (specially on Windows). [sidnei]0.9 (2008-12-05)Windows path compatibility fix.  [Juan A. Diaz]0.8 (2008-09-23)Changed the default for gzipping to True. Addinggzip = trueto all our
server deployment configs gets tired pretty quickly, so doing it by default
is the best default. Stuff like this needs to be changedbeforea 1.0
release :-) [reinout]Backup of additional databases (if you have configured them) now takes place
before the backup of the main database (same with restore). [reinout]0.7 (2008-09-19)Added $BACKUP-style enviroment variable subsitution in addition to the tilde
expansion offered by 0.6. [reinout, idea by Fred van Dijk]0.6 (2008-09-19)Fixed the test setup so both bin/test and python setup.py test
work. [reinout+maurits]Added support for ~ in path names. And fixed a bug at the same time that
would occur if you call the backup script from a different location than
your buildout directory in combination with a non-absolute backup
location. [reinout]0.5 (2008-09-18)Added support for additional_filestorages option, needed for for instance a
split-out catalog.fs. [reinout]Test setup fixes. [reinout+maurits]0.4 (2008-08-19)Allowed the user to make the script more quiet (say in a cronjob)
by using ‘bin/backup -q’ (or –quiet).  [maurits]Refactored initialization template so it is easier to change.  [maurits]0.3.1 (2008-07-04)Added ‘gzip’ option, including changes to the cleanup functionality that
treats .fsz also as a full backup like .fs. [reinout]Fixed typo: repoze is now repozo everywhere… [reinout]0.2 (2008-07-03)Extra tests and documentation change for ‘keep’: the default is to keep 2
backups instead of all backups. [reinout]If debug=true, then repozo is also run in –verbose mode. [reinout]0.1 (2008-07-03)Added bin/restore. [reinout]Added snapshot backups. [reinout]Enabled cleaning up of older backups. [reinout]First working version that runs repozo and that creates a backup dir if
needed. [reinout]Started project based on zopeskel template. [reinout]