Repository: borgbackup/borg

Issue:
prune: keeping last X number of backups
`prune` currently allows one to specify to keep all backups created inside a time interval via `--keep-within` or a more complex regime via `-H -d -w -m -y`. But what is missing (and what I need right now) is the possibility to keep the last X number of backups disregarding any time interval: `--keep-last N`



Discussion:
If you don't back up more than once an hour, then -H N would be equivalent.  (Hours without backups don't count towards the total.)

In this case I use borg to backup several VMs. Sometimes I backup multiple times an hour, sometimes I don't backup for several days. I just want to keep the last 30 (or so) backups around without accidentally deleting one (which then later turns out to be the important one) because it was the first of two backups during one hour. 

well, assuming you backed up 30 times in a single hour, would you just want to keep these 30 backups and delete the one from the hour before and from yesterday?

In this case: yes. I always want to keep the last X backups.

I see that this option conflicts/may conflict with `--keep-within` and the other time-based pruning options, which can be combined.

I would like this too.
I want to backup to a local server which then transmits the backups to an offsite location. The offsite location has only limited space available so I'd like to write a script that checks the folder size and calls prune with a decremented count until it fits. If you like to add a --keep-within XX MB size option, that would be nice too but would likely raise other questions for what to throw. 

I think, that while the prune options are very sophisticated, this scheme misses the most simple cases.
In my use case, it's a bunch of VMs, that are used on a **very irregular** base. Any time based prune policy will fail sooner or later, given, the backup task is: "just keep a certain number of revisions, no matter how old they are".

Also, I would like to note, that the documentation for pruning shows: it is complex to explain and to use. As time limited, as most of us are, it's hard to swallow in a short time, when getting started with borg. A simple --keep # option would help to satisfy the simple needs here, and leave the complex scheme for later (when it's really needed).

So, what do you think, would adding a --keep # option be acceptable, given, I would take care of the initial implementation?

From the comments above, one could conclude that --keep N is equivalent to `--keep-<small_timeunit> N` (e.g. we could just add minutes and seconds to existing theme). So it could be done by same mechanism and also would not cause any problem if mixed. Needs checking.

I agree that for some simple cases this is useful. It doesn't help with grasping the more complex scheme though and you will need the more complex scheme in many cases (as just keeping last N is often a bad option).


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -775,10 +775,10 @@ def format_cmdline(cmdline):
     @with_repository()
     def do_prune(self, args, repository, manifest, key):
         """Prune repository archives according to specified rules"""
-        if not any((args.hourly, args.daily,
+        if not any((args.secondly, args.minutely, args.hourly, args.daily,
                     args.weekly, args.monthly, args.yearly, args.within)):
             self.print_error('At least one of the "keep-within", "keep-last", '
-                             '"keep-hourly", "keep-daily", '
+                             '"keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", '
                              '"keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified.')
             return self.exit_code
         archives = manifest.list_archive_infos(sort_by='ts', reverse=True)  # just a ArchiveInfo list
@@ -787,6 +787,10 @@ def do_prune(self, args, repository, manifest, key):
         keep = []
         if args.within:
             keep += prune_within(archives, args.within)
+        if args.secondly:
+            keep += prune_split(archives, '%Y-%m-%d %H:%M:%S', args.secondly, keep)
+        if args.minutely:
+            keep += prune_split(archives, '%Y-%m-%d %H:%M', args.minutely, keep)
         if args.hourly:
             keep += prune_split(archives, '%Y-%m-%d %H', args.hourly, keep)
         if args.daily:
@@ -798,21 +802,21 @@ def do_prune(self, args, repository, manifest, key):
         if args.yearly:
             keep += prune_split(archives, '%Y', args.yearly, keep)
 
-        keep.sort(key=attrgetter('ts'), reverse=True)
-        to_delete = [a for a in archives if a not in keep]
+        to_delete = set(archives) - set(keep)
         stats = Statistics()
         with Cache(repository, key, manifest, do_files=args.cache_files, lock_wait=self.lock_wait) as cache:
-            for archive in keep:
-                if args.output_list:
-                    logger.info('Keeping archive: %s' % format_archive(archive))
-            for archive in to_delete:
-                if args.dry_run:
-                    if args.output_list:
-                        logger.info('Would prune:     %s' % format_archive(archive))
+            for archive in archives:
+                if archive in to_delete:
+                    if args.dry_run:
+                        if args.output_list:
+                            logger.info('Would prune:     %s' % format_archive(archive))
+                    else:
+                        if args.output_list:
+                            logger.info('Pruning archive: %s' % format_archive(archive))
+                        Archive(repository, key, manifest, archive.name, cache).delete(stats)
                 else:
                     if args.output_list:
-                        logger.info('Pruning archive: %s' % format_archive(archive))
-                    Archive(repository, key, manifest, archive.name, cache).delete(stats)
+                        logger.info('Keeping archive: %s' % format_archive(archive))
             if to_delete and not args.dry_run:
                 manifest.write()
                 repository.commit(save_space=args.save_space)
@@ -1587,13 +1591,10 @@ def build_parser(self, args=None, prog=None):
         any of the specified retention options. This command is normally used by
         automated backup scripts wanting to keep a certain number of historic backups.
 
-        As an example, "-d 7" means to keep the latest backup on each day, up to 7
-        most recent days with backups (days without backups do not count).
-        The rules are applied from hourly to yearly, and backups selected by previous
-        rules do not count towards those of later rules. The time that each backup
-        completes is used for pruning purposes. Dates and times are interpreted in
-        the local timezone, and weeks go from Monday to Sunday. Specifying a
-        negative number of archives to keep means that there is no limit.
+        If a prefix is set with -P, then only archives that start with the prefix are
+        considered for deletion and only those archives count towards the totals
+        specified by the rules.
+        Otherwise, *all* archives in the repository are candidates for deletion!
 
         The "--keep-within" option takes an argument of the form "<int><char>",
         where char is "H", "d", "w", "m", "y". For example, "--keep-within 2d" means
@@ -1601,10 +1602,18 @@ def build_parser(self, args=None, prog=None):
         "1m" is taken to mean "31d". The archives kept with this option do not
         count towards the totals specified by any other options.
 
-        If a prefix is set with -P, then only archives that start with the prefix are
-        considered for deletion and only those archives count towards the totals
-        specified by the rules.
-        Otherwise, *all* archives in the repository are candidates for deletion!
+        A good procedure is to thin out more and more the older your backups get.
+        As an example, "--keep-daily 7" means to keep the latest backup on each day,
+        up to 7 most recent days with backups (days without backups do not count).
+        The rules are applied from secondly to yearly, and backups selected by previous
+        rules do not count towards those of later rules. The time that each backup
+        completes is used for pruning purposes. Dates and times are interpreted in
+        the local timezone, and weeks go from Monday to Sunday. Specifying a
+        negative number of archives to keep means that there is no limit.
+
+        The "--keep-last N" option is doing the same as "--keep-secondly N" (and it will
+        keep the last N archives under the assumption that you do not create more than one
+        backup archive in the same second).
         """)
         subparser = subparsers.add_parser('prune', parents=[common_parser], add_help=False,
                                           description=self.do_prune.__doc__,
@@ -1623,6 +1632,10 @@ def build_parser(self, args=None, prog=None):
                                help='output verbose list of archives it keeps/prunes')
         subparser.add_argument('--keep-within', dest='within', type=str, metavar='WITHIN',
                                help='keep all archives within this time interval')
+        subparser.add_argument('--keep-last', '--keep-secondly', dest='secondly', type=int, default=0,
+                               help='number of secondly archives to keep')
+        subparser.add_argument('--keep-minutely', dest='minutely', type=int, default=0,
+                               help='number of minutely archives to keep')
         subparser.add_argument('-H', '--keep-hourly', dest='hourly', type=int, default=0,
                                help='number of hourly archives to keep')
         subparser.add_argument('-d', '--keep-daily', dest='daily', type=int, default=0,


Test patch: