Repository: borgbackup/borg

Issue:
data loss: borg prune -P-- affects all archives
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

borg 1.1.10

#### Operating system (distribution) and version.

Debian testing, Debian stable (both)

#### Hardware / network configuration, and filesystems used.

ext4, tmpfs

#### How much data is handled by borg?

several TB, a few MB

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```
$ borg prune -d7 --stats -P-- io
```

## Describe the problem you're observing.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Yes:

```
$ borg init -e none test
$ borg create "test::test-$(date --iso-8601=seconds)" icons
$ borg create "test::--$(date --iso-8601=seconds)" icons
$ borg create "test::test-$(date --iso-8601=seconds)" icons
$ borg create "test::--$(date --iso-8601=seconds)" icons
$ borg prune -n --list --keep-hourly 1 -Ptest- test
Keeping archive: test-2019-09-29T18:11:06+02:00       Sun, 2019-09-29 18:11:06 [9d1c048374200473d4c2155488e24e8e6ea37c1b257e2bd684dff0c49f1f73cf]
Would prune:     test-2019-09-29T18:10:57+02:00       Sun, 2019-09-29 18:10:57 [f2d53c6752713f68d7e1409784ee22cdb2e8ca7094286e0199bbf919c5548c86]
$ borg prune -n --list --keep-hourly 1 -P-- test
Keeping archive: --2019-09-29T18:11:09+02:00          Sun, 2019-09-29 18:11:09 [6f59a4b2575626a2a8fa748be6e44abd6bb4db17ee1db0dfe67ab10b095ea71e]
Would prune:     test-2019-09-29T18:11:06+02:00       Sun, 2019-09-29 18:11:06 [9d1c048374200473d4c2155488e24e8e6ea37c1b257e2bd684dff0c49f1f73cf]
Would prune:     --2019-09-29T18:11:02+02:00          Sun, 2019-09-29 18:11:03 [795d5c84deb4d646135ae7fcf853a2f7537418b81f91aefba74a8e9bc097e5bb]
Would prune:     test-2019-09-29T18:10:57+02:00       Sun, 2019-09-29 18:10:57 [f2d53c6752713f68d7e1409784ee22cdb2e8ca7094286e0199bbf919c5548c86]
```

As you can see, ``borg prune`` will prune *all* archives when invoked with ``-P--``. This does not happen with ``-P-`` or ``-P---``. It also happens with  ``--prefix=--``, but not with ``'-a--*'``.

I would expect borg to either:

- Complain about a malformed argument if it can’t deal with ``--`` for whatever reason, or
- Operate correctly only pruning archives starting with ``--``

Instead, it pruned away many my backups because I assumed this would work :/ and did not do another ``--dry-run``.


Discussion:
Uuh, that is a strange bug.

I suspect you somehow confused python's `argparse` command line argument parsing, so that it thought `--` is another option and not the value for `--prefix` / `-P` and used an empty string there.

I'm sorry it pruned all your old backups.

Yeah, doing another `prune --dry-run` could have avoided it (or not using strange prefixes that can be confused with options).
As this might be a python argparse bug, can you please give the python version used by the borg client?

Did you use the borgbackup package from debian or the fat binary downloaded from us from github releases page?
BTW, you maybe don't need the `$(date ...)` call, have a look at borg's `{now}` placeholder.
Python: Python 3.7.4+. I use the debian package.

```
python3:
  Installed: 3.7.3-1
  Candidate: 3.7.3-1
  Version table:
 *** 3.7.3-1 990
        990 http://ftp.de.debian.org/debian testing/main amd64 Packages
        500 http://ftp.de.debian.org/debian unstable/main amd64 Packages
        100 /var/lib/dpkg/status
borgbackup:
  Installed: 1.1.10-2
  Candidate: 1.1.10-2
  Version table:
 *** 1.1.10-2 990
        990 http://ftp.de.debian.org/debian testing/main amd64 Packages
        500 http://ftp.de.debian.org/debian unstable/main amd64 Packages
        100 /var/lib/dpkg/status
```

FTR: This was luckily a rather unimportant backup repository, so no harm done!
So you use py 3.7.3? (first sentence contradicts output below)
This is indeed an argparse issue:

##### foo.py
```python
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("-P", "--prefix")
parser.add_argument("foo")
parser.add_argument("bar")

print(parser.parse_args())
```

```console
$ python3 foo.py -P xyz foo bar
Namespace(bar='bar', foo='foo', prefix='xyz')
$ python3 foo.py -P'-- foo' foo bar
Namespace(bar='bar', foo='foo', prefix='-- foo')
$ python3 foo.py -P'--' foo bar
Namespace(bar='bar', foo='foo', prefix=[])
```

This is super creepy. As a first measure, I suggest that borg rejects ``-P`` if it’s an empty list. As a second measure, we should figure out if this is intended behaviour.
Also noticeable: see how the prefix value changes from a str to an empty list when `--` is encountered.
> So you use py 3.7.3? (first sentence contradicts output below)

Huh, indeed.

```console
$ python3 --version
Python 3.7.4+
$ apt policy python3
python3:
  Installed: 3.7.3-1
  Candidate: 3.7.3-1
  Version table:
     3.7.5-1 500
        500 http://ftp.de.debian.org/debian unstable/main amd64 Packages
 *** 3.7.3-1 990
        990 http://ftp.de.debian.org/debian testing/main amd64 Packages
        100 /var/lib/dpkg/status
```

Not sure what to make of that.

> Also noticeable: see how the prefix value changes from a str to an empty list when ``--`` is encountered.

Yes, this is very concerning. Maybe as a first-aid measure, borg could detect that ``prefix`` is an empty list (as opposed to an empty string or None) and error out in that case. I think this is a terrible hack, but silently deleting all the archives is worse :/.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -302,7 +302,8 @@ def do_check(self, args, repository):
                        truish=('YES', ), retry=False,
                        env_var_override='BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'):
                 return EXIT_ERROR
-        if args.repo_only and any((args.verify_data, args.first, args.last, args.prefix)):
+        if args.repo_only and any(
+           (args.verify_data, args.first, args.last, args.prefix is not None, args.glob_archives)):
             self.print_error("--repository-only contradicts --first, --last, --prefix and --verify-data arguments.")
             return EXIT_ERROR
         if args.repair and args.max_duration:
@@ -318,7 +319,7 @@ def do_check(self, args, repository):
         if not args.archives_only:
             if not repository.check(repair=args.repair, save_space=args.save_space, max_duration=args.max_duration):
                 return EXIT_WARNING
-        if args.prefix:
+        if args.prefix is not None:
             args.glob_archives = args.prefix + '*'
         if not args.repo_only and not ArchiveChecker().check(
                 repository, repair=args.repair, archive=args.location.archive,
@@ -1092,7 +1093,7 @@ def do_rename(self, args, repository, manifest, key, cache, archive):
     @with_repository(exclusive=True, manifest=False)
     def do_delete(self, args, repository):
         """Delete an existing repository or archives"""
-        archive_filter_specified = args.first or args.last or args.prefix or args.glob_archives
+        archive_filter_specified = any((args.first, args.last, args.prefix is not None, args.glob_archives))
         explicit_archives_specified = args.location.archive or args.archives
         if archive_filter_specified and explicit_archives_specified:
             self.print_error('Mixing archive filters and explicitly named archives is not supported.')
@@ -1301,7 +1302,7 @@ def _list_repository(self, args, repository, manifest, key):
     @with_repository(cache=True, compatibility=(Manifest.Operation.READ,))
     def do_info(self, args, repository, manifest, key, cache):
         """Show archive details such as disk space used"""
-        if any((args.location.archive, args.first, args.last, args.prefix, args.glob_archives)):
+        if any((args.location.archive, args.first, args.last, args.prefix is not None, args.glob_archives)):
             return self._info_archives(args, repository, manifest, key, cache)
         else:
             return self._info_repository(args, repository, manifest, key, cache)
@@ -1397,7 +1398,7 @@ def do_prune(self, args, repository, manifest, key):
                              '"keep-secondly", "keep-minutely", "keep-hourly", "keep-daily", '
                              '"keep-weekly", "keep-monthly" or "keep-yearly" settings must be specified.')
             return self.exit_code
-        if args.prefix:
+        if args.prefix is not None:
             args.glob_archives = args.prefix + '*'
         checkpoint_re = r'\.checkpoint(\.\d+)?'
         archives_checkpoints = manifest.archives.list(glob=args.glob_archives,
@@ -2582,7 +2583,7 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
             filters_group = subparser.add_argument_group('Archive filters',
                                                          'Archive filters can be applied to repository targets.')
             group = filters_group.add_mutually_exclusive_group()
-            group.add_argument('-P', '--prefix', metavar='PREFIX', dest='prefix', type=PrefixSpec, default='',
+            group.add_argument('-P', '--prefix', metavar='PREFIX', dest='prefix', type=PrefixSpec, default=None,
                                help='only consider archive names starting with this prefix.')
             group.add_argument('-a', '--glob-archives', metavar='GLOB', dest='glob_archives',
                                type=GlobSpec, default=None,
diff --git a/src/borg/helpers/manifest.py b/src/borg/helpers/manifest.py
--- a/src/borg/helpers/manifest.py
+++ b/src/borg/helpers/manifest.py
@@ -103,7 +103,7 @@ def list_considering(self, args):
         """
         if args.location.archive:
             raise Error('The options --first, --last, --prefix and --glob-archives can only be used on repository targets.')
-        if args.prefix:
+        if args.prefix is not None:
             args.glob_archives = args.prefix + '*'
         return self.list(sort_by=args.sort_by.split(','), glob=args.glob_archives, first=args.first, last=args.last)
 


Test patch: