Repository: borgbackup/borg

Issue:
borgfs include/exclude pattern matching broken
Version: 1.1.4 (Debian/stretch-backports, amd64)


The attached test script creates a fresh repo, one archive and then tries to mount the repo with three different command-lines.

Case #1:
- Mounting works
- Accessing the archive list within the repo mount point works
- Acessing individual archives leads to an exception, see 1-*.log


Cases ##2, 3:
Mounting immediately leads to an exception, both with shell-style globbing (```'*'```) and a simple regex (```'.*'```) - see the 2nd and 3rd log.

[borgfsfailure.sh.txt](https://github.com/borgbackup/borg/files/1635185/borgfsfailure.sh.txt)
[1-borgfs-nopattern.log](https://github.com/borgbackup/borg/files/1635187/1-borgfs-nopattern.log)
[1-ls.log](https://github.com/borgbackup/borg/files/1635188/1-ls.log)
[2-borgfs-asterisk.log](https://github.com/borgbackup/borg/files/1635189/2-borgfs-asterisk.log)
[3-borgfs-dot-asterisk.log](https://github.com/borgbackup/borg/files/1635190/3-borgfs-dot-asterisk.log)









Discussion:
A quick workaround is

```
# diff -Naur /usr/lib/python3/dist-packages/borg/patterns.py{.orig,}
--- /usr/lib/python3/dist-packages/borg/patterns.py.orig        2018-01-17 17:58:41.352862857 +0100
+++ /usr/lib/python3/dist-packages/borg/patterns.py     2018-01-17 17:58:49.120876127 +0100
@@ -133,6 +133,8 @@
     def add_inclexcl(self, patterns):
         """Add list of patterns (of type CmdTuple) to internal list.
         """
+        if patterns is None:
+               return
         for pattern, cmd in patterns:
             self._add(pattern, cmd)
```

which allows to at least access all the archives. Actual usage of --exclude and/or --pattern still has no (obvious) effect. I'm not sure though whether this is to attribute to an implementation error or to insufficient documentation about PATTERN usage.
Is there a special reason why you use `borgfs ...` and not `borg mount ...`?
I have my backup repos integrated in `/etc/fstab` so I can access them via a simple `mount`. I've gotten used to that so much that I did not in the least think that `borg mount` might make a difference.

I just tried again and can confirm that `borg mount` actually works without patching `patterns.py`.

So it seems that argument handling somewhat differs between `borg` and `borgfs` invocations...
> So it seems that argument handling somewhat differs between borg and borgfs invocations...

Found the culprit: the argument parser taking effect when `sys.argv[0]` is `borgfs`, lacks a default value for `patterns` which leads to the reported exception.

If one considers `pattern.py` implemented by-contract, another possible fix is
```
--- archiver.py 2018-01-22 01:01:19.894241848 +0100
+++ archiver.py2        2018-01-22 01:01:17.782230054 +0100
@@ -2505,6 +2505,7 @@
 
         common_parser = argparse.ArgumentParser(add_help=False, prog=self.prog)
         # some empty defaults for all subparsers
+        parser.set_defaults(patterns=[])
         common_parser.set_defaults(paths=[], patterns=[])
         parser.common_options.add_common_group(common_parser, '_subcommand')
 ```
Hmm, shouldn't it also default paths=[]? And maybe only set these defaults for borgfs?
The latter would require quite some re-structuring of the argparser objects, I guess. And I'm not sure about the first one. You probably should ask the one who implemented the include/exclude cmdline-options and associated code.

Personally, I very much prefer patching patterns.py.
in the calls seen in the traceback, it processes both paths and patterns and it tries to iterate over both.
guess this is why it defaults both to [], just one line below the set_defaults call you added.

and we need to be consistent here: either we always go the set_defaults way, or we kill set_defaults and adapt patterns.py code to cope with getting None.
working on this.

about case # 1: valid, fix via set_defaults for both path and patterns. i also tried to fix via patterns.py and remove the set_defaults, but other code also relies on the empty list default for these.

about cases # 2 and 3: invalid pattern, patterns must start with + / - / !.
the error msg it gave was not really helpful, but I already fixed that recently so that it is more clear now what's going wrong.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -45,7 +45,7 @@
 from .crypto.keymanager import KeyManager
 from .helpers import EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR
 from .helpers import Error, NoManifestError, set_ec
-from .helpers import positive_int_validator, location_validator, archivename_validator, ChunkerParams
+from .helpers import positive_int_validator, location_validator, archivename_validator, ChunkerParams, Location
 from .helpers import PrefixSpec, SortBySpec, HUMAN_SORT_KEYS, FilesCacheMode
 from .helpers import BaseFormatter, ItemFormatter, ArchiveFormatter
 from .helpers import format_timedelta, format_file_size, parse_file_size, format_archive
@@ -1755,9 +1755,11 @@ def repo_validate(section, name, value=None, check_value=True):
         def cache_validate(section, name, value=None, check_value=True):
             if section not in ['cache', ]:
                 raise ValueError('Invalid section')
-            # I looked at the cache config and did not see anything a user would want to edit,
-            # so, for now, raise for any key name
-            raise ValueError('Invalid name')
+            if name in ['previous_location', ]:
+                if check_value:
+                    Location(value)
+            else:
+                raise ValueError('Invalid name')
 
         try:
             section, name = args.name.split('.')
@@ -2562,7 +2564,9 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
 
         parser = argparse.ArgumentParser(prog=self.prog, description='Borg - Deduplicated Backups',
                                          add_help=False)
-        parser.set_defaults(fallback2_func=functools.partial(self.do_maincommand_help, parser))
+        # paths and patterns must have an empty list as default everywhere
+        parser.set_defaults(fallback2_func=functools.partial(self.do_maincommand_help, parser),
+                            paths=[], patterns=[])
         parser.common_options = self.CommonOptions(define_common_options,
                                                    suffix_precedence=('_maincommand', '_midcommand', '_subcommand'))
         parser.add_argument('-V', '--version', action='version', version='%(prog)s ' + __version__,
@@ -2570,7 +2574,6 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
         parser.common_options.add_common_group(parser, '_maincommand', provide_defaults=True)
 
         common_parser = argparse.ArgumentParser(add_help=False, prog=self.prog)
-        # some empty defaults for all subparsers
         common_parser.set_defaults(paths=[], patterns=[])
         parser.common_options.add_common_group(common_parser, '_subcommand')
 
@@ -3754,12 +3757,17 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
                                    type=CompressionSpec, default=CompressionSpec('lz4'),
                                    help='select compression algorithm, see the output of the '
                                         '"borg help compression" command for details.')
-        archive_group.add_argument('--recompress', dest='recompress', nargs='?', default='never', const='if-different',
-                                   choices=('never', 'if-different', 'always'),
-                                   help='recompress data chunks according to ``--compression`` if `if-different`. '
-                                        'When `always`, chunks that are already compressed that way are not skipped, '
-                                        'but compressed again. Only the algorithm is considered for `if-different`, '
-                                        'not the compression level (if any).')
+        archive_group.add_argument('--recompress', metavar='MODE', dest='recompress', nargs='?',
+                                   default='never', const='if-different', choices=('never', 'if-different', 'always'),
+                                   help='recompress data chunks according to ``--compression``. '
+                                        'MODE `if-different`: '
+                                        'recompress if current compression is with a different compression algorithm '
+                                        '(the level is not considered). '
+                                        'MODE `always`: '
+                                        'recompress even if current compression is with the same compression algorithm '
+                                        '(use this to change the compression level). '
+                                        'MODE `never` (default): '
+                                        'do not recompress.')
         archive_group.add_argument('--chunker-params', metavar='PARAMS', dest='chunker_params',
                                    type=ChunkerParams, default=CHUNKER_PARAMS,
                                    help='specify the chunker parameters (CHUNK_MIN_EXP, CHUNK_MAX_EXP, '


Test patch: