Repository: borgbackup/borg

Issue:
excluded directory still accessed, causing backup to fail on missing permissions
This is a regression in borg 1.1.0.

I'm running backups as `root` while regular users have some directories mounted via FUSE. `root` is not allowed to access those directories. Therefore I exclude them from being backed up with `--exclude /path/to/fuse-mount-point`.

In borg 1.0.9 this used to work fine. The directories were not backed up, borg did not emit an error message, and barring any other errors borg exited with code 0 signalling success.

In borg 1.1.0 it still tries to access that mount point. That fails, and borg emits an error message "Permission denied". The whole borg run exits with error code 1 in that case signalling an error.

Here's how to reproduce:

1. Create a repository
2. Set up a tree to back up with some files and a directory to be used as the FUSE mount point
3. Configure FUSE not to allow access to mounted directories by other users (in `/etc/fuse.conf`: option `user_allow_other` must not be active)
4. Mount a file system as a regular user on said mount point, e.g. with `sshfs`
5. Try to create a borg backup as root while excluding said mount point

Here's an example I've just run:

``` sh
[0 mbunkus@chai-latte ~] mkdir -p tmp/test/{tree,repo}
[0 mbunkus@chai-latte ~] cd ~/tmp/test/tree
[0 mbunkus@chai-latte ~/tmp/test/tree] borg init -e none ~/tmp/test/repo
[0 mbunkus@chai-latte ~/tmp/test/tree] echo chunky > bacon.txt
[0 mbunkus@chai-latte ~/tmp/test/tree] echo foo > bar.txt
[0 mbunkus@chai-latte ~/tmp/test/tree] mkdir fuse
[0 mbunkus@chai-latte ~/tmp/test/tree] sshfs sweet-chili: fuse
[0 mbunkus@chai-latte ~/tmp/test/tree] ls -l
total 12
-rw-rw-r-- 1 mbunkus Domain Users    7 Oct 23 12:41 bacon.txt
-rw-rw-r-- 1 mbunkus Domain Users    4 Oct 23 12:43 bar.txt
drwxr-xr-x 1    1000        10002 2468 Oct 23 10:38 fuse/
[2 mbunkus@chai-latte ~/tmp/test/tree] sudo borg create --exclude ~/tmp/test/tree/bar.txt --exclude ~/tmp/test/tree/fuse $HOME/tmp/test/repo::test ~/tmp/test/tree
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] y
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~/tmp/test/tree] sudo borg list $HOME/tmp/test/repo::test
drwxrwxr-x mbunkus Domain Users        0 Mon, 2017-10-23 12:43:30 home/mbunkus/tmp/test/tree
-rw-rw-r-- mbunkus Domain Users        7 Mon, 2017-10-23 12:41:55 home/mbunkus/tmp/test/tree/bacon.txt
[0 mbunkus@chai-latte ~/tmp/test/tree]
```

The test shows that the `--exclude` statements are correct. The first exclusion of the file `bar.txt` is handled correctly by borg; the file isn't present in the archive. The second exclusion isn't handled correctly as the error message shows.



Discussion:
did you try 1.1.1 version?
I wasn't aware that 1.1.1 had been released. Sorry about that.

A quick re-run of my procedure shown above with 1.1.1 shows that the problem still exists in 1.1.1.
Likely this is due to some changes in 1.1 branch - we added the option that you can exclude some directory and include some stuff below it again - for that, the excluded dir needs to be traversed and files/dirs encountered matched against the other patterns.

IIRC this was supposed to be a `--patterns` (new) thing only, so the behaviour change of the already existing --exclude was unintended (and unknown yet).

For a quick workaround, you could use `--patterns` with the pattern that unconditionally excludes (not recursing into the exclusion). See the docs for details.
I've read the whole section "borg help patterns" in the documentation, but I'm not sure I really understand which pattern you mean. I tried `--pattern '- /home/…`, but that didn't change anything, just like `--pattern '- pf:/home/…`:

```
[0 mbunkus@chai-latte ~/tmp/test/tree] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "- $HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test2 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~/tmp/test/tree] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "- pf:$HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test3 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~/tmp/test/tree] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "- pp:$HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test4 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~/tmp/test/tree]
```

Hmm, is the norecurse variant "!" of exclude (instead of recursing variant "-") not documented?
I'd say it isn't:

```
[0 mbunkus@chai-latte ~] ./borg-1.1.1 help patterns | grep -F '!'
    matching any single character specified, including ranges, and '[!...]'
[0 mbunkus@chai-latte ~]
```

However, using `!` doesn't change anything either:

```
[0 mbunkus@chai-latte ~] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "! pp:$HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test6 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~] sudo ~/borg-1.1.1 create --pattern "- $HOME/tmp/test/tree/bar.txt" --pattern "! $HOME/tmp/test/tree/fuse" $HOME/tmp/test/repo::test7 ~/tmp/test/tree
/home/mbunkus/tmp/test/tree: stat: [Errno 13] Permission denied: '/home/mbunkus/tmp/test/tree/fuse'
[1 mbunkus@chai-latte ~]
```
Looks like it does a stat() (which fails) on the excluded dir before deciding to not recurse into it.
BTW, rc 1 means "warning", 2 is "error".

For "borg create" failing to read some items is not an error, it will just log that and continue with the backup (that will eventually reach its normal end). Only some major, fatal issue will give rc 2.
That's good to know, thanks (how did I overlook this?). I'll adjust my scripts accordingly.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -537,20 +537,27 @@ def _process(self, fso, cache, matcher, exclude_caches, exclude_if_present,
 
         This should only raise on critical errors. Per-item errors must be handled within this method.
         """
-        if st is None:
-            with backup_io('stat'):
-                st = os.stat(path, follow_symlinks=False)
-
-        recurse_excluded_dir = False
-        if not matcher.match(path):
-            self.print_file_status('x', path)
-
-            if stat.S_ISDIR(st.st_mode) and matcher.recurse_dir:
-                recurse_excluded_dir = True
+        try:
+            recurse_excluded_dir = False
+            if matcher.match(path):
+                if st is None:
+                    with backup_io('stat'):
+                        st = os.stat(path, follow_symlinks=False)
             else:
-                return
+                self.print_file_status('x', path)
+                # get out here as quickly as possible:
+                # we only need to continue if we shall recurse into an excluded directory.
+                # if we shall not recurse, then do not even touch (stat()) the item, it
+                # could trigger an error, e.g. if access is forbidden, see #3209.
+                if not matcher.recurse_dir:
+                    return
+                if st is None:
+                    with backup_io('stat'):
+                        st = os.stat(path, follow_symlinks=False)
+                recurse_excluded_dir = stat.S_ISDIR(st.st_mode)
+                if not recurse_excluded_dir:
+                    return
 
-        try:
             if (st.st_ino, st.st_dev) in skip_inodes:
                 return
             # if restrict_dev is given, we do not want to recurse into a new filesystem,
@@ -1823,10 +1830,12 @@ def do_break_lock(self, args, repository):
             may specify the backup roots (starting points) and patterns for inclusion/exclusion.
             A root path starts with the prefix `R`, followed by a path (a plain path, not a
             file pattern). An include rule starts with the prefix +, an exclude rule starts
-            with the prefix -, both followed by a pattern.
+            with the prefix -, an exclude-norecurse rule starts with !, all followed by a pattern.
             Inclusion patterns are useful to include paths that are contained in an excluded
             path. The first matching pattern is used so if an include pattern matches before
-            an exclude pattern, the file is backed up.
+            an exclude pattern, the file is backed up. If an exclude-norecurse pattern matches
+            a directory, it won't recurse into it and won't discover any potential matches for
+            include rules below that directory.
 
             Note that the default pattern style for ``--pattern`` and ``--patterns-from`` is
             shell style (`sh:`), so those patterns behave similar to rsync include/exclude


Test patch: