Repository: borgbackup/borg

Issue:
Always getting "Permission denied" on a folder that has been excluded
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### borg 1.1.17

#### macOS 11.5.2

## Describe the problem you're observing.

My borg command is as follow:

```
borg create \
    --exclude-from borg_excluded.txt \
    --progress \
    --one-file-system \
    --compression auto,zlib,6 \
    $BACKUP_DRIVE::$DATE \
    /Users/USERNAME/ \
    /Library
```

And my exclude file as follow:

```
...
/Library/Application Support/ApplePushService/
...
```

So the important part is that the "ApplePushService" folder is excluded. However, whenever I run the backup I'm getting the error:

> /Library/Application Support/ApplePushService: scandir: [Errno 13] Permission denied: '/Library/Application Support/ApplePushService'

So it seems Borg is still trying to access the folder even though I've excluded it. To be sure I've also tried excluding "**/ApplePushService" but that didn't make any difference and I still got the same error.

Any idea what might be the issue? Is that possibly a bug in Borg, or am I using the wrong exclude pattern?


Discussion:
Can you try to remove the trailing slash?
Also, if you use patterns, always be careful about the kind of patterns and what's the default kind, see `borg help patterns`.
Yes thank you, removing the trailing slash fixed it! Would that be considered a bug, or is it working as expected? (although in any case it's a bit confusing to get an error on an excluded folder)
Well, borg's internal matcher tries to match the path it is currently processing against the patterns you have given.

If you give "path/" and borg is currently at "path", that does not (yet) match.

IIRC we had some trailing-slash related change, could be that this is fixed already in borg 1.2 (rc1 available already).

I would call this a feature.  "path/to/folder/" matches the *contents* of folder, while "path/to/folder" matches the folder itself.  You might want to exclude the contents but backup the folder...

That's true, like you might want to copy a cache folder but not its content. Should this behaviour be documented in the exclude pattern doc? Happy to create a pull request for this if that can help.
Guess we should check this with borg 1.2 first ...
Tried to reproduce with borg master branch (borg 1.2.0rc1+):
```
(borg-env) tw@mba2020 borg % borg create --show-rc --exclude '/Library/Application Support/ApplePushService/' repo::a1 '/Library/Application Support'
/Library/Application Support/ApplePushService: dir_open: [Errno 13] Permission denied: 'ApplePushService'
/Library/Application Support/Apple/AssetCache/Data: dir_open: [Errno 13] Permission denied: 'Data'
/Library/Application Support/Apple/ParentalControls/Users: dir_open: [Errno 13] Permission denied: 'Users'
terminating with warning status, rc 1

(borg-env) tw@mba2020 borg % borg create --show-rc --exclude '/Library/Application Support/ApplePushService' repo::a2 '/Library/Application Support' 
/Library/Application Support/Apple/AssetCache/Data: dir_open: [Errno 13] Permission denied: 'Data'
/Library/Application Support/Apple/ParentalControls/Users: dir_open: [Errno 13] Permission denied: 'Users'
terminating with warning status, rc 1
```

So, in 1.2.0, it depends on presence of the trailing slash.

Reproduce with borg 1.1-maint branch (borg 1.1.17+):
```
(borg-env) tw@mba2020 borg % borg create --show-rc --exclude '/Library/Application Support/ApplePushService/' repo::a11 '/Library/Application Support'
/Library/Application Support/ApplePushService: scandir: [Errno 13] Permission denied: '/Library/Application Support/ApplePushService'
/Library/Application Support/Apple/AssetCache/Data: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/AssetCache/Data'
/Library/Application Support/Apple/ParentalControls/Users: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/ParentalControls/Users'
terminating with warning status, rc 1

(borg-env) tw@mba2020 borg % borg create --show-rc --exclude '/Library/Application Support/ApplePushService' repo::a12 '/Library/Application Support' 
/Library/Application Support/Apple/AssetCache/Data: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/AssetCache/Data'
/Library/Application Support/Apple/ParentalControls/Users: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/ParentalControls/Users'
terminating with warning status, rc 1
```

So, same thing in borg 1.1.

I agree with @jdchristensen, it's a feature! :-)

For the other include/exclude mechanism (`--pattern`) I'ld like to note that there are 2 kinds of exclusion:
- `-` excludes, but still recurses to match any potential includes
- `!` excludes and does not recurse (good if access is forbidden / problematic)

1.1-maint:
```
(borg-env) tw@mba2020 borg % borg create --show-rc --pattern '- /Library/Application Support/ApplePushService/' repo::a21 '/Library/Application Support'
/Library/Application Support/ApplePushService: scandir: [Errno 13] Permission denied: '/Library/Application Support/ApplePushService'
/Library/Application Support/Apple/AssetCache/Data: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/AssetCache/Data'
/Library/Application Support/Apple/ParentalControls/Users: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/ParentalControls/Users'
terminating with warning status, rc 1

(borg-env) tw@mba2020 borg % borg create --show-rc --pattern '- /Library/Application Support/ApplePushService' repo::a22 '/Library/Application Support' 
/Library/Application Support/ApplePushService: scandir: [Errno 13] Permission denied: '/Library/Application Support/ApplePushService'
/Library/Application Support/Apple/AssetCache/Data: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/AssetCache/Data'
/Library/Application Support/Apple/ParentalControls/Users: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/ParentalControls/Users'
terminating with warning status, rc 1

(borg-env) tw@mba2020 borg % borg create --show-rc --pattern '! /Library/Application Support/ApplePushService/' repo::a31 '/Library/Application Support'
/Library/Application Support/ApplePushService: scandir: [Errno 13] Permission denied: '/Library/Application Support/ApplePushService'
/Library/Application Support/Apple/AssetCache/Data: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/AssetCache/Data'
/Library/Application Support/Apple/ParentalControls/Users: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/ParentalControls/Users'
terminating with warning status, rc 1

(borg-env) tw@mba2020 borg % borg create --show-rc --pattern '! /Library/Application Support/ApplePushService' repo::a32 '/Library/Application Support' 
/Library/Application Support/Apple/AssetCache/Data: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/AssetCache/Data'
/Library/Application Support/Apple/ParentalControls/Users: scandir: [Errno 13] Permission denied: '/Library/Application Support/Apple/ParentalControls/Users'
terminating with warning status, rc 1
```

master: same behaviour.

@laurent22 if you could check the docs about this and make a PR, that would be great!

When editing docs, read the 1st line of the file, quite some stuff is autogenerated from `src/borg/archiver.py`.

Sure I'll take a look

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2234,6 +2234,11 @@ def do_break_lock(self, args, repository):
         /path/to/repo /home/user`` will store all files as
         `home/user/.../file.ext`.
 
+        A directory exclusion pattern can end either with or without a slash ('/').
+        If it ends with a slash, such as `some/path/`, the directory will be
+        included but not its content. If it does not end with a slash, such as
+        `some/path`, both the directory and content will be excluded.
+
         File patterns support these styles: fnmatch, shell, regular expressions,
         path prefixes and path full-matches. By default, fnmatch is used for
         ``--exclude`` patterns and shell-style is used for the ``--pattern``


Test patch: