Repository: borgbackup/borg

Issue:
Include patterns with excluded parent dir show entries in backup for all cache directories in parent dir
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG report

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

Borg version 1.1.6 Linux64 on both machines

#### Operating system (distribution) and version.

Both Debian Jessie

#### Hardware / network configuration, and filesystems used.

/var/lib/docker is ext4

#### How much data is handled by borg?

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

Excludes trimmed down (just regular --exclude to unrelated directories), destination and backup name anonymized, otherwise exactly as used

ionice -c 3 /etc/scripts/borg create -s --remote-path /etc/scripts/borg-linux64-1.1.6 --show-rc --exclude-caches --keep-exclude-tags \
         --pattern=+/var/lib/docker/volumes \
         --pattern=-/var/lib/docker \
         user@backuphost:/backup_path::backupname /

## Describe the problem you're observing.

Directories under /var/lib/docker/overlay2  that contain a CACHEDIR.TAG appear in the archive, though not the CACHEDIR.TAG files themselves (which would make sense due to --keep-exclude-tags), e.g. 

drwxr-xr-x root   root          0 Sun, 2018-07-08 01:33:05 var/lib/docker/overlay2/4716af2331e06a6a89d5de837cee369760e07ebc02579dbc3d6f6523c9904b0e/diff/var/cache/fontconfig
drwxr-xr-x root   root          0 Sun, 2018-03-18 00:23:42 var/lib/docker/overlay2/fcd658b1daddbd8d0787a1a17b08aa0ff00091da9201362cfd1f9354e5df512a/diff/var/cache/fontconfig
drwxr-xr-x root   root          0 Mon, 2017-07-10 11:13:21 var/lib/docker/overlay2/2891a096e4caa4ec612fb91598ef1193ed239472195dcd110fc9354566c310fa/diff/var/cache/fontconfig
drwxr-xr-x root   root          0 Sun, 2018-07-08 02:05:09 var/lib/docker/overlay2/5bf47430958e0ff9b68326b68314c3822713f4faa3b9c9fc6d0ea78d2c99290e/diff/var/cache/fontconfig
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:06:50 var/lib/docker/overlay2/5bf47430958e0ff9b68326b68314c3822713f4faa3b9c9fc6d0ea78d2c99290e/diff/var/cache/man       
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/cs    
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/da
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/de    
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/es    
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/fi    
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/fr    
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/hu    
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/id    
drwxr-sr-x man    root          0 Sun, 2018-07-08 02:09:24 var/lib/docker/overlay2/324e46dba0ccfb3fe1a7b1ed6218bc9f076bff4709e22f732044028d56093d8e/diff/var/cache/man/it    


#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

The problem appears in all (2) archives since I started using the patterns for /var/lib/docker and /var/lib/docker/volumes



Discussion:
To reproduce, this is easier:
```
(borg-env) tw@tux:~/w/borg$ mkdir inp
(borg-env) tw@tux:~/w/borg$ mkdir inp/excl
(borg-env) tw@tux:~/w/borg$ mkdir inp/excl/incl
(borg-env) tw@tux:~/w/borg$ mkdir inp/incl
(borg-env) tw@tux:~/w/borg$ mkdir inp/excl/check
(borg-env) tw@tux:~/w/borg$ echo 'Signature: 8a477f597d28d172789f06886806bc55' > inp/excl/check/CACHEDIR.TAG

(borg-env) tw@tux:~/w/borg$ borg init -e none repo

(borg-env) tw@tux:~/w/borg$ borg create repo::arch inp --list --exclude-caches --keep-exclude-tags --pattern='+inp/excl/incl' --pattern='-inp/excl'
x inp/excl
d inp/excl/incl
x inp/excl/check
x inp/excl/check/CACHEDIR.TAG
x inp/excl/check
d inp/incl
d inp

(borg-env) tw@tux:~/w/borg$ borg list repo::arch
drwxrwxr-x tw     tw            0 Sat, 2018-08-04 19:25:49 inp
drwxrwxr-x tw     tw            0 Sat, 2018-08-04 19:25:34 inp/excl/incl
drwxrwxr-x tw     tw            0 Sat, 2018-08-04 19:30:08 inp/excl/check
drwxrwxr-x tw     tw            0 Sat, 2018-08-04 19:25:49 inp/incl
```

Notable: `inp/excl/check` is archived, `inp/excl/check/CACHEDIR.TAG` is not archived.

The question now is whether the bug is
- that the check directory is archived, or
- that the CACHEDIR.TAG is not archived.
Well, considering nothing else in that explicitly excluded tree is archived I would say the cache directories and their CACHEDIR.TAG files should not be archived either if they are under an explicitly excluded path.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -583,13 +583,16 @@ def _process(self, fso, cache, matcher, exclude_caches, exclude_if_present,
                 if recurse:
                     tag_paths = dir_is_tagged(path, exclude_caches, exclude_if_present)
                     if tag_paths:
-                        if keep_exclude_tags and not dry_run:
-                            fso.process_dir(path, st)
-                            for tag_path in tag_paths:
-                                self._process(fso, cache, matcher, exclude_caches, exclude_if_present,
-                                              keep_exclude_tags, skip_inodes, tag_path, restrict_dev,
-                                              read_special=read_special, dry_run=dry_run)
-                        self.print_file_status('x', path)
+                        # if we are already recursing in an excluded dir, we do not need to do anything else than
+                        # returning (we do not need to archive or recurse into tagged directories), see #3991:
+                        if not recurse_excluded_dir:
+                            if keep_exclude_tags and not dry_run:
+                                fso.process_dir(path, st)
+                                for tag_path in tag_paths:
+                                    self._process(fso, cache, matcher, exclude_caches, exclude_if_present,
+                                                  keep_exclude_tags, skip_inodes, tag_path, restrict_dev,
+                                                  read_special=read_special, dry_run=dry_run)
+                            self.print_file_status('x', path)
                         return
                 if not dry_run:
                     if not recurse_excluded_dir:


Test patch: