Repository: borgbackup/borg

Issue:
Running `borg recreate --exclude-caches` on a specific archive raises a KeyError
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

```
$ borg -V
borg 1.2.0a8.dev61+gfc96fc4b
```

#### Operating system (distribution) and version.

Arch Linux

#### Hardware / network configuration, and filesystems used.

- borg host: btrfs RAID5 on HDD
- borg client: btrfs on SSD
- `borg create` over SSH, `borg recreate` locally on host

#### How much data is handled by borg?

```
$ du -hs /mnt/data/Backups/Hosts/able/borg
609G    /mnt/data/Backups/Hosts/able/borg
```

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```
$ export BORG_PASSCOMMAND="pass misc/borg"
$ export BORG_REPO="/mnt/data/Backups/Hosts/able/borg"
$ borg recreate --exclude-caches --keep-exclude-tags --progress --stats ::able-2019-08-21
```

## Describe the problem you're observing.

Using `borg recreate --exclude-caches` on a single specific archive in my repository raises a KeyError.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

The problem can be reproduced on this specific archive 100% of the time.

#### Include any warning/errors/backtraces from the system logs

```
$ export BORG_PASSCOMMAND="pass misc/borg"
$ export BORG_REPO="/mnt/data/Backups/Hosts/able/borg"
$ borg recreate --exclude-caches --keep-exclude-tags --progress --stats ::able-2019-08-21
recreate is an experimental feature.
Type 'YES' if you understand this and want to continue: YES
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 4476, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 4407, in run
    return set_ec(func(args))
  File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 165, in wrapper
    return method(self, args, repository=repository, cache=cache_, **kwargs)
  File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 1570, in do_recreate
    if not recreater.recreate(name, args.comment, args.target):
  File "/usr/lib/python3.8/site-packages/borg/archive.py", line 1887, in recreate
    self.matcher_add_tagged_dirs(archive)
  File "/usr/lib/python3.8/site-packages/borg/archive.py", line 2042, in matcher_add_tagged_dirs
    file = open_item(archive, cachedir_masters[item.source])
KeyError: 'arch/var/lib/flatpak/repo/objects/50/7741d5f44a0b8c6d81e907afb1160bb7f3aba3e44779d2bcb0b04a556a70f7.file'

Platform: Linux stratofortress.nexus.i.intelfx.name 5.4.7-arch1-1 #1 SMP PREEMPT Tue, 31 Dec 2019 17:20:16 +0000 x86_64
Linux: Unknown Linux
Borg: 1.2.0a8.dev61+gfc96fc4b  Python: CPython 3.8.1 msgpack: 0.6.2
PID: 2148059  CWD: /mnt/data/Backups/Hosts/able/borg
sys.argv: ['/usr/bin/borg', 'recreate', '--exclude-caches', '--keep-exclude-tags', '--progress', '--stats', '::able-2019-08-21']
SSH_ORIGINAL_COMMAND: None
```

## Additional notes:

Mounting the problematic archive reveals that `arch/var/lib/flatpak/repo/objects/50/7741d5f44a0b8c6d81e907afb1160bb7f3aba3e44779d2bcb0b04a556a70f7.file` contains valid cachedir tag contents, and adding a `print()` to the crash site reveals that this file is a hardlink destination for a properly named cachedir tag:

```
$ borg recreate --exclude-caches --keep-exclude-tags --progress --stats ::able-2019-08-21
recreate is an experimental feature.                                                                               
Type 'YES' if you understand this and want to continue: YES                                                        
XXX: going to crash:
        item.path=arch/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/18.08/365d62760000772f8cf401cfb6389252a2c1e2a0ebb39d99302fb730abfd8b74/files/cache/fontconfig/CACHEDIR.TAG
        item.source=arch/var/lib/flatpak/repo/objects/50/7741d5f44a0b8c6d81e907afb1160bb7f3aba3e44779d2bcb0b04a556a70f7.file
Local Exception
<...>
```

As I understand the situation, the crash occurs when a `CACHEDIR.TAG` is a hardlink to another file that is _not_ named `CACHEDIR.TAG` (which can and will happen with deduplicating content-addressable stores such as flatpak, ostree etc).

I have worked around this issue locally by simply skipping tag content verification when the hardlink cannot be resolved, but I don't see a good way to fix this properly without either loading the whole archive contents into the hashmap or making `matcher_add_tagged_dirs()` quadratic.

Any ideas?


Discussion:

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -2032,24 +2032,33 @@ def exclude(dir, tag_item):
         matcher = self.matcher
         tag_files = []
         tagged_dirs = []
-        # build hardlink masters, but only for paths ending in CACHE_TAG_NAME, so we can read hard-linked TAGs
+
+        # to support reading hard-linked CACHEDIR.TAGs (aka CACHE_TAG_NAME), similar to hardlink_masters:
         cachedir_masters = {}
 
+        if self.exclude_caches:
+            # sadly, due to how CACHEDIR.TAG works (filename AND file [header] contents) and
+            # how borg deals with hardlinks (slave hardlinks referring back to master hardlinks),
+            # we need to pass over the archive collecting hardlink master paths.
+            # as seen in issue #4911, the master paths can have an arbitrary filenames,
+            # not just CACHEDIR.TAG.
+            for item in archive.iter_items(filter=lambda item: os.path.basename(item.path) == CACHE_TAG_NAME):
+                if stat.S_ISREG(item.mode) and 'chunks' not in item and 'source' in item:
+                    # this is a hardlink slave, referring back to its hardlink master (via item.source)
+                    cachedir_masters[item.source] = None  # we know the key (path), but not the value (item) yet
+
         for item in archive.iter_items(
-                filter=lambda item: item.path.endswith(CACHE_TAG_NAME) or matcher.match(item.path)):
-            if item.path.endswith(CACHE_TAG_NAME):
+                filter=lambda item: os.path.basename(item.path) == CACHE_TAG_NAME or matcher.match(item.path)):
+            if self.exclude_caches and item.path in cachedir_masters:
                 cachedir_masters[item.path] = item
             dir, tag_file = os.path.split(item.path)
             if tag_file in self.exclude_if_present:
                 exclude(dir, item)
-            if stat.S_ISREG(item.mode):
-                if self.exclude_caches and tag_file == CACHE_TAG_NAME:
-                    if 'chunks' in item:
-                        file = open_item(archive, item)
-                    else:
-                        file = open_item(archive, cachedir_masters[item.source])
-                    if file.read(len(CACHE_TAG_CONTENTS)).startswith(CACHE_TAG_CONTENTS):
-                        exclude(dir, item)
+            elif self.exclude_caches and tag_file == CACHE_TAG_NAME and stat.S_ISREG(item.mode):
+                content_item = item if 'chunks' in item else cachedir_masters[item.source]
+                file = open_item(archive, content_item)
+                if file.read(len(CACHE_TAG_CONTENTS)) == CACHE_TAG_CONTENTS:
+                    exclude(dir, item)
         matcher.add(tag_files, IECommand.Include)
         matcher.add(tagged_dirs, IECommand.ExcludeNoRecurse)
 
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -1279,6 +1279,20 @@ def test_recreate_exclude_keep_tagged(self):
                  '--exclude-caches', '--keep-exclude-tags', self.repository_location + '::test')
         self._assert_test_keep_tagged()
 
+    @pytest.mark.skipif(not are_hardlinks_supported(), reason='hardlinks not supported')
+    def test_recreate_hardlinked_tags(self):  # test for issue #4911
+        self.cmd('init', '--encryption=none', self.repository_location)
+        self.create_regular_file('file1', contents=CACHE_TAG_CONTENTS)  # "wrong" filename, but correct tag contents
+        os.mkdir(os.path.join(self.input_path, 'subdir'))  # to make sure the tag is encountered *after* file1
+        os.link(os.path.join(self.input_path, 'file1'),
+                os.path.join(self.input_path, 'subdir', CACHE_TAG_NAME))  # correct tag name, hardlink to file1
+        self.cmd('create', self.repository_location + '::test', 'input')
+        # in the "test" archive, we now have, in this order:
+        # - a regular file item for "file1"
+        # - a hardlink item for "CACHEDIR.TAG" referring back to file1 for its contents
+        self.cmd('recreate', '--exclude-caches', '--keep-exclude-tags', self.repository_location + '::test')
+        # if issue #4911 is present, the recreate will crash with a KeyError for "input/file1"
+
     @pytest.mark.skipif(not xattr.XATTR_FAKEROOT, reason='Linux capabilities test, requires fakeroot >= 1.20.2')
     def test_extract_capabilities(self):
         fchown = os.fchown


Test patch: