Repository: borgbackup/borg

Issue:
Errors in check of repo converted from attic
Hi, I made a "borg check" on my compressed unencrypted repo stored on an USB external hard drive because of #1138 (with borg 1.0.8), and though repository analysis shows no error, during archives analysis I am getting plenty of errors such as :
`Did not get expected metadata dict when unpacking item metadata [chunk: 000026_a40edec797ea5a6c57452c3c5400b2566fffd75d86699fa6a144d207946237de]`
(with tens of errors for the same chunk, and thousands of chunks). Orphaned objects have been found as well.

I verified each archive independently, and it appears that only the older archives that were created with attic show this problem, not the archives that were created with borg after the conversion. The complete history of my repo is the following :
- September 2014 : created repo with attic 0.13
- September 2014 - October 2014 : created several archives with attic 0.13
- August 2015 : converted the repo with borg 0.23.0
- August 2015 - December 2015 : created several archives with borg 0.23.0 (and maybe other versions <= 0.29.0)

I ran "attic check" (with attic 0.16) on the original attic repository (which I kept), and it did not find any problem.

I focused on the latest archive created with attic (also the latest that has the problem). I attached the complete output of "borg check -v --archives-only" ([check.log.gz](https://github.com/borgbackup/borg/files/587393/check.log.gz)), and the result of "borg debug-dump-archive-items" ([items.log.gz](https://github.com/borgbackup/borg/files/587394/items.log.gz)). There are 4237 chunks, with 6 groups of duplicated chunks, each group with 2 or 3 duplicated chunks :
`
% ls | sort -k 1.8,1.24 | uniq -D -s 7
000752_06e338c1e28099ca2ad07ec728ba265f842db9754d21a94eb5f1e96bc320c517.items
000784_06e338c1e28099ca2ad07ec728ba265f842db9754d21a94eb5f1e96bc320c517.items
000444_1a5a6c364d47290cb2c776542ff3b8cd8158f31867eb46f2a396ea96ef2090d6.items
001098_1a5a6c364d47290cb2c776542ff3b8cd8158f31867eb46f2a396ea96ef2090d6.items
001137_1a5a6c364d47290cb2c776542ff3b8cd8158f31867eb46f2a396ea96ef2090d6.items
000761_2eabd1d3873fd061e0a24614e4d3ac9775421caa2ab96a37dfac1e3d69d7e454.items
000793_2eabd1d3873fd061e0a24614e4d3ac9775421caa2ab96a37dfac1e3d69d7e454.items
000750_4909761a9103abb057d075b9a4b4cf1c25971cb107feac064ca1362e4ff786a8.items
000782_4909761a9103abb057d075b9a4b4cf1c25971cb107feac064ca1362e4ff786a8.items
000751_fb370bb6b3edaa9e6c79fd75a7fca949041e655e4e8a6f39c17d6c6d727c9410.items
000783_fb370bb6b3edaa9e6c79fd75a7fca949041e655e4e8a6f39c17d6c6d727c9410.items
000450_fbd07dcf5e9e67fc579446d4e66b010afd7398a0384779afe8353935323541eb.items
001100_fbd07dcf5e9e67fc579446d4e66b010afd7398a0384779afe8353935323541eb.items
001139_fbd07dcf5e9e67fc579446d4e66b010afd7398a0384779afe8353935323541eb.items
`

Two of these groups are less than 4kB (2.4kB and 3.9kB).

Any idea of what could have happened ?


Discussion:
Is this the first check you ran, or did it previously report no errors, but now started to report these errors?

I probably ran a check after the conversion, but I don't remember for sure. I did not run another check afterwards, until today.

Hmm, a bit unusual for USB, but is this maybe same bug we fixed in 1.0.4? (see big warning at beginning of changes.rst)

This looks like we broke valid item detection in one of the more recent versions, because the item/chunk count is quite plausible and it was changed recently (also the recognized-as-broken indices are continuous).

Can you take a look at one of these .items files it gave you, whether there is some private data in there (like file names /paths, xattrs, these are plain-text). If you could find one that you're comfortable sharing with us it could be very helpful. If you don't want to upload it publicly you can send it via mail to Thomas or me (emails in the AUTHORS file in the repo). From an analysis perspective a 00000 file would be easiest. You can probably truncate it after a few kb.

Sent some items by email. I forgot to mention that the repo is compressed (the archives are created with parameters "--compression zlib,6 --chunker-params 10,23,16,4095").
Let me know if there is anything else I can do or try!

Thanks a lot for the data, it would've been quite difficult to find the issue without it! :)

After some hex-pickaxing my eye caught onto 'acl'. It seems that early attic releases up to and including 0.13 had a bug that always, unconditionally stored the value None in the 'acl' key. This was fixed in https://github.com/borgbackup/borg/commit/8f4e0f7506eee280d5489412aa5dde955dd6fed6#diff-1c7a2045a0364f6333ae24761429cb5eL357 :

``` py
    def stat_attrs(self, st, path):
        ...
        item[b'acl'] = acl_get(path, item, self.numeric_owner)
```

Note that acl_get() never returned the ACL, it was always "fetch ACL and put it into the item".

This then results in:

```
{b'path': b'redacted', ... b'acl': None}
```

Which is not a subset of the valid keys as currently defined.

So this is just a false positive in borg-check and no actual issue with the archives / repository.

Suggested fix:
1. Add a test case for this; since we know exactly now how this looks the data is easily generated for that
2. Add to valid keys, and just ignore it.

Maybe don't add it to ITEM_KEYS, it was never valid.

We could just obj.pop(b'acl', None) in valid_item() function, before the final check.

And add a comment above ITEM_KEYS constant that we shall never add a 'acl' key.

With the Item class in master we'll just add it to the in-line definition there (plus a comment what it's doing there). Then we probably don't need the pop in master, only in 1.0-maint.

``` py
class Item(PropDict):
    VALID_KEYS = ITEM_KEYS | {'deleted', 'nlink', }
```

Fantastic, I tested your PR and I confirm that it does not find any problem on my repo anymore. I also ran a "borg extract --dry-run" with borg 1.0.8 on the first attic archive, last attic archive, first borg archive, and last borg archive to confirm that there were no other problem. Concerning the duplicate items of "borg debug-dump-archive-items", it seems to correspond at least for some of them to large duplicated files in the archive, so I guess it is normal. Thank you for solving this quickly!

@cyril42e thanks for testing the PR!


Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -1017,11 +1017,21 @@ def report(msg, chunk_id, chunk_no):
                 self.error_found = True
                 logger.error(msg)
 
+            def list_keys_safe(keys):
+                return ', '.join((k.decode() if isinstance(k, bytes) else str(k) for k in keys))
+
             def valid_item(obj):
                 if not isinstance(obj, StableDict):
-                    return False
+                    return False, 'not a dictionary'
+                # A bug in Attic up to and including release 0.13 added a (meaningless) b'acl' key to every item.
+                # We ignore it here, should it exist. See test_attic013_acl_bug for details.
+                obj.pop(b'acl', None)
                 keys = set(obj)
-                return REQUIRED_ITEM_KEYS.issubset(keys) and keys.issubset(item_keys)
+                if not REQUIRED_ITEM_KEYS.issubset(keys):
+                    return False, 'missing required keys: ' + list_keys_safe(REQUIRED_ITEM_KEYS - keys)
+                if not keys.issubset(item_keys):
+                    return False, 'invalid keys: ' + list_keys_safe(keys - item_keys)
+                return True, ''
 
             i = 0
             for state, items in groupby(archive[b'items'], missing_chunk_detector):
@@ -1037,10 +1047,11 @@ def valid_item(obj):
                     unpacker.feed(self.key.decrypt(chunk_id, cdata))
                     try:
                         for item in unpacker:
-                            if valid_item(item):
+                            valid, reason = valid_item(item)
+                            if valid:
                                 yield item
                             else:
-                                report('Did not get expected metadata dict when unpacking item metadata', chunk_id, i)
+                                report('Did not get expected metadata dict when unpacking item metadata (%s)' % reason, chunk_id, i)
                     except RobustUnpacker.UnpackerCrashed as err:
                         report('Unpacker crashed while unpacking item metadata, trying to resync...', chunk_id, i)
                         unpacker.resync()
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -1451,6 +1451,25 @@ def test_empty_repository(self):
             repository.commit()
         self.cmd('check', self.repository_location, exit_code=1)
 
+    def test_attic013_acl_bug(self):
+        # Attic up to release 0.13 contained a bug where every item unintentionally received
+        # a b'acl'=None key-value pair.
+        # This bug can still live on in Borg repositories (through borg upgrade).
+        archive, repository = self.open_archive('archive1')
+        with repository:
+            manifest, key = Manifest.load(repository)
+            with Cache(repository, key, manifest) as cache:
+                archive = Archive(repository, key, manifest, '0.13', cache=cache, create=True)
+                archive.items_buffer.add({
+                    # path and mtime are required.
+                    b'path': '1234',
+                    b'mtime': 0,
+                    # acl is the offending key.
+                    b'acl': None
+                })
+                archive.save()
+        self.cmd('check', self.repository_location, exit_code=0)
+
 
 @pytest.mark.skipif(sys.platform == 'cygwin', reason='remote is broken on cygwin and hangs')
 class RemoteArchiverTestCase(ArchiverTestCase):


Test patch: