Repository: borgbackup/borg

Issue:
borg check: false negative for item metadata qualifier
While analyzing #1130, I found a bug in borg check's internal "resync" code.

If an item metadata chunk is missing somehow, resync mode is triggered.
While it is active, it skips byte-wise over input data as long as the metadata dict qualifier returns a negative result.

The qualifier works by looking directly at the bytes in the input data (which is msgpacked serialized data) and looks for the bytes that introduce a mapping (dict) and a bytestring (key name inside the dict).

The problem was that it only checked for fixmap and that is only used for metadata dicts up to 15 items. If an item had more metadata, map16 is used and the qualifier did not check for that and thus returned a false negative, skipping over the (valid) data.

Another (theoretical) problem is that it only checked for fixstr and that is only used for bytestrings up to 31 bytes. If a metadata dict key were longer than that, str16 (or str8 or str32) would be used and the qualifier would not check for that and thus return a false negative, skipping over the (valid) data. This is not a real problem for borg, as (currently and in the past) all dict keys are shorter than 32 bytes.



Discussion:
len(borg.constants.ITEM_KEYS) = 18, but not all of them can occur on one item:
- acl_default/acl_access and acl_extended and acl_nfs4 are mutually exclusive. Making the count 15/16 (osx / linux+freebsd).
- bsdflags is not on Linux (in 1.0), making it 15 there. Still 16 on FreeBSD.
- rdev and source are mutually exclusive, making it 14/15 (rest / FreeBSD).

So for any 1.0.x it's safe. But not for 1.1+ archives, there we need the fix.

Are you implying the fix for this (potential) issue should not be applied to 1.0-maint?

I'd say it's not a bugfix in 1.0-maint ("check may break things"), but more like a, err, "compatibility-with-future-versions enhancement".

Hmm, if an archive is made with 1.1+ (assuming repo format is still compatible), running borg-1.0 check on it might trigger this "bug". With --repair, I guess it could remove valid data then. 

Yes and that's about what the changelog should say I think. Note that accessing a 1.1+ repo from 1.0.x already requires some minor manual intervention (deleting hints+index), if it's local (remote ofc. not). (Also I think that the scenario in general isn't really going to happen often.)


Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -592,6 +592,34 @@ def _open_rb(path):
                  b'xattrs', b'bsdflags', b'acl_nfs4', b'acl_access', b'acl_default', b'acl_extended', ])
 
 
+def valid_msgpacked_item(d, item_keys_serialized):
+    """check if the data <d> looks like a msgpacked item metadata dict"""
+    d_len = len(d)
+    if d_len == 0:
+        return False
+    if d[0] & 0xf0 == 0x80:  # object is a fixmap (up to 15 elements)
+        offs = 1
+    elif d[0] == 0xde:  # object is a map16 (up to 2^16-1 elements)
+        offs = 3
+    else:
+        # object is not a map (dict)
+        # note: we must not have item dicts with > 2^16-1 elements
+        return False
+    if d_len <= offs:
+        return False
+    # is the first dict key a bytestring?
+    if d[offs] & 0xe0 == 0xa0:  # key is a small bytestring (up to 31 chars)
+        pass
+    elif d[offs] in (0xd9, 0xda, 0xdb):  # key is a str8, str16 or str32
+        pass
+    else:
+        # key is not a bytestring
+        return False
+    # is the bytestring any of the expected key names?
+    key_serialized = d[offs:]
+    return any(key_serialized.startswith(pattern) for pattern in item_keys_serialized)
+
+
 class RobustUnpacker:
     """A restartable/robust version of the streaming msgpack unpacker
     """
@@ -622,18 +650,10 @@ def __next__(self):
             while self._resync:
                 if not data:
                     raise StopIteration
-                # Abort early if the data does not look like a serialized dict
-                if len(data) < 2 or ((data[0] & 0xf0) != 0x80) or ((data[1] & 0xe0) != 0xa0):
-                    data = data[1:]
-                    continue
-                # Make sure it looks like an item dict
-                for pattern in self.item_keys:
-                    if data[1:].startswith(pattern):
-                        break
-                else:
+                # Abort early if the data does not look like a serialized item dict
+                if not valid_msgpacked_item(data, self.item_keys):
                     data = data[1:]
                     continue
-
                 self._unpacker = msgpack.Unpacker(object_hook=StableDict)
                 self._unpacker.feed(data)
                 try:
diff --git a/borg/testsuite/archive.py b/borg/testsuite/archive.py
--- a/borg/testsuite/archive.py
+++ b/borg/testsuite/archive.py
@@ -2,8 +2,9 @@
 from unittest.mock import Mock
 
 import msgpack
+import pytest
 
-from ..archive import Archive, CacheChunkBuffer, RobustUnpacker
+from ..archive import Archive, CacheChunkBuffer, RobustUnpacker, valid_msgpacked_item, ITEM_KEYS
 from ..key import PlaintextKey
 from ..helpers import Manifest
 from . import BaseTestCase
@@ -112,3 +113,35 @@ def test_corrupt_chunk(self):
         input = [(False, chunks[:3]), (True, [b'gar', b'bage'] + chunks[3:])]
         result = self.process(input)
         self.assert_equal(result, [{b'path': b'foo'}, {b'path': b'boo'}, {b'path': b'baz'}])
+
+
+@pytest.fixture
+def item_keys_serialized():
+    return [msgpack.packb(name) for name in ITEM_KEYS]
+
+
+@pytest.mark.parametrize('packed',
+    [b'', b'x', b'foobar', ] +
+    [msgpack.packb(o) for o in (
+        [None, 0, 0.0, False, '', {}, [], ()] +
+        [42, 23.42, True, b'foobar', {b'foo': b'bar'}, [b'foo', b'bar'], (b'foo', b'bar')]
+    )])
+def test_invalid_msgpacked_item(packed, item_keys_serialized):
+    assert not valid_msgpacked_item(packed, item_keys_serialized)
+
+
+@pytest.mark.parametrize('packed',
+    [msgpack.packb(o) for o in [
+        {b'path': b'/a/b/c'},  # small (different msgpack mapping type!)
+        dict((k, b'') for k in ITEM_KEYS),  # as big (key count) as it gets
+        dict((k, b'x' * 1000) for k in ITEM_KEYS),  # as big (key count and volume) as it gets
+    ]])
+def test_valid_msgpacked_items(packed, item_keys_serialized):
+    assert valid_msgpacked_item(packed, item_keys_serialized)
+
+
+def test_key_length_msgpacked_items():
+    key = b'x' * 32  # 31 bytes is the limit for fixstr msgpack type
+    data = {key: b''}
+    item_keys_serialized = [msgpack.packb(key), ]
+    assert valid_msgpacked_item(msgpack.packb(data), item_keys_serialized)


Test patch: