Repository: borgbackup/borg

Issue:
Error during prune command
Hi, I have an archive that seems to be corrupted. Both prune and check gives an error for the specific archive. I use the --list option for prune and it stops at the same archive every time. 

The repository is on a samba share mounted with smbnetfs.

Error with prune:

```
Traceback (most recent call last):
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 1520, in main
    exit_code = archiver.run(args)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 1465, in run
    return args.func(args)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 81, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 610, in do_prune
    Archive(repository, key, manifest, archive.name, cache).delete(stats)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archive.py", line 448, in delete
    if b'chunks' in item:
TypeError: argument of type 'int' is not iterable

Platform: Linux xps15 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64
Linux: Ubuntu 16.04 xenial
Borg: 1.0.3  Python: CPython 3.5.1+
PID: 19383  CWD: /home/jan
sys.argv: ['/home/jan/borg-env/bin/borg', 'prune', '-H', '99999', '--list', '-v', '/home/jan/sambadrive']
SSH_ORIGINAL_COMMAND: None
```

Error with check: 

```
Traceback (most recent call last):
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 1520, in main
    exit_code = archiver.run(args)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 1465, in run
    return args.func(args)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 81, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 148, in do_check
    last=args.last, prefix=args.prefix, save_space=args.save_space):
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archive.py", line 672, in check
    self.rebuild_refcounts(archive=archive, last=last, prefix=prefix)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archive.py", line 858, in rebuild_refcounts
    items_buffer.add(item)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/archive.py", line 84, in add
    self.buffer.write(self.packer.pack(StableDict(item)))
  File "msgpack/_packer.pyx", line 231, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:231)
  File "msgpack/_packer.pyx", line 233, in msgpack._packer.Packer.pack (msgpack/_packer.cpp:233)
  File "msgpack/_packer.pyx", line 200, in msgpack._packer.Packer._pack (msgpack/_packer.cpp:200)
  File "/home/jan/borg-env/lib/python3.5/site-packages/borg/helpers.py", line 868, in items
    return sorted(super().items())
TypeError: unorderable types: int() < bytes()

Platform: Linux xps15 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64
Linux: Ubuntu 16.04 xenial
Borg: 1.0.3  Python: CPython 3.5.1+
PID: 19534  CWD: /home/jan
sys.argv: ['/home/jan/borg-env/bin/borg', 'check', '/home/jan/sambadrive']
SSH_ORIGINAL_COMMAND: None
```



Discussion:
prune: seems like the msgpack unpacker unpacked an integer object instead of a dictionary with the item metadata - so it seems it got corrupted data. that's a bit strange as all the data is checked using a crc32 in the repo layer (or even hmac / sha on upper layers).

could you check the RAM of the machines that are involved, like running 1 full pass of memtest86+?
is it non-ECC or ECC RAM?

is your repo using encryption?

is your repo accessed by multiple machines? did you use break-lock recently?

did your borg repository disk run full?

borg check should be able to deal with corrupted data, so maybe there is some error handling missing, I'll have a look. if you could keep (a copy of) the repo in the reported state it would be nice so we can check the new code whether borg check is fixed.

@nesneros can you try the patch in pull request #1133 with `borg check -v` on your repo?

The output of that would be interesting, it should report the defect item metadata.

I have not yet check the ram but will do that.

My repo is not using encryption.

The repo is not accessed by multiple machines, but I have used break-lock recently. The borg disk is not full.

The repo is only a couple of weeks old, and I'm kind of evaluating using to take regular backup of my home dir. The backup is running as a cron job, at one point I added /etc which resulted as a lot of read permission errors, since the cron job is running as normal user and not root. I might also very well have killed borg a couple of times when it was creating a backup

I have executed check -v with the patch. As (relevant) output I get:

```
 Skipping broken item metadata dict [unorderable types: bytes() < int()]: {101: 1463765037673899107, b'gid': 1000, b'atime': 1463904905708518881, b'\x8a\xa5ati': 109, b'chunks': [[b'"\xa2{\x1a\xfbD/R\xc7\x17I\x93\xe4\x12i\xe0\x14\xf9\xcb\xea^\xf3\x88\xe2\x93\xc1\xf2\xcd\x873\xad-', 49, 98]], b'ctime': 1462111069680416304, b'group': b'jan'} [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
Did not get expected metadata dict when unpacking item metadata [chunk: 001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1]
```

Can you run `borg debug-dump-archive-items <repo>::<archive in question>`? You should get a couple files and one with the same SHA-256 hash in its name (72442c69...). Check with e.g. a hexeditor / hexdump whether it contains sensitive data and whether you feel comfortable sharing it for analysis.

Random data corruption: Has to not trigger CRC32 on these fairly short (~16 K) blocks while making a valid msgpack stream. The latter isn't too hard though, the distance between the object types in msgpack is very small. The output you show does look like it though.

I have attach the file dumped with debug-dump-archive-items (with txt extension to make github accept the file)

[001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1.items.txt](https://github.com/borgbackup/borg/files/298976/001477_72442c69343dfc6ec9361c47cb877f43210be52232c08ab7cefa24234111fcf1.items.txt)

@enkore you say "a couple of files". I actually got 1800. I tried with the archive made 12 minutes earlier and 12 minutes later, and both had 2000+ files. Did you literally only expect a couple of files?

```
Skipping broken item metadata dict [unorderable types: bytes() < int()]: {
b'\x8a\xa5ati': 109,  <- broken key - could have been "atime"?
101: 1463765037673899107,     <- broken key
b'gid': 1000,
b'atime': 1463904905708518881,
b'chunks': [[...]],
b'ctime': 1462111069680416304,
b'group': b'jan'
}
```

All keys are readable ascii strings.

btw, i have found a resync logic bug that occurs if item metadata dicts have more than 15 keys. updated PR coming soon.

Not really, depends on the archive size (the number of metadata chunks [what's dumped here as files] is proportional to the number of files).

Ok, this is chunk no 1477 and you say you got 1800 total. So this isn't the last chunk of the stream, therefore it can't be shorter than the minimum chunk length (4K for metadata). Therefore this chunk is truncated. [Take a peek at the files under the data/n/ directory in your repository: there's probably a suspiciously small file with an mtime around the time this archive was created?]

Looking at the data I think that two Borg instances may tripped over each other toes here. The data makes sense, but either there where concurrent writes to the same file or the packer state was corrupted or something like that.

Look at the start:

`00000000  8a a5 61 74 69 6d 65 cf  14 50 56 8b e1 30 64 63  |..atime..PV..0dc|`

 Notice how 0x8aa5 _seemingly_ appears in a key above - only because msgpack must have been already processing a map when reading this chunk. Specifically, the parser state must have been set to fixstr-5 by the last byte in the preceding chunk. _Note:_ So this chunk could be okay if it weren't too short.

Normally 0x8a starts a fixmap and 0xa5 is a 5-byte fixstr (`Xtime` etc). Since the parser state was fixstr-5 we get `0x8a 0xa5 ati` as a key. The value is 109, or 0x6d (`m`). Then follows 0x65 = 101 (`e`), which is then interpreted as the next key (101). Then we have 0xcf which indicates an uint64 which then follows etc.

What I find curious looking at this is that python-msgpack packs 32 bytes with a str16 instead of str8. Hu?

(`0x91 0x93 0xda <32 bytes>`)

Edit: Aha, str8 is a relatively recent msgpack addition.

@enkore @nesneros guess we also need chunk 001476_*.

@enkore If I understand you correctly you say that all files (except the last) should be at least 4K. I can inform you that file 1478 is also less than 4K. 
[001478_88ef6c30b6fb09bc6f55a158960b430d6eb6c3965293537404349c5da22b448a.items.txt](https://github.com/borgbackup/borg/files/299008/001478_88ef6c30b6fb09bc6f55a158960b430d6eb6c3965293537404349c5da22b448a.items.txt)

@enkore about str16: true, fixing that...

Attached is file 001476 

[001476_5befd541a1fa80cad1400368bfcb3cfc59cbdcd98ae15cab6fdb0e53e684149b.items.txt](https://github.com/borgbackup/borg/files/299010/001476_5befd541a1fa80cad1400368bfcb3cfc59cbdcd98ae15cab6fdb0e53e684149b.items.txt)

@nesneros can you run borg check again with the updated PR code?

Ok, from the data so far I think this is very plausible: 

> Looking at the data I think that two Borg instances may tripped over each other toes here. 

Look at the last items in 001476. At that point it's in the ~/projects/borg directory. The last item in 001476 is at setup.cfg. In 001477 it's ~/projects/sqlpark/sql/hive/test/.../ - this cannot happen if Borg traversed ~/projects normally (all the parent directory items are missing).

The last item in 001476 starts at 0x9020 (0x8a [=fixmap 10 elements] 0xa5 [=fixstr 5 bytes, atime]). It immediately starts with the atime, chunks, ctime, gid, EOF

Then comes 001477 and starts again with 0x8a 0xa5, but the fixmap started in 001476 has still a few elements remaining, which confirms my guess from above that the unpacker state must have been still in map mode at the beginning of 001477.

Edit:

> Specifically, the parser state must have been set to fixstr-5 by the last byte in the preceding chunk.

Indeed. `a3 6769 64 cd 03 e8 a5` => 0xa3 => fixstr 3, gid, 0xcd => 16 bit uint, 0x03e8 (1000), **0xa5** => fixstr-5

Re. fixing issues like this "the fast way": --force for `borg delete`, so that you can delete broken archives. I.e. it tries to delete normally, but doesn't fail if the metadata is corrupted, and the manifest entry is always removed at least [and the archive superblock deleted from the repo].

@nesneros could it be that 2 `borg create` ran at the same time?

like starting first one, starting second one (doesn't start due to the locks), using `borg break-lock`, starting second one (now really).

I am almost certain I have not used break-lock and had two borgs running at the same time.

As I mentioned earlier I trigger the backup via cron (at the time of the corrupted archive with 12 minutes interval), and I use the timestamp as part of the name. I still have the syslog for the creation of the corrupted archive. The borg create command completed succesful, and before cron triggered the next execution. From syslog with respect to the creation of the corrupted archive I have: 

```
May 30 19:42:33 xps15 RBACKUP: Archive name: homedir-2016-05-30T17:36:01
May 30 19:42:33 xps15 RBACKUP: Archive fingerprint: b3d69c949697587773b0e19a83bfd6125687b886dc84d18c0ded9ae5d2c08f33
May 30 19:42:33 xps15 RBACKUP: Time (start): Mon, 2016-05-30 19:36:31
May 30 19:42:33 xps15 RBACKUP: Time (end):   Mon, 2016-05-30 19:42:33
May 30 19:42:33 xps15 RBACKUP: Duration: 6 minutes 1.48 seconds
May 30 19:42:33 xps15 RBACKUP: Number of files: 146040
May 30 19:42:33 xps15 RBACKUP: ------------------------------------------------------------------------------
May 30 19:42:33 xps15 RBACKUP:                        Original size      Compressed size    Deduplicated size
May 30 19:42:33 xps15 RBACKUP: This archive:               19.04 GB             11.27 GB             93.82 kB
May 30 19:42:33 xps15 RBACKUP: All archives:              684.85 GB            405.05 GB              7.75 GB
May 30 19:42:33 xps15 RBACKUP: 
May 30 19:42:33 xps15 RBACKUP:                        Unique chunks         Total chunks
May 30 19:42:33 xps15 RBACKUP: Chunk index:                  110150              5239787
May 30 19:42:33 xps15 RBACKUP: ------------------------------------------------------------------------------
```

The running time of 6 minutes is very long. The next backup took 23 seconds (which is typically range) and had a dedup size of 288K (as opposed to 93K). The timestamp generated when  cron starts is about 30 seconds earlier than when borg tells the backup started. Normally that would be 1-2 seconds. Maybe the machine was on heavy load? Backup 12 minutes earlier and later is in "normal" range.

The corrupted backup generated log messages from borg like: 

```
/home/jan/projects/spark/core/target/scala-2.11/test-classes/org/apache/spark/rdd/AsyncRDDActionsSuite$$anonfun$1.class: [Errno 5] Input/output error
```

I might very well have been executing a compilation during the backup, creating and deleting class files. Would you expect messages like that if files disappear underneath borg?

> Would you expect messages like that if files disappear underneath borg?

No; if a file is truncated while open Borg will just save the truncated part. If it's deleted in a race from "backup this specific file" and "opening this specific file" then you'd get ENOENT (on the Python level this would be FileNotFoundError)

EIO means a (non-benign) file system or media failure. These are normally logged: if the file system generated the failure you'll find it in dmesg normally. If it's from the media then SMART should have it. This could also have caused the long duration as well.

Can you check the SMART data for both the backup source and repository location? `# smartctl -a /dev/sd..`

The target is a LaCie Cloudbox. I only have web ui, I can't see any option to run any kind of disk check. It is mounted with smbnetfs.

I backup mu home dir on a Dell XPS 15 laptop. After playing around with the options for smartctl I came up with the following:

```
$ suo smartctl -a -T permissive -d scsi /dev/nvme0n1
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-22-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               NVMe
Product:              PM951 NVMe SAMSU
Revision:             7D0Q
Compliance:           SPC-4
User Capacity:        512,110,190,592 bytes [512 GB]
Logical block size:   512 bytes
Rotation Rate:        Solid State Device
Logical Unit id:      0x0025384bf1b01bc5
Serial number:        S29PNXAGB36828
Device type:          disk
Local Time is:        Sun Jun  5 11:10:45 2016 CEST
device is NOT READY (e.g. spun down, busy)
SMART support is:     Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===

Current Drive Temperature:     41 C
Drive Trip Temperature:        82 C

Error Counter logging not supported


[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
Device does not support Self Test logging
```

Note that is says the drive lacks smart capabilities.

I use the laptop quite heavily, and have not in any other case seen anything that make me suspect a bad HD. I have grepped in syslog (7 days back) trying to find other I/O errors but could only find the one related to this backup. I suspected myself the I/O errors from borg  mean media failure. I strongly believe the disk is in good health, so I find it quite mysterious these errors.

I was looking more at the syslog around the execution of the backup. I was multiple time reconnecting to the wifi hotspot, and even trynig to use my phone as hotspot. (I believe I was experience some problems with internet connection). The Lacie drive is connected with wifi, so the connection to the target drive would have been disrupted during the backup.

I have probably around 50 borg backups in my syslog. All but one is running fine. I can't see how media errors on the source and disrupted connection to the target can be related, but either it is a weird coincidence or they are somehow related.

(note: 2 posts moved to #1138 -- tw)

I have been able to reproduce the issue by  disconnecting and reconnecting wifi during backup (and hence disrupting the connection to the target). It took 4-5 tries before I got the corrupted archive. The other tries it gave errors, but no corruption. I also saw the I/O error reporting errors on a source file. 

If should be any value the output of check is: 

```
Skipping broken item metadata dict [unorderable types: int() < bytes()]: {b'atime': 1465100933910358283, b'gid': 1000, b'path': b'home/jan/opt/idea-IC-145.597.3\x8a\xa5atime\xcf\x14PVi\xf2\x1c\x9d\xbe\xa6c', 104: 117, 110: 107, b'group': b'jan', b'mode': 16893, b'ctime': 1461340513601531707, b'mtime': 1461340513601531707} [chunk: 001482_d410191fae8488341cde102fe082e50f27fe97785b9bd9eb991e4b8a05846b81]
```

For information regarding smbnetfs

```
$ smbnetfs -V
SMBNetFS version 0.6.0
libsmbclient version 4.3.9-Ubuntu
FUSE library version: 2.9.4
fusermount version: 2.9.4
using FUSE kernel interface version 7.19
```

@nesneros `smartctl -son -a /dev/sdx` might help if device says there is no SMART.

Interesting that you could reproduce the problem by disconnecting wifi.

My current hypothesis on the exact events leading to this:

When the ChunkBuffer, which buffers item metadata and writes it out to the repository periodically, gets an OSError from the repository because e.g. an EIO happened, then the internal buffer is truncated without all data being written to the repository. This can then make these truncated/short chunks in the repo.

Because it is not possible to detect at which point the IO operation failed fatally this error has to be fatal and the operation must terminate and must not complete with a warning.

This should be addressed by patches for #1138 

@nesneros : thanks for the report and your help debugging it, much appreciated :)


Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -224,7 +224,7 @@ def iter_items(self, filter=None, preload=False):
             yield item
 
     def add_item(self, item):
-        unknown_keys = set(item) - ITEM_KEYS
+        unknown_keys = set(item) - self.manifest.item_keys
         assert not unknown_keys, ('unknown item metadata keys detected, please update ITEM_KEYS: %s',
                                   ','.join(k.decode('ascii') for k in unknown_keys))
         if self.show_progress:
@@ -587,9 +587,12 @@ def _open_rb(path):
 
 
 # this set must be kept complete, otherwise the RobustUnpacker might malfunction:
-ITEM_KEYS = set([b'path', b'source', b'rdev', b'chunks',
-                 b'mode', b'user', b'group', b'uid', b'gid', b'mtime', b'atime', b'ctime',
-                 b'xattrs', b'bsdflags', b'acl_nfs4', b'acl_access', b'acl_default', b'acl_extended', ])
+ITEM_KEYS = frozenset([b'path', b'source', b'rdev', b'chunks',
+                       b'mode', b'user', b'group', b'uid', b'gid', b'mtime', b'atime', b'ctime',
+                       b'xattrs', b'bsdflags', b'acl_nfs4', b'acl_access', b'acl_default', b'acl_extended', ])
+
+# this is the set of keys that are always present in items:
+REQUIRED_ITEM_KEYS = frozenset([b'path', b'mtime', ])
 
 
 def valid_msgpacked_item(d, item_keys_serialized):
@@ -623,9 +626,9 @@ def valid_msgpacked_item(d, item_keys_serialized):
 class RobustUnpacker:
     """A restartable/robust version of the streaming msgpack unpacker
     """
-    def __init__(self, validator):
+    def __init__(self, validator, item_keys):
         super().__init__()
-        self.item_keys = [msgpack.packb(name) for name in ITEM_KEYS]
+        self.item_keys = [msgpack.packb(name) for name in item_keys]
         self.validator = validator
         self._buffered_data = []
         self._resync = False
@@ -729,6 +732,11 @@ def rebuild_manifest(self):
         Iterates through all objects in the repository looking for archive metadata blocks.
         """
         logger.info('Rebuilding missing manifest, this might take some time...')
+        # as we have lost the manifest, we do not know any more what valid item keys we had.
+        # collecting any key we encounter in a damaged repo seems unwise, thus we just use
+        # the hardcoded list from the source code. thus, it is not recommended to rebuild a
+        # lost manifest on a older borg version than the most recent one that was ever used
+        # within this repository (assuming that newer borg versions support more item keys).
         manifest = Manifest(self.key, self.repository)
         for chunk_id, _ in self.chunks.iteritems():
             cdata = self.repository.get(chunk_id)
@@ -806,7 +814,8 @@ def robust_iterator(archive):
 
             Missing item chunks will be skipped and the msgpack stream will be restarted
             """
-            unpacker = RobustUnpacker(lambda item: isinstance(item, dict) and b'path' in item)
+            item_keys = self.manifest.item_keys
+            unpacker = RobustUnpacker(lambda item: isinstance(item, dict) and b'path' in item, item_keys)
             _state = 0
 
             def missing_chunk_detector(chunk_id):
@@ -821,6 +830,12 @@ def report(msg, chunk_id, chunk_no):
                 self.error_found = True
                 logger.error(msg)
 
+            def valid_item(obj):
+                if not isinstance(obj, StableDict):
+                    return False
+                keys = set(obj)
+                return REQUIRED_ITEM_KEYS.issubset(keys) and keys.issubset(item_keys)
+
             i = 0
             for state, items in groupby(archive[b'items'], missing_chunk_detector):
                 items = list(items)
@@ -835,7 +850,7 @@ def report(msg, chunk_id, chunk_no):
                     unpacker.feed(self.key.decrypt(chunk_id, cdata))
                     try:
                         for item in unpacker:
-                            if isinstance(item, dict):
+                            if valid_item(item):
                                 yield item
                             else:
                                 report('Did not get expected metadata dict when unpacking item metadata', chunk_id, i)
diff --git a/borg/helpers.py b/borg/helpers.py
--- a/borg/helpers.py
+++ b/borg/helpers.py
@@ -85,16 +85,19 @@ class Manifest:
 
     MANIFEST_ID = b'\0' * 32
 
-    def __init__(self, key, repository):
+    def __init__(self, key, repository, item_keys=None):
+        from .archive import ITEM_KEYS
         self.archives = {}
         self.config = {}
         self.key = key
         self.repository = repository
+        self.item_keys = frozenset(item_keys) if item_keys is not None else ITEM_KEYS
 
     @classmethod
     def load(cls, repository, key=None):
         from .key import key_factory
         from .repository import Repository
+        from .archive import ITEM_KEYS
         try:
             cdata = repository.get(cls.MANIFEST_ID)
         except Repository.ObjectNotFound:
@@ -112,6 +115,8 @@ def load(cls, repository, key=None):
         if manifest.timestamp:
             manifest.timestamp = manifest.timestamp.decode('ascii')
         manifest.config = m[b'config']
+        # valid item keys are whatever is known in the repo or every key we know
+        manifest.item_keys = frozenset(m.get(b'item_keys', [])) | ITEM_KEYS
         return manifest, key
 
     def write(self):
@@ -121,6 +126,7 @@ def write(self):
             'archives': self.archives,
             'timestamp': self.timestamp,
             'config': self.config,
+            'item_keys': tuple(self.item_keys),
         }))
         self.id = self.key.id_hash(data)
         self.repository.put(self.MANIFEST_ID, self.key.encrypt(data))
diff --git a/borg/testsuite/archive.py b/borg/testsuite/archive.py
--- a/borg/testsuite/archive.py
+++ b/borg/testsuite/archive.py
@@ -68,7 +68,7 @@ def _validator(self, value):
         return isinstance(value, dict) and value.get(b'path') in (b'foo', b'bar', b'boo', b'baz')
 
     def process(self, input):
-        unpacker = RobustUnpacker(validator=self._validator)
+        unpacker = RobustUnpacker(validator=self._validator, item_keys=ITEM_KEYS)
         result = []
         for should_sync, chunks in input:
             if should_sync:


Test patch: