Repository: borgbackup/borg

Issue:
per archive total item count limit
the archive item (which has a list of "item metadata stream" chunks) is currently stored as a single object.

we limit size of repo objects to MAX_OBJ_SIZE (20MB).
note: we did only enforce the limit on get() - not on put().

each list entry takes ~40 bytes (32 sha256 id + 2 \* 4 for the size / csize), msgpacked a bit less maybe.

so, that means we can reference ~500.000 item metadata stream chunks per archive.

each item metadata stream chunk is ~16kiB (due to special, more fine grained and hardcoded item metadata chunker settings).

so that means the whole item metadata stream is limited to ~8GB.

the item entry size is determined by:
- file size (number of chunks \* 40B)
- path lengths
- amount of ACLs, xattrs
- a constant small amount of file metadata (timestamps, uid/gid, mode, ...)

if the medium size of an item entry is 100B (small size file, no ACLs/xattrs), that means we have a limit of 80M files/directories per archive. if the medium size of an item entry is 2kB (~100MB size files), that means we have a limit of 4M files/directories per archive.

we need a short term change to relax this limit and also a long term change to resolve it completely, this issue is about the short term change.

also, this limit should be documented.

per archive total item count limit
the archive item (which has a list of "item metadata stream" chunks) is currently stored as a single object.

we limit size of repo objects to MAX_OBJ_SIZE (20MB).
note: we did only enforce the limit on get() - not on put().

each list entry takes ~40 bytes (32 sha256 id + 2 \* 4 for the size / csize), msgpacked a bit less maybe.

so, that means we can reference ~500.000 item metadata stream chunks per archive.

each item metadata stream chunk is ~16kiB (due to special, more fine grained and hardcoded item metadata chunker settings).

so that means the whole item metadata stream is limited to ~8GB.

the item entry size is determined by:
- file size (number of chunks \* 40B)
- path lengths
- amount of ACLs, xattrs
- a constant small amount of file metadata (timestamps, uid/gid, mode, ...)

if the medium size of an item entry is 100B (small size file, no ACLs/xattrs), that means we have a limit of 80M files/directories per archive. if the medium size of an item entry is 2kB (~100MB size files), that means we have a limit of 4M files/directories per archive.

we need a short term change to relax this limit and also a long term change to resolve it completely, this issue is about the short term change.

also, this limit should be documented.



Discussion:
Note: attic used same 64KB chunk size for data and metadata. at some time, borg made the data chunker params user configurable and raised the default data chunk size to 2MB. at some time, the item metadata chunker got separate (hardcoded) chunker params and 16KB chunk size for better / more fine grained metadata dedup.

So, we could raise the chunk size for the metadata stream, e.g. to 128KB - increases limit by 8x and is easy to do.

related: #1453

> https://mail.python.org/pipermail/borgbackup/2016q3/000357.html

Bump max_object_size and such? Or change the metadata chunker params. The latter is more compatible.

In this case we have only 4.6 million files.

Hmm, 8GB max item metadata stream size for 4.6M files means almost 2kB per msgpacked item.

Why are items that big in this case?

Oh, guess we forgot there is of course also a item chunks list in the item metadata, with also ~40B per chunk. So if we have ~50 chunks per item, that is 2kB. 50 chunks with default chunker settings are ~100MB file size.

Since lots of these should be hard links they should have less chunks. 320 GB / 4.6 million is ~70 kB, and the 4.6 million even includes hard links. Metadata isn't shared among hard links, maybe ACLs or xattrs?

Note: attic used same 64KB chunk size for data and metadata. at some time, borg made the data chunker params user configurable and raised the default data chunk size to 2MB. at some time, the item metadata chunker got separate (hardcoded) chunker params and 16KB chunk size for better / more fine grained metadata dedup.

So, we could raise the chunk size for the metadata stream, e.g. to 128KB - increases limit by 8x and is easy to do.

related: #1453

> https://mail.python.org/pipermail/borgbackup/2016q3/000357.html

Bump max_object_size and such? Or change the metadata chunker params. The latter is more compatible.

In this case we have only 4.6 million files.

Hmm, 8GB max item metadata stream size for 4.6M files means almost 2kB per msgpacked item.

Why are items that big in this case?

Oh, guess we forgot there is of course also a item chunks list in the item metadata, with also ~40B per chunk. So if we have ~50 chunks per item, that is 2kB. 50 chunks with default chunker settings are ~100MB file size.

Since lots of these should be hard links they should have less chunks. 320 GB / 4.6 million is ~70 kB, and the 4.6 million even includes hard links. Metadata isn't shared among hard links, maybe ACLs or xattrs?


Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -39,7 +39,7 @@
 CHUNKER_PARAMS = (CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE)
 
 # chunker params for the items metadata stream, finer granularity
-ITEMS_CHUNKER_PARAMS = (12, 16, 14, HASH_WINDOW_SIZE)
+ITEMS_CHUNKER_PARAMS = (15, 19, 17, HASH_WINDOW_SIZE)
 
 has_lchmod = hasattr(os, 'lchmod')
 has_lchflags = hasattr(os, 'lchflags')


Test patch: