Repository: borgbackup/borg

Issue:
add exception handler for "Data integrity error: More than allowed put data"
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Issue

## System information. For client/server mode post info for both machines.
CentOS 7.6.1810
#### Your borg version (borg -V).
borg 1.1.13
#### Operating system (distribution) and version.

#### Hardware / network configuration, and filesystems used.

#### How much data is handled by borg?
87TB
#### Full borg commandline that lead to the problem (leave away excludes and passwords)

sys.argv: ['/usr/bin/borg', 'create', '/backups6/roarchive::20200818092212', '/vnas00/data/roarchive', '-v', '--stats', '--one-file-system', '--show-rc', '--exclude-caches', '-e', '*.delme', '-e', '**/.cache', '--files-cache', 'ctime,size', '--numeric-owner', '--checkpoint-interval', '300']

## Describe the problem you're observing.
I was trying to backup 87TB of data, it always gets stuck around 56TB mark, then thows the error message.

```
Data integrity error: More than allowed put data [20996131 > 20971479]
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4565, in main
    exit_code = archiver.run(args)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4497, in run
    return set_ec(func(args))
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 176, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 584, in do_create
    create_inner(archive, cache)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 547, in create_inner
    read_special=args.read_special, dry_run=dry_run, st=st)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 661, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 661, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 661, in _process
    read_special=read_special, dry_run=dry_run)
  [Previous line repeated 6 more times]
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 635, in _process
    status = archive.process_file(path, st, cache)
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line 1071, in process_file
    self.chunk_file(item, cache, self.stats, backup_io_iter(self.chunker.chunkify(fd, fh)))
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line 1003, in chunk_file
    from_chunk, part_number = self.write_part_file(item, from_chunk, part_number)
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line 981, in write_part_file
    self.write_checkpoint()
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line 478, in write_checkpoint
    self.save(self.checkpoint_name)
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line 512, in save
    self.cache.add_chunk(self.id, data, self.stats)
  File "/usr/lib64/python3.6/site-packages/borg/cache.py", line 899, in add_chunk
    self.repository.put(id, data, wait=wait)
  File "/usr/lib64/python3.6/site-packages/borg/repository.py", line 1145, in put
    segment, offset = self.io.write_put(id, data)
  File "/usr/lib64/python3.6/site-packages/borg/repository.py", line 1545, in write_put
    raise IntegrityError('More than allowed put data [{} > {}]'.format(data_size, MAX_DATA_SIZE))
borg.helpers.IntegrityError: Data integrity error: More than allowed put data [20996131 > 20971479]

Platform: Linux vnasbak02.int.voleon.net 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64
Linux: CentOS Linux 7.6.1810 Core
Borg: 1.1.13  Python: CPython 3.6.8 msgpack: 0.5.6
PID: 15443  CWD: /usr/local/sbin
sys.argv: ['/usr/bin/borg', 'create', '/backups6/roarchive::20200818092212', '/vnas00/data/roarchive', '-v', '--stats', '--one-file-system', '--show-rc', '--exclude-caches', '-e', '*.delme', '-e', '**/.cache', '--files-cache', 'ctime,size', '--numeric-owner', '--checkpoint-interval', '300']
SSH_ORIGINAL_COMMAND: None

terminating with error status, rc 2
```

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
yes
#### Include any warning/errors/backtraces from the system logs
No system logs with regard to this issue.

<!--

If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html

*IMPORTANT* - Please mark logs and text output from terminal commands 
or else Github will not display them correctly. 
An example is provided below.

Example:
```
this is an example how log text should be marked (wrap it with ```)
```
-->



Discussion:
uhoh. that's an interesting failure, borg should never produce that big chunks.

is there anything special with your data? it seems you are the first finding this issue.

also please note that checkpoint-interval 300s is a rather bad idea if you are dealing with bigger data sets, that might add a huge overhead. if you have a rather stable connection to your repo, maybe rather use the default or an even longer interval.
Thanks for your advice. I'll change the interval back to default. Nothing special with my data, it just has a lot of small files and some hard links.
Is that your first borg backup of that data set?

borg maintains a "percent of max archive size used" value and displays it in `borg create --stats` and `borg info` output, iirc.

my first suspicion is that you maybe have gone beyond the limits (maybe due to the lots of little files), so the toplevel archive chunk got too big. if that is the case, borg currently can not do that, there is an open issue #1473 about this.

You could try excluding some files so the total amount of files per archive gets much lower (like 50% maybe) and have an eye on that "percentage of max. archive size used" value while you exclude less and less stuff.

Or rather partition your input data files set into multiple subsets, so each is significantly smaller than the overall set.

see also: https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#archives for more details.

Tagged as duplicate because this is a known and documented design limitation and we already have an open issue about it.

For the same reason, it's not a bug.
You are right, it's been 100% utilized. 

```
borg info .::20200818092212.checkpoint
Archive name: 20200818092212.checkpoint
Archive fingerprint: 25dcb5b0ad6013975778554c7c66721508a3058f5ec92847e57c359d9b088086
Comment:
Hostname: vnasbak02
Username: root
Time (start): Tue, 2020-08-18 09:22:16
Time (end): Thu, 2020-08-20 04:36:30
Duration: 1 days 19 hours 14 minutes 13.68 seconds
Number of files: 57080703
Command line: /usr/bin/borg create /backups6/roarchive::20200818092212 /vnas00/data/roarchive -v --stats --one-file-system --show-rc --exclude-caches -e '*.delme' -e '**/.cache' --files-cache ctime,size --numeric-owner --checkpoint-interval 300
Utilization of maximum supported archive size: 100%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               72.79 TB             67.99 TB             11.07 GB
All archives:              220.96 TB            205.67 TB             61.05 TB

                       Unique chunks         Total chunks
Chunk index:                70224481            235687551
```

Thanks for the confirmation!

Wow, > 57 Million files, that's quite a lot.
Borg should handle this situation more gracefully, giving a clear error message instead of a backtrace.
Guess we could add an exception handler in `save`.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -576,7 +576,15 @@ def save(self, name=None, comment=None, timestamp=None, stats=None, additional_m
         metadata = ArchiveItem(metadata)
         data = self.key.pack_and_authenticate_metadata(metadata.as_dict(), context=b'archive')
         self.id = self.key.id_hash(data)
-        self.cache.add_chunk(self.id, data, self.stats)
+        try:
+            self.cache.add_chunk(self.id, data, self.stats)
+        except IntegrityError as err:
+            err_msg = str(err)
+            # hack to avoid changing the RPC protocol by introducing new (more specific) exception class
+            if 'More than allowed put data' in err_msg:
+                raise Error('%s - archive too big (issue #1473)!' % err_msg)
+            else:
+                raise
         while self.repository.async_response(wait=True) is not None:
             pass
         self.manifest.archives[name] = (self.id, metadata.time)


Test patch: