Repository: borgbackup/borg

Issue:
borg check --repair: Data integrity error: id verification failed
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes, many hardware issues seem related, but this appears _after_ a hardware issue was resolved.

## Is this a BUG / ISSUE report or a QUESTION?

Bug

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

borg 1.1.9 (Debian Buster package) for Debian client/server. I've also tested with borg 1.1.10 binary from the site.

#### Operating system (distribution) and version.

Backup severs Debian Buster (Proxmox 6.0)
Backup clients: Debian Buster (Proxmox 6.0), macOS X 10.13 and 10.14.

#### Hardware / network configuration, and filesystems used.

Backup server is a Proxmox/Debian machine, actual backups are made to a Buster-based Proxmox container. No overclocking, ECC RAM. Storage disk is a USB3 volume, using dmcrypt for block-level encryption. Encrypted device is managed as Linux ZFS.

Server memtest86 passed (4 full passes), ZFS scrub of backup disk (and all disks for that matter) report no issues.

Backup clients:

- The onsite Proxmox server itself (the actual hypervisor, not a container).
- Various Macs (2 laptops, 1 iMac).

All connectivity is via SSH, even when the hypervisor is backing up to a container it hosts itself. 

#### How much data is handled by borg?

Roughly 3TB total across all clients stored at each location.

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

Several affected commands:

check --repair --verbose ::
check --repair --verbose --verify-data  ::

Current failing command:

check --repair --verbose ::

## Describe the problem you're observing.

I appear to have had widespread backup corruption on this server. 3 Mac clients and the hypervisor backup itself were reporting errors during "borg check", even though memtest86 and ZFS scrub were clean. All 4 clients were reporting no check errors for the same backups to an offsite server (separate Borg destination, not a rsync or clone of the onsite backup).

Despite the lack of memory or disk errors, I have replaced most of the hardware on this machine. After one pass of check --repair, no errors are now reported on the Macs.

However the Debian/Proxmox backup cannot complete its check repair. Issues DB indicates that its expected that "Data integrity error" will block some operations, but check --repair is called out as the fix for these.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

I've been repeatedly running check --repair for several days, and other repairs have been made. I believe any hardware issues are resolved as all 3 other clients are clean, the prior instability in detecting errors (errors would be reported in different places) is gone.

At this point I appear to be stuck as several runs have all halted with the same error.

I will try delete --force to remove the affected archive next, but I believe based on documentation that check --repair should not fail due to data integrity.

#### Include any warning/errors/backtraces from the system logs

Traceback encountered with borg 1.1.10:

```
Full-1568690224: <filename redacted>: New missing file chunk detected (Byte 2226083-3571440). Replacing with all-zero chunk.
Data integrity error: Chunk 5d767a5c23c6c6ca22478db09312b3181ec2f0b25e610a3f86b1b5ba325e4caf: id verification failed
Traceback (most recent call last):
  File "borg/archiver.py", line 4501, in main
  File "borg/archiver.py", line 4433, in run
  File "borg/archiver.py", line 166, in wrapper
  File "borg/archiver.py", line 334, in do_check
  File "borg/archive.py", line 1225, in check
  File "borg/archive.py", line 1612, in rebuild_refcounts
  File "borg/archive.py", line 1555, in robust_iterator
  File "borg/crypto/key.py", line 396, in decrypt
  File "borg/crypto/key.py", line 197, in assert_id
borg.helpers.IntegrityError: Data integrity error: Chunk 5d767a5c23c6c6ca22478db09312b3181ec2f0b25e610a3f86b1b5ba325e4caf: id verification failed
```



Discussion:
What it does in `assert_id` is to compare the **wanted* id (which is the address of the chunk in the repo, and should be the content hash/MAC of the plaintext data in that chunk) with the **computed** id (the content hash/MAC of the actual decrypted/decompressed plaintext data).

That should of course match. If it does not, something is wrong.

Before failing there, it already has checked the stored MAC against the computed MAC of the stored chunk ciphertext (in `decrypt`) and as that did not raise an IntegrityError in your case, these both seem to be consistent. Then it has decrypted the ciphertext and also decompressed it, resulting in the plaintext. Then it called assert_id and failed.

So, one possible scenario leading to that behaviour is that we got valid chunk content of **some** chunk, but (according to the failing assert_id) **not the chunk we wanted**.

Potential causes: corrupt repo index, data corruption in CPU, on bus or in memory, ... ?

When you used `borg check --repair`, did you do a full repair or did you somehow limit it using options like `--archives-only` or `--repository-only`? I'ld recommend a full repair, so the repo index gets rebuilt.

Original code (borg 1.1.10 line 197 in `borg/crypto/key.py`):
```
raise IntegrityError('Chunk %s: id verification failed' % bin_to_hex(id))
```

If you can reproduce the issue, can you do that after changing this to:

```
raise IntegrityError('Chunk %s: id verification failed: %s' % (bin_to_hex(id), bin_to_hex(id_computed)))
```

It would be interesting in case the 2 values shown only differ in a few bits.
The archives were created with the Debian package for 1.1.9. However, I have been trying to repair with 1.1.10 from the site just in case it was a problem with the Debian packaging. Both client and server we running 1.1.10 in the traceback I sent, though I can switch to 1.1.9 if you like.

Early in the investigation of the corruption, I did run check with --archives-only and --repository-only, but I do not believe I ran --repair with those options. The current command is just 'check --repair --verbose ::'

I did, however, previously use 'borg delete --force' to remove archives that were throwing "borg.helpers.DecompressionError: Decompression error: Corrupt input data" during prior checks. If you need those tracebacks I have some samples, but I'm willing to believe those were hardware errors.

Since I don't know how to patch your change into the 1.1.10 binary, I made the borg/crypto/key.py patch in the 1.1.9 version installed (from the Debian package). Rather than run the whole verify, I used 'borg delete  --force Full-1568690224' which I know trips the same error, here's the result:

Data integrity error: Chunk 5d767a5c23c6c6ca22478db09312b3181ec2f0b25e610a3f86b1b5ba325e4caf: id verification failed: a89f1d88ebfc3104734414e50be3470978243d545f818da492428fcceb3ff283

While I cannot completely discount hardware (again, it's the only plausible path for the errors I see), at this point most hardware has been changed and memtest86 and ZFS are not reporting errors. I'm also suspicious that the problem is repeatable at the same chunk 3 times in a row, with different commands. Prior errors that I concluded were hardware were more random (different errors every pass).

That leaves the repo index, is rebuilding that separate from check --repair? 

`borg check --repair` rebuilds the repo index in its first phase, when it is checking the repo (before 2nd phase, when it is checking archives).
OK, that is the phase that is failing with the exception above. So it appears there's a catch here that there are conditions where the repo index is so corrupt it can't be rebuilt?
I have now brought a second drive out from offline cold storage, and it has similar corruption, again only for this machine. Unlike the current hot disk, other clients on the same backup disk pass check --verbose --verify-data without problems. It is only the server backup to itself that is affected.

The server itself fails with a similar chunk failure on this disk, running 'check --verbose --repair',  the repository check reports:

`Remote: Completed repository check, no problems found.`

However during the same run archive analysis fails:

```
Analyzing archive Full-1567941424 (246/285)
Analyzing archive Full-1567948623 (247/285)
Analyzing archive Full-1567955863 (248/285)
Analyzing archive Full-1567963023 (249/285)
Data integrity error: Chunk 1ba0725b28954a237ff04eb22491781c4400b610277c07c473b1f2202e9b7952: id verification failed: b1e5d25b7ac6e697cf1ac73b752e5bee21c590d17907b21b2bf7a62a7011f297
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4455, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4387, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 154, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 322, in do_check
    verify_data=args.verify_data, save_space=args.save_space):
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1196, in check
    self.rebuild_refcounts(archive=archive, first=first, last=last, sort_by=sort_by, glob=glob)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1583, in rebuild_refcounts
    for item in robust_iterator(archive):
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1526, in robust_iterator
    data = self.key.decrypt(chunk_id, cdata)
  File "/usr/lib/python3/dist-packages/borg/crypto/key.py", line 398, in decrypt
    self.assert_id(id, data)
  File "/usr/lib/python3/dist-packages/borg/crypto/key.py", line 198, in assert_id
    raise IntegrityError('Chunk %s: id verification failed: %s' % (bin_to_hex(id), bin_to_hex(id_computed)))
borg.helpers.IntegrityError: Data integrity error: Chunk 1ba0725b28954a237ff04eb22491781c4400b610277c07c473b1f2202e9b7952: id verification failed: b1e5d25b7ac6e697cf1ac73b752e5bee21c590d17907b21b2bf7a62a7011f297
```

This is again 1.1.9 from Debian with the patch requested above inserted.

What is potentially interesting about this example is that the failure happens during archive verification, after the point at which I believe the repo index is  already supposed to have been rebuilt?

What usually happens when borg is storing `data` into the repo:

```
id = H(data)
repo[id] = data
```

I suspect that on your machine with the hardware issues, the following happened:

```
id = H(data)
# here it corrupts data (or the id, or both id and data) due to a hw defect
repo[id] = data  # also: compresses, encrypts, authenticates data, adds crc32
```

In this corruption scenario, the bad data gets a matching crc32 and also the authentication matches.
It is just not the data that should be stored under that id, and `H(data) <> id`.

And the only thing we know is that something is wrong, we do not know whether the data is bad or the id is bad.

What you could try (please make a backup of the repo first):
- get the problematic object `id` from the repo, store it into a file `badobj` (maybe runs into same issue)
- delete object `id` from the repo (this will cause data loss of that data, of course, if you could not get it in the previous step)
- in case you succeeded getting `badobj`, put it into the repo again (this will recompute the id)
- run `borg check --repair` again (guess it will replace the missing chunk by an all-zero chunk if it is a content chunk, could also damage archive(s) if it is a metadata chunk)

`borg debug get-obj / delete-obj / put-obj` are the respective commands.
I believe the problem was with the original computed hash for the chunk. Going through a get-obj / delete-obj / put-obj cycle for the bad chunk resulted in this change in ID:

1ba0725b28954a237ff04eb22491781c4400b610277c07c473b1f2202e9b7952 -> 44e475957b7fe5d039af6692f6ed1f5331b4320f05595456e31f3f87cc0ad3b6

I also tried the other chunk ID (the id_computed in the hash), expecting it to be computed the same, with this result:

b1e5d25b7ac6e697cf1ac73b752e5bee21c590d17907b21b2bf7a62a7011f297 -> f187a68716c7a8eeaef63a8e6d711925f8b88ee68f9055ce67d1e85e6957b1f5

I'm not sure if that's significant. 

Running check --verbose then reported errors of the form:

```
Analyzing archive Full-1568189823 (284/285)
item metadata chunk missing [chunk: 000439_b1e5d25b7ac6e697cf1ac73b752e5bee21c590d17907b21b2bf7a62a7011f297]
```
(Many archives were impacted).

Running check --repair --verbose appears to have fixed it, but I assume missing metadata means those archives are damaged still? I'm inclined to delete them if that's the case (they are all recent).







Yes, if an archive metadata (stream) chunk is missing, it means that quite some files might be missing in that archive now. The problem is that the badly computed id is referenced from all these archives.
Sorry for the extended silence, I had to put this on hold for a bit, and once I resumed it takes several days to verify each archive. 

The basic workflow was:
- 'borg check --verbose' to find a bad chunk, which stops the check due to exception above. Make a note of the affected archive.
- 'borg debug delete-obj'  to remove the bad chunk.
- Repeat prior steps till 'borg check' completes without exceptions (but reporting lots of missing chunks).
- 'borg check --repair' to fix repo for the missing chunks, so other steps will work.
- 'borg delete' to remove the affected archives previously identified as referencing the bad chunks
- 'borg check --verify-data' for paranoia.

This seems to have returned me to a stable state (I hope). Problematic chunks were all located in very recent archives, which I presume correspond roughly with the hardware failure.

From the overall perspective, however, I believe there is still an issue with how 'borg check' handles mismatched chunks. In particular the unhandled exception seems like its something that borg check should report, but continue. Even if manual intervention is necessary to repair, having 'borg check' not bail out would have reduced the number of (slow) cycles in steps 1 and 2 to identify the bad chunks.
Hmm, was this maybe related to the repo index corruption issue we fixed in 1.1.11 (see close to the top of the changelog for an advisory about that).
but: `borg check [--repair]` should not crash, check if we can do anything about that.

I have also run into this issue, with borg 1.1.17. `borg check --repository-only` showed no issues.

~~~
Data integrity error: Chunk 00ed881c60b95ff2ab13c7f85707b2ee855470d14dca99e43da4f80e470cc8ad: id verification failed
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/borg/archiver.py", line 4703, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.9/site-packages/borg/archiver.py", line 4635, in run
    return set_ec(func(args))
  File "/usr/lib/python3.9/site-packages/borg/archiver.py", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.9/site-packages/borg/archiver.py", line 192, in wrapper
    return method(self, args, repository=repository, manifest=manifest, key=key, archive=archive, **kwargs)
  File "/usr/lib/python3.9/site-packages/borg/archiver.py", line 815, in do_extract
    archive.extract_item(item, stdout=stdout, sparse=sparse, hardlink_masters=hardlink_masters,
  File "/usr/lib/python3.9/site-packages/borg/archive.py", line 658, in extract_item
    for data in self.pipeline.fetch_many(ids, is_preloaded=True):
  File "/usr/lib/python3.9/site-packages/borg/archive.py", line 252, in fetch_many
    yield self.key.decrypt(id_, data)
  File "/usr/lib/python3.9/site-packages/borg/crypto/key.py", line 396, in decrypt
    self.assert_id(id, data)
  File "/usr/lib/python3.9/site-packages/borg/crypto/key.py", line 197, in assert_id
    raise IntegrityError('Chunk %s: id verification failed' % bin_to_hex(id))
borg.helpers.IntegrityError: Data integrity error: Chunk 00ed881c60b95ff2ab13c7f85707b2ee855470d14dca99e43da4f80e470cc8ad: id verification failed
~~~

It took me a while to get this error, because using `borg mount` and accessing the broken file kills the whole mount (`transport is not connected`) instead of just returning read errors for unreadable files. Maybe that would be nice to have.
@Artefact2 check if you have hardware issues, esp. RAM - e.g. run memtest86+ for some rounds.

The error you got should never happen if everything works correctly. borg uses the hash / MAC of data as address to store the data (using a key-value store, the key is hash(data), the value is the data). When wanting to read some data, borg already knows the key (== the hash), then reads the data and then verifies if the data it got really matches that hash (computed_hash == wanted_hash?).

The easiest explanation for it not working is that some bits flipped somewhere, e.g. due to bad RAM.
Could be also bad CPU, overheating, mainboard issue, disk, controller or cabling issue.

The reason why `borg check --repository-only` worked for you could be:
- the hw issue not always happening / borg not using the defective RAM area
- the error is not found by the simpler crc32 check that part of the check run uses

You can also try out some stuff I already mentioned in older comments above.
Hi, sorry if this is considered spam, but I got the same error as OP and wanted to have this logged somewhere. In my case it seems the issue was caused by a segfault in borg (for which I don't know the reason), see log at the end.

While investigating the situation I think I got an error related to the cache (`/root/.cache/borg/<hash>`), so I ended up removing it. The next `borg check` worked fine.

<details><summary>Log from my borg backup script</summary>
<p>

```
Started Borg Backup Service default.
Running preHook
Running 'borg create [...]'
Creating archive at "backup@srv1.local:mini.borg::mini-20220123T011506.IN_PROGRESS"
[...]
M /home/bf/.mozilla/firefox/913u97a5.default-1629308750536/storage/default/moz-extension+++c776d706-879d-4dfd-a158-ff0abd2ea991^userContextId=4294967295/idb/3647222921wleabcEoxlt-eengsairo.sqlite
M /home/bf/.mozilla/firefox/913u97a5.default-1629308750536/storage/default/moz-extension+++d1d3466a-f9d6-4c22-8fa2-2174e7a432fe^userContextId=4294967295/idb/3647222921wleabcEoxlt-eengsairo.sqlite
M /home/bf/.mozilla/firefox/913u97a5.default-1629308750536/storage/permanent/chrome/idb/3870112724rsegmnoittet-es.sqlite
Fatal Python error: Segmentation fault
Current thread 0x00007f42d955b340 (most recent call first):
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archive.py", line 1098 in process_file
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 649 in _process
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 674 in _process
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 674 in _process
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 674 in _process
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 674 in _process
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 674 in _process
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 674 in _process
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 674 in _process
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 556 in create_inner
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 598 in do_create
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 177 in wrapper
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 4635 in run
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 4703 in main
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/bin/.borg-wrapped", line 9 in <module>
/nix/store/xwl2z4mpyznk1qp9yxrn5n9y8bkfymx7-borg-backup: line 70: 838861 Segmentation fault      (core dumped) borg create --stats --verbose --list --filter ME --show-rc --one-file-system $autofs_excludes --exclude-caches --exclude /tmp/ --exclude /var/tmp/ --exclude /var/swapfile --exclude '/home/*/.cache/' --exclude '/home/*/.thumbnails/' --exclude '/home/*/.nox/' --exclude '*/.Trash*/' --exclude '*/$RECYCLE.BIN' --exclude '*/System Volume Information' --exclude /mnt/data/.glusterfs --compression lz4 "$repository::$archive_in_progress" / /home /mnt/data
Renaming archive: mini-20220123T011506.IN_PROGRESS -> mini-20220123T011506.UNSUCCESSFUL
Failed to create/acquire the lock /root/.cache/borg/3286f10ee37e8605a20aea82a262c241acc1aab345fff0e3cbfb17cdb931e5e7/lock.exclusive (timeout).
Running 'borg prune [...]'
Failed to create/acquire the lock /root/.cache/borg/3286f10ee37e8605a20aea82a262c241acc1aab345fff0e3cbfb17cdb931e5e7/lock.exclusive (timeout).
terminating with error status, rc 2
Running 'borg check [...]' (since a check is due and today is Sunday)
Remote: Starting repository check
Remote: Starting repository index check
Remote: Index object count match.
Remote: Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive mini-20210626T011506 (1/30)
Analyzing archive mini-20210731T011505 (2/30)
Data integrity error: Chunk fde14d0d8a481639c96dc6aee99364768b24fc4d93bda793d0811f4e2592d1d0: id verification failed
Traceback (most recent call last):
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 4703, in main
    exit_code = archiver.run(args)
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 4635, in run
    return set_ec(func(args))
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archiver.py", line 348, in do_check
    if not args.repo_only and not ArchiveChecker().check(
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archive.py", line 1249, in check
    self.rebuild_refcounts(archive=archive, first=first, last=last, sort_by=sort_by, glob=glob)
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archive.py", line 1639, in rebuild_refcounts
    for item in robust_iterator(archive):
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/archive.py", line 1579, in robust_iterator
    data = self.key.decrypt(chunk_id, cdata)
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/crypto/key.py", line 299, in decrypt
    self.assert_id(id, data)
  File "/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/lib/python3.9/site-packages/borg/crypto/key.py", line 197, in assert_id
    raise IntegrityError('Chunk %s: id verification failed' % bin_to_hex(id))
borg.helpers.IntegrityError: Data integrity error: Chunk fde14d0d8a481639c96dc6aee99364768b24fc4d93bda793d0811f4e2592d1d0: id verification failed
Platform: Linux mini 5.16.0 #1-NixOS SMP PREEMPT Sun Jan 9 22:55:34 UTC 2022 x86_64
Linux: Unknown Linux
Borg: 1.1.17  Python: CPython 3.9.6 msgpack: 0.5.6.+borg1
PID: 866181  CWD: /
sys.argv: ['/nix/store/q44lpmaggpifpsbsph2a3k625cczfgbg-borgbackup-1.1.17/bin/borg', 'check', '--verbose', '--show-rc', 'backup@srv1.local:mini.borg']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
borg create, prune and/or check operation failed. Exiting with error.
Running postHook
Done, exiting with status 1.
borg-backup-default.service: Main process exited, code=exited, status=1/FAILURE
borg-backup-default.service: Failed with result 'exit-code'.
borg-backup-default.service: Consumed 20min 16.695s CPU time, received 689.6M IP traffic, sent 40.2M IP traffic.
```

</p>
</details>

@bjornfor I had a look at the source:

archiver:1098 - it's calling `item.get_size(memorize=True)` there. Nothing special, except that that is cython code.

So, no idea about what has caused the segfault - maybe a hw issue?

But the main question is what's causing the IntegrityError.

Do you remember more details about  the error related to the cache (/root/.cache/borg/<hash>)? Error message?
Or did you mean the leftover **lock** (which is somehow expected for severe errors like segfaults).
Yes, it was probably the lock file. I was running some commands interactively and didn't keep a record, sorry.
I also have a data integrity error. Can I somehow see which one of my file(s) are affected via the chunk id? Or maybe which archive is affected.

```
Starting cryptographic data integrity verification...
chunk d671b1174965693d86df425e552c7aa344eb3c76f22673c19cb093da1f717b61, integrity error: Data integrity error: Chunk d671b1174965693d86df425e552c7aa344eb3c76f22673c19cb093da1f717b61: id verification failed
Found defect chunks. With --repair, they would get deleted, so affected files could get repaired then and maybe healed later.
```
Well, you don't need to find/fix the file yourself, you can just do what it says on the 3rd line.
check --repair, create (same backup as usual), another check --repair.

Update: ... and before running repair, make sure your hardware works OK (RAM, disk), see FAQ about IntegrityError.
I ran check with --verify-data --repair -v -p and it showed me all archives and which file was damaged in the end. The chunk was then replaced with an all-zero chunk. 
Which means in the next backup the file should be detected as changed and I will have a good file again starting with my new backup? The file in the old Backups will still be "broken", correct?
If a missing chunk (that was replaced by an all-zero replacement chunk) is re-created (because the input files still create that chunk), the next `borg check --repair` after that will heal all files in all archives that miss this chunk.
Had a deeper look at the code:

From `borg check ----> robust_iterator`:
```
                for chunk_id, cdata in zip(items, repository.get_many(items)):
                    data = self.key.decrypt(chunk_id, cdata)      # <----- could throw IntegrityError
                    unpacker.feed(data)
                    try:
                        for item in unpacker:
                            valid, reason = valid_item(item)
                            if valid:
                                yield Item(internal_dict=item)
                            else:
                                report('Did not get expected metadata dict when unpacking item metadata (%s)' % reason, chunk_id, i)
                    except RobustUnpacker.UnpackerCrashed:
                        report('Unpacker crashed while unpacking item metadata, trying to resync...', chunk_id, i)
                        unpacker.resync()
                    except Exception:
                        report('Exception while unpacking item metadata', chunk_id, i)
                        raise
                    i += 1
```

As seen above, the `decrypt` can throw an `borg.helpers.IntegrityError` if the `cdata` is toast or does not match the data we wanted (== `chunk_id`).

As suggested by @axxelh, we could catch that exception, report an error and continue with the check. Likely the unpacker will be out of sync then when we feed the next chunk (due to the skipped corrupt metadata chunk), but the code should be able to deal with it and resync quickly (see `valid_item` and exception handlers).
Bounty: https://app.bountysource.com/issues/81463140-borg-check-repair-data-integrity-error-id-verification-failed

Try to write a (failing) test that triggers this problem:
- you will need to corrupt the item metadata stream of an archive for this.
- running borg check shall detect the issue, but not crash with an unhandled `IntegrityError`
- running `borg check --repair` shall repair as good as it gets. It will lose some data due to the corrupt metadata chunk, but not crash with an unhandled `IntegrityError`.

Fix the issue (see previous comment) to make the test pass.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1937,20 +1937,25 @@ def valid_item(obj):
                 if state > 0:
                     unpacker.resync()
                 for chunk_id, cdata in zip(items, repository.get_many(items)):
-                    data = self.key.decrypt(chunk_id, cdata)
-                    unpacker.feed(data)
                     try:
+                        data = self.key.decrypt(chunk_id, cdata)
+                        unpacker.feed(data)
                         for item in unpacker:
                             valid, reason = valid_item(item)
                             if valid:
                                 yield Item(internal_dict=item)
                             else:
                                 report('Did not get expected metadata dict when unpacking item metadata (%s)' % reason, chunk_id, i)
+                    except IntegrityError as integrity_error:
+                        # key.decrypt() detected integrity issues.
+                        # maybe the repo gave us a valid cdata, but not for the chunk_id we wanted.
+                        # or the authentication of cdata failed, meaning the encrypted data was corrupted.
+                        report(str(integrity_error), chunk_id, i)
                     except msgpack.UnpackException:
                         report('Unpacker crashed while unpacking item metadata, trying to resync...', chunk_id, i)
                         unpacker.resync()
                     except Exception:
-                        report('Exception while unpacking item metadata', chunk_id, i)
+                        report('Exception while decrypting or unpacking item metadata', chunk_id, i)
                         raise
                     i += 1
 
@@ -1984,13 +1989,19 @@ def valid_item(obj):
                 logger.info(f'Analyzing archive {info.name} ({i + 1}/{num_archives})')
                 archive_id = info.id
                 if archive_id not in self.chunks:
-                    logger.error('Archive metadata block is missing!')
+                    logger.error('Archive metadata block %s is missing!', bin_to_hex(archive_id))
                     self.error_found = True
                     del self.manifest.archives[info.name]
                     continue
                 mark_as_possibly_superseded(archive_id)
                 cdata = self.repository.get(archive_id)
-                data = self.key.decrypt(archive_id, cdata)
+                try:
+                    data = self.key.decrypt(archive_id, cdata)
+                except IntegrityError as integrity_error:
+                    logger.error('Archive metadata block %s is corrupted: %s', bin_to_hex(archive_id), integrity_error)
+                    self.error_found = True
+                    del self.manifest.archives[info.name]
+                    continue
                 archive = ArchiveItem(internal_dict=msgpack.unpackb(data))
                 if archive.version != 1:
                     raise Exception('Unknown archive metadata version')


Test patch: