Repository: borgbackup/borg

Issue:
no automatic rebuild of a deleted chunks file
borg does not tolerate a deleted chunks file (deleted because it was corrupted) due to the integrity checks:
```
u0_a151@localhost ~> ./borgbackup.sh

Fri Jun  5 21:11:53 EDT 2020 Starting backup

Killed stale lock localhost@75173968615802.3665-0.
Removed stale exclusive roster lock for host localhost@75173968615802 pid 3665 thread 0.
Removed stale exclusive roster lock for host localhost@75173968615802 pid 9194 thread 0.
Removed stale exclusive roster lock for host localhost@75173968615802 pid 3665 thread 0.
Removed stale exclusive roster lock for host localhost@75173968615802 pid 9194 thread 0.
Creating archive at "aelius@192.168.1.143:/media/borg-monolith::Monolith-2020-06-05T21:11:55"
Local Exception
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/archiver.py", line 4529, in main
    exit_code = archiver.run(args)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/archiver.py", line 4461, in run
    return set_ec(func(args))
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/archiver.py", line 166, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/archiver.py", line 565, in do_create
    with Cache(repository, key, manifest, do_files=args.cache_files, progress=args.progress,
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/cache.py", line 380, in __new__
    return local()
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/cache.py", line 372, in local
    return LocalCache(repository=repository, key=key, manifest=manifest, path=path, sync=sync,
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/cache.py", line 457, in __init__
    self.open()
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/cache.py", line 505, in open
    self.rollback()
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/cache.py", line 610, in rollback
    self._do_open()
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/cache.py", line 493, in _do_open
    with IntegrityCheckedFile(path=os.path.join(self.path, 'chunks'), write=False,
  File "/data/data/com.termux/files/usr/lib/python3.8/site-packages/borgbackup-1.1.11-py3.8-linux-x86_64.egg/borg/crypto/file_integrity.py", line 129, in __init__
    self.file_fd = override_fd or open(path, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/data/data/com.termux/files/home/.cache/borg/6d24295b115c5b4eb74bace9521ab7b09de7f82f096b8b39f8fcd1411fe4b34e/chunks'

Platform: Linux localhost 4.9.197-arter97-r12-g29a968fb09d85 #1 SMP PREEMPT Mon Oct 28 19:22:17 KST 2019 aarch64
Linux: Unknown Linux
Borg: 1.1.11  Python: CPython 3.8.3 msgpack: 0.5.6
PID: 11175  CWD: /data/data/com.termux/files/home
sys.argv: ['/data/data/com.termux/files/usr/bin/borg', 'create', '--verbose', '--filter', 'AME', '--list', '--stats', '--show-rc', '--compression', 'lz4', '--exclude-caches', '--pattern=+/storage/emulated/0/Android/data/org.tasks', '--pattern=+/storage/emulated/0/Android/data/net.sourceforge.opencamera', '--pattern=-/storage/emulated/0/Android', '--pattern=-/storage/emulated/0/DCIM', '--exclude', '*/.sync', '::Monolith-{now}', 'borgbackup.sh', '/storage/emulated/0/']
SSH_ORIGINAL_COMMAND: None

terminating with error status, rc 2
```

Discussion:
codecov-commenter: :warning: Please install the !['codecov app svg image'](https://github.com/codecov/engineering-team/assets/152432831/e90313f4-9d3a-4b63-8b54-cfe14e7ec20d) to ensure uploads and comments are reliably processed by Codecov.

## [Codecov](https://app.codecov.io/gh/borgbackup/borg/pull/8257?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=borgbackup) Report
Attention: Patch coverage is `88.88889%` with `2 lines` in your changes missing coverage. Please review.
> Project coverage is 83.19%. Comparing base [(`00962f9`)](https://app.codecov.io/gh/borgbackup/borg/commit/00962f9d5da83533f0762568f214668be5bae1f6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=borgbackup) to head [(`0992fad`)](https://app.codecov.io/gh/borgbackup/borg/commit/0992fad802d83718ac359d6a3d08a13c3a351598?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=borgbackup).
> Report is 6 commits behind head on master.

> :exclamation: **Current head 0992fad differs from pull request most recent head d4d658d**
> 
> Please [upload](https://docs.codecov.com/docs/codecov-uploader) reports for the commit d4d658d to get more accurate results.

| [Files](https://app.codecov.io/gh/borgbackup/borg/pull/8257?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=borgbackup) | Patch % | Lines |
|---|---|---|
| [src/borg/cache.py](https://app.codecov.io/gh/borgbackup/borg/pull/8257?src=pr&el=tree&filepath=src%2Fborg%2Fcache.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=borgbackup#diff-c3JjL2JvcmcvY2FjaGUucHk=) | 88.88% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/borgbackup/borg/pull/8257?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=borgbackup) |

:exclamation: Your organization needs to install the [Codecov GitHub app](https://github.com/apps/codecov/installations/select_target) to enable full functionality.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #8257      +/-   ##
==========================================
- Coverage   83.75%   83.19%   -0.56%     
==========================================
  Files          67       67              
  Lines       12060    12045      -15     
  Branches     2189     1903     -286     
==========================================
- Hits        10101    10021      -80     
- Misses       1367     1416      +49     
- Partials      592      608      +16     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/borgbackup/borg/pull/8257?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=borgbackup).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=borgbackup).


Patch:
diff --git a/src/borg/cache.py b/src/borg/cache.py
index b6cd5c0a..c36572c0 100644
--- a/src/borg/cache.py
+++ b/src/borg/cache.py
@@ -531,7 +531,12 @@ def __init__(
             self.security_manager.assert_access_unknown(warn_if_unencrypted, manifest, self.key)
             self.create()
 
-        self.open()
+        try:
+            self.open()
+        except (FileNotFoundError, FileIntegrityError):
+            self.wipe_cache()
+            self.open()
+
         try:
             self.security_manager.assert_secure(manifest, self.key, cache_config=self.cache_config)
 
@@ -924,19 +929,31 @@ def check_cache_compatibility(self):
         return True
 
     def wipe_cache(self):
-        logger.warning("Discarding incompatible cache and forcing a cache rebuild")
+        logger.warning("Discarding incompatible or corrupted cache and forcing a cache rebuild")
         archive_path = os.path.join(self.path, "chunks.archive.d")
         if os.path.isdir(archive_path):
             shutil.rmtree(os.path.join(self.path, "chunks.archive.d"))
             os.makedirs(os.path.join(self.path, "chunks.archive.d"))
         self.chunks = ChunkIndex()
-        with SaveFile(os.path.join(self.path, files_cache_name()), binary=True):
+        with IntegrityCheckedFile(path=os.path.join(self.path, "chunks"), write=True) as fd:
+            self.chunks.write(fd)
+        self.cache_config.integrity["chunks"] = fd.integrity_data
+        with IntegrityCheckedFile(path=os.path.join(self.path, files_cache_name()), write=True) as fd:
             pass  # empty file
+        self.cache_config.integrity[files_cache_name()] = fd.integrity_data
         self.cache_config.manifest_id = ""
         self.cache_config._config.set("cache", "manifest", "")
+        if not self.cache_config._config.has_section("integrity"):
+            self.cache_config._config.add_section("integrity")
+        for file, integrity_data in self.cache_config.integrity.items():
+            self.cache_config._config.set("integrity", file, integrity_data)
+        # This is needed to pass the integrity check later on inside CacheConfig.load()
+        self.cache_config._config.set("integrity", "manifest", "")
 
         self.cache_config.ignored_features = set()
         self.cache_config.mandatory_features = set()
+        with SaveFile(self.cache_config.config_path) as fd:
+            self.cache_config._config.write(fd)
 
     def update_compatibility(self):
         operation_to_features_map = self.manifest.get_all_mandatory_features()


Test patch:
diff --git a/src/borg/testsuite/archiver/corruption.py b/src/borg/testsuite/archiver/corruption.py
index a604a478..cb8b5541 100644
--- a/src/borg/testsuite/archiver/corruption.py
+++ b/src/borg/testsuite/archiver/corruption.py
@@ -2,13 +2,15 @@
 import json
 import os
 from configparser import ConfigParser
+from unittest.mock import patch
 
 import pytest
 
 from ...constants import *  # NOQA
-from ...crypto.file_integrity import FileIntegrityError
 from ...helpers import bin_to_hex, Error
 from . import cmd, create_src_archive, create_test_files, RK_ENCRYPTION
+from ...hashindex import ChunkIndex
+from ...cache import LocalCache
 
 
 def test_check_corrupted_repository(archiver):
@@ -43,15 +45,31 @@ def corrupt(file, amount=1):
         fd.write(corrupted)
 
 
+@pytest.mark.allow_cache_wipe
 def test_cache_chunks(archiver):
     corrupt_archiver(archiver)
-    corrupt(os.path.join(archiver.cache_path, "chunks"))
-    if archiver.FORK_DEFAULT:
-        out = cmd(archiver, "rinfo", exit_code=2)
-        assert "failed integrity check" in out
-    else:
-        with pytest.raises(FileIntegrityError):
-            cmd(archiver, "rinfo")
+    create_src_archive(archiver, "test")
+    chunks_path = os.path.join(archiver.cache_path, "chunks")
+    chunks_before_corruption = set(ChunkIndex(path=chunks_path).iteritems())
+    corrupt(chunks_path)
+
+    assert not archiver.FORK_DEFAULT  # test does not support forking
+
+    chunks_in_memory = None
+    sync_chunks = LocalCache.sync
+
+    def sync_wrapper(cache):
+        nonlocal chunks_in_memory
+        sync_chunks(cache)
+        chunks_in_memory = set(cache.chunks.iteritems())
+
+    with patch.object(LocalCache, "sync", sync_wrapper):
+        out = cmd(archiver, "rinfo")
+
+    assert chunks_in_memory == chunks_before_corruption
+    assert "forcing a cache rebuild" in out
+    chunks_after_repair = set(ChunkIndex(path=chunks_path).iteritems())
+    assert chunks_after_repair == chunks_before_corruption
 
 
 def test_cache_files(archiver):