Repository: borgbackup/borg

Issue:
Aborting check --repair in recover_segment() may leave segment partially recovered
I have not tested this, so may be missing something, but looking at the code:

[`recover_segment`](https://github.com/borgbackup/borg/blob/85e6c0afd82f52182c56bfcc0b795aef33adc037/src/borg/repository.py#L1485) does rename a segment adding the [`.beforerecover`](https://github.com/borgbackup/borg/blob/85e6c0afd82f52182c56bfcc0b795aef33adc037/src/borg/repository.py#L1489) extension and then creates a new file with the original name, [**slowly**](https://github.com/borgbackup/borg/issues/4243#issuecomment-451702272) iterates over the renamed original segment, copying any parts with correct CRC over.

This code is the only occurrence of the string `beforerecover` within the source, so I conclude, that such a renamed original segment is never considered for repair again.

Thus, when a long running `borg check --repair` is interrupted for some reason and that interruption happens within above process, the original (renamed to `.beforerecover`) segment as well as the new segment, which however only has parts of the original data recovered yet, are left behind and a new run of `borg check --repair` would check than new segment only and skip on to the next, if its fine in itself (or if not (or is the new partial one always self-consistent?), maybe even overwrite the original by again renaming the partial new one to `.beforerecover`?).

Either way, with no new errors showing up, the user will assume the `borg check --repair` to have resumed just fine, when it eventually skipped over what worst case could be a large part of a segment containing data that could have been recovered.

[This comment](https://github.com/borgbackup/borg/issues/4243#issuecomment-451702455) seems to have observed that happening in practice, but seems to have went unnoticed within that question.


Discussion:
Possible steps to improve:

1. Rename the `.beforerecover` to something like `.afterecover` or `.recovered` (adding numbered extensions when that already exists to avoid ever overwriting a original segment) within `recover_segment` as soon as it is iterated over.
2. In the check code, do look for left behind `.beforerecover` files, warn/error about their existence in all cases.
3. Restart `recover_segment` on a left behind `.beforerecover` file in case of check repair.

To not over-complicate things, 3 could be skipped (with an according warning) if nonsense/unsafe combinations of files exist, like e.g. both a `.beforerecover` and `.recovered`.
Uh, good catch. I didn't check the source yet, but guess you're right.

Another idea would be to re-use the existing pattern/code, where we have a context manager that helps with reading the original file, writing to a temp file and only if successful, atomically renaming the temp file over the original file.

Or copy instead of rename, in case we want to keep the original.


Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -1486,23 +1486,21 @@ def recover_segment(self, segment, filename):
         logger.info('attempting to recover ' + filename)
         if segment in self.fds:
             del self.fds[segment]
-        backup_filename = filename + '.beforerecover'
-        os.rename(filename, backup_filename)
-        if os.path.getsize(backup_filename) < MAGIC_LEN + self.header_fmt.size:
+        if os.path.getsize(filename) < MAGIC_LEN + self.header_fmt.size:
             # this is either a zero-byte file (which would crash mmap() below) or otherwise
             # just too small to be a valid non-empty segment file, so do a shortcut here:
-            with open(filename, 'wb') as fd:
+            with SaveFile(filename, binary=True) as fd:
                 fd.write(MAGIC)
             return
-        with open(backup_filename, 'rb') as backup_fd:
-            # note: file must not be 0 size or mmap() will crash.
-            with mmap.mmap(backup_fd.fileno(), 0, access=mmap.ACCESS_READ) as mm:
-                # memoryview context manager is problematic, see https://bugs.python.org/issue35686
-                data = memoryview(mm)
-                d = data
-                try:
-                    with open(filename, 'wb') as fd:
-                        fd.write(MAGIC)
+        with SaveFile(filename, binary=True) as dst_fd:
+            with open(filename, 'rb') as src_fd:
+                # note: file must not be 0 size or mmap() will crash.
+                with mmap.mmap(src_fd.fileno(), 0, access=mmap.ACCESS_READ) as mm:
+                    # memoryview context manager is problematic, see https://bugs.python.org/issue35686
+                    data = memoryview(mm)
+                    d = data
+                    try:
+                        dst_fd.write(MAGIC)
                         while len(d) >= self.header_fmt.size:
                             crc, size, tag = self.header_fmt.unpack(d[:self.header_fmt.size])
                             if size < self.header_fmt.size or size > len(d):
@@ -1511,11 +1509,11 @@ def recover_segment(self, segment, filename):
                             if crc32(d[4:size]) & 0xffffffff != crc:
                                 d = d[1:]
                                 continue
-                            fd.write(d[:size])
+                            dst_fd.write(d[:size])
                             d = d[size:]
-                finally:
-                    del d
-                    data.release()
+                    finally:
+                        del d
+                        data.release()
 
     def read(self, segment, offset, id, read_data=True):
         """


Test patch: