Repository: borgbackup/borg

Issue:
Aborting check --repair in recover_segment() may leave segment partially recovered
I have not tested this, so may be missing something, but looking at the code:

[`recover_segment`](https://github.com/borgbackup/borg/blob/85e6c0afd82f52182c56bfcc0b795aef33adc037/src/borg/repository.py#L1485) does rename a segment adding the [`.beforerecover`](https://github.com/borgbackup/borg/blob/85e6c0afd82f52182c56bfcc0b795aef33adc037/src/borg/repository.py#L1489) extension and then creates a new file with the original name, [**slowly**](https://github.com/borgbackup/borg/issues/4243#issuecomment-451702272) iterates over the renamed original segment, copying any parts with correct CRC over.

This code is the only occurrence of the string `beforerecover` within the source, so I conclude, that such a renamed original segment is never considered for repair again.

Thus, when a long running `borg check --repair` is interrupted for some reason and that interruption happens within above process, the original (renamed to `.beforerecover`) segment as well as the new segment, which however only has parts of the original data recovered yet, are left behind and a new run of `borg check --repair` would check than new segment only and skip on to the next, if its fine in itself (or if not (or is the new partial one always self-consistent?), maybe even overwrite the original by again renaming the partial new one to `.beforerecover`?).

Either way, with no new errors showing up, the user will assume the `borg check --repair` to have resumed just fine, when it eventually skipped over what worst case could be a large part of a segment containing data that could have been recovered.

[This comment](https://github.com/borgbackup/borg/issues/4243#issuecomment-451702455) seems to have observed that happening in practice, but seems to have went unnoticed within that question.
Aborting check --repair in recover_segment() may leave segment partially recovered
I have not tested this, so may be missing something, but looking at the code:

[`recover_segment`](https://github.com/borgbackup/borg/blob/85e6c0afd82f52182c56bfcc0b795aef33adc037/src/borg/repository.py#L1485) does rename a segment adding the [`.beforerecover`](https://github.com/borgbackup/borg/blob/85e6c0afd82f52182c56bfcc0b795aef33adc037/src/borg/repository.py#L1489) extension and then creates a new file with the original name, [**slowly**](https://github.com/borgbackup/borg/issues/4243#issuecomment-451702272) iterates over the renamed original segment, copying any parts with correct CRC over.

This code is the only occurrence of the string `beforerecover` within the source, so I conclude, that such a renamed original segment is never considered for repair again.

Thus, when a long running `borg check --repair` is interrupted for some reason and that interruption happens within above process, the original (renamed to `.beforerecover`) segment as well as the new segment, which however only has parts of the original data recovered yet, are left behind and a new run of `borg check --repair` would check than new segment only and skip on to the next, if its fine in itself (or if not (or is the new partial one always self-consistent?), maybe even overwrite the original by again renaming the partial new one to `.beforerecover`?).

Either way, with no new errors showing up, the user will assume the `borg check --repair` to have resumed just fine, when it eventually skipped over what worst case could be a large part of a segment containing data that could have been recovered.

[This comment](https://github.com/borgbackup/borg/issues/4243#issuecomment-451702455) seems to have observed that happening in practice, but seems to have went unnoticed within that question.


Discussion:
Possible steps to improve:

1. Rename the `.beforerecover` to something like `.afterecover` or `.recovered` (adding numbered extensions when that already exists to avoid ever overwriting a original segment) within `recover_segment` as soon as it is iterated over.
2. In the check code, do look for left behind `.beforerecover` files, warn/error about their existence in all cases.
3. Restart `recover_segment` on a left behind `.beforerecover` file in case of check repair.

To not over-complicate things, 3 could be skipped (with an according warning) if nonsense/unsafe combinations of files exist, like e.g. both a `.beforerecover` and `.recovered`.
Uh, good catch. I didn't check the source yet, but guess you're right.

Another idea would be to re-use the existing pattern/code, where we have a context manager that helps with reading the original file, writing to a temp file and only if successful, atomically renaming the temp file over the original file.

Or copy instead of rename, in case we want to keep the original.

Possible steps to improve:

1. Rename the `.beforerecover` to something like `.afterecover` or `.recovered` (adding numbered extensions when that already exists to avoid ever overwriting a original segment) within `recover_segment` as soon as it is iterated over.
2. In the check code, do look for left behind `.beforerecover` files, warn/error about their existence in all cases.
3. Restart `recover_segment` on a left behind `.beforerecover` file in case of check repair.

To not over-complicate things, 3 could be skipped (with an according warning) if nonsense/unsafe combinations of files exist, like e.g. both a `.beforerecover` and `.recovered`.
Uh, good catch. I didn't check the source yet, but guess you're right.

Another idea would be to re-use the existing pattern/code, where we have a context manager that helps with reading the original file, writing to a temp file and only if successful, atomically renaming the temp file over the original file.

Or copy instead of rename, in case we want to keep the original.


Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -1438,23 +1438,21 @@ def recover_segment(self, segment, filename):
         logger.info('attempting to recover ' + filename)
         if segment in self.fds:
             del self.fds[segment]
-        backup_filename = filename + '.beforerecover'
-        os.rename(filename, backup_filename)
-        if os.path.getsize(backup_filename) < MAGIC_LEN + self.header_fmt.size:
+        if os.path.getsize(filename) < MAGIC_LEN + self.header_fmt.size:
             # this is either a zero-byte file (which would crash mmap() below) or otherwise
             # just too small to be a valid non-empty segment file, so do a shortcut here:
-            with open(filename, 'wb') as fd:
+            with SaveFile(filename, binary=True) as fd:
                 fd.write(MAGIC)
             return
-        with open(backup_filename, 'rb') as backup_fd:
-            # note: file must not be 0 size or mmap() will crash.
-            with mmap.mmap(backup_fd.fileno(), 0, access=mmap.ACCESS_READ) as mm:
-                # memoryview context manager is problematic, see https://bugs.python.org/issue35686
-                data = memoryview(mm)
-                d = data
-                try:
-                    with open(filename, 'wb') as fd:
-                        fd.write(MAGIC)
+        with SaveFile(filename, binary=True) as dst_fd:
+            with open(filename, 'rb') as src_fd:
+                # note: file must not be 0 size or mmap() will crash.
+                with mmap.mmap(src_fd.fileno(), 0, access=mmap.ACCESS_READ) as mm:
+                    # memoryview context manager is problematic, see https://bugs.python.org/issue35686
+                    data = memoryview(mm)
+                    d = data
+                    try:
+                        dst_fd.write(MAGIC)
                         while len(d) >= self.header_fmt.size:
                             crc, size, tag = self.header_fmt.unpack(d[:self.header_fmt.size])
                             if size < self.header_fmt.size or size > len(d):
@@ -1463,11 +1461,11 @@ def recover_segment(self, segment, filename):
                             if crc32(d[4:size]) & 0xffffffff != crc:
                                 d = d[1:]
                                 continue
-                            fd.write(d[:size])
+                            dst_fd.write(d[:size])
                             d = d[size:]
-                finally:
-                    del d
-                    data.release()
+                    finally:
+                        del d
+                        data.release()
 
     def read(self, segment, offset, id, read_data=True):
         """


Test patch: