Repository: borgbackup/borg

Issue:
borg check --repair does not repair object count mismatch
Tried to repair the broken repo from #1395, but couldn't:

```
(borg-env)tw@tux:~/w/borg$ borg check -v repo
Starting repository check
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive 3.checkpoint (3/3)
Analyzing archive 2.checkpoint (2/3)
Analyzing archive 1.checkpoint (1/3)
Archive consistency check complete, no problems found.
```

```
(borg-env)tw@tux:~/w/borg$ borg check -v --repair repo
'check --repair' is an experimental feature that might result in data loss.
Type 'YES' if you understand this and want to continue: YES
Starting repository check
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive 3.checkpoint (3/3)
Analyzing archive 2.checkpoint (2/3)
Analyzing archive 1.checkpoint (1/3)
Archive consistency check complete, no problems found.
```

```
(borg-env)tw@tux:~/w/borg$ borg check -v repo
Starting repository check
Index object count mismatch. 92 != 93
Completed repository check, errors found.
```

```
(borg-env)tw@tux:~/w/borg$ borg check -v --repair repo
'check --repair' is an experimental feature that might result in data loss.
Type 'YES' if you understand this and want to continue: YES
Starting repository check
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive 3.checkpoint (3/3)
Analyzing archive 2.checkpoint (2/3)
Analyzing archive 1.checkpoint (1/3)
1 orphaned objects found!
Archive consistency check complete, problems found.
```

```
(borg-env)tw@tux:~/w/borg$ borg check -v --repair repo
'check --repair' is an experimental feature that might result in data loss.
Type 'YES' if you understand this and want to continue: YES
Starting repository check
Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive 3.checkpoint (3/3)
Analyzing archive 2.checkpoint (2/3)
Analyzing archive 1.checkpoint (1/3)
1 orphaned objects found!
Archive consistency check complete, problems found.
```

```
(borg-env)tw@tux:~/w/borg$ borg check -v repo
Starting repository check
Index object count mismatch. 92 != 93
Completed repository check, errors found.
```



Discussion:
What I don't get here is that the index mismatch is a repository-level error. This really shouldn't happen. Esp. since it did not happen at first. This seems to imply that the higher-level borg-check somehow managed to corrupt the repository index.

A bug in compact_segments.

This is _not_ critical. **It does not lead to data loss**. In fact, quite the opposite. Borg stumbles over supposedly-deleted data, we just "forgot" the fact that we deleted it ;)  (or, IOW, that fact is now only in the index, not the segments anymore).

More in-depth analysis and fix will be coming.

A simple example:

```
State 1:

S0: put A, put B, put C
S1: delete A, commit

Objects := {B, C}
Index := {B, C}

compact_segments

S0.... not compacting S0, it's not sparse enough
       (only A deleted, B/C are still live)
S1...
    Asking the question "if segment > index_transaction_id"
    to bring "delete A" into a fresh segment (S2). Answer is no.

Complete compaction:
S2: commit
Delete S1
Compaction done.

State 2:

S0: put A, put B, put C
S1 (does not exist)
S2: commit

Objects := {A, B, C}
Index := {B, C}
```

---

(IRC)

> <enkore> the question for a proper fix is somewhat difficult. the 1st order case is possible to tackle with no further changes needed. but 2nd+ order not.
> <enkore> we need to have a "shadow index" of sorts, were TAG_PUTs of deleted objects are tracked.
> <enkore> this could live in the hints file (like FreeSpace)
> <enkore> hmmmmm
> <enkore> so the 1st order case is the picture i made in the ticket, right.
> <enkore> so inside the same transaction we have a PUT and then a DELETE. we can track this in-memory, no problems.
> <enkore> then we know we cannot drop the DELETE if we did not compact the PUT.
> <enkore> but if we consider 2nd order, spanning 2 transactions, then it's already impossible with no helper data structure. to know whether a DELETE can be dropped we have to know whether there is still a PUT somewhere around, and that we don't know, so we'd have to scan whole repository. that does not work.
> <enkore> so, when we commit we still have the old index on disk, we could use that (with the according disadvantage in memory usage, because we now have two indices in memory both tracking live data ie. tha majority of the data)
> <enkore> so i see three solutions here
> <enkore> 1.) shadow index in the hints file. the hints can be lost, but if we preserved the PUT/DELETE ordering in the transactions _using_ the shadow index then that's not a problem at all.
> <enkore> but: 1.0 compat, borg check 1.0 would still raise the error. a --repair would fix it though, by making the dead data live again.
> <enkore> 2.) use two index solution. baaaad. so much memory usage.
> <enkore> 3.) remove smart handling of compaction
> <enkore> 4.) like 3.) but special-case 00000000...00000 to not compact because of that one.
> <enkore> since 000..000 is never deleted, only overwritten there is no put/delete ordering issue and no delete's that could be dropped.

WIP @ https://github.com/enkore/borg/tree/issue/1396

@enkore so, is this fixed by #1448?

No, 86d95fe is the draft-fix for this specific issue. I'll polish it and add some tests.


Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -22,6 +22,7 @@
 from .helpers import ProgressIndicatorPercent
 from .helpers import bin_to_hex
 from .locking import Lock, LockError, LockErrorT
+from .logger import create_logger
 from .lrucache import LRUCache
 from .platform import SaveFile, SyncFile, sync_dir
 
@@ -110,6 +111,10 @@ def __init__(self, path, create=False, exclusive=False, lock_wait=None, lock=Tru
         self.io = None
         self.lock = None
         self.index = None
+        # This is an index of shadowed log entries during this transaction. Consider the following sequence:
+        # segment_n PUT A, segment_x DELETE A
+        # After the "DELETE A" in segment_x the shadow index will contain "A -> [n]".
+        self.shadow_index = {}
         self._active_txn = False
         self.lock_wait = lock_wait
         self.do_lock = lock
@@ -308,6 +313,7 @@ def prepare_txn(self, transaction_id, do_cleanup=True):
         if transaction_id is None:
             self.segments = {}  # XXX bad name: usage_count_of_segment_x = self.segments[x]
             self.compact = FreeSpace()  # XXX bad name: freeable_space_of_segment_x = self.compact[x]
+            self.shadow_index.clear()
         else:
             if do_cleanup:
                 self.io.cleanup(transaction_id)
@@ -338,6 +344,11 @@ def prepare_txn(self, transaction_id, do_cleanup=True):
             else:
                 self.segments = hints[b'segments']
                 self.compact = FreeSpace(hints[b'compact'])
+            # Drop uncommitted segments in the shadow index
+            for key, shadowed_segments in self.shadow_index.items():
+                for segment in list(shadowed_segments):
+                    if segment > transaction_id:
+                        shadowed_segments.remove(segment)
 
     def write_index(self):
         hints = {b'version': 2,
@@ -413,31 +424,40 @@ def compact_segments(self):
         index_transaction_id = self.get_index_transaction_id()
         segments = self.segments
         unused = []  # list of segments, that are not used anymore
+        logger = create_logger('borg.debug.compact_segments')
 
         def complete_xfer(intermediate=True):
             # complete the current transfer (when some target segment is full)
             nonlocal unused
             # commit the new, compact, used segments
-            self.io.write_commit(intermediate=intermediate)
+            segment = self.io.write_commit(intermediate=intermediate)
+            logger.debug('complete_xfer: wrote %scommit at segment %d', 'intermediate ' if intermediate else '', segment)
             # get rid of the old, sparse, unused segments. free space.
             for segment in unused:
+                logger.debug('complete_xfer: deleting unused segment %d', segment)
                 assert self.segments.pop(segment) == 0
                 self.io.delete_segment(segment)
                 del self.compact[segment]
             unused = []
 
+        logger.debug('compaction started.')
         for segment, freeable_space in sorted(self.compact.items()):
             if not self.io.segment_exists(segment):
+                logger.warning('segment %d not found, but listed in compaction data', segment)
                 del self.compact[segment]
                 continue
             segment_size = self.io.segment_size(segment)
             if segment_size > 0.2 * self.max_segment_size and freeable_space < 0.15 * segment_size:
-                logger.debug('not compacting segment %d for later (only %d bytes are sparse)',
-                             segment, freeable_space)
+                logger.debug('not compacting segment %d (only %d bytes are sparse)', segment, freeable_space)
                 continue
             segments.setdefault(segment, 0)
+            logger.debug('compacting segment %d with usage count %d and %d freeable bytes',
+                         segment, segments[segment], freeable_space)
             for tag, key, offset, data in self.io.iter_objects(segment, include_data=True):
-                if tag == TAG_PUT and self.index.get(key, (-1, -1)) == (segment, offset):
+                if tag == TAG_COMMIT:
+                    continue
+                in_index = self.index.get(key) == (segment, offset)
+                if tag == TAG_PUT and in_index:
                     try:
                         new_segment, offset = self.io.write_put(key, data, raise_full=True)
                     except LoggedIO.SegmentFull:
@@ -447,8 +467,22 @@ def complete_xfer(intermediate=True):
                     segments.setdefault(new_segment, 0)
                     segments[new_segment] += 1
                     segments[segment] -= 1
+                elif tag == TAG_PUT and not in_index:
+                    # If this is a PUT shadowed by a later tag, then it will be gone when this segment is deleted after
+                    # this loop. Therefore it is removed from the shadow index.
+                    try:
+                        self.shadow_index[key].remove(segment)
+                    except (KeyError, ValueError):
+                        pass
                 elif tag == TAG_DELETE:
-                    if index_transaction_id is None or segment > index_transaction_id:
+                    # If the shadow index doesn't contain this key, then we can't say if there's a shadowed older tag,
+                    # therefore we do not drop the delete, but write it to a current segment.
+                    shadowed_put_exists = key not in self.shadow_index or any(
+                        # If the key is in the shadow index and there is any segment with an older PUT of this
+                        # key, we have a shadowed put.
+                        shadowed < segment for shadowed in self.shadow_index[key])
+
+                    if shadowed_put_exists or index_transaction_id is None or segment > index_transaction_id:
                         # (introduced in 6425d16aa84be1eaaf88)
                         # This is needed to avoid object un-deletion if we crash between the commit and the deletion
                         # of old segments in complete_xfer().
@@ -492,6 +526,7 @@ def complete_xfer(intermediate=True):
             assert segments[segment] == 0
             unused.append(segment)
         complete_xfer(intermediate=False)
+        logger.debug('compaction completed.')
 
     def replay_segments(self, index_transaction_id, segments_transaction_id):
         # fake an old client, so that in case we do not have an exclusive lock yet, prepare_txn will upgrade the lock:
@@ -705,6 +740,7 @@ def delete(self, id, wait=True):
             segment, offset = self.index.pop(id)
         except KeyError:
             raise self.ObjectNotFound(id, self.path) from None
+        self.shadow_index.setdefault(id, []).append(segment)
         self.segments[segment] -= 1
         size = self.io.read(segment, offset, id, read_data=False)
         self.compact[segment] += size
@@ -1017,6 +1053,7 @@ def write_commit(self, intermediate=False):
         crc = self.crc_fmt.pack(crc32(header) & 0xffffffff)
         fd.write(b''.join((crc, header)))
         self.close_segment()
+        return self.segment - 1  # close_segment() increments it
 
 
 MAX_DATA_SIZE = MAX_OBJECT_SIZE - LoggedIO.put_header_fmt.size
diff --git a/src/borg/testsuite/repository.py b/src/borg/testsuite/repository.py
--- a/src/borg/testsuite/repository.py
+++ b/src/borg/testsuite/repository.py
@@ -295,6 +295,56 @@ def test_moved_deletes_are_tracked(self):
             for tag, key, offset, size in self.repository.io.iter_objects(segment):
                 assert tag != TAG_DELETE
 
+    def test_shadowed_entries_are_preserved(self):
+        get_latest_segment = self.repository.io.get_latest_segment
+        self.repository.put(H(1), b'1')
+        # This is the segment with our original PUT of interest
+        put_segment = get_latest_segment()
+        self.repository.commit()
+
+        # We now delete H(1), and force this segment to not be compacted, which can happen
+        # if it's not sparse enough (symbolized by H(2) here).
+        self.repository.delete(H(1))
+        self.repository.put(H(2), b'1')
+        delete_segment = get_latest_segment()
+
+        # We pretend these are mostly dense (not sparse) and won't be compacted
+        del self.repository.compact[put_segment]
+        del self.repository.compact[delete_segment]
+
+        self.repository.commit()
+
+        # Now we perform an unrelated operation on the segment containing the DELETE,
+        # causing it to be compacted.
+        self.repository.delete(H(2))
+        self.repository.commit()
+
+        assert self.repository.io.segment_exists(put_segment)
+        assert not self.repository.io.segment_exists(delete_segment)
+
+        # Basic case, since the index survived this must be ok
+        assert H(1) not in self.repository
+        # Nuke index, force replay
+        os.unlink(os.path.join(self.repository.path, 'index.%d' % get_latest_segment()))
+        # Must not reappear
+        assert H(1) not in self.repository
+
+    def test_shadow_index_rollback(self):
+        self.repository.put(H(1), b'1')
+        self.repository.delete(H(1))
+        assert self.repository.shadow_index[H(1)] == [0]
+        self.repository.commit()
+        # note how an empty list means that nothing is shadowed for sure
+        assert self.repository.shadow_index[H(1)] == []
+        self.repository.put(H(1), b'1')
+        self.repository.delete(H(1))
+        # 0 put/delete; 1 commit; 2 compacted; 3 commit; 4 put/delete
+        assert self.repository.shadow_index[H(1)] == [4]
+        self.repository.rollback()
+        self.repository.put(H(2), b'1')
+        # After the rollback segment 4 shouldn't be considered anymore
+        assert self.repository.shadow_index[H(1)] == []
+
 
 class RepositoryAppendOnlyTestCase(RepositoryTestCaseBase):
     def open(self, create=False):


Test patch: