Repository: borgbackup/borg

Issue:
Index object count mismatch. 51251 != 51250
I am currently having 18 borg-repos. Backups from 6 different servers, 3 repos per server. My idea is to rotate them for different weekdays (Mon+Thu, TueSat, Wed+Fri+Sun) in case a repo breaks. My way of protecting against the downside of deduplication. Even with three mirrors of the data I am saving lots of space. The backup scripts does a create, prune, check.

I have been running this for 1-2 weeks now, with the last 4 days or so using the 1.1.0-b1 version. After I started to use the beta version I started to get the Index object count mismatch from the check-jobs. I currently have the issue with 4 of my 18 repos. The interesting fact is that this far, all of them has a mismatch of **one**.

At first I only noticed the issue with one host. I deleted a repo to make sure I started over from scratch, and removed the prune job from the backup script, hoping the prune was the issue. This did not help.

Earlier I was using version 1.0.7 and did not get these issues, so I assume it's something with the new code, though I can't be certain.

I assume this isn't a known issue since I couldn't find anything about it. Any ideas what could be the cause? Anything I could do to help finding the issue? Any other ideas?

One of my simplest repos are backed up like this:
${BORG} create --compression lzma,1 borg@${DESTHOST}:${DESTPATH}${BORGREPO}.borg::{now} /etc /usr/local/bin /root /home
Eg, no weird options at all, and it happened to this one early this morning.

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/37897778-index-object-count-mismatch-51251-51250)

Index object count mismatch. 51251 != 51250
I am currently having 18 borg-repos. Backups from 6 different servers, 3 repos per server. My idea is to rotate them for different weekdays (Mon+Thu, TueSat, Wed+Fri+Sun) in case a repo breaks. My way of protecting against the downside of deduplication. Even with three mirrors of the data I am saving lots of space. The backup scripts does a create, prune, check.

I have been running this for 1-2 weeks now, with the last 4 days or so using the 1.1.0-b1 version. After I started to use the beta version I started to get the Index object count mismatch from the check-jobs. I currently have the issue with 4 of my 18 repos. The interesting fact is that this far, all of them has a mismatch of **one**.

At first I only noticed the issue with one host. I deleted a repo to make sure I started over from scratch, and removed the prune job from the backup script, hoping the prune was the issue. This did not help.

Earlier I was using version 1.0.7 and did not get these issues, so I assume it's something with the new code, though I can't be certain.

I assume this isn't a known issue since I couldn't find anything about it. Any ideas what could be the cause? Anything I could do to help finding the issue? Any other ideas?

One of my simplest repos are backed up like this:
${BORG} create --compression lzma,1 borg@${DESTHOST}:${DESTPATH}${BORGREPO}.borg::{now} /etc /usr/local/bin /root /home
Eg, no weird options at all, and it happened to this one early this morning.

---

:moneybag: [there is a bounty for this](https://www.bountysource.com/issues/37897778-index-object-count-mismatch-51251-51250)



Discussion:

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -481,8 +481,9 @@ def complete_xfer(intermediate=True):
             for tag, key, offset, data in self.io.iter_objects(segment, include_data=True):
                 if tag == TAG_COMMIT:
                     continue
-                in_index = self.index.get(key) == (segment, offset)
-                if tag == TAG_PUT and in_index:
+                in_index = self.index.get(key)
+                is_index_object = in_index == (segment, offset)
+                if tag == TAG_PUT and is_index_object:
                     try:
                         new_segment, offset = self.io.write_put(key, data, raise_full=True)
                     except LoggedIO.SegmentFull:
@@ -492,22 +493,23 @@ def complete_xfer(intermediate=True):
                     segments.setdefault(new_segment, 0)
                     segments[new_segment] += 1
                     segments[segment] -= 1
-                elif tag == TAG_PUT and not in_index:
+                elif tag == TAG_PUT and not is_index_object:
                     # If this is a PUT shadowed by a later tag, then it will be gone when this segment is deleted after
                     # this loop. Therefore it is removed from the shadow index.
                     try:
                         self.shadow_index[key].remove(segment)
                     except (KeyError, ValueError):
                         pass
-                elif tag == TAG_DELETE:
+                elif tag == TAG_DELETE and not in_index:
                     # If the shadow index doesn't contain this key, then we can't say if there's a shadowed older tag,
                     # therefore we do not drop the delete, but write it to a current segment.
                     shadowed_put_exists = key not in self.shadow_index or any(
                         # If the key is in the shadow index and there is any segment with an older PUT of this
                         # key, we have a shadowed put.
                         shadowed < segment for shadowed in self.shadow_index[key])
+                    delete_is_not_stable = index_transaction_id is None or segment > index_transaction_id
 
-                    if shadowed_put_exists or index_transaction_id is None or segment > index_transaction_id:
+                    if shadowed_put_exists or delete_is_not_stable:
                         # (introduced in 6425d16aa84be1eaaf88)
                         # This is needed to avoid object un-deletion if we crash between the commit and the deletion
                         # of old segments in complete_xfer().
@@ -650,18 +652,25 @@ def report_error(msg):
         try:
             transaction_id = self.get_transaction_id()
             current_index = self.open_index(transaction_id)
-        except Exception:
+            logger.debug('Read committed index of transaction %d', transaction_id)
+        except Exception as exc:
             transaction_id = self.io.get_segments_transaction_id()
             current_index = None
+            logger.debug('Failed to read committed index (%s)', exc)
         if transaction_id is None:
+            logger.debug('No segments transaction found')
             transaction_id = self.get_index_transaction_id()
         if transaction_id is None:
+            logger.debug('No index transaction found, trying latest segment')
             transaction_id = self.io.get_latest_segment()
         if repair:
             self.io.cleanup(transaction_id)
         segments_transaction_id = self.io.get_segments_transaction_id()
+        logger.debug('Segment transaction is    %s', segments_transaction_id)
+        logger.debug('Determined transaction is %s', transaction_id)
         self.prepare_txn(None)  # self.index, self.compact, self.segments all empty now!
         segment_count = sum(1 for _ in self.io.segment_iterator())
+        logger.debug('Found %d segments', segment_count)
         pi = ProgressIndicatorPercent(total=segment_count, msg="Checking segments %3.1f%%", step=0.1)
         for i, (segment, filename) in enumerate(self.io.segment_iterator()):
             pi.show(i)
@@ -683,11 +692,28 @@ def report_error(msg):
             report_error('Adding commit tag to segment {}'.format(transaction_id))
             self.io.segment = transaction_id + 1
             self.io.write_commit()
+        logger.info('Starting repository index check')
         if current_index and not repair:
             # current_index = "as found on disk"
             # self.index = "as rebuilt in-memory from segments"
             if len(current_index) != len(self.index):
-                report_error('Index object count mismatch. {} != {}'.format(len(current_index), len(self.index)))
+                report_error('Index object count mismatch.')
+                logger.error('committed index: %d objects', len(current_index))
+                logger.error('rebuilt index:   %d objects', len(self.index))
+
+                line_format = '%-64s %-16s %-16s'
+                not_found = '<not found>'
+                logger.warning(line_format, 'ID', 'rebuilt index', 'committed index')
+                for key, value in self.index.iteritems():
+                    current_value = current_index.get(key, not_found)
+                    if current_value != value:
+                        logger.warning(line_format, bin_to_hex(key), value, current_value)
+                for key, current_value in current_index.iteritems():
+                    if key in self.index:
+                        continue
+                    value = self.index.get(key, not_found)
+                    if current_value != value:
+                        logger.warning(line_format, bin_to_hex(key), value, current_value)
             elif current_index:
                 for key, value in self.index.iteritems():
                     if current_index.get(key, (-1, -1)) != value:


Test patch: