Repository: borgbackup/borg

Issue:
Can borg repair broken repository chunks
In the case of bit-rot in the repository where a chunk no longer matches its hash value, is it possible for the borg client to upload a new chunk to replace/fix the broken chunk.

It would not help in the case where a file has changed over time and the chunk came from an older version of the file.  But for files that are slow to change, repairing the chunk in the repository would keep older archives from being broken.  So the usefulness would depend on how fast your data set changes, and whether the bit-rot affected a newer chunk or a really ancient chunk.



Discussion:
Duplicity can use par2 error correction files in order to repair broken and even missing chunks of old backup files. I think that something like that would be even more important with borg because of the deduplication. With duplicity one pretty much ends up with multiple copies of files due to the full-incremental scheme, which borg avoids. Despite being pretty careful, I have had bit rot happen in my backups due to HD failure, rsync network errors, and other accidents. If I could dial-in a % overhead to add some level of redundancy then I'd do that.

my understandign is, that all that is needed is force-writing said chunk,
then in the segments it is repressented as delete/update pair and would replace the corrupt data completely on the next vacuum,

this needs a implementation

please continue discussion about adding redundancy there: https://github.com/borgbackup/borg/issues/225
and stay here on topic, see first post.

From the mailing list:

```
On 06.07.2016 17:57, Tomasz Melcer wrote:
> On 06.07.2016 17:44, Marian Beermann wrote:
>> The data files is another story. There's no forward error correction in
>> Borg itself, so errors can be detected but only some minor errors can be
>> corrected. "borg check --repair" will replace corrupted data chunks with
>> runs of zeroes of the same length, while saying where it did that.
>> Corrupted commit tags can take some more data with them to nirvana.
>
> So, please correct me if I'm wrong:
>
> Let say there was a bad block inside one of the data files. After
> recovery I can just run `borg check --repair`, and while some of the
> data will be lost, other chunks will still be there. Therefore, I can
> fully recover all past backups except for the chunk(s) hit by the bad
> blocks.

Well it always depends on what block is exactly hit and how.

You may lose a lot of FS structure through one unlucky bad block
equating to tons of data loss, or maybe just a data block somewhere.
If it's only in the data, then check--repair has an easy job and it'd be
really only that block, if it also hit's structure metadata more chunks
in the same file may be lost.
If it hit's a commit more may be lost.
If it hit's the one chunk you're interested in right now you have a
problem and so on.
If it hits the metadata of an archive the archive may loose some files
upon repair or may be un-repairable, but no data *per se* would be lost.

But in principle, yes, and from a pure statistics view the likelihood of
data itself being affected instead of metadata is high for a single block.

>
> One more question. If the next backup will happen to have the same
> chunk, will it be added to the backup, filling the missing part for
> older backups? In some scenarios I find it likely that a possible bad
> block could just hit a chunk of one of the files that are still
> available on the live system.
>

Yes, but the old backup archive will still have a run of zeroes in it.

To explain why, a rough sketch how data storage works in Borg (also
explained in the internals docs in more detail). A file item has a chunk
ID list, which lists the chunks containing the file data in order (and
some metadata). If check finds that one of these is gone, then it
creates a *new* chunk, same size as the corrupted one, made from zeroes,
stores it, and *edits* the chunk ID list of the affected file to refer
to the zeroes chunk instead.

If you do a new backup after that, then the corrupted chunk simply won't
be in the repository any more and will be newly stored by the backup.
But since creating a new backup doesn't touch old archives the old
archive will still have a run of zeroes there.

Arguably this could be improved (e.g. when repairing a file, edit the
chunk ID list, but also store how it was edited - when the affected
chunks are stored back in the repository these files may be healed
"retrospectively" then.)

Cheers, Marian
```

How about adding a "chunks_healthy" key to the item metadata dict with a copy of the original (correct, but partly unavailable) chunk ids in case we choose to "repair" it (by replacing some chunks with all-zero chunks and modifying the "chunks" key with these replacement IDs)?

The copy into the chunks_healthy key would be only made if there is no such key already - so that we do not lose information in case such a repair happens multiple times.

A healing operation could then check for the chunks_healthy key, if it is present, compute the set difference with chunks and check if all the missing chunks are in the repo NOW, if so, copy chunks_healthy value into chunks and remove chunks_healthy key.

1.0.4 will already remember the good/original chunk IDs.
Healing functionality based on that is still TODO.


Resolving patch:
diff --git a/borg/archive.py b/borg/archive.py
--- a/borg/archive.py
+++ b/borg/archive.py
@@ -912,31 +912,56 @@ def add_reference(id_, size, csize, cdata=None):
                     self.repository.put(id_, cdata)
 
         def verify_file_chunks(item):
-            """Verifies that all file chunks are present
+            """Verifies that all file chunks are present.
 
-            Missing file chunks will be replaced with new chunks of the same
-            length containing all zeros.
+            Missing file chunks will be replaced with new chunks of the same length containing all zeros.
+            If a previously missing file chunk re-appears, the replacement chunk is replaced by the correct one.
             """
             offset = 0
             chunk_list = []
             chunks_replaced = False
-            for chunk_id, size, csize in item[b'chunks']:
+            has_chunks_healthy = b'chunks_healthy' in item
+            chunks_current = item[b'chunks']
+            chunks_healthy = item[b'chunks_healthy'] if has_chunks_healthy else chunks_current
+            assert len(chunks_current) == len(chunks_healthy)
+            for chunk_current, chunk_healthy in zip(chunks_current, chunks_healthy):
+                chunk_id, size, csize = chunk_healthy
                 if chunk_id not in self.chunks:
-                    # If a file chunk is missing, create an all empty replacement chunk
-                    logger.error('{}: Missing file chunk detected (Byte {}-{})'.format(item[b'path'].decode('utf-8', 'surrogateescape'), offset, offset + size))
-                    self.error_found = chunks_replaced = True
-                    data = bytes(size)
-                    chunk_id = self.key.id_hash(data)
-                    cdata = self.key.encrypt(data)
-                    csize = len(cdata)
-                    add_reference(chunk_id, size, csize, cdata)
+                    # a chunk of the healthy list is missing
+                    if chunk_current == chunk_healthy:
+                        logger.error('{}: New missing file chunk detected (Byte {}-{}). '
+                                     'Replacing with all-zero chunk.'.format(
+                                     item[b'path'].decode('utf-8', 'surrogateescape'), offset, offset + size))
+                        self.error_found = chunks_replaced = True
+                        data = bytes(size)
+                        chunk_id = self.key.id_hash(data)
+                        cdata = self.key.encrypt(data)
+                        csize = len(cdata)
+                        add_reference(chunk_id, size, csize, cdata)
+                    else:
+                        logger.info('{}: Previously missing file chunk is still missing (Byte {}-{}). '
+                                    'It has a all-zero replacement chunk already.'.format(
+                                    item[b'path'].decode('utf-8', 'surrogateescape'), offset, offset + size))
+                        chunk_id, size, csize = chunk_current
+                        add_reference(chunk_id, size, csize)
                 else:
-                    add_reference(chunk_id, size, csize)
-                chunk_list.append((chunk_id, size, csize))
+                    if chunk_current == chunk_healthy:
+                        # normal case, all fine.
+                        add_reference(chunk_id, size, csize)
+                    else:
+                        logger.info('{}: Healed previously missing file chunk! (Byte {}-{}).'.format(
+                            item[b'path'].decode('utf-8', 'surrogateescape'), offset, offset + size))
+                        add_reference(chunk_id, size, csize)
+                        mark_as_possibly_superseded(chunk_current[0])  # maybe orphaned the all-zero replacement chunk
+                chunk_list.append([chunk_id, size, csize])  # list-typed element as chunks_healthy is list-of-lists
                 offset += size
-            if chunks_replaced and b'chunks_healthy' not in item:
+            if chunks_replaced and not has_chunks_healthy:
                 # if this is first repair, remember the correct chunk IDs, so we can maybe heal the file later
                 item[b'chunks_healthy'] = item[b'chunks']
+            if has_chunks_healthy and chunk_list == chunks_healthy:
+                logger.info('{}: Completely healed previously damaged file!'.format(
+                            item[b'path'].decode('utf-8', 'surrogateescape')))
+                del item[b'chunks_healthy']
             item[b'chunks'] = chunk_list
 
         def robust_iterator(archive):
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -973,9 +973,12 @@ def build_parser(self, args=None, prog=None):
         - Check if archive metadata chunk is present. if not, remove archive from
           manifest.
         - For all files (items) in the archive, for all chunks referenced by these
-          files, check if chunk is present (if not and we are in repair mode, replace
-          it with a same-size chunk of zeros). This requires reading of archive and
-          file metadata, but not data.
+          files, check if chunk is present.
+          If a chunk is not present and we are in repair mode, replace it with a same-size
+          replacement chunk of zeros.
+          If a previously lost chunk reappears (e.g. via a later backup) and we are in
+          repair mode, the all-zero replacement chunk will be replaced by the correct chunk.
+          This requires reading of archive and file metadata, but not data.
         - If we are in repair mode and we checked all the archives: delete orphaned
           chunks from the repo.
         - if you use a remote repo server via ssh:, the archive check is executed on
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -1142,12 +1142,45 @@ def test_missing_file_chunk(self):
         with repository:
             for item in archive.iter_items():
                 if item[b'path'].endswith('testsuite/archiver.py'):
-                    repository.delete(item[b'chunks'][-1][0])
+                    valid_chunks = item[b'chunks']
+                    killed_chunk = valid_chunks[-1]
+                    repository.delete(killed_chunk[0])
                     break
+            else:
+                self.assert_true(False)  # should not happen
             repository.commit()
         self.cmd('check', self.repository_location, exit_code=1)
-        self.cmd('check', '--repair', self.repository_location, exit_code=0)
+        output = self.cmd('check', '--repair', self.repository_location, exit_code=0)
+        self.assert_in('New missing file chunk detected', output)
         self.cmd('check', self.repository_location, exit_code=0)
+        # check that the file in the old archives has now a different chunk list without the killed chunk
+        for archive_name in ('archive1', 'archive2'):
+            archive, repository = self.open_archive(archive_name)
+            with repository:
+                for item in archive.iter_items():
+                    if item[b'path'].endswith('testsuite/archiver.py'):
+                        self.assert_not_equal(valid_chunks, item[b'chunks'])
+                        self.assert_not_in(killed_chunk, item[b'chunks'])
+                        break
+                else:
+                    self.assert_true(False)  # should not happen
+        # do a fresh backup (that will include the killed chunk)
+        with patch.object(ChunkBuffer, 'BUFFER_SIZE', 10):
+            self.create_src_archive('archive3')
+        # check should be able to heal the file now:
+        output = self.cmd('check', '-v', '--repair', self.repository_location, exit_code=0)
+        self.assert_in('Healed previously missing file chunk', output)
+        self.assert_in('testsuite/archiver.py: Completely healed previously damaged file!', output)
+        # check that the file in the old archives has the correct chunks again
+        for archive_name in ('archive1', 'archive2'):
+            archive, repository = self.open_archive(archive_name)
+            with repository:
+                for item in archive.iter_items():
+                    if item[b'path'].endswith('testsuite/archiver.py'):
+                        self.assert_equal(valid_chunks, item[b'chunks'])
+                        break
+                else:
+                    self.assert_true(False)  # should not happen
 
     def test_missing_archive_item_chunk(self):
         archive, repository = self.open_archive('archive1')


Test patch: