Repository: borgbackup/borg

Issue:
consider adding repo cleanup for failed files
there is a known issue that if a file somehow fails in the middle of processing (e.g. due to an I/O error within the input file), borg will have written some of the content chunks to the repo, but then will abort processing that single file without adding a file item to the archive that references these chunks. borg will process all other remaining files and then commit.

this is how orphaned chunks can get created. borg check will then report an issue and will clean up these orphaned chunks.

in `item.chunks` we have the list of chunkids we'ld need to remove again within the exception handler or a `finally:`. 

Bounty: https://app.bountysource.com/issues/108737475-consider-adding-repo-cleanup-for-failed-files



Discussion:
Low severity bug. The biggest issue is that users are getting irritated about where the orphans come from.

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -1300,13 +1300,31 @@ def chunk_processor(chunk):
         # to get rid of .chunks_healthy, as it might not correspond to .chunks any more.
         if self.rechunkify and "chunks_healthy" in item:
             del item.chunks_healthy
-        for chunk in chunk_iter:
-            cle = chunk_processor(chunk)
-            item.chunks.append(cle)
-            self.current_volume += cle[1]
-            if show_progress:
-                stats.show_progress(item=item, dt=0.2)
-            self.maybe_checkpoint(item)
+        try:
+            for chunk in chunk_iter:
+                cle = chunk_processor(chunk)
+                item.chunks.append(cle)
+                self.current_volume += cle[1]
+                if show_progress:
+                    stats.show_progress(item=item, dt=0.2)
+                self.maybe_checkpoint(item)
+        except BackupOSError:
+            # something went wrong (e.g. an I/O error while reading a source file), try to avoid orphan content chunks:
+            # case A: "no checkpoint archive has been created yet":
+            # we have incref'd (written) some chunks, no commit yet, no file item for these chunks yet.
+            # -> item.chunks has a list of orphaned content chunks, we need to decref them.
+            # case B: "some checkpoint archives have been created already":
+            # at the time we commit them, everything is fine and consistent:
+            # we have incref'd (written) some chunks, created a part file item referencing them, committed.
+            # directly after commit, we have removed the part file item, but kept chunks in the repo, kept refcounts.
+            # maybe we have incref'd (written) some more chunks after the commit, no file item for these chunks yet.
+            # -> item.chunks has a list of orphaned content chunks, we need to decref them.
+            # So, cases A and B need same treatment.
+            for chunk in item.chunks:
+                cache.chunk_decref(chunk.id, stats, wait=False)
+            # now that we have cleaned up the chunk references, we can re-raise the exception
+            # this will skip THIS processing of this file, but continue with the next one.
+            raise
 
 
 class FilesystemObjectProcessors:


Test patch: