Repository: borgbackup/borg

Issue:
with-lock crashes with 1.1.0-rc2
I just upgraded from borgbackup-1.1.0-b3 to rc2, found an issue within minutes.

A simple example, borgbackup is version 1.1.0-rc2
> $ borgbackup --version
borgbackup 1.1.0rc2
> $ borgbackup with-lock mail-01.borg ls
[Files are listed here]
> Data integrity error: Segment entry data short read [segment 391, offset 1590]: expected 1890652, got 2497 bytes
> Traceback (most recent call last):
>   File "borg/archiver.py", line 3968, in main
>   File "borg/archiver.py", line 3896, in run
>   File "borg/archiver.py", line 148, in wrapper
>   File "borg/archiver.py", line 1677, in do_with_lock
>   File "borg/repository.py", line 411, in commit
>   File "borg/repository.py", line 685, in compact_segments
>   File "borg/repository.py", line 1292, in iter_objects
>   File "borg/repository.py", line 1373, in _read
> borg.helpers.IntegrityError: Data integrity error: Segment entry data short read [segment 391, offset 1590]: expected 1890652, got 2497 bytes
> 
> Platform: Linux thor 4.8.0-1-amd64 #1 SMP Debian 4.8.5-1 (2016-10-28) x86_64 
> Linux: debian stretch/sid 
> Borg: 1.1.0rc2  Python: CPython 3.5.3
> PID: 31905  CWD: /data/borg-WedFriSun
> sys.argv: ['borgbackup', 'with-lock', 'mail-01.borg', 'ls']
> SSH_ORIGINAL_COMMAND: None
> 

With 1.1.0-b3 it works fine.

No errors when running check
> $ borgbackup check mail-01.borg
> Enter passphrase for key /data/borg-WedFriSun/mail-01.borg: 
>


Discussion:

Patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
index ca55c698..dada3c14 100644
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -1176,6 +1176,8 @@ def cleanup(self, transaction_id):
         count = 0
         for segment, filename in self.segment_iterator(reverse=True):
             if segment > transaction_id:
+                if segment in self.fds:
+                    del self.fds[segment]
                 truncate_and_unlink(filename)
                 count += 1
             else:
@@ -1232,6 +1234,12 @@ def get_write_fd(self, no_new=False, raise_full=False):
             self._write_fd = SyncFile(self.segment_filename(self.segment), binary=True)
             self._write_fd.write(MAGIC)
             self.offset = MAGIC_LEN
+            if self.segment in self.fds:
+                # we may have a cached fd for a segment file we already deleted and
+                # we are writing now a new segment file to same file name. get rid of
+                # of the cached fd that still refers to the old file, so it will later
+                # get repopulated (on demand) with a fd that refers to the new file.
+                del self.fds[self.segment]
         return self._write_fd
 
     def get_fd(self, segment):


Test patch:
diff --git a/src/borg/testsuite/repository.py b/src/borg/testsuite/repository.py
index e29c7d53..1bec1400 100644
--- a/src/borg/testsuite/repository.py
+++ b/src/borg/testsuite/repository.py
@@ -210,6 +210,23 @@ def test_sparse_delete(self):
         self.repository.commit()
         assert 0 not in [segment for segment, _ in self.repository.io.segment_iterator()]
 
+    def test_uncommitted_garbage(self):
+        # uncommitted garbage should be no problem, it is cleaned up automatically.
+        # we just have to be careful with invalidation of cached FDs in LoggedIO.
+        self.repository.put(H(0), b'foo')
+        self.repository.commit()
+        # write some crap to a uncommitted segment file
+        last_segment = self.repository.io.get_latest_segment()
+        with open(self.repository.io.segment_filename(last_segment + 1), 'wb') as f:
+            f.write(MAGIC + b'crapcrapcrap')
+        self.repository.close()
+        # usually, opening the repo and starting a transaction should trigger a cleanup.
+        self.repository = self.open()
+        with self.repository:
+            self.repository.put(H(0), b'bar')  # this may trigger compact_segments()
+            self.repository.commit()
+        # the point here is that nothing blows up with an exception.
+
 
 class RepositoryCommitTestCase(RepositoryTestCaseBase):