Repository: borgbackup/borg

Issue:
with-lock crashes with 1.1.0-rc2
I just upgraded from borgbackup-1.1.0-b3 to rc2, found an issue within minutes.

A simple example, borgbackup is version 1.1.0-rc2
> $ borgbackup --version
borgbackup 1.1.0rc2
> $ borgbackup with-lock mail-01.borg ls
[Files are listed here]
> Data integrity error: Segment entry data short read [segment 391, offset 1590]: expected 1890652, got 2497 bytes
> Traceback (most recent call last):
>   File "borg/archiver.py", line 3968, in main
>   File "borg/archiver.py", line 3896, in run
>   File "borg/archiver.py", line 148, in wrapper
>   File "borg/archiver.py", line 1677, in do_with_lock
>   File "borg/repository.py", line 411, in commit
>   File "borg/repository.py", line 685, in compact_segments
>   File "borg/repository.py", line 1292, in iter_objects
>   File "borg/repository.py", line 1373, in _read
> borg.helpers.IntegrityError: Data integrity error: Segment entry data short read [segment 391, offset 1590]: expected 1890652, got 2497 bytes
> 
> Platform: Linux thor 4.8.0-1-amd64 #1 SMP Debian 4.8.5-1 (2016-10-28) x86_64 
> Linux: debian stretch/sid 
> Borg: 1.1.0rc2  Python: CPython 3.5.3
> PID: 31905  CWD: /data/borg-WedFriSun
> sys.argv: ['borgbackup', 'with-lock', 'mail-01.borg', 'ls']
> SSH_ORIGINAL_COMMAND: None
> 

With 1.1.0-b3 it works fine.

No errors when running check
> $ borgbackup check mail-01.borg
> Enter passphrase for key /data/borg-WedFriSun/mail-01.borg: 
>



Discussion:
hmm, does not always happen, with-lock worked fine for me here:
```
$ borg -V
borg 1.1.0rc2
$ borg with-lock repo ls -l repo
total 672
-rw------- 1 tw tw    209 Aug 14 17:34 config
drwx------ 3 tw tw   4096 Aug 14 17:34 data
-rw------- 1 tw tw     64 Aug 31 00:37 hints.43
-rw------- 1 tw tw 656458 Aug 31 00:37 index.43
-rw------- 1 tw tw    190 Aug 31 00:37 integrity.43
drwx------ 2 tw tw   4096 Aug 31 00:39 lock.exclusive
-rw------- 1 tw tw     50 Aug 31 00:39 lock.roster
-rw------- 1 tw tw     73 Aug 14 17:34 README
$ borg create repo::foo1 .
$ borg with-lock repo ls -l repo
total 672
-rw------- 1 tw tw    209 Aug 14 17:34 config
drwx------ 3 tw tw   4096 Aug 14 17:34 data
-rw------- 1 tw tw     66 Aug 31 00:40 hints.51
-rw------- 1 tw tw 656458 Aug 31 00:40 index.51
-rw------- 1 tw tw    190 Aug 31 00:40 integrity.51
drwx------ 2 tw tw   4096 Aug 31 00:40 lock.exclusive
-rw------- 1 tw tw     50 Aug 31 00:40 lock.roster
-rw------- 1 tw tw     73 Aug 14 17:34 README
```

btw, this is (again) a case where having a separate header (without data) crc would have been nice.

as we only have a crc for header+data, we need to rely on correctness of the length information from the header, then try to read that length and only when successfull, we can check the crc.

we already have a ticket for that.
I can't reproduce. @magma1447 does it still happen on your system?
If it still happens: reproduce the problem and give the current traceback.

Also give `ls repo_dir/index.*` (quick) or `borg check --debug --repository-only repo` (depends on repo size).
@ThomasWaldmann I will leave tomorrow morning for Czech Republic, so if I am not responsive before next week, you know why.

I had issues with mail-01.borg yesterday, that seems to work fine now. I then tried all with a for loop over them. The first one failed.

UPDATE: I do however have 3 mail-01.borg (rotating depending on weekday), not sure which I tried on yesterday. Might have been one of the others.

Not sure if you wanted the ls with or without lock
```
$ ls db-xbmc-01.borg/index.* -l
-rw------- 1 borg root 82138 Aug 31 10:33 db-xbmc-01.borg/index.633
```

```
$ /usr/local/bin/borgbackup-1.1.0-rc2 with-lock db-xbmc-01.borg ls db-xbmc-01.borg/index.*
ls: cannot access 'db-xbmc-01.borg/index.631': No such file or directory
Data integrity error: Segment entry data short read [segment 634, offset 8]:
     expected 966583, got 4079 bytes
Traceback (most recent call last):
  File "borg/archiver.py", line 3968, in main
  File "borg/archiver.py", line 3896, in run
  File "borg/archiver.py", line 148, in wrapper
  File "borg/archiver.py", line 1677, in do_with_lock
  File "borg/repository.py", line 411, in commit
  File "borg/repository.py", line 685, in compact_segments
  File "borg/repository.py", line 1292, in iter_objects
  File "borg/repository.py", line 1373, in _read
borg.helpers.IntegrityError: Data integrity error:
     Segment entry data short read [segment 634, offset 8]: expected 966583, got 4079 bytes

Platform: Linux thor 4.8.0-1-amd64 #1 SMP Debian 4.8.5-1 (2016-10-28) x86_64 
Linux: debian stretch/sid 
Borg: 1.1.0rc2  Python: CPython 3.5.3
PID: 5617  CWD: /data/borg-TueSat
sys.argv: ['/usr/local/bin/borgbackup-1.1.0-rc2', 'with-lock',
     'db-xbmc-01.borg', 'ls', 'db-xbmc-01.borg/index.631']
SSH_ORIGINAL_COMMAND: None
```

Also ran the check, without with-lock

```
$ /usr/local/bin/borgbackup-1.1.0-rc2 check --debug --repository-only db-xbmc-01.borg
using builtin fallback logging configuration
35 self tests completed in 0.09 seconds
Starting repository check
Verified integrity of /data/borg-TueSat/db-xbmc-01.borg/index.633
Verified integrity of /data/borg-TueSat/db-xbmc-01.borg/hints.633
Verified integrity of /data/borg-TueSat/db-xbmc-01.borg/index.635
Read committed index of transaction 635
Segment transaction is    635
Determined transaction is 635
Found 326 segments
Starting repository index check
Completed repository check, no problems found.
```

Interesting: check verifies integrity of `index.633` and `index.635`. why do we have 2? bug? feature?

Guess that segment 635 will be a small commit-only segment, so segment 634 is the last "real" segment (with data). And it blew up in 634.
BTW, issue does not happen in b3 because there we still have `rollback()` at the end of `do_with_lock` - but we needed to change that to `commit()` for other reasons.
@magma1447 just to avoid chasing bugs already fixed, see #1867:

Does the repo in question get rsynced (or similar) using `borg with-lock` of borg < 1.1.0rc2?

If so, please upgrade borg everywhere and at least once use rsync in a way that makes sure files have at least same size in the end (so, *not* `--ignore-existing` in case you used that). Then try to reproduce the issue.
@ThomasWaldmann It does get a regular borg with-lock rsync ... with the repo as a **source**. This is the original repo. So rsync in itself ain't writing to it, only borg with-lock. That lock has been with version < 1.1.0-rc2.

Is that what you asked for? And since it's my source, I assume your second paragraph isn't relevant? Or did I misunderstand?
oops, seems i got the bug number wrong in 9fc4d00.

@magma1447 using this repo as a src (only) for rsync sounds unproblematic (IF the borg version is 1.1.0rc2).

Can you try the following?:
- create a new backup in the repo (using rc2)
- retry `borg with-lock ...` (using rc2)
(and stop using old borg with `with-lock`)

@ThomasWaldmann My first initial test shows that it works. That's only 1 of 3*17 repositories tested though. I will upgrade to -rc2 again and test them all. It will take a a few days for all repositories to get new backups added to them.

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -1176,6 +1176,8 @@ def cleanup(self, transaction_id):
         count = 0
         for segment, filename in self.segment_iterator(reverse=True):
             if segment > transaction_id:
+                if segment in self.fds:
+                    del self.fds[segment]
                 truncate_and_unlink(filename)
                 count += 1
             else:
@@ -1232,6 +1234,12 @@ def get_write_fd(self, no_new=False, raise_full=False):
             self._write_fd = SyncFile(self.segment_filename(self.segment), binary=True)
             self._write_fd.write(MAGIC)
             self.offset = MAGIC_LEN
+            if self.segment in self.fds:
+                # we may have a cached fd for a segment file we already deleted and
+                # we are writing now a new segment file to same file name. get rid of
+                # of the cached fd that still refers to the old file, so it will later
+                # get repopulated (on demand) with a fd that refers to the new file.
+                del self.fds[self.segment]
         return self._write_fd
 
     def get_fd(self, segment):
diff --git a/src/borg/testsuite/repository.py b/src/borg/testsuite/repository.py
--- a/src/borg/testsuite/repository.py
+++ b/src/borg/testsuite/repository.py
@@ -210,6 +210,23 @@ def test_sparse_delete(self):
         self.repository.commit()
         assert 0 not in [segment for segment, _ in self.repository.io.segment_iterator()]
 
+    def test_uncommitted_garbage(self):
+        # uncommitted garbage should be no problem, it is cleaned up automatically.
+        # we just have to be careful with invalidation of cached FDs in LoggedIO.
+        self.repository.put(H(0), b'foo')
+        self.repository.commit()
+        # write some crap to a uncommitted segment file
+        last_segment = self.repository.io.get_latest_segment()
+        with open(self.repository.io.segment_filename(last_segment + 1), 'wb') as f:
+            f.write(MAGIC + b'crapcrapcrap')
+        self.repository.close()
+        # usually, opening the repo and starting a transaction should trigger a cleanup.
+        self.repository = self.open()
+        with self.repository:
+            self.repository.put(H(0), b'bar')  # this may trigger compact_segments()
+            self.repository.commit()
+        # the point here is that nothing blows up with an exception.
+
 
 class RepositoryCommitTestCase(RepositoryTestCaseBase):
 


Test patch: