Repository: borgbackup/borg

Issue:
with-lock crashes with 1.1.0-rc2
I just upgraded from borgbackup-1.1.0-b3 to rc2, found an issue within minutes.

A simple example, borgbackup is version 1.1.0-rc2
> $ borgbackup --version
borgbackup 1.1.0rc2
> $ borgbackup with-lock mail-01.borg ls
[Files are listed here]
> Data integrity error: Segment entry data short read [segment 391, offset 1590]: expected 1890652, got 2497 bytes
> Traceback (most recent call last):
>   File "borg/archiver.py", line 3968, in main
>   File "borg/archiver.py", line 3896, in run
>   File "borg/archiver.py", line 148, in wrapper
>   File "borg/archiver.py", line 1677, in do_with_lock
>   File "borg/repository.py", line 411, in commit
>   File "borg/repository.py", line 685, in compact_segments
>   File "borg/repository.py", line 1292, in iter_objects
>   File "borg/repository.py", line 1373, in _read
> borg.helpers.IntegrityError: Data integrity error: Segment entry data short read [segment 391, offset 1590]: expected 1890652, got 2497 bytes
> 
> Platform: Linux thor 4.8.0-1-amd64 #1 SMP Debian 4.8.5-1 (2016-10-28) x86_64 
> Linux: debian stretch/sid 
> Borg: 1.1.0rc2  Python: CPython 3.5.3
> PID: 31905  CWD: /data/borg-WedFriSun
> sys.argv: ['borgbackup', 'with-lock', 'mail-01.borg', 'ls']
> SSH_ORIGINAL_COMMAND: None
> 

With 1.1.0-b3 it works fine.

No errors when running check
> $ borgbackup check mail-01.borg
> Enter passphrase for key /data/borg-WedFriSun/mail-01.borg: 
>



Discussion:
hmm, does not always happen, with-lock worked fine for me here:
```
$ borg -V
borg 1.1.0rc2
$ borg with-lock repo ls -l repo
total 672
-rw------- 1 tw tw    209 Aug 14 17:34 config
drwx------ 3 tw tw   4096 Aug 14 17:34 data
-rw------- 1 tw tw     64 Aug 31 00:37 hints.43
-rw------- 1 tw tw 656458 Aug 31 00:37 index.43
-rw------- 1 tw tw    190 Aug 31 00:37 integrity.43
drwx------ 2 tw tw   4096 Aug 31 00:39 lock.exclusive
-rw------- 1 tw tw     50 Aug 31 00:39 lock.roster
-rw------- 1 tw tw     73 Aug 14 17:34 README
$ borg create repo::foo1 .
$ borg with-lock repo ls -l repo
total 672
-rw------- 1 tw tw    209 Aug 14 17:34 config
drwx------ 3 tw tw   4096 Aug 14 17:34 data
-rw------- 1 tw tw     66 Aug 31 00:40 hints.51
-rw------- 1 tw tw 656458 Aug 31 00:40 index.51
-rw------- 1 tw tw    190 Aug 31 00:40 integrity.51
drwx------ 2 tw tw   4096 Aug 31 00:40 lock.exclusive
-rw------- 1 tw tw     50 Aug 31 00:40 lock.roster
-rw------- 1 tw tw     73 Aug 14 17:34 README
```

btw, this is (again) a case where having a separate header (without data) crc would have been nice.

as we only have a crc for header+data, we need to rely on correctness of the length information from the header, then try to read that length and only when successfull, we can check the crc.

we already have a ticket for that.
I can't reproduce. @magma1447 does it still happen on your system?
If it still happens: reproduce the problem and give the current traceback.

Also give `ls repo_dir/index.*` (quick) or `borg check --debug --repository-only repo` (depends on repo size).
@ThomasWaldmann I will leave tomorrow morning for Czech Republic, so if I am not responsive before next week, you know why.

I had issues with mail-01.borg yesterday, that seems to work fine now. I then tried all with a for loop over them. The first one failed.

UPDATE: I do however have 3 mail-01.borg (rotating depending on weekday), not sure which I tried on yesterday. Might have been one of the others.

Not sure if you wanted the ls with or without lock
```
$ ls db-xbmc-01.borg/index.* -l
-rw------- 1 borg root 82138 Aug 31 10:33 db-xbmc-01.borg/index.633
```

```
$ /usr/local/bin/borgbackup-1.1.0-rc2 with-lock db-xbmc-01.borg ls db-xbmc-01.borg/index.*
ls: cannot access 'db-xbmc-01.borg/index.631': No such file or directory
Data integrity error: Segment entry data short read [segment 634, offset 8]:
     expected 966583, got 4079 bytes
Traceback (most recent call last):
  File "borg/archiver.py", line 3968, in main
  File "borg/archiver.py", line 3896, in run
  File "borg/archiver.py", line 148, in wrapper
  File "borg/archiver.py", line 1677, in do_with_lock
  File "borg/repository.py", line 411, in commit
  File "borg/repository.py", line 685, in compact_segments
  File "borg/repository.py", line 1292, in iter_objects
  File "borg/repository.py", line 1373, in _read
borg.helpers.IntegrityError: Data integrity error:
     Segment entry data short read [segment 634, offset 8]: expected 966583, got 4079 bytes

Platform: Linux thor 4.8.0-1-amd64 #1 SMP Debian 4.8.5-1 (2016-10-28) x86_64 
Linux: debian stretch/sid 
Borg: 1.1.0rc2  Python: CPython 3.5.3
PID: 5617  CWD: /data/borg-TueSat
sys.argv: ['/usr/local/bin/borgbackup-1.1.0-rc2', 'with-lock',
     'db-xbmc-01.borg', 'ls', 'db-xbmc-01.borg/index.631']
SSH_ORIGINAL_COMMAND: None
```

Also ran the check, without with-lock

```
$ /usr/local/bin/borgbackup-1.1.0-rc2 check --debug --repository-only db-xbmc-01.borg
using builtin fallback logging configuration
35 self tests completed in 0.09 seconds
Starting repository check
Verified integrity of /data/borg-TueSat/db-xbmc-01.borg/index.633
Verified integrity of /data/borg-TueSat/db-xbmc-01.borg/hints.633
Verified integrity of /data/borg-TueSat/db-xbmc-01.borg/index.635
Read committed index of transaction 635
Segment transaction is    635
Determined transaction is 635
Found 326 segments
Starting repository index check
Completed repository check, no problems found.
```

Interesting: check verifies integrity of `index.633` and `index.635`. why do we have 2? bug? feature?

Guess that segment 635 will be a small commit-only segment, so segment 634 is the last "real" segment (with data). And it blew up in 634.
BTW, issue does not happen in b3 because there we still have `rollback()` at the end of `do_with_lock` - but we needed to change that to `commit()` for other reasons.
@magma1447 just to avoid chasing bugs already fixed, see #1867:

Does the repo in question get rsynced (or similar) using `borg with-lock` of borg < 1.1.0rc2?

If so, please upgrade borg everywhere and at least once use rsync in a way that makes sure files have at least same size in the end (so, *not* `--ignore-existing` in case you used that). Then try to reproduce the issue.

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -1,4 +1,5 @@
 import errno
+import mmap
 import os
 import shutil
 import struct
@@ -1305,25 +1306,28 @@ def iter_objects(self, segment, offset=0, include_data=False, read_data=True):
             header = fd.read(self.header_fmt.size)
 
     def recover_segment(self, segment, filename):
+        logger.info('attempting to recover ' + filename)
         if segment in self.fds:
             del self.fds[segment]
-        with open(filename, 'rb') as fd:
-            # XXX: Rather use mmap, this loads the entire segment (up to 500 MB by default) into memory.
-            data = memoryview(fd.read())
-        os.rename(filename, filename + '.beforerecover')
-        logger.info('attempting to recover ' + filename)
-        with open(filename, 'wb') as fd:
-            fd.write(MAGIC)
-            while len(data) >= self.header_fmt.size:
-                crc, size, tag = self.header_fmt.unpack(data[:self.header_fmt.size])
-                if size < self.header_fmt.size or size > len(data):
-                    data = data[1:]
-                    continue
-                if crc32(data[4:size]) & 0xffffffff != crc:
-                    data = data[1:]
-                    continue
-                fd.write(data[:size])
-                data = data[size:]
+        backup_filename = filename + '.beforerecover'
+        os.rename(filename, backup_filename)
+        with open(backup_filename, 'rb') as backup_fd:
+            # note: file must not be 0 size (windows can't create 0 size mapping)
+            with mmap.mmap(backup_fd.fileno(), 0, access=mmap.ACCESS_READ) as mm:
+                data = memoryview(mm)
+                with open(filename, 'wb') as fd:
+                    fd.write(MAGIC)
+                    while len(data) >= self.header_fmt.size:
+                        crc, size, tag = self.header_fmt.unpack(data[:self.header_fmt.size])
+                        if size < self.header_fmt.size or size > len(data):
+                            data = data[1:]
+                            continue
+                        if crc32(data[4:size]) & 0xffffffff != crc:
+                            data = data[1:]
+                            continue
+                        fd.write(data[:size])
+                        data = data[size:]
+                data.release()
 
     def read(self, segment, offset, id, read_data=True):
         """


Test patch: