Repository: borgbackup/borg

Issue:
Power-loss during backup corrupts repository index
Apparently my backup host lost power in the middle of backup today,.

Aside from understandably stuck lock the repo itself is corrupted, all operations are failing with:

```
hashindex: /backup1/mordor/index.146096: fread header failed (expected 18, got 0)
```

File index.146096 does indeed exist and is of zero size.

I am trying borg check --repair now to see if that helps, but in general data creating code seems to need an audit to ensure atomicity of transactions at the very least.

The setup is:
Server running borg 1.0.1 ("binary package" from releases page) in serve mode, client is running borg 1.0.3 from Fedora23. x86 Linux on both ends (centos7.2 on the server). ssh is the pipe to communicate.



Discussion:
The repository isn't corrupted, it's the index that is truncated. I added some code to master a while back that automatically handles this kind of situation. Delete the offending index+hints file and it should recover with no further problems (which is what the code in master also does, just without the user-annoyance :)

If that solves the problem then I think this can be closed (since fixed in master and no real "_lochfraß_" bug, just annyoing)

If the repository cannot be accessed with an error and to clear the error you need to perform some recovery (manual) steps, then it is corrupted in my book.

It's interestign that if the zero-sized file is removed, it's automatically rebuilt without a hitch even with 1.0.3
So it sounds like detecting a zero-sized file and deleting it should be a good workaround?

On a different note, I use ext4 in ordered journal mode. Looking at the code and at strace for the sequence of events, the truncated file should not be possible at all. I wonder if there's an ext4 bug at play.

So, apparently my understanding of ext4 had a gap - the delayed allocation did not come into the picture, meaning the dirty pages were not part of transaction commit.
Here's a longish commend from Ted Tso  from 2009th explaining the problem:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45

I guess doing fdatasync on the index file before rename or doing a hackish
"create index.tmp2, create and write index.tmp; rename index.tmp into index.tmp2 ; rename index.tmp2 (to force the allocation) into proper Index.XXX." workaround are too messy for the problem at hand.
Instead it looks like just testing that the index file is zero length and removing it is the most straightforward solution here.

I guess we could discuss whether a backport of the changeset I mentioned above (#858) to 1.0-maint for release with 1.0.4 is worthwhile (or if documenting it is enough / needed). Otherwise this can be closed I think.

I clarified the issue title that we are talking about the repo index (not the data in the repo), so while it is inconvenient to currently have to fix that manually, past backups are still safe.

So I tried a somewhat straightforward fix of:

```
--- a/borg/repository.py
+++ b/borg/repository.py
@@ -129,7 +129,7 @@ class Repository:
         shutil.rmtree(self.path)

     def get_index_transaction_id(self):
-        indices = sorted((int(name[6:]) for name in os.listdir(self.path) if name.startswith('index.') and name[6:].isdigit()))
+        indices = sorted((int(name[6:]) for name in os.listdir(self.path) if name.startswith('index.') and name[6:].isdigit()) and os.stat(os.path.join(self.path, name)).st_size != 0)
         if indices:
             return indices[-1]
         else:
```

But because my python-fu is weak, that gives me a compile error.
I suspect a trivial fix like that is all that's needed for this ticket here (or an additional extra light symmetric addition in the "prune the old indexes) loop in write_index().


Resolving patch:
diff --git a/borg/repository.py b/borg/repository.py
--- a/borg/repository.py
+++ b/borg/repository.py
@@ -129,7 +129,9 @@ def destroy(self):
         shutil.rmtree(self.path)
 
     def get_index_transaction_id(self):
-        indices = sorted((int(name[6:]) for name in os.listdir(self.path) if name.startswith('index.') and name[6:].isdigit()))
+        indices = sorted(int(fn[6:])
+                         for fn in os.listdir(self.path)
+                         if fn.startswith('index.') and fn[6:].isdigit() and os.stat(os.path.join(self.path, fn)).st_size != 0)
         if indices:
             return indices[-1]
         else:


Test patch: