Repository: borgbackup/borg

Issue:
borg with-lock being incompatible with rsync --ignore-existing
I'm planning on doing a backup in borg on a server that is configured to `append-only`.
The backup from this server will then be cloned to a second (off-site) server regularly, using rsync with `ignore-existing`. The aim of this is to keep a copy all files, even if they get changed/deleted on the local backup server.

So, in my scenario, any file will only be synced once and all modifications or deletions won't get synced with the remote server. Will this cause any trouble for borg? I understand that the transaction log won't be synced, but I can live with that if it doesn't cause any problems for backup integrity.



Discussion:
> Will this cause any trouble for borg?

No. Segments are only written to once. After that they will only be deleted or not modified at all.

When doing this it should be ensured that the last operation completed successfully, so that you can't get into the situation where uncommitted segments are synced to the replica. Borg 1.1.x `with-lock` ensures that.

>  I understand that the transaction log won't be synced, but I can live with that if it doesn't cause any problems for backup integrity.

From Borgs point of view it's just a text file that is only appended to, never read back.

It might be useful to have it on the replica, too, perhaps there is some rsync option to only copy appended parts or something like that.

How should `with-lock` ensure that no uncommitted segments get synced?

If `borg create` gets interrupted, our cleanup code will usually remove the lock, but there will be uncommitted segments.

with-lock has that PUT/GET manifest thing going on, which will trigger a transaction on the repository, cleaning leftovers up. I think the reason for it was a different one though, so maybe we want to explain this other reason as well in the code.

hmm, I've seen that rather as a implementation detail caused by compatibility requirements of old servers (IIRC), not as a feature.

Actually the PUT and following rollback would mean that the segment created by the PUT is uncommitted and will later contain different data in the next transaction. So we need to repository.commit() in with_lock, not .rollback().

Special case local repositories here to avoid that? It would seem to me that the typical use of with-lock is only with local repositories anyway.

OK, so on <1.1.0 I just have to trigger the sync manually after the last backup has completed successfully.

So I didn't fully understand the essence of that `with-lock` discussion. Will the upgrade to 1.1.x ensure I can do my rsync thing?

Also, are the segments terminated in a defined way? Then I could use the rsync switch `--append` instead of `--ignore-existing`, which would enable me to sync the updated transaction log without risking corrupted segment files.

I'm adding this to 1.1.0rc1 milestone - so we'll look at the proposed with-lock modification until then.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1666,7 +1666,13 @@ def do_with_lock(self, args, repository):
             # we exit with the return code we get from the subprocess
             return subprocess.call([args.command] + args.args)
         finally:
-            repository.rollback()
+            # we need to commit the "no change" operation we did to the manifest
+            # because it created a new segment file in the repository. if we would
+            # roll back, the same file would be later used otherwise (for other content).
+            # that would be bad if somebody uses rsync with ignore-existing (or
+            # any other mechanism relying on existing segment data not changing).
+            # see issue #1867.
+            repository.commit()
 
     def do_debug_info(self, args):
         """display system information for debugging / bug reports"""


Test patch: