Repository: borgbackup/borg

Issue:
race condition, when calling borg init first time for two repositories simultaneously
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

borg 1.1.15

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

#### Hardware / network configuration, and filesystems used.

It's on a cloud system, using ext4. The bug is probably unrelated to the hardware.

#### How much data is handled by borg?

None, bug happens at `init` time.

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

    BORG_PASSCOMMAND="cat borgpasswd-client1.pwd" borg init --encryption=keyfile ssh://backupstore-key/./client1

`backupstore-key` is a ssh configuration in `.ssh/config`:

```
Host backupstore-key
  Hostname u0000.server.de
  Port 23
  User u0000
  IdentityFile ~/.ssh/backup-host00
  IdentitiesOnly yes
```

## Describe the problem you're observing.

The host has multiple client VMs using LXC/LXD, which are all backuped with borg running from the host. When initializing the host with *Ansible* the above `borg init` is run for every client. Due to the scripted nature of Ansible this happens - more or less - simultaneously. Sometimes Ansible stops with an exception:

    FileExistsError: [Errno 17] File exists: '/root/.config/borg'

This is on a freshly installed system that is setup from scratch with two client VMs. My sight on what happens is, as the two `borg init` commands are issued shortly after another, both find the `.config/borg/` directory missing, so both try to create it. This obvoiusly leads to one process creating the directory first and the second trying to create it again with the already existing directory. Apparently python does not like that. It crashes with the followin stack trace:

```
Local Exception
Traceback (most recent call last):
  File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 4621, in main
    exit_code = archiver.run(args)
  File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 4553, in run
    return set_ec(func(args))
  File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File \"/usr/lib/python3/dist-packages/borg/archiver.py\", line 291, in do_init
    key = key_creator(repository, args)
  File \"/usr/lib/python3/dist-packages/borg/crypto/key.py\", line 110, in key_creator
    return key.create(repository, args)
  File \"/usr/lib/python3/dist-packages/borg/crypto/key.py\", line 688, in create
    key.init_ciphers()
  File \"/usr/lib/python3/dist-packages/borg/crypto/key.py\", line 419, in init_ciphers
    self.nonce_manager = NonceManager(self.repository, self.enc_cipher, manifest_nonce)
  File \"/usr/lib/python3/dist-packages/borg/crypto/nonces.py\", line 22, in __init__
    self.nonce_file = os.path.join(get_security_dir(self.repository.id_str), 'nonce')
  File \"/usr/lib/python3/dist-packages/borg/helpers.py\", line 542, in get_security_dir
    security_dir = os.environ.get('BORG_SECURITY_DIR', os.path.join(get_config_dir(), 'security'))
  File \"/usr/lib/python3/dist-packages/borg/helpers.py\", line 585, in get_config_dir
    os.makedirs(config_dir)
  File \"/usr/lib/python3.8/os.py\", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/root/.config/borg'

Platform: Linux host00 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64
Linux: Unknown Linux  
Borg: 1.1.15  Python: CPython 3.8.10 msgpack: 0.5.6
PID: 94179  CWD: /root
sys.argv: ['/usr/bin/borg', 'init', '--encryption=keyfile', 'ssh://backupstore-key/./client1']
SSH_ORIGINAL_COMMAND: Non
```

The Ansible message, (omitting the stack trace, for brevity) is:

```
{
  "changed": true,
  "cmd": "BORG_PASSCOMMAND=\"cat .borgpasswd/client1.pwd\" borg init --encryption=keyfile ssh://backupstore-key/./client1",
  "delta": "0:00:01.983646",
  "end": "2021-07-26 00:33:58.498176",
  "msg": "non-zero return code",
  "rc": 2,
  "start": "2021-07-26 00:33:56.514530",
  "stderr": "Local Exception\nTraceback (most recent call last):...  [truncated, stack trace]",
  "stderr_lines": [
    "Local Exception",
    "Traceback (most recent call last):",
    [truncated, stack trace]
    "Platform: Linux host00 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64",
    "Linux: Unknown Linux  ",
    "Borg: 1.1.15  Python: CPython 3.8.10 msgpack: 0.5.6",
    "PID: 94179  CWD: /root",
    "sys.argv: ['/usr/bin/borg', 'init', '--encryption=keyfile', 'ssh://backupstore-key/./client1']",
    "SSH_ORIGINAL_COMMAND: None"
  ],
  "stdout": "",
  "stdout_lines": []
}
```

I solved the problem on the outside by setting the Ansible switch `serial` to 1, thus serializing the calls to `borg init`. While this works for me, it does not fix the problem itself.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Due to the nature of the problem it's hard to reproduce. It happens only randomly, when the timing is just right.


Discussion:
https://docs.python.org/3/library/os.html#os.makedirs seems like we need `exist_ok=True`.
https://github.com/borgbackup/borg/blob/1.1-maint/src/borg/helpers.py#L528

I guess this is already fixed if you use a current borg 1.1.x release (like 1.1.17), see #5216.

For Ubuntu you could try the package from the maintainer's PPA, which is usually more recent than the one in the stable release. As it is usually based on a newer release from us, there should be less bugs in that.

https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/borgbackup
Considering the above, I am closing this. If you still have this issue with borg 1.1.17, please re-open.
Tnx, I'm pretty sure this fixes my current issue. I'm probably the only one who came across this in real life anyway.

I dug into the code a little, to check if there are any possible similar issues, as init does some inter-repository work, the first time it is called. Unfortunately I'm more a C++/C# guy but your Python code is quite readable nevertheless.

There seems to be a similar "if not exists then write a file" issue in `get_cache_dir` line 96:

https://github.com/borgbackup/borg/blob/450ab38e4227773db83ad45edf5d5e8df7d4e1b8/src/borg/helpers/fs.py#L96

```python
cache_fn = os.path.join(cache_dir, CACHE_TAG_NAME)
if not os.path.exists(cache_fn):
    with open(cache_fn, 'wb') as fd:
        fd.write(CACHE_TAG_CONTENTS)
        fd.write(textwrap.dedent("""
        # This file is a cache directory tag created by Borg.
        # For information about cache directory tags, see:
        #       http://www.bford.info/cachedir/spec.html
        """).encode('ascii'))
return cache_dir
```

I'm not sure of the implications here, most probably it would just corrupt the cache tag file, but might as well crash. Is it possible to do a similar trick to `ensure_dir` like one would in `C`, open the file in create mode and if was already existing, don't do anything?

I only glanced over the other parts, of the init code, but everything that's repo specific should not be a problem I guess.
@hmb good catch! yeah, that place looks racy.

Should use this code from there:

https://github.com/borgbackup/borg/blob/5543e3b74b8d69f2300629b4059eef3d3e08857a/src/borg/platform/base.py#L187
@hmb see #6056 - guess this improves the situation a bit.

But I noticed that the `SaveFile` code always uses the same temporary file, so that's not pretty. OTOH, if multiple processes run that code in parallel, the last one would still succeed writing the correct content to the file.

Hopefully this is good enough, see above. Closing for now.

Resolving patch:
diff --git a/src/borg/helpers/fs.py b/src/borg/helpers/fs.py
--- a/src/borg/helpers/fs.py
+++ b/src/borg/helpers/fs.py
@@ -100,14 +100,8 @@ def get_cache_dir():
         #       http://www.bford.info/cachedir/spec.html
         """).encode('ascii')
         from ..platform import SaveFile
-        try:
-            with SaveFile(cache_tag_fn, binary=True) as fd:
-                fd.write(cache_tag_contents)
-        except FileExistsError:
-            # if we have multiple SaveFile calls running in parallel for same cache_tag_fn,
-            # it is fine if just one (usually first/quicker one) of them run gets through
-            # and all others raise FileExistsError.
-            pass
+        with SaveFile(cache_tag_fn, binary=True) as fd:
+            fd.write(cache_tag_contents)
     return cache_dir
 
 
diff --git a/src/borg/platform/base.py b/src/borg/platform/base.py
--- a/src/borg/platform/base.py
+++ b/src/borg/platform/base.py
@@ -1,6 +1,7 @@
 import errno
 import os
 import socket
+import tempfile
 import uuid
 
 from borg.helpers import safe_unlink
@@ -147,10 +148,22 @@ class SyncFile:
     TODO: A Windows implementation should use CreateFile with FILE_FLAG_WRITE_THROUGH.
     """
 
-    def __init__(self, path, binary=False):
+    def __init__(self, path, *, fd=None, binary=False):
+        """
+        Open a SyncFile.
+
+        :param path: full path/filename
+        :param fd: additionally to path, it is possible to give an already open OS-level fd
+               that corresponds to path (like from os.open(path, ...) or os.mkstemp(...))
+        :param binary: whether to open in binary mode, default is False.
+        """
         mode = 'xb' if binary else 'x'  # x -> raise FileExists exception in open() if file exists already
-        self.fd = open(path, mode)
-        self.fileno = self.fd.fileno()
+        self.path = path
+        if fd is None:
+            self.f = open(path, mode=mode)  # python file object
+        else:
+            self.f = os.fdopen(fd, mode=mode)
+        self.fd = self.f.fileno()  # OS-level fd
 
     def __enter__(self):
         return self
@@ -159,7 +172,7 @@ def __exit__(self, exc_type, exc_val, exc_tb):
         self.close()
 
     def write(self, data):
-        self.fd.write(data)
+        self.f.write(data)
 
     def sync(self):
         """
@@ -167,21 +180,21 @@ def sync(self):
         after sync().
         """
         from .. import platform
-        self.fd.flush()
-        platform.fdatasync(self.fileno)
+        self.f.flush()
+        platform.fdatasync(self.fd)
         # tell the OS that it does not need to cache what we just wrote,
         # avoids spoiling the cache for the OS and other processes.
-        safe_fadvise(self.fileno, 0, 0, 'DONTNEED')
+        safe_fadvise(self.fd, 0, 0, 'DONTNEED')
 
     def close(self):
         """sync() and close."""
         from .. import platform
         dirname = None
         try:
-            dirname = os.path.dirname(self.fd.name)
+            dirname = os.path.dirname(self.path)
             self.sync()
         finally:
-            self.fd.close()
+            self.f.close()
             if dirname:
                 platform.sync_dir(dirname)
 
@@ -196,36 +209,41 @@ class SaveFile:
     atomically and won't become corrupted, even on power failures or
     crashes (for caveats see SyncFile).
 
-    Calling SaveFile(path) in parallel for same path is safe (even when using the same SUFFIX), but the
-    caller needs to catch potential FileExistsError exceptions that may happen in this racy situation.
-    The caller executing SaveFile->SyncFile->open() first will win.
-    All other callers will raise a FileExistsError in open(), at least until the os.replace is executed.
+    SaveFile can safely by used in parallel (e.g. by multiple processes) to write
+    to the same target path. Whatever writer finishes last (executes the os.replace
+    last) "wins" and has successfully written its content to the target path.
+    Internally used temporary files are created in the target directory and are
+    named <BASENAME>-<RANDOMCHARS>.tmp and cleaned up in normal and error conditions.
     """
-
-    SUFFIX = '.tmp'
-
     def __init__(self, path, binary=False):
         self.binary = binary
         self.path = path
-        self.tmppath = self.path + self.SUFFIX
+        self.dir = os.path.dirname(path)
+        self.tmp_prefix = os.path.basename(path) + '-'
+        self.tmp_fd = None  # OS-level fd
+        self.tmp_fname = None  # full path/filename corresponding to self.tmp_fd
+        self.f = None  # python-file-like SyncFile
 
     def __enter__(self):
         from .. import platform
-        try:
-            safe_unlink(self.tmppath)
-        except FileNotFoundError:
-            pass
-        self.fd = platform.SyncFile(self.tmppath, self.binary)
-        return self.fd
+        self.tmp_fd, self.tmp_fname = tempfile.mkstemp(prefix=self.tmp_prefix, suffix='.tmp', dir=self.dir)
+        self.f = platform.SyncFile(self.tmp_fname, fd=self.tmp_fd, binary=self.binary)
+        return self.f
 
     def __exit__(self, exc_type, exc_val, exc_tb):
         from .. import platform
-        self.fd.close()
+        self.f.close()  # this indirectly also closes self.tmp_fd
+        self.tmp_fd = None
         if exc_type is not None:
-            safe_unlink(self.tmppath)
-            return
-        os.replace(self.tmppath, self.path)
-        platform.sync_dir(os.path.dirname(self.path))
+            safe_unlink(self.tmp_fname)  # with-body has failed, clean up tmp file
+            return  # continue processing the exception normally
+        try:
+            os.replace(self.tmp_fname, self.path)  # POSIX: atomic rename
+        except OSError:
+            safe_unlink(self.tmp_fname)  # rename has failed, clean up tmp file
+            raise
+        finally:
+            platform.sync_dir(self.dir)
 
 
 def swidth(s):


Test patch: