Repository: borgbackup/borg

Issue:
recreate: drop the "experimental feature" warning?
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

More of a feature request, really :)

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

```
backup ~ # borg -V
borg 1.1.9
```

#### Operating system (distribution) and version.

```
backup ~ # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster
backup ~ # cat /etc/debian_version
10.3
```

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```
borg recreate --dry-run --verbose --list -e 'pp:home/some.user' '::archive...'
```

## Describe the problem you're observing.

`borg recreate` currently prints a warning indicating that the `recreate` feature is still experimental:

```
recreate is an experimental feature.
Type 'YES' if you understand this and want to continue:
```

[This warning has been in the code for 4 years](https://github.com/borgbackup/borg/blame/08a12cc40523b58bdfe92be9e9d937179287e805/src/borg/archiver.py#L1569) (which was the point where `recreate` was originally implemented). As far as I can tell from the currently open bug reports, there are currently no serious issues regarding `recreate`, and looking at the closed pull requests, most of the work on it happened back in 2017.

Is this warning still relevant today, or can it safely be removed?

Discussion:
codecov-commenter: # [Codecov](https://codecov.io/gh/borgbackup/borg/pull/5241?src=pr&el=h1) Report
> Merging [#5241](https://codecov.io/gh/borgbackup/borg/pull/5241?src=pr&el=desc) into [master](https://codecov.io/gh/borgbackup/borg/commit/5b5bd9f7f21b9f06b890906bb6b9eb47733c1772&el=desc) will **decrease** coverage by `0.45%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/borgbackup/borg/pull/5241/graphs/tree.svg?width=650&height=150&src=pr&token=LssEpmW3o4)](https://codecov.io/gh/borgbackup/borg/pull/5241?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5241      +/-   ##
==========================================
- Coverage   83.97%   83.52%   -0.46%     
==========================================
  Files          37       37              
  Lines        9873     9873              
  Branches     1647     1647              
==========================================
- Hits         8291     8246      -45     
- Misses       1103     1139      +36     
- Partials      479      488       +9     
```


| [Impacted Files](https://codecov.io/gh/borgbackup/borg/pull/5241?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/borg/archiver.py](https://codecov.io/gh/borgbackup/borg/pull/5241/diff?src=pr&el=tree#diff-c3JjL2JvcmcvYXJjaGl2ZXIucHk=) | `80.84% <100.00%> (-0.36%)` | :arrow_down: |
| [src/borg/platform/\_\_init\_\_.py](https://codecov.io/gh/borgbackup/borg/pull/5241/diff?src=pr&el=tree#diff-c3JjL2JvcmcvcGxhdGZvcm0vX19pbml0X18ucHk=) | `72.00% <0.00%> (-12.00%)` | :arrow_down: |
| [src/borg/platform/xattr.py](https://codecov.io/gh/borgbackup/borg/pull/5241/diff?src=pr&el=tree#diff-c3JjL2JvcmcvcGxhdGZvcm0veGF0dHIucHk=) | `82.53% <0.00%> (-6.35%)` | :arrow_down: |
| [src/borg/xattr.py](https://codecov.io/gh/borgbackup/borg/pull/5241/diff?src=pr&el=tree#diff-c3JjL2JvcmcveGF0dHIucHk=) | `48.83% <0.00%> (-3.49%)` | :arrow_down: |
| [src/borg/platform/base.py](https://codecov.io/gh/borgbackup/borg/pull/5241/diff?src=pr&el=tree#diff-c3JjL2JvcmcvcGxhdGZvcm0vYmFzZS5weQ==) | `77.58% <0.00%> (-3.45%)` | :arrow_down: |
| [src/borg/helpers/misc.py](https://codecov.io/gh/borgbackup/borg/pull/5241/diff?src=pr&el=tree#diff-c3JjL2JvcmcvaGVscGVycy9taXNjLnB5) | `77.02% <0.00%> (-2.03%)` | :arrow_down: |
| [src/borg/archive.py](https://codecov.io/gh/borgbackup/borg/pull/5241/diff?src=pr&el=tree#diff-c3JjL2JvcmcvYXJjaGl2ZS5weQ==) | `82.02% <0.00%> (-1.44%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/borgbackup/borg/pull/5241?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/borgbackup/borg/pull/5241?src=pr&el=footer). Last update [5b5bd9f...12a057e](https://codecov.io/gh/borgbackup/borg/pull/5241?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/docs/faq.rst b/docs/faq.rst
index 6df7b035..5585d7d0 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -894,7 +894,7 @@ I am having troubles with some network/FUSE/special filesystem, why?
 --------------------------------------------------------------------
 
 Borg is doing nothing special in the filesystem, it only uses very
-common and compatible operations (even the locking is just "mkdir").
+common and compatible operations (even the locking is just "rename").
 
 So, if you are encountering issues like slowness, corruption or malfunction
 when using a specific filesystem, please try if you can reproduce the issues
@@ -1055,7 +1055,7 @@ Here's a (incomplete) list of some major changes:
   nor the pbkdf2 iteration count in "passphrase" mode)
 * simple sparse file support, great for virtual machine disk files
 * can read special files (e.g. block devices) or from stdin, write to stdout
-* mkdir-based locking is more compatible than attic's posix locking
+* rename-based locking is more compatible than attic's posix locking
 * uses fadvise to not spoil / blow up the fs cache
 * better error messages / exception handling
 * better logging, screen output, progress indication
diff --git a/docs/internals/data-structures.rst b/docs/internals/data-structures.rst
index 241a25de..3f54da93 100644
--- a/docs/internals/data-structures.rst
+++ b/docs/internals/data-structures.rst
@@ -977,16 +977,19 @@ Lock files
 Borg uses locks to get (exclusive or shared) access to the cache and
 the repository.
 
-The locking system is based on creating a directory `lock.exclusive` (for
-exclusive locks). Inside the lock directory, there is a file indicating
+The locking system is based on renaming a temporary directory
+to `lock.exclusive` (for
+exclusive locks). Inside this directory, there is a file indicating
 hostname, process id and thread id of the lock holder.
 
 There is also a json file `lock.roster` that keeps a directory of all shared
 and exclusive lockers.
 
-If the process can create the `lock.exclusive` directory for a resource, it has
-the lock for it. If creation fails (because the directory has already been
-created by some other process), lock acquisition fails.
+If the process is able to rename a temporary directory (with the
+host/process/thread identifier prepared inside it) in the resource directory
+to `lock.exclusive`, it has the lock for it. If renaming fails
+(because this directory already exists and its host/process/thread identifier
+denotes a thread on the host which is still alive), lock acquisition fails.
 
 The cache lock is usually in `~/.cache/borg/REPOID/lock.*`.
 The repository lock is in `repository/lock.*`.
diff --git a/docs/internals/frontends.rst b/docs/internals/frontends.rst
index 1ade9058..09a943ad 100644
--- a/docs/internals/frontends.rst
+++ b/docs/internals/frontends.rst
@@ -575,8 +575,6 @@ Prompts
     BORG_RELOCATED_REPO_ACCESS_IS_OK
         For "Warning: The repository at location ... was previously located at ..."
     BORG_CHECK_I_KNOW_WHAT_I_AM_DOING
-        For "Warning: 'check --repair' is an experimental feature that might result in data loss."
+        For "This is a potentially dangerous function..." (check --repair)
     BORG_DELETE_I_KNOW_WHAT_I_AM_DOING
         For "You requested to completely DELETE the repository *including* all archives it contains:"
-    BORG_RECREATE_I_KNOW_WHAT_I_AM_DOING
-        For "recreate is an experimental feature."
diff --git a/docs/man/borg.1 b/docs/man/borg.1
index f2f69906..2cd4fb64 100644
--- a/docs/man/borg.1
+++ b/docs/man/borg.1
@@ -585,7 +585,10 @@ Large repositories may require large files (>2 GB).
 .IP \(bu 2
 Up to 1000 files per directory (10000 for repositories initialized with Borg 1.0)
 .IP \(bu 2
-mkdir(2) should be atomic, since it is used for locking
+rename(2) / MoveFile(Ex) should work as specified, i.e. on the same file system
+it should be a move (not a copy) operation, and in case of a directory
+it should fail if the destination exists and is not an empty directory,
+since this is used for locking
 .IP \(bu 2
 Hardlinks are needed for \fIborg_upgrade\fP \fB\-\-inplace\fP
 .UNINDENT
diff --git a/docs/usage/general/environment.rst.inc b/docs/usage/general/environment.rst.inc
index a3d3060e..87a3f84b 100644
--- a/docs/usage/general/environment.rst.inc
+++ b/docs/usage/general/environment.rst.inc
@@ -83,11 +83,9 @@ Some automatic "answerers" (if set, they automatically answer confirmation quest
     BORG_RELOCATED_REPO_ACCESS_IS_OK=no (or =yes)
         For "Warning: The repository at location ... was previously located at ..."
     BORG_CHECK_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
-        For "Warning: 'check --repair' is an experimental feature that might result in data loss."
+        For "This is a potentially dangerous function..." (check --repair)
     BORG_DELETE_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
         For "You requested to completely DELETE the repository *including* all archives it contains:"
-    BORG_RECREATE_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
-        For "recreate is an experimental feature."
 
     Note: answers are case sensitive. setting an invalid answer value might either give the default
     answer or ask you interactively, depending on whether retries are allowed (they by default are
diff --git a/docs/usage/general/file-systems.rst.inc b/docs/usage/general/file-systems.rst.inc
index 5a7d75e8..137697ca 100644
--- a/docs/usage/general/file-systems.rst.inc
+++ b/docs/usage/general/file-systems.rst.inc
@@ -22,5 +22,8 @@ and readable after one of the failures mentioned above occurred, run
 - Typically, file sizes up to a few hundred MB.
   Large repositories may require large files (>2 GB).
 - Up to 1000 files per directory (10000 for repositories initialized with Borg 1.0)
-- mkdir(2) should be atomic, since it is used for locking
+- rename(2) / MoveFile(Ex) should work as specified, i.e. on the same file system
+  it should be a move (not a copy) operation, and in case of a directory
+  it should fail if the destination exists and is not an empty directory,
+  since this is used for locking.
 - Hardlinks are needed for :ref:`borg_upgrade` ``--inplace``
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
index 6dcef8da..976faaf7 100644
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -45,7 +45,7 @@
     from .compress import CompressionSpec
     from .crypto.key import key_creator, key_argument_names, tam_required_file, tam_required, RepoKey, PassphraseKey
     from .crypto.keymanager import KeyManager
-    from .helpers import EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR
+    from .helpers import EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR, EXIT_SIGNAL_BASE
     from .helpers import Error, NoManifestError, set_ec
     from .helpers import positive_int_validator, location_validator, archivename_validator, ChunkerParams, Location
     from .helpers import PrefixSpec, GlobSpec, CommentSpec, SortBySpec, FilesCacheMode
@@ -306,8 +306,10 @@ def do_init(self, args, repository):
     def do_check(self, args, repository):
         """Check repository consistency"""
         if args.repair:
-            msg = ("'check --repair' is an experimental feature that might result in data loss." +
-                   "\n" +
+            msg = ("This is a potentially dangerous function.\n"
+                   "check --repair might lead to data loss (for kinds of corruption it is not\n"
+                   "capable of dealing with). BE VERY CAREFUL!\n"
+                   "\n"
                    "Type 'YES' if you understand this and want to continue: ")
             if not yes(msg, false_msg="Aborting.", invalid_msg="Invalid answer, aborting.",
                        truish=('YES', ), retry=False,
@@ -1587,12 +1589,6 @@ def do_upgrade(self, args, repository, manifest=None, key=None):
     @with_repository(cache=True, exclusive=True, compatibility=(Manifest.Operation.CHECK,))
     def do_recreate(self, args, repository, manifest, key, cache):
         """Re-create archives"""
-        msg = ("recreate is an experimental feature.\n"
-               "Type 'YES' if you understand this and want to continue: ")
-        if not yes(msg, false_msg="Aborting.", truish=('YES',),
-                   env_var_override='BORG_RECREATE_I_KNOW_WHAT_I_AM_DOING'):
-            return EXIT_ERROR
-
         matcher = self.build_matcher(args.patterns, args.paths)
         self.output_list = args.output_list
         self.output_filter = args.output_filter
@@ -2861,6 +2857,9 @@ def define_borg_mount(parser):
         check_epilog = process_epilog("""
         The check command verifies the consistency of a repository and the corresponding archives.
 
+        check --repair is a potentially dangerous function and might lead to data loss
+        (for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL!
+
         First, the underlying repository data files are checked:
 
         - For all segments, the segment magic header is checked.
@@ -4088,7 +4087,8 @@ def define_borg_mount(parser):
         recreate_epilog = process_epilog("""
         Recreate the contents of existing archives.
 
-        This is an *experimental* feature. Do *not* use this on your only backup.
+        recreate is a potentially dangerous function and might lead to data loss
+        (if used wrongly). BE VERY CAREFUL!
 
         Important: Repository disk space is **not** freed until you run ``borg compact``.
 
@@ -4415,27 +4415,27 @@ def get_args(self, argv, cmd):
                 # make sure we only process like normal if the client is executing
                 # the same command as specified in the forced command, otherwise
                 # just skip this block and return the forced command (== result).
-                # client is allowed to specify the whitelisted options,
+                # client is allowed to specify the allowlisted options,
                 # everything else comes from the forced "borg serve" command (or the defaults).
-                # stuff from blacklist must never be used from the client.
-                blacklist = {
+                # stuff from denylist must never be used from the client.
+                denylist = {
                     'restrict_to_paths',
                     'restrict_to_repositories',
                     'append_only',
                     'storage_quota',
                     'umask',
                 }
-                whitelist = {
+                allowlist = {
                     'debug_topics',
                     'lock_wait',
                     'log_level',
                 }
                 not_present = object()
-                for attr_name in whitelist:
-                    assert attr_name not in blacklist, 'whitelist has blacklisted attribute name %s' % attr_name
+                for attr_name in allowlist:
+                    assert attr_name not in denylist, 'allowlist has denylisted attribute name %s' % attr_name
                     value = getattr(client_result, attr_name, not_present)
                     if value is not not_present:
-                        # note: it is not possible to specify a whitelisted option via a forced command,
+                        # note: it is not possible to specify a allowlisted option via a forced command,
                         # it always gets overridden by the value specified (or defaulted to) by the client commmand.
                         setattr(result, attr_name, value)
 
@@ -4644,17 +4644,17 @@ def main():  # pragma: no cover
             msg = 'Keyboard interrupt'
             tb_log_level = logging.DEBUG
             tb = '%s\n%s' % (traceback.format_exc(), sysinfo())
-            exit_code = EXIT_ERROR
+            exit_code = EXIT_SIGNAL_BASE + 2
         except SigTerm:
             msg = 'Received SIGTERM'
             msgid = 'Signal.SIGTERM'
             tb_log_level = logging.DEBUG
             tb = '%s\n%s' % (traceback.format_exc(), sysinfo())
-            exit_code = EXIT_ERROR
+            exit_code = EXIT_SIGNAL_BASE + 15
         except SigHup:
             msg = 'Received SIGHUP.'
             msgid = 'Signal.SIGHUP'
-            exit_code = EXIT_ERROR
+            exit_code = EXIT_SIGNAL_BASE + 1
         if msg:
             logger.error(msg, msgid=msgid)
         if tb:
@@ -4668,6 +4668,8 @@ def main():  # pragma: no cover
                 rc_logger.warning(exit_msg % ('warning', exit_code))
             elif exit_code == EXIT_ERROR:
                 rc_logger.error(exit_msg % ('error', exit_code))
+            elif exit_code >= EXIT_SIGNAL_BASE:
+                rc_logger.error(exit_msg % ('signal', exit_code))
             else:
                 rc_logger.error(exit_msg % ('abnormal', exit_code or 666))
         sys.exit(exit_code)
diff --git a/src/borg/constants.py b/src/borg/constants.py
index a2eff43f..b265ec6b 100644
--- a/src/borg/constants.py
+++ b/src/borg/constants.py
@@ -80,6 +80,7 @@
 EXIT_SUCCESS = 0  # everything done, no problems
 EXIT_WARNING = 1  # reached normal end of operation, but there were issues
 EXIT_ERROR = 2  # terminated abruptly, did not reach end of operation
+EXIT_SIGNAL_BASE = 128  # terminated due to signal, rc = 128 + sig_no
 
 # never use datetime.isoformat(), it is evil. always use one of these:
 # datetime.strftime(ISO_FORMAT)  # output always includes .microseconds
diff --git a/src/borg/fuse.py b/src/borg/fuse.py
index 46d491c4..bc8b59be 100644
--- a/src/borg/fuse.py
+++ b/src/borg/fuse.py
@@ -18,7 +18,7 @@
 from .archiver import Archiver
 from .archive import Archive
 from .hashindex import FuseVersionsIndex
-from .helpers import daemonize, hardlinkable, signal_handler, format_file_size
+from .helpers import daemonize, daemonizing, hardlinkable, signal_handler, format_file_size
 from .helpers import msgpack
 from .item import Item
 from .lrucache import LRUCache
@@ -510,10 +510,13 @@ def pop_option(options, key, present, not_present, wanted_type, int_base=0):
         self._create_filesystem()
         llfuse.init(self, mountpoint, options)
         if not foreground:
-            old_id, new_id = daemonize()
-            if not isinstance(self.repository_uncached, RemoteRepository):
-                # local repo and the locking process' PID just changed, migrate it:
-                self.repository_uncached.migrate_lock(old_id, new_id)
+            if isinstance(self.repository_uncached, RemoteRepository):
+                daemonize()
+            else:
+                with daemonizing() as (old_id, new_id):
+                    # local repo: the locking process' PID is changing, migrate it:
+                    logger.debug('fuse: mount local repo, going to background: migrating lock.')
+                    self.repository_uncached.migrate_lock(old_id, new_id)
 
         # If the file system crashes, we do not want to umount because in that
         # case the mountpoint suddenly appears to become empty. This can have
diff --git a/src/borg/helpers/msgpack.py b/src/borg/helpers/msgpack.py
index 10d0e222..84c1301f 100644
--- a/src/borg/helpers/msgpack.py
+++ b/src/borg/helpers/msgpack.py
@@ -183,7 +183,7 @@ def is_supported_msgpack():
     # DO NOT CHANGE OR REMOVE! See also requirements and comments in setup.py.
     import msgpack
     return (0, 5, 6) <= msgpack.version <= (1, 0, 0) and \
-           msgpack.version not in []  # < blacklist bad releases here
+           msgpack.version not in []  # < add bad releases here to deny list
 
 
 def get_limited_unpacker(kind):
diff --git a/src/borg/helpers/parseformat.py b/src/borg/helpers/parseformat.py
index 0e58ca0a..7331fc99 100644
--- a/src/borg/helpers/parseformat.py
+++ b/src/borg/helpers/parseformat.py
@@ -271,16 +271,17 @@ def parse_file_size(s):
 
 
 def sizeof_fmt(num, suffix='B', units=None, power=None, sep='', precision=2, sign=False):
-    prefix = '+' if sign and num > 0 else ''
-
+    sign = '+' if sign and num > 0 else ''
+    fmt = '{0:{1}.{2}f}{3}{4}{5}'
+    prec = 0
     for unit in units[:-1]:
         if abs(round(num, precision)) < power:
-            if isinstance(num, int):
-                return "{}{}{}{}{}".format(prefix, num, sep, unit, suffix)
-            else:
-                return "{}{:3.{}f}{}{}{}".format(prefix, num, precision, sep, unit, suffix)
+            break
         num /= float(power)
-    return "{}{:.{}f}{}{}{}".format(prefix, num, precision, sep, units[-1], suffix)
+        prec = precision
+    else:
+        unit = units[-1]
+    return fmt.format(num, sign, prec, sep, unit, suffix)
 
 
 def sizeof_fmt_iec(num, suffix='B', sep='', precision=2, sign=False):
diff --git a/src/borg/helpers/process.py b/src/borg/helpers/process.py
index 763a8c68..2e61ad0a 100644
--- a/src/borg/helpers/process.py
+++ b/src/borg/helpers/process.py
@@ -6,6 +6,8 @@
 import signal
 import subprocess
 import sys
+import time
+import traceback
 
 from .. import __version__
 
@@ -13,17 +15,23 @@
 from ..logger import create_logger
 logger = create_logger()
 
+from ..helpers import EXIT_SUCCESS, EXIT_WARNING, EXIT_SIGNAL_BASE
 
-def daemonize():
-    """Detach process from controlling terminal and run in background
 
-    Returns: old and new get_process_id tuples
-    """
+@contextlib.contextmanager
+def _daemonize():
     from ..platform import get_process_id
     old_id = get_process_id()
     pid = os.fork()
     if pid:
-        os._exit(0)
+        exit_code = EXIT_SUCCESS
+        try:
+            yield old_id, None
+        except _ExitCodeException as e:
+            exit_code = e.exit_code
+        finally:
+            logger.debug('Daemonizing: Foreground process (%s, %s, %s) is now dying.' % old_id)
+            os._exit(exit_code)
     os.setsid()
     pid = os.fork()
     if pid:
@@ -31,13 +39,101 @@ def daemonize():
     os.chdir('/')
     os.close(0)
     os.close(1)
-    os.close(2)
     fd = os.open(os.devnull, os.O_RDWR)
     os.dup2(fd, 0)
     os.dup2(fd, 1)
-    os.dup2(fd, 2)
     new_id = get_process_id()
-    return old_id, new_id
+    try:
+        yield old_id, new_id
+    finally:
+        # Close / redirect stderr to /dev/null only now
+        # for the case that we want to log something before yield returns.
+        os.close(2)
+        os.dup2(fd, 2)
+
+
+def daemonize():
+    """Detach process from controlling terminal and run in background
+
+    Returns: old and new get_process_id tuples
+    """
+    with _daemonize() as (old_id, new_id):
+        return old_id, new_id
+
+
+@contextlib.contextmanager
+def daemonizing(*, timeout=5):
+    """Like daemonize(), but as context manager.
+
+    The with-body is executed in the background process,
+    while the foreground process survives until the body is left
+    or the given timeout is exceeded. In the latter case a warning is
+    reported by the foreground.
+    Context variable is (old_id, new_id) get_process_id tuples.
+    An exception raised in the body is reported by the foreground
+    as a warning as well as propagated outside the body in the background.
+    In case of a warning, the foreground exits with exit code EXIT_WARNING
+    instead of EXIT_SUCCESS.
+    """
+    with _daemonize() as (old_id, new_id):
+        if new_id is None:
+            # The original / parent process, waiting for a signal to die.
+            logger.debug('Daemonizing: Foreground process (%s, %s, %s) is waiting for background process...' % old_id)
+            exit_code = EXIT_SUCCESS
+            # Indeed, SIGHUP and SIGTERM handlers should have been set on archiver.run(). Just in case...
+            with signal_handler('SIGINT', raising_signal_handler(KeyboardInterrupt)), \
+                 signal_handler('SIGHUP', raising_signal_handler(SigHup)), \
+                 signal_handler('SIGTERM', raising_signal_handler(SigTerm)):
+                try:
+                    if timeout > 0:
+                        time.sleep(timeout)
+                except SigTerm:
+                    # Normal termination; expected from grandchild, see 'os.kill()' below
+                    pass
+                except SigHup:
+                    # Background wants to indicate a problem; see 'os.kill()' below,
+                    # log message will come from grandchild.
+                    exit_code = EXIT_WARNING
+                except KeyboardInterrupt:
+                    # Manual termination.
+                    logger.debug('Daemonizing: Foreground process (%s, %s, %s) received SIGINT.' % old_id)
+                    exit_code = EXIT_SIGNAL_BASE + 2
+                except BaseException as e:
+                    # Just in case...
+                    logger.warning('Daemonizing: Foreground process received an exception while waiting:\n' +
+                                   ''.join(traceback.format_exception(e.__class__, e, e.__traceback__)))
+                    exit_code = EXIT_WARNING
+                else:
+                    logger.warning('Daemonizing: Background process did not respond (timeout). Is it alive?')
+                    exit_code = EXIT_WARNING
+                finally:
+                    # Don't call with-body, but die immediately!
+                    # return would be sufficient, but we want to pass the exit code.
+                    raise _ExitCodeException(exit_code)
+
+        # The background / grandchild process.
+        sig_to_foreground = signal.SIGTERM
+        logger.debug('Daemonizing: Background process (%s, %s, %s) is starting...' % new_id)
+        try:
+            yield old_id, new_id
+        except BaseException as e:
+            sig_to_foreground = signal.SIGHUP
+            logger.warning('Daemonizing: Background process raised an exception while starting:\n' +
+                           ''.join(traceback.format_exception(e.__class__, e, e.__traceback__)))
+            raise e
+        else:
+            logger.debug('Daemonizing: Background process (%s, %s, %s) has started.' % new_id)
+        finally:
+            try:
+                os.kill(old_id[1], sig_to_foreground)
+            except BaseException as e:
+                logger.error('Daemonizing: Trying to kill the foreground process raised an exception:\n' +
+                             ''.join(traceback.format_exception(e.__class__, e, e.__traceback__)))
+
+
+class _ExitCodeException(BaseException):
+    def __init__(self, exit_code):
+        self.exit_code = exit_code
 
 
 class SignalException(BaseException):
diff --git a/src/borg/locking.py b/src/borg/locking.py
index 37d1f8f7..34f5281f 100644
--- a/src/borg/locking.py
+++ b/src/borg/locking.py
@@ -1,6 +1,7 @@
 import errno
 import json
 import os
+import tempfile
 import time
 
 from . import platform
@@ -124,22 +125,42 @@ def acquire(self, timeout=None, sleep=None):
             timeout = self.timeout
         if sleep is None:
             sleep = self.sleep
-        timer = TimeoutTimer(timeout, sleep).start()
-        while True:
-            try:
-                os.mkdir(self.path)
-            except FileExistsError:  # already locked
-                if self.by_me():
+        parent_path, base_name = os.path.split(self.path)
+        unique_base_name = os.path.basename(self.unique_name)
+        temp_path = None
+        try:
+            temp_path = tempfile.mkdtemp(".tmp", base_name + '.', parent_path)
+            temp_unique_name = os.path.join(temp_path, unique_base_name)
+            with open(temp_unique_name, "wb"):
+                pass
+        except OSError as err:
+            raise LockFailed(self.path, str(err)) from None
+        else:
+            timer = TimeoutTimer(timeout, sleep).start()
+            while True:
+                try:
+                    os.rename(temp_path, self.path)
+                except OSError:  # already locked
+                    if self.by_me():
+                        return self
+                    self.kill_stale_lock()
+                    if timer.timed_out_or_sleep():
+                        raise LockTimeout(self.path) from None
+                else:
+                    temp_path = None  # see finally:-block below
                     return self
-                self.kill_stale_lock()
-                if timer.timed_out_or_sleep():
-                    raise LockTimeout(self.path)
-            except OSError as err:
-                raise LockFailed(self.path, str(err)) from None
-            else:
-                with open(self.unique_name, "wb"):
+        finally:
+            if temp_path is not None:
+                # Renaming failed for some reason, so temp_dir still exists and
+                # should be cleaned up anyway. Try to clean up, but don't crash.
+                try:
+                    os.unlink(temp_unique_name)
+                except:
+                    pass
+                try:
+                    os.rmdir(temp_path)
+                except:
                     pass
-                return self
 
     def release(self):
         if not self.is_locked():
@@ -147,7 +168,15 @@ def release(self):
         if not self.by_me():
             raise NotMyLock(self.path)
         os.unlink(self.unique_name)
-        os.rmdir(self.path)
+        try:
+            os.rmdir(self.path)
+        except OSError as err:
+            if err.errno not in (errno.ENOTEMPTY, errno.ENOENT):
+                # EACCES or EIO or ... = we cannot operate anyway, so re-throw
+                raise err
+            # else:
+            # Directory is not empty or doesn't exist any more.
+            # this means we lost the race to somebody else -- which is ok.
 
     def is_locked(self):
         return os.path.exists(self.path)
@@ -156,42 +185,47 @@ def by_me(self):
         return os.path.exists(self.unique_name)
 
     def kill_stale_lock(self):
-        for name in os.listdir(self.path):
-            try:
-                host_pid, thread_str = name.rsplit('-', 1)
-                host, pid_str = host_pid.rsplit('.', 1)
-                pid = int(pid_str)
-                thread = int(thread_str)
-            except ValueError:
-                # Malformed lock name? Or just some new format we don't understand?
-                logger.error("Found malformed lock %s in %s. Please check/fix manually.", name, self.path)
-                return False
-
-            if platform.process_alive(host, pid, thread):
-                return False
-
-            if not self.kill_stale_locks:
-                if not self.stale_warning_printed:
-                    # Log this at warning level to hint the user at the ability
-                    logger.warning("Found stale lock %s, but not deleting because self.kill_stale_locks = False.", name)
-                    self.stale_warning_printed = True
-                return False
-
-            try:
-                os.unlink(os.path.join(self.path, name))
-                logger.warning('Killed stale lock %s.', name)
-            except OSError as err:
-                if not self.stale_warning_printed:
-                    # This error will bubble up and likely result in locking failure
-                    logger.error('Found stale lock %s, but cannot delete due to %s', name, str(err))
-                    self.stale_warning_printed = True
-                return False
+        try:
+            names = os.listdir(self.path)
+        except FileNotFoundError:  # another process did our job in the meantime.
+            pass
+        else:
+            for name in names:
+                try:
+                    host_pid, thread_str = name.rsplit('-', 1)
+                    host, pid_str = host_pid.rsplit('.', 1)
+                    pid = int(pid_str)
+                    thread = int(thread_str)
+                except ValueError:
+                    # Malformed lock name? Or just some new format we don't understand?
+                    logger.error("Found malformed lock %s in %s. Please check/fix manually.", name, self.path)
+                    return False
+
+                if platform.process_alive(host, pid, thread):
+                    return False
+
+                if not self.kill_stale_locks:
+                    if not self.stale_warning_printed:
+                        # Log this at warning level to hint the user at the ability
+                        logger.warning("Found stale lock %s, but not deleting because self.kill_stale_locks = False.", name)
+                        self.stale_warning_printed = True
+                    return False
+
+                try:
+                    os.unlink(os.path.join(self.path, name))
+                    logger.warning('Killed stale lock %s.', name)
+                except OSError as err:
+                    if not self.stale_warning_printed:
+                        # This error will bubble up and likely result in locking failure
+                        logger.error('Found stale lock %s, but cannot delete due to %s', name, str(err))
+                        self.stale_warning_printed = True
+                    return False
 
         try:
             os.rmdir(self.path)
         except OSError as err:
-            if err.errno == errno.ENOTEMPTY:
-                # Directory is not empty = we lost the race to somebody else
+            if err.errno == errno.ENOTEMPTY or err.errno == errno.ENOENT:
+                # Directory is not empty or doesn't exist any more = we lost the race to somebody else--which is ok.
                 return False
             # EACCES or EIO or ... = we cannot operate anyway
             logger.error('Failed to remove lock dir: %s', str(err))
diff --git a/src/borg/remote.py b/src/borg/remote.py
index fa04c7b2..b23b9153 100644
--- a/src/borg/remote.py
+++ b/src/borg/remote.py
@@ -107,7 +107,7 @@ def __init__(self, data):
 # For the client the return of the negotiate method is either 2 if the server is in the version range [0.29.0, 1.0.x]
 # inclusive, or it is a dict which includes the server version.
 #
-# All method calls on the remote repository object must be whitelisted in RepositoryServer.rpc_methods and have api
+# All method calls on the remote repository object must be allowlisted in RepositoryServer.rpc_methods and have api
 # stubs in RemoteRepository. The @api decorator on these stubs is used to set server version requirements.
 #
 # Method parameters are identified only by name and never by position. Unknown parameters are ignored by the server side.


Test patch:
diff --git a/src/borg/testsuite/__init__.py b/src/borg/testsuite/__init__.py
index 5e204728..844db8a8 100644
--- a/src/borg/testsuite/__init__.py
+++ b/src/borg/testsuite/__init__.py
@@ -238,30 +238,55 @@ def _assert_dirs_equal_cmp(self, diff, ignore_flags=False, ignore_xattrs=False,
             self._assert_dirs_equal_cmp(sub_diff, ignore_flags=ignore_flags, ignore_xattrs=ignore_xattrs, ignore_ns=ignore_ns)
 
     @contextmanager
-    def fuse_mount(self, location, mountpoint=None, *options, **kwargs):
+    def fuse_mount(self, location, mountpoint=None, *options, fork=True, os_fork=False, **kwargs):
+        # For a successful mount, `fork = True` is required for
+        # the borg mount daemon to work properly or the tests
+        # will just freeze. Therefore, if argument `fork` is not
+        # specified, the default value is `True`, regardless of
+        # `FORK_DEFAULT`. However, leaving the possibilty to run
+        # the command with `fork = False` is still necessary for
+        # testing for mount failures, for example attempting to
+        # mount a read-only repo.
+        #    `os_fork = True` is needed for testing (the absence of)
+        # a race condition of the Lock during lock migration when
+        # borg mount (local repo) is daemonizing (#4953). This is another
+        # example where we need `fork = False`, because the test case
+        # needs an OS fork, not a spawning of the fuse mount.
+        # `fork = False` is implied if `os_fork = True`.
         if mountpoint is None:
             mountpoint = tempfile.mkdtemp()
         else:
             os.mkdir(mountpoint)
-        if 'fork' not in kwargs:
-            # For a successful mount, `fork = True` is required for
-            # the borg mount daemon to work properly or the tests
-            # will just freeze. Therefore, if argument `fork` is not
-            # specified, the default value is `True`, regardless of
-            # `FORK_DEFAULT`. However, leaving the possibilty to run
-            # the command with `fork = False` is still necessary for
-            # testing for mount failures, for example attempting to
-            # mount a read-only repo.
-            kwargs['fork'] = True
-        self.cmd('mount', location, mountpoint, *options, **kwargs)
-        if kwargs.get('exit_code', EXIT_SUCCESS) == EXIT_ERROR:
-            # If argument `exit_code = EXIT_ERROR`, then this call
-            # is testing the behavior of an unsuccessful mount and
-            # we must not continue, as there is no mount to work
-            # with. The test itself has already failed or succeeded
-            # with the call to `self.cmd`, above.
-            yield
-            return
+        args = ['mount', location, mountpoint] + list(options)
+        if os_fork:
+            # Do not spawn, but actually (OS) fork.
+            if os.fork() == 0:
+                # The child process.
+                # Decouple from parent and fork again.
+                # Otherwise, it becomes a zombie and pretends to be alive.
+                os.setsid()
+                if os.fork() > 0:
+                    os._exit(0)
+                # The grandchild process.
+                try:
+                    self.cmd(*args, fork=False, **kwargs)  # borg mount not spawning.
+                finally:
+                    # This should never be reached, since it daemonizes,
+                    # and the grandchild process exits before cmd() returns.
+                    # However, just in case...
+                    print('Fatal: borg mount did not daemonize properly. Force exiting.',
+                          file=sys.stderr, flush=True)
+                    os._exit(0)
+        else:
+            self.cmd(*args, fork=fork, **kwargs)
+            if kwargs.get('exit_code', EXIT_SUCCESS) == EXIT_ERROR:
+                # If argument `exit_code = EXIT_ERROR`, then this call
+                # is testing the behavior of an unsuccessful mount and
+                # we must not continue, as there is no mount to work
+                # with. The test itself has already failed or succeeded
+                # with the call to `self.cmd`, above.
+                yield
+                return
         self.wait_for_mountstate(mountpoint, mounted=True)
         yield
         umount(mountpoint)
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
index 3b970071..a234d4c9 100644
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -251,7 +251,6 @@ class ArchiverTestCaseBase(BaseTestCase):
     def setUp(self):
         os.environ['BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'] = 'YES'
         os.environ['BORG_DELETE_I_KNOW_WHAT_I_AM_DOING'] = 'YES'
-        os.environ['BORG_RECREATE_I_KNOW_WHAT_I_AM_DOING'] = 'YES'
         os.environ['BORG_PASSPHRASE'] = 'waytooeasyonlyfortests'
         self.archiver = not self.FORK_DEFAULT and Archiver() or None
         self.tmpdir = tempfile.mkdtemp()
@@ -2503,6 +2502,94 @@ def test_fuse_mount_options(self):
         with self.fuse_mount(self.repository_location, mountpoint, '--prefix=nope'):
             assert sorted(os.listdir(os.path.join(mountpoint))) == []
 
+    @unittest.skipUnless(has_llfuse, 'llfuse not installed')
+    def test_migrate_lock_alive(self):
+        """Both old_id and new_id must not be stale during lock migration / daemonization."""
+        from functools import wraps
+        import pickle
+        import traceback
+
+        # Check results are communicated from the borg mount background process
+        # to the pytest process by means of a serialized dict object stored in this file.
+        assert_data_file = os.path.join(self.tmpdir, 'migrate_lock_assert_data.pickle')
+
+        # Decorates Lock.migrate_lock() with process_alive() checks before and after.
+        # (We don't want to mix testing code into runtime.)
+        def write_assert_data(migrate_lock):
+            @wraps(migrate_lock)
+            def wrapper(self, old_id, new_id):
+                wrapper.num_calls += 1
+                assert_data = {
+                    'num_calls': wrapper.num_calls,
+                    'old_id': old_id,
+                    'new_id': new_id,
+                    'before': {
+                        'old_id_alive': platform.process_alive(*old_id),
+                        'new_id_alive': platform.process_alive(*new_id)},
+                    'exception': None,
+                    'exception.extr_tb': None,
+                    'after': {
+                        'old_id_alive': None,
+                        'new_id_alive': None}}
+                try:
+                    with open(assert_data_file, 'wb') as _out:
+                        pickle.dump(assert_data, _out)
+                except:
+                    pass
+                try:
+                    return migrate_lock(self, old_id, new_id)
+                except BaseException as e:
+                    assert_data['exception'] = e
+                    assert_data['exception.extr_tb'] = traceback.extract_tb(e.__traceback__)
+                finally:
+                    assert_data['after'].update({
+                        'old_id_alive': platform.process_alive(*old_id),
+                        'new_id_alive': platform.process_alive(*new_id)})
+                    try:
+                        with open(assert_data_file, 'wb') as _out:
+                            pickle.dump(assert_data, _out)
+                    except:
+                        pass
+            wrapper.num_calls = 0
+            return wrapper
+
+        # Decorate
+        borg.locking.Lock.migrate_lock = write_assert_data(borg.locking.Lock.migrate_lock)
+        try:
+            self.cmd('init', '--encryption=none', self.repository_location)
+            self.create_src_archive('arch')
+            mountpoint = os.path.join(self.tmpdir, 'mountpoint')
+            # In order that the decoration is kept for the borg mount process, we must not spawn, but actually fork;
+            # not to be confused with the forking in borg.helpers.daemonize() which is done as well.
+            with self.fuse_mount(self.repository_location, mountpoint, os_fork=True):
+                pass
+            with open(assert_data_file, 'rb') as _in:
+                assert_data = pickle.load(_in)
+            print('\nLock.migrate_lock(): assert_data = %r.' % (assert_data, ), file=sys.stderr, flush=True)
+            exception = assert_data['exception']
+            if exception is not None:
+                extracted_tb = assert_data['exception.extr_tb']
+                print(
+                    'Lock.migrate_lock() raised an exception:\n',
+                    'Traceback (most recent call last):\n',
+                    *traceback.format_list(extracted_tb),
+                    *traceback.format_exception(exception.__class__, exception, None),
+                    sep='', end='', file=sys.stderr, flush=True)
+
+            assert assert_data['num_calls'] == 1, "Lock.migrate_lock() must be called exactly once."
+            assert exception is None, "Lock.migrate_lock() may not raise an exception."
+
+            assert_data_before = assert_data['before']
+            assert assert_data_before['old_id_alive'], "old_id must be alive (=must not be stale) when calling Lock.migrate_lock()."
+            assert assert_data_before['new_id_alive'], "new_id must be alive (=must not be stale) when calling Lock.migrate_lock()."
+
+            assert_data_after = assert_data['after']
+            assert assert_data_after['old_id_alive'], "old_id must be alive (=must not be stale) when Lock.migrate_lock() has returned."
+            assert assert_data_after['new_id_alive'], "new_id must be alive (=must not be stale) when Lock.migrate_lock() has returned."
+        finally:
+            # Undecorate
+            borg.locking.Lock.migrate_lock = borg.locking.Lock.migrate_lock.__wrapped__
+
     def verify_aes_counter_uniqueness(self, method):
         seen = set()  # Chunks already seen
         used = set()  # counter values already used
@@ -3570,6 +3657,10 @@ def test_debug_put_get_delete_obj(self):
     def test_config(self):
         pass
 
+    @unittest.skip('only works locally')
+    def test_migrate_lock_alive(self):
+        pass
+
     def test_strip_components_doesnt_leak(self):
         self.cmd('init', '--encryption=repokey', self.repository_location)
         self.create_regular_file('dir/file', contents=b"test file contents 1")
diff --git a/src/borg/testsuite/locking.py b/src/borg/testsuite/locking.py
index de14e394..7a0f219c 100644
--- a/src/borg/testsuite/locking.py
+++ b/src/borg/testsuite/locking.py
@@ -1,5 +1,7 @@
 import random
 import time
+from threading import Thread, Lock as ThreadingLock
+from traceback import format_exc
 
 import pytest
 
@@ -10,6 +12,8 @@
 
 ID1 = "foo", 1, 1
 ID2 = "bar", 2, 2
+RACE_TEST_NUM_THREADS = 40
+RACE_TEST_DURATION = 0.4  # seconds
 
 
 @pytest.fixture()
@@ -90,6 +94,78 @@ def test_migrate_lock(self, lockpath):
         assert lock.by_me()  # we still have the lock
         assert old_unique_name != new_unique_name  # locking filename is different now
 
+    def test_race_condition(self, lockpath):
+
+        class SynchronizedCounter:
+
+            def __init__(self, count=0):
+                self.lock = ThreadingLock()
+                self.count = count
+                self.maxcount = count
+
+            def value(self):
+                with self.lock:
+                    return self.count
+
+            def maxvalue(self):
+                with self.lock:
+                    return self.maxcount
+
+            def incr(self):
+                with self.lock:
+                    self.count += 1
+                    if self.count > self.maxcount:
+                        self.maxcount = self.count
+                    return self.count
+
+            def decr(self):
+                with self.lock:
+                    self.count -= 1
+                    return self.count
+
+        def print_locked(msg):
+            with print_lock:
+                print(msg)
+
+        def acquire_release_loop(id, timeout, thread_id, lock_owner_counter, exception_counter, print_lock, last_thread=None):
+            print_locked("Thread %2d: Starting acquire_release_loop(id=%s, timeout=%d); lockpath=%s" % (thread_id, id, timeout, lockpath))
+            timer = TimeoutTimer(timeout, -1).start()
+            cycle = 0
+
+            while not timer.timed_out():
+                cycle += 1
+                try:
+                    with ExclusiveLock(lockpath, id=id, timeout=timeout/20, sleep=-1):  # This timeout is only for not exceeding the given timeout by more than 5%. With sleep<0 it's constantly polling anyway.
+                        lock_owner_count = lock_owner_counter.incr()
+                        print_locked("Thread %2d: Acquired the lock. It's my %d. loop cycle. I am the %d. who has the lock concurrently." % (thread_id, cycle, lock_owner_count))
+                        time.sleep(0.005)
+                        lock_owner_count = lock_owner_counter.decr()
+                        print_locked("Thread %2d: Releasing the lock, finishing my %d. loop cycle. Currently, %d colleagues still have the lock." % (thread_id, cycle, lock_owner_count))
+                except LockTimeout:
+                    print_locked("Thread %2d: Got LockTimeout, finishing my %d. loop cycle." % (thread_id, cycle))
+                except:
+                    exception_count = exception_counter.incr()
+                    e = format_exc()
+                    print_locked("Thread %2d: Exception thrown, finishing my %d. loop cycle. It's the %d. exception seen until now: %s" % (thread_id, cycle, exception_count, e))
+
+            print_locked("Thread %2d: Loop timed out--terminating after %d loop cycles." % (thread_id, cycle))
+            if last_thread is not None:  # joining its predecessor, if any
+                last_thread.join()
+
+        print('')
+        lock_owner_counter = SynchronizedCounter()
+        exception_counter = SynchronizedCounter()
+        print_lock = ThreadingLock()
+        thread = None
+        for thread_id in range(RACE_TEST_NUM_THREADS):
+            thread = Thread(target=acquire_release_loop, args=(('foo', thread_id, 0), RACE_TEST_DURATION, thread_id, lock_owner_counter, exception_counter, print_lock, thread))
+            thread.start()
+        thread.join()  # joining the last thread
+
+        assert lock_owner_counter.maxvalue() > 0, 'Never gained the lock? Something went wrong here...'
+        assert lock_owner_counter.maxvalue() <= 1, "Maximal number of concurrent lock holders was %d. So exclusivity is broken." % (lock_owner_counter.maxvalue())
+        assert exception_counter.value() == 0, "ExclusiveLock threw %d exceptions due to unclean concurrency handling." % (exception_counter.value())
+
 
 class TestLock:
     def test_shared(self, lockpath):