Repository: borgbackup/borg

Issue:
FileExistsError/FileNotFoundError when locking
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

bug

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

```
$ borg -V
borg 1.1.16
```

#### Operating system (distribution) and version.

```
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal
```

#### Hardware / network configuration, and filesystems used.

BTRFS on target host

## Describe the problem you're observing.

I've been seeing locking-related errors for some time. See backtraces below. So far, I've only seen this issue on one out of two servers. The only difference I'm aware of is that the machine is using BTRFS as filesystem.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

No, but the issue reappears sporadically. 

#### Include any warning/errors/backtraces from the system logs

```
Traceback (most recent call last):
  File "borg/locking.py", line 130, in acquire
FileExistsError: [Errno 17] File exists: '/backup/borg_repos/backup01/lock.exclusive'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "borg/remote.py", line 247, in serve
  File "borg/remote.py", line 375, in open
  File "borg/repository.py", line 190, in __enter__
  File "borg/repository.py", line 421, in open
  File "borg/locking.py", line 350, in acquire
  File "borg/locking.py", line 363, in _wait_for_readers_finishing
  File "borg/locking.py", line 134, in acquire
  File "borg/locking.py", line 159, in kill_stale_lock
FileNotFoundError: [Errno 2] No such file or directory: '/backup/borg_repos/backup01/lock.exclusive'

Borg server: Platform: Linux backup02 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64
Borg server: Linux: debian bullseye/sid 
Borg server: Borg: 1.1.16  Python: CPython 3.7.9 msgpack: 0.5.6.+borg1
Borg server: PID: 1808518  CWD: /backup/borg_repos/backup01
Borg server: sys.argv: ['borg', 'serve', '--restrict-to-path', '/backup/borg_repos/backup01']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --error'
Platform: Linux backup01 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64
Linux: debian bullseye/sid 
Borg: 1.1.16  Python: CPython 3.7.9 msgpack: 0.5.6.+borg1
PID: 67423  CWD: /root
sys.argv: ['borg', 'create', '--error', '--compression', 'zstd', '--lock-wait', '43200', '--noatime', '--patterns-from', '/etc/borg/patterns-main', '--', 'borg-backup01@backup02.tocco.ch:~::main-{now:%Y-%m-%dT%H:%M:%S}']
SSH_ORIGINAL_COMMAND: None
```

```
Traceback (most recent call last):
  File "borg/locking.py", line 130, in acquire
FileExistsError: [Errno 17] File exists: '/backup/borg_repos/backup01/lock.exclusive'


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "borg/remote.py", line 247, in serve
  File "borg/remote.py", line 375, in open
  File "borg/repository.py", line 190, in __enter__
  File "borg/repository.py", line 421, in open
  File "borg/locking.py", line 350, in acquire
  File "borg/locking.py", line 363, in _wait_for_readers_finishing
  File "borg/locking.py", line 134, in acquire
  File "borg/locking.py", line 159, in kill_stale_lock
FileNotFoundError: [Errno 2] No such file or directory: '/backup/borg_repos/backup01/lock.exclusive'

Borg server: Platform: Linux backup02 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64
Borg server: Linux: debian bullseye/sid 
Borg server: Borg: 1.1.16  Python: CPython 3.7.9 msgpack: 0.5.6.+borg1
Borg server: PID: 1808518  CWD: /backup/borg_repos/backup01
Borg server: sys.argv: ['borg', 'serve', '--restrict-to-path', '/backup/borg_repos/backup01']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077 --error'
Platform: Linux backup01 5.4.0-73-generic #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64
Linux: debian bullseye/sid 
Borg: 1.1.16  Python: CPython 3.7.9 msgpack: 0.5.6.+borg1
PID: 67423  CWD: /root
sys.argv: ['borg', 'create', '--error', '--compression', 'zstd', '--lock-wait', '43200', '--noatime', '--patterns-from', '/etc/borg/patterns-main', '--', 'borg-backup01@backup02.tocco.ch:~::main-{now:%Y-%m-%dT%H:%M:%S}']
SSH_ORIGINAL_COMMAND: None
```

<!--

If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html

*IMPORTANT* - Please mark logs and text output from terminal commands 
or else Github will not display them correctly. 
An example is provided below.

Example:
```
this is an example how log text should be marked (wrap it with ```)
```
-->



Discussion:
Do you have scripts running that use `borg break-lock`?

Do you have multiple borg client processes running against that one repo? At the same time?

Analyzing this:
- locking:130 exclusive lock directory (ELD) exists already when we try creating it
- locking:134 we only get here if it is NOT our own lock and try to kill stale locks now
- locking:159 ELD is gone when we try looking into it

So, as ELD is gone in 159 then something must have removed it (which is why I was asking for break-lock and other borg processes which maybe raced for deleting a stale lock).
Hmm, no matter how this happened, I think you have found a bug here.

borg should not crash when trying to look into a not-any-more-existing exclusive lock directory.

Resolving patch:
diff --git a/src/borg/locking.py b/src/borg/locking.py
--- a/src/borg/locking.py
+++ b/src/borg/locking.py
@@ -156,7 +156,13 @@ def by_me(self):
         return os.path.exists(self.unique_name)
 
     def kill_stale_lock(self):
-        for name in os.listdir(self.path):
+        try:
+            names = os.listdir(self.path)
+        except FileNotFoundError:
+            # if another borg process won the race for killing a stale lock, we get here.
+            return False
+
+        for name in names:
             try:
                 host_pid, thread_str = name.rsplit('-', 1)
                 host, pid_str = host_pid.rsplit('.', 1)


Test patch: