Repository: borgbackup/borg

Issue:
Call to `sync_file_range` syscall raises invalid exception (EPERM)
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes.

## Is this a BUG / ISSUE report or a QUESTION?

BUG / ISSUE

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

A: 1.1.9 (`borg create` instance)
B: 1.1.9 (`borg serve` instance)

#### Operating system (distribution) and version.

Debian buster (Problem occurred after upgrading to buster and the recent borg version.)

#### Hardware / network configuration, and filesystems used.

Raspberry Pi 4, GbE, btrfs (also tested ext4)

The borg repository server (`borg@backup.${DOMAIN}` below) is running in a systemd-nspawn container with user namespaces enabled. The container has all the standard capabilities.

#### How much data is handled by borg?

~ 60 GiB, multiple repositories

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```bash
export BORG_REPO="borg@backup.${DOMAIN}:repos/new_repo"
export BORG_PASSPHRASE="*****************************************************"
borg init -e repokey
borg create -x "::boot-${IDENTIFIER}" /boot
```

## Describe the problem you're observing.

After creating a new repository on my backup host, I cannot create any backups with `borg create`. I always receive this traceback from the remote borg process:

```
Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/borg/remote.py", line 248, in serve
    res = f(**args)

  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1101, in put
    segment, offset = self.io.write_put(id, data)

  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1482, in write_put
    fd.write(b''.join((crc, header, id, data)))

  File "src/borg/platform/linux.pyx", line 283, in borg.platform.linux.SyncFile.write

  File "src/borg/platform/linux.pyx", line 236, in borg.platform.linux._sync_file_range

PermissionError: [Errno 1] Operation not permitted

Borg server: Platform: Linux backup 4.19.50-v7l+ #895 SMP Thu Jun 20 16:03:42 BST 2019 armv7l
Borg server: Linux: debian 10.0
Borg server: Borg: 1.1.9  Python: CPython 3.7.3
Borg server: PID: 1226  CWD: /var/lib/borg
Borg server: sys.argv: ['/usr/bin/borg', 'serve', '--append-only', '--restrict-to-path',
+'/var/lib/borg/repos/new_repo']
Borg server: SSH_ORIGINAL_COMMAND: 'borg serve --umask=077'
Platform: Linux rpi4 4.19.50-v7l+ #895 SMP Thu Jun 20 16:03:42 BST 2019 armv7l
Linux: debian 10.0
Borg: 1.1.9  Python: CPython 3.7.3
PID: 10925  CWD: /
sys.argv: ['/usr/bin/borg', 'create', '-x', '::root-20190803013801', '/']
SSH_ORIGINAL_COMMAND: None
```


#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Fully reproducible on my system. I checked all file permission and everything is correct. The same command works on existing borg repos without any problems.

After stracing the remote borg process, I figured out that the syscall `sync_file_range2` has a return value of != 0.

I think [this exception here](https://github.com/borgbackup/borg/blob/1.1.9/src/borg/platform/linux.pyx#L236) is thrown and somehow converted from an OSError to a PermissionError.

According to the manpage of `sync_file_range` the syscall should set errno, but EPERM is not in the list of valid error codes. Maybe borgs error handling code is wrong?

#### Changes I applied to circumvent the problem

The problem goes away if I comment out the usage of `sync_file_range`. I do not know if this has severe consequences for the durability of my backups. (According to the manpage the syscall is not especially safe to use anyway.)

Just for reference, this is the change that I applied for debugging the problem:

```diff
--- borgbackup-1.1.9.orig/src/borg/platform/linux.pyx
+++ borgbackup-1.1.9/src/borg/platform/linux.pyx
@@ -280,10 +280,10 @@ else:
             offset = self.offset & ~PAGE_MASK
             if offset >= self.last_sync + self.write_window:
                 self.fd.flush()
-                _sync_file_range(self.fileno, self.last_sync, offset - self.last_sync, SYNC_FILE_RANGE_WRITE)
-                if self.pending_sync is not None:
-                    _sync_file_range(self.fileno, self.pending_sync, self.last_sync - self.pending_sync,
-                                     SYNC_FILE_RANGE_WRITE | SYNC_FILE_RANGE_WAIT_BEFORE | SYNC_FILE_RANGE_WAIT_AFTER)
                 self.pending_sync = self.last_sync
                 self.last_sync = offset
```


Discussion:
from errno.h:
```
#define EPERM            1      /* Operation not permitted */
```
To make analyzing this easier you need to simplify it:
- repo url is missing, is it in BORG_REPO env var?
- you could try the latest borg version
- please reproduce as a normal user, accessing some files in that user's home directory as source (do all commands as that user, never use sudo or root)
- you could reproduce it with a local repository, also located in that user's home dir
- you could try to reproduce on x86 / x64

As nobody else encountered this yet, it could be maybe also a bug in the OS (libc / kernel), maybe only on ARM?

Note: i fixed the link to line 237 in your post (do not link to master branch, but 1.1.9 tag).


You could also use the same workaround as is implemented for WSL (use `if True:`):

https://github.com/borgbackup/borg/blob/1.1.10/src/borg/platform/linux.pyx#L255

It will affect "streaming behaviour" (disk access could be more bursty), but does not affect data safety.
Thanks for your quick response. I am happy to add all necessary information that I forgot in my initial report.
> To make analyzing this easier you need to simplify it:
> 
>     * repo url is missing, is it in BORG_REPO env var?

Yes. I use borg always via ssh to not run into permission problems. So there are two borg instances, both running the same version and Debian buster.

There is one thing special: My borg repo is located in a systemd-nspawn container managed by machinectl. I am using user namespaces. Potentially this could result in syscalls giving permission errors. I think this is not very likely for a sync call on a FD owned by the container and also no required capabilities are mentioned in the documentation of the call.

I will try reproducing this problem on a spare Raspberry Pi with and without using containers.

>     * you could try the latest borg version

I will try this. I might need a few days before I can test it.

>     * please reproduce as a normal user, accessing some files in that user's home directory as source (do all commands as that user, never use sudo or root)
>     * you could reproduce it with a local repository, also located in that user's home dir
>     * you could try to reproduce on x86 / x64
> As nobody else encountered this yet, it could be maybe also a bug in the OS (libc / kernel), maybe only on ARM?

I will try to do all of the above. This might take a few days.

What I do not understand: Why is the problem only appearing for some repos? I distribute exactly the same backup configuration and backup scripts to my hosts/containers and only after provisioning a new computer and initializing a new repo the problem manifests. On the server side the same code path should be used for all invocations of `borg serve`...
OK, so I suggest you reproduce without namespaces, containers, systemd, ...

I suspect this is an issue related to that.

As this issue tracker is for borg bugs (and not issues in other software), I am closing this.

Feel free to reopen if you find out that it is in fact a borg issue and you can reproduce it on a simple setup.
Hmm, just hit this - I assume it's a bug in systemd-nspawn's syscall filter? And architecture specific I assume, or it would have been noticed more widely. Weirdly, I swear tested this in a VM before putting it live and it was fine - can't think what is different about current setup.
People have seen a related issue with postgres: https://www.postgresql.org/message-id/flat/CA%2BhUKG%2BydOUT4zjxb6QmJWy8U9WbC-q%2BJWV7wLsEY9Df%3Dmw0Mw%40mail.gmail.com#ac8f14897647dc7eae3c7e7cbed36d93
Simplest possible test case:

```
#define _GNU_SOURCE
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>

void main()
{
        int f = open("/tmp/syncrange.test",O_CREAT|O_RDWR,0666);
        int r=sync_file_range(f, 0, 0, 0);
        if (r)
                perror("sync_file_range");
        close(f);
}
```

EPERMs for me on arm, not on x86_64.
I have moaned: https://lists.freedesktop.org/archives/systemd-devel/2019-August/043276.html
Possible alternative work-around to rebuilding borg (evil, untested):

```
gcc -shared -fPIC -x c - -o syncfr.o <<EOS
#define _GNU_SOURCE         /* See feature_test_macros(7) */
#include <fcntl.h>

int sync_file_range(int fd, off64_t offset, off64_t nbytes,
                                          unsigned int flags)
{
        return 0;
}
EOS
```

Then `LD_PRELOAD=$PWD/syncfr.o ./test` or whatever..

> As this issue tracker is for borg bugs (and not issues in other software), I am closing this.

Yes, you are right and I was also aware of this. In the beginning I was thinking this was a borg issue.

I also did additional tests and it is exactly as as @srd424 says. The problem occurs only inside `systemd-nspawn` and not on an x64 architecture. So it's not related to borg.
upstream have merged the obvious fix: https://github.com/systemd/systemd/pull/13352
I am trying to persuade Ubuntu to SRU it: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1840640
@fgrsnau, you might want to do the same for Deb?
Good that it got fixed at the right place. \o/

Given that sync_file_range seems to be iffy in various settings, wonder if it's worth working around generally though .. will leave that decision to you! maybe an environment variable to fallback to more simplistic handling?
Well, then everybody would use the workaround and it would not get fixed where it should be fixed.
Just thinking it might take quite while for the upstream fix to trickle back into stable releases of distributions, etc. Ubuntu have said they will SRU but others may not. borg's heavy-duty enough to have become "key infrastructure" for a lot of us :)
Reopening it, consider adding a workaround.

The workaround could be also used to address the WSL issue that is currently fixed by some special detection code.

Resolving patch:
diff --git a/src/borg/helpers.py b/src/borg/helpers.py
--- a/src/borg/helpers.py
+++ b/src/borg/helpers.py
@@ -81,6 +81,12 @@
 from .constants import *  # NOQA
 
 
+# generic mechanism to enable users to invoke workarounds by setting the
+# BORG_WORKAROUNDS environment variable to a list of comma-separated strings.
+# see the docs for a list of known workaround strings.
+workarounds = tuple(os.environ.get('BORG_WORKAROUNDS', '').split(','))
+
+
 '''
 The global exit_code variable is used so that modules other than archiver can increase the program exit code if a
 warning or error occurred during their operation. This is different from archiver.exit_code, which is only accessible


Test patch: