Repository: borgbackup/borg

Issue:
freebsd 12 FUSE xattr issues
The xattrs in the native freebsd filesystem are set like this, then backed up with borg:
```
   xattr.setxattr(os.path.join(self.input_path, 'fusexattr'), 'user.foo', b'bar')
   xattr.setxattr(os.path.join(self.input_path, 'fusexattr'), 'user.empty', b'')
```

Then the archive is fuse mounted and test_fuse executed:

freebsd 12 amd64 VM (see Vagrantfile), current master branch, no fakeroot:

```
freebsd64: >           assert sorted(no_selinux(xattr.listxattr(out_fn))) == ['user.empty', 'user.foo', ]
freebsd64: E           AssertionError: assert ['empty', 'foo'] == ['user.empty', 'user.foo']
freebsd64: E             At index 0 diff: 'empty' != 'user.empty'
```

borg 1.2.0.dev569+g23bd2345.d20180704

----
freebsd 12 amd64 VM (see Vagrantfile), current platform-xattr branch, no fakeroot:

borg 1.2.0.dev560+gadb298d1

behaves the same after setting `borg.xattr.XATTR_FAKEROOT = False` so that same test code is executed.

----

So it seems that something is stripping away the "user." from the xattr keys.


Discussion:
freebsd 12 amd64 VM (see Vagrantfile), current platform-xattr branch, no fakeroot:

When having XATTR_FAKEROOT = True, this happens:
```
                if not xattr.XATTR_FAKEROOT and xattr.is_enabled(self.input_path):
                    assert sorted(no_selinux(xattr.listxattr(out_fn))) == ['user.empty', 'user.foo', ]
                    assert xattr.getxattr(out_fn, 'user.foo') == b'bar'
                    # Special case: getxattr returns None (not b'') when reading an empty xattr.
                    assert xattr.getxattr(out_fn, 'user.empty') is None
                else:
>                   assert xattr.listxattr(out_fn) == []

src/borg/testsuite/archiver.py:2192: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/borg/platform/freebsd.pyx:62: in borg.platform.freebsd.listxattr
    n, buf = _listxattr_inner(func, path)
src/borg/platform/xattr.py:72: in _listxattr_inner
    n = _check(func(path, buf, size), path, detect_buffer_too_small=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

rv = -1, path = b'/tmp/tmpyrlor7jv/mountpoint/input/fusexattr', detect_buffer_too_small = True

    def _check(rv, path=None, detect_buffer_too_small=False):
        if rv < 0:
            e = get_errno()
            if detect_buffer_too_small and e == errno.ERANGE:
                # listxattr and getxattr signal with ERANGE that they need a bigger result buffer.
                # setxattr signals this way that e.g. a xattr key name is too long / inacceptable.
                raise BufferTooSmallError
            else:
                try:
                    msg = os.strerror(e)
                except ValueError:
                    msg = ''
                if isinstance(path, int):
                    path = '<FD %d>' % path
>               raise OSError(e, msg, path)
E               OSError: [Errno 22] Invalid argument: b'/tmp/tmpyrlor7jv/mountpoint/input/fusexattr'

src/borg/platform/xattr.py:55: OSError
```
So listxattr on a file that is visible via the fuse mount and has not xattrs set gives errno 22.

A quick test in the native freebsd filesystem with the platform-xattr code showed nothing special, works as expected.
So, looks like some FreeBSD 12 fuse bugs.
Still seeing the xattr name issue on a fresh freebsd 12 release install with all updates.
This sounds to me like a bug in borgbackup, not FreeBSD.  But I can't be sure unless you can post a reproduction case that uses lsextattr(1) or extattr_list_file(2).  I say that I don't think it's a bug in FreeBSD, because extattr_list_file is supposed to return just the attribute names, not prefixed with their namespaces.
Well, we are running this test on different platforms (fbsd, macOS, linux) and IIRC also used to run it on FreeBSD < 12.

And IIRC it only fails on FBSD 12, this is why I filed this ticket.
 
If you think it's a bug in FreeBSD, could you please try to reproduce it independently of borg-backup so a FreeBSD developer like myself can debug it?
I can try, but I usually don't write C code for such stuff and also usually do not work on fbsd, I just use it for platform testing.
Also, as this is about borg's fuse code working together with fuse code of freebsd, guess it will need borg for that part.
C code isn't necessary.  sh code will work just fine using the command-line utilities lsextattr, getextattr, rmextattr, and setextattr.
script
======
```
mkdir input
touch input/file-with-xattr
setextattr user user.foo bar input/file-with-xattr
setextattr user user.empty '' input/file-with-xattr
setextattr user xxx.yyy zzz input/file-with-xattr

echo "os level xattrs (NS 'user'):"
lsextattr user input/file-with-xattr

borg init -e none repo
borg create repo::archive input

mkdir mnt
borg mount repo::archive mnt
echo "fuse level xattrs (NS 'user'):"
lsextattr user mnt/input/file-with-xattr
umount mnt

rm -rf mnt
borg delete repo
rm -rf input
```

testing
=======

using borg 1.2.0 ~ alpha5 (alpha4 should also work). i also checked borg 1.1.7 from freebsd, showed same behaviour (although our xattr code had a major refactoring in between).

output freebsd 12
-----------------
```
os level xattrs (NS 'user'):
input/file-with-xattr	user.foo	user.empty	xxx.yyy

fuse level xattrs (NS 'user'):
mnt/input/file-with-xattr	empty	foo
```
output freebsd 10.3
-------------------
```
os level xattrs (NS 'user'):
input/file-with-xattr	user.foo	user.empty	xxx.yyy

fuse level xattrs (NS 'user'):
lsextattr: mnt/input/file-with-xattr: failed: Operation not supported
```
notes
=====

10.3: hmm, maybe the failure for 10.3 somehow triggered a different path in our test suite (like "fuse does not support xattrs")

12: see how it reports key names with "user." on OS level, but not on fuse level. also, the "xxx.yyy" key is completely missing.

I admit the key names are a bit weird because they put a key named "user.foo" into the "user" namespace, but I guess at least the FUSE level should have same result as OS level.

The weirdness comes from linux not having an extra param for the namespace and we have the same tests for all platforms.

I think the confusion might be because FreeBSD extattr deals with the namespace separately from the attribute name as opposed to FUSE / Linux which have the namespace encoded in the name. The platform code for FreeBSD in borg specifies the user namespace in its calls to the exattr functions, and it should not expect the namespace to also be present in the attribute names.

https://github.com/freebsd/freebsd/blob/master/sys/fs/fuse/fuse_vnops.c#L2182-L2203
https://github.com/freebsd/freebsd/blob/master/sys/fs/fuse/fuse_vnops.c#L2327-L2332
https://reviews.freebsd.org/D12485
Does the bug happen with borg 1.1.8 as well?
To demonstrate this, I created and attached a disk image, formatted it as ext4, mounted on FreeBSD 12 with e2fuse (from e2fsprogs port with FUSE config option enabled), set up my permissions, then:
```
$ touch mnt/foo
$ setextattr user user.foo 'bar' mnt/foo
$ setextattr user user.empty '' mnt/foo
$ setextattr user abc xyz mnt/foo
$ lsextattr user mnt/foo
mnt/foo abc     user.foo        user.empty
```
Then I unmounted and detached the image and moved it over to a Linux system, where I mounted it and inspected the attributes:
```
$ sudo mount foo.img /mnt -o loop=/dev/loop0
$ getfattr /mnt/foo
getfattr: Removing leading '/' from absolute path names
# file: mnt/foo
user.abc
user.user.empty
user.user.foo
```
Notice the "user.user.*" as a result of specifying the namespace twice on FreeBSD, as opposed to "user.abc" for the attribute that was correctly named without the extra namespace prefixing.
@freqlabs i realise that NS is dealt with differently on fbsd than on linux, but it should not be different on fbsd and fbsd-fuse, right?
> @freqlabs i realise that NS is dealt with differently on fbsd than on linux, but it should not be different on fbsd and fbsd-fuse, right?

I suspect it has something to do with how the test file is being created by borg. When manually setting and listing attributes, I am seeing consistent behavior between native and fuse filesystems on FreeBSD.
@ThomasWaldmann it looks like borg implements its own FUSE filesystem?  That's probably where the problem lies.  As @freqlabs showed, there doesn't seem to be a problem with fusefs-ext4fuse.
Well, it works on linux, but I'ld say there is something on fbsd stripping "user." from the keys (and that something is not our fuse code). And it also drops the "xxx.yyy" key.
Yes that's what the links in my initial post were showing. The namespace prefixes are stripped from FUSE filesystems to translate them to the FreeBSD convention.
@freqlabs but lsextattr lists the keys WITH "user." on the normal fs and without on the fuse fs.
That's why I suspect that the problem is in borg's FUSE implementation, because lsextattr does show the "user." with ext4fuse.  Does borg 1.1.8 exhibit the problem?  If so I'll try it out myself.
I am not familiar with the inner workings of borg but if I were to hazard a guess, it seems like borg is failing to take this difference into account.

I would expect here that borg should be prepending "user." to each of the names it gets back in the list in order to be consistent with the behavior of the other platforms:
https://github.com/borgbackup/borg/blob/master/src/borg/platform/freebsd.pyx#L64

And likewise here I would expect "user." to be stripped from attribute names before getting/setting the value:
https://github.com/borgbackup/borg/blob/master/src/borg/platform/freebsd.pyx#L77
https://github.com/borgbackup/borg/blob/master/src/borg/platform/freebsd.pyx#L91

Assuming these functions have some role in how borg implements FUSE, I think that would explain the problem.
I guess 1.1.8 should behave the same as 1.1.7 (which i tried), so yes.
Yeah, guess we need to prepend / strip "user." at some places for FreeBSD because of FreeBSD's FUSE code doing it the "linux way".

While we are at it, how about the other namespaces than "user": shall a backup tool try archiving and restoring them and for what are they used on fbsd?
You can use `extattr_namespace_to_string(3)` and `extattr_string_to_namespace(3)` for that.  The only other namespace used in current versions of FreeBSD is "system".  It's used primarily for POSIX ACLs, but also by pnfsd.
I'm working on some code, will post link to PR when ready for first review.

Resolving patch:
diff --git a/src/borg/testsuite/xattr.py b/src/borg/testsuite/xattr.py
--- a/src/borg/testsuite/xattr.py
+++ b/src/borg/testsuite/xattr.py
@@ -44,13 +44,13 @@ def test(self):
     def test_listxattr_buffer_growth(self):
         # make it work even with ext4, which imposes rather low limits
         buffer.resize(size=64, init=True)
-        # xattr raw key list will be size 9 * (10 + 1), which is > 64
-        keys = ['user.attr%d' % i for i in range(9)]
+        # xattr raw key list will be > 64
+        keys = ['user.attr%d' % i for i in range(20)]
         for key in keys:
             setxattr(self.tmpfile.name, key, b'x')
         got_keys = listxattr(self.tmpfile.name)
         self.assert_equal_se(got_keys, keys)
-        self.assert_equal(len(buffer), 128)
+        self.assert_true(len(buffer) > 64)
 
     def test_getxattr_buffer_growth(self):
         # make it work even with ext4, which imposes rather low limits
diff --git a/src/borg/xattr.py b/src/borg/xattr.py
--- a/src/borg/xattr.py
+++ b/src/borg/xattr.py
@@ -323,6 +323,7 @@ def func(path, name, value, size):
     libc.extattr_set_file.argtypes = (c_char_p, c_int, c_char_p, c_char_p, c_size_t)
     libc.extattr_set_file.restype = c_int
     ns = EXTATTR_NAMESPACE_USER = 0x0001
+    prefix = 'user.'
 
     def listxattr(path, *, follow_symlinks=True):
         def func(path, buf, size):
@@ -335,7 +336,7 @@ def func(path, buf, size):
                     return libc.extattr_list_link(path, ns, buf, size)
 
         n, buf = _listxattr_inner(func, path)
-        return [os.fsdecode(name) for name in split_lstring(buf[:n]) if name]
+        return [prefix + os.fsdecode(name) for name in split_lstring(buf[:n]) if name]
 
     def getxattr(path, name, *, follow_symlinks=True):
         def func(path, name, buf, size):
@@ -347,6 +348,10 @@ def func(path, name, buf, size):
                 else:
                     return libc.extattr_get_link(path, ns, name, buf, size)
 
+        # strip namespace if there, but ignore if not there.
+        # older borg / attic versions did not prefix the namespace to the names.
+        if name.startswith(prefix):
+            name = name[len(prefix):]
         n, buf = _getxattr_inner(func, path, name)
         return buf[:n] or None
 
@@ -360,6 +365,10 @@ def func(path, name, value, size):
                 else:
                     return libc.extattr_set_link(path, ns, name, value, size)
 
+        # strip namespace if there, but ignore if not there.
+        # older borg / attic versions did not prefix the namespace to the names.
+        if name.startswith(prefix):
+            name = name[len(prefix):]
         _setxattr_inner(func, path, name, value)
 
 else:  # pragma: unknown platform only


Test patch: