Repository: borgbackup/borg

Issue:
broken error handling for broken xattr FS implementation
Backing up a FUSE based ObjectiveFS filesystem results in the following error:

```Local Exception.
Traceback (most recent call last):
  File "borg/archiver.py", line 2052, in main
  File "borg/archiver.py", line 1997, in run
  File "borg/archiver.py", line 90, in wrapper
  File "borg/archiver.py", line 305, in do_create
  File "borg/archiver.py", line 278, in create_inner
  File "borg/archiver.py", line 348, in _process
  File "borg/archive.py", line 605, in process_dir
  File "borg/archive.py", line 594, in stat_attrs
  File "borg/xattr.py", line 43, in get_all
  File "borg/xattr.py", line 220, in getxattr
  File "borg/xattr.py", line 164, in _getxattr_inner
  File "borg/helpers.py", line 735, in get
  File "borg/helpers.py", line 725, in resize
ValueError: Requested buffer size 33554432 is above the limit of 16777216.

Platform: Linux ip-172-31-12-165 3.13.0-101-lowlatency #148-Ubuntu SMP PREEMPT Thu Oct 20 22:48:47 UTC 2016 x86_64 x86_64
Linux: debian jessie/sid
Borg: 1.0.9  Python: CPython 3.5.2
PID: 2542  CWD: /tmp
sys.argv: ['./borg-linux64', 'create', '--stats', '--debug', '/mnt/borg_backup::OFS-Backup-now', '/ofs/']
SSH_ORIGINAL_COMMAND: None
```

This is the relevant strace:

```
[pid  2542] 04:52:48 lstat("/ofs", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=0, ...}) = 0
[pid  2542] 04:52:48 open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 5
[pid  2542] 04:52:48 open("/etc/group", O_RDONLY|O_CLOEXEC) = 5
[pid  2542] 04:52:48 llistxattr("/ofs", "\0\0\0\0\0\0\0\0", 4096) = 8
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x1e9fc80, 4096) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x1f1d5b0, 8192) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x1f1f5c0, 16384) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x1f235d0, 32768) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x1f2b5e0, 65536) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x1f3b5f0, 131072) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x7f2922027010, 262144) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x7f2921fa6010, 524288) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x7f2921ea5010, 1048576) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x7f2921ca4010, 2097152) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x7f29218a3010, 4194304) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x7f29210a2010, 8388608) = -1 ERANGE (Numerical result out of range)
[pid  2542] 04:52:48 lgetxattr("/ofs", "", 0x7f29200a1010, 16777216) = -1 ERANGE (Numerical result out of range)
```

From my understanding it looks like the array of null-terminated character strings \0\0\0\0\0\0\0\0 is causing the issue? 

Not sure if this is a bug in Borg or ObjectiveFS, as the linux attr command works as expected:
```
strace -f -t -e trace=file attr -l /ofs
06:04:31 execve("/usr/bin/attr", ["attr", "-l", "/ofs"], [/* 23 vars */]) = 0
06:04:31 llistxattr("/ofs", "\0\0\0\0\0\0\0\0", 65536) = 8
06:04:31 +++ exited with 0 +++
```

```
strace -f -t -e trace=file attr -l /ofs/all_dbs.sql.gz
05:57:04 execve("/usr/bin/attr", ["attr", "-l", "/ofs/all_dbs.sql.gz"], [/* 23 vars */]) = 0
05:57:04 llistxattr("/ofs/all_dbs.sql.gz", "user.user.name\0", 65536) = 15
05:57:04 lgetxattr("/ofs/all_dbs.sql.gz", "user.user.name", 0x0, 0) = 4

Attribute "user.name" has a 4 byte value for /ofs/all_dbs.sql.gz
05:57:04 +++ exited with 0 +++
```

Backing up a file with an extended attribute set results in a successful backup:

```
attr -s user.name -V test /ofs/all_dbs.sql.gz

./borg-linux64 create --stats --debug /mnt/borg_backup::OFS-Backup-now /ofs/all_dbs.sql.gz
using builtin fallback logging configuration
Enter passphrase for key /mnt/borg_backup:
TAM-verified manifest
Reading files cache ...
Processing files ...
------------------------------------------------------------------------------
Archive name: OFS-Backup-now
Archive fingerprint: 565567fefdd19bbe577b88d49ec7f7c2d2fb138d71e99709751573cf0c2edb78
Time (start): Tue, 2017-01-03 05:30:45
Time (end):   Tue, 2017-01-03 05:30:52
Duration: 6.98 seconds
Number of files: 1
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               76.94 MB             76.94 MB             76.94 MB
All archives:               76.94 MB             76.94 MB             76.94 MB

                       Unique chunks         Total chunks
Chunk index:                      29                   29
------------------------------------------------------------------------------
```

Thank you very much in advance for your help and if you need any further data please let me know.


Discussion:
> From my understanding it looks like the array of null-terminated character strings \0\0\0\0\0\0\0\0 is causing the issue?

Yes, it looks like four issues working together:

1. The FS doesn't have any xattrs on the object, but does not return an empty list from llistxattr; instead a list of 7 empty strings is returned (FS bug)

2. Borg does not ignore empty attribute names (but it probably should, Borg bug)

3. When asked to retrieve the empty attributes the FS earlier said existed it always fails with ERANGE, indicating that the supplied buffer is too small, instead of ENOATTR (FS bug)

4. Running out of buffer space in xattr retrieval should probably be contained to the file in question and not abort the backup. (Borg bug)
From your testing it looks like this special FS behaviour only happens on the mountpoint /ofs, and not on files in the FS? If so, `/ofs/*` might work for you as a workaround.

Resolving patch:
diff --git a/borg/xattr.py b/borg/xattr.py
--- a/borg/xattr.py
+++ b/borg/xattr.py
@@ -10,7 +10,7 @@
 from ctypes.util import find_library
 from distutils.version import LooseVersion
 
-from .helpers import Buffer
+from .helpers import Buffer, Error
 
 
 try:
@@ -113,8 +113,11 @@ def split_lstring(buf):
     return result
 
 
-class BufferTooSmallError(Exception):
-    """the buffer given to an xattr function was too small for the result"""
+class BufferTooSmallError(OSError):
+    """insufficient buffer memory for completing a xattr operation."""
+
+    def __str__(self):
+        return self.__doc__
 
 
 def _check(rv, path=None, detect_buffer_too_small=False):
@@ -205,7 +208,7 @@ def func(path, buf, size):
 
         n, buf = _listxattr_inner(func, path)
         return [os.fsdecode(name) for name in split_string0(buf[:n])
-                if not name.startswith(b'system.posix_acl_')]
+                if name and not name.startswith(b'system.posix_acl_')]
 
     def getxattr(path, name, *, follow_symlinks=True):
         def func(path, name, buf, size):
@@ -261,7 +264,7 @@ def func(path, buf, size):
                     return libc.listxattr(path, buf, size, XATTR_NOFOLLOW)
 
         n, buf = _listxattr_inner(func, path)
-        return [os.fsdecode(name) for name in split_string0(buf[:n])]
+        return [os.fsdecode(name) for name in split_string0(buf[:n]) if name]
 
     def getxattr(path, name, *, follow_symlinks=True):
         def func(path, name, buf, size):
@@ -320,7 +323,7 @@ def func(path, buf, size):
                     return libc.extattr_list_link(path, ns, buf, size)
 
         n, buf = _listxattr_inner(func, path)
-        return [os.fsdecode(name) for name in split_lstring(buf[:n])]
+        return [os.fsdecode(name) for name in split_lstring(buf[:n]) if name]
 
     def getxattr(path, name, *, follow_symlinks=True):
         def func(path, name, buf, size):


Test patch: