Repository: borgbackup/borg

Issue:
borg create throws os.fsync error on cifs mounted repo
Observed both on Arch linux systems running the binary package and Debian GNU/Linux systems compiling from pip. With borg 1.05, any attempt to create a new archive in a repository that is accessible via cifs begins to run the create and then errors as soon as it reaches the first new file that needs to be written.  Backups to purely local filesystems or remote backups via ssh with the same repositories appear to work perfectly.

The systems in question had no similar problem with any prior versions of borg, and a repository check on the relevant repositories comes back clean.

An example of the errors thrown:

```
Exception ignored in: <bound method Repository.__del__ of <Repository /mnt/backups/Saturnine.borg>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 72, in __del__
    self.close()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 189, in close
    self.io.close()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 532, in close
    self.close_segment()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 752, in close_segment
    self._write_fd.flush()
ValueError: flush of closed file
Local Exception.
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 81, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 247, in do_create
    create_inner(archive, cache)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 219, in create_inner
    read_special=args.read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 301, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 275, in _process
    status = archive.process_file(path, st, cache, self.ignore_inode)
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 681, in process_file
    self.add_item(item)
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 279, in add_item
    self.items_buffer.add(item)
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 133, in add
    self.flush()
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 148, in flush
    self.chunks.append(self.write_chunk(chunk))
  File "/usr/local/lib/python3.4/dist-packages/borg/archive.py", line 164, in write_chunk
    id_, _, _ = self.cache.add_chunk(self.key.id_hash(chunk), chunk, self.stats)
  File "/usr/local/lib/python3.4/dist-packages/borg/cache.py", line 365, in add_chunk
    self.repository.put(id, data, wait=False)
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 481, in put
    segment, offset = self.io.write_put(id, data)
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 721, in write_put
    fd = self.get_write_fd(raise_full=raise_full)
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 608, in get_write_fd
    self.close_segment()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 759, in close_segment
    sync_dir(os.path.dirname(self._write_fd.name))
  File "/usr/local/lib/python3.4/dist-packages/borg/platform.py", line 9, in sync_dir
    os.fsync(fd)
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 1601, in main
    exit_code = archiver.run(args)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 1538, in run
    return args.func(args)
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 81, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 94, in __exit__
    self.close()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 189, in close
    self.io.close()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 532, in close
    self.close_segment()
  File "/usr/local/lib/python3.4/dist-packages/borg/repository.py", line 752, in close_segment
    self._write_fd.flush()
ValueError: flush of closed file

Platform: Linux Saturnine 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u2 (2016-06-25) x86_64 
Linux: debian 8.5 
Borg: 1.0.5  Python: CPython 3.4.2
PID: 19650  CWD: /home/alraban
sys.argv: ['/usr/local/bin/borg', 'create', '-sv', '-C', 'lz4', '-e', '/proc/', '-e', '/sys/', '-e', '/run/', '-e', '/tmp/', '-e', '/dev/', '-e', '/mnt/', '-e', '/home/pi/backups', '/mnt/backups/Saturnine.borg::2016-07-08-09', '/']

```

The error on the Arch systems is substantially similar. Let me know if I can provide any additional information or testing.



Discussion:
Looks like smbfs does not like a flush on a directory fd (this is new in 1.0.4/1.0.5 to make sure the directory is really on-disk to be more robust, e.g. in case of crashes or power failures).

Workaround: go back to 1.0.3 until a fix is available.

Can you test a sourcecode patch for 1.0.5?

Maybe we just use the diaper pattern here.

(Previous comment deleted, since not accurate).

Ignoring EINVAL should be ok. sshfs issue isn't related, just tested. I have no traceback for that one, just an error code, so ...

@enkore can we be sure that EINVAL is all we get to see from this? And not some other fs throwing another error and it blows up again?

btw, we need more beta testers.

fsync has EBADF¹, EINTR², EINVAL, EIO³ (and those of read/write with the same meanings). Anything else means the FS/Kernel is _severely_ broken.

¹ can't happen - open succeeded
² handled by Python.
³ don't want to shadow EIO

> btw, we need more beta testers.

Yes.

Also a checklist when we touch IO code. Works on proper FS. Works on NTFS. Works on non-compliant FS (sshfs, cifs/smb, nfs, ...).

@alraban can you test #1288?


Resolving patch:
diff --git a/borg/platform.py b/borg/platform.py
--- a/borg/platform.py
+++ b/borg/platform.py
@@ -1,3 +1,4 @@
+import errno
 import os
 import sys
 
@@ -7,6 +8,11 @@ def sync_dir(path):
     fd = os.open(path, os.O_RDONLY)
     try:
         os.fsync(fd)
+    except OSError as os_error:
+        # Some network filesystems don't support this and fail with EINVAL.
+        # Other error codes (e.g. EIO) shouldn't be silenced.
+        if os_error.errno != errno.EINVAL:
+            raise
     finally:
         os.close(fd)
 


Test patch: