Repository: borgbackup/borg

Issue:
stale file handle when checkpointing
Since upgrading this error keeps happening when I hit large zip files. tried all types of compression, no difference. this was working fine with 1.0.1
```
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4024, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 3952, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 148, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 528, in do_create
    create_inner(archive, cache)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 492, in create_inner
    read_special=args.read_special, dry_run=dry_run, st=st)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 593, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 593, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 593, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 593, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 571, in _process
    status = archive.process_file(path, st, cache, self.ignore_inode, self.files_cache_mode)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 991, in process_file
    self.chunk_file(item, cache, self.stats, backup_io_iter(self.chunker.chunkify(fd, fh)))
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 923, in chunk_file
    from_chunk, part_number = self.write_part_file(item, from_chunk, part_number)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 905, in write_part_file
    self.write_checkpoint()
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 438, in write_checkpoint
    self.save(self.checkpoint_name)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 476, in save
    self.manifest.write()
  File "/usr/lib/python3/dist-packages/borg/helpers.py", line 410, in write
    self.repository.put(self.MANIFEST_ID, self.key.encrypt(data))
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1048, in put
    size = self.io.read(segment, offset, id, read_data=False)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1356, in read
    header = fd.read(self.put_header_fmt.size)
OSError: [Errno 116] Stale file handle

Platform: Linux Alexis-Machine 4.11.0-14-generic #20~16.04.1-Ubuntu SMP Wed Aug 9 09:06:22 UTC 2017 x86_64 x86_64
Linux: LinuxMint 18.2 sonya
Borg: 1.1.0  Python: CPython 3.5.2
PID: 22212  CWD: /etc/cron.monthly
sys.argv: ['/usr/bin/borg', 'create', '--debug', '--exclude-caches', '--list', '--one-file-system', '/mnt/backup/borg-data-repo::data-{now:%Y-%m-%d}', '/media/ironStorage/mail', '/media/ironStorage/Users', '/media/ironStorage/media/_Books', '/media/ironStorage/media/_NewMusic', '/media/ironStorage/Files', '/media/ironStorage/sys-backups', '/media/ironStorage/media/mp3', '/media/ironStorage/Temp', '/media/redStorage/Public', '/media/redStorage/HockeyVideos']
SSH_ORIGINAL_COMMAND: None
```
Archive data-2017-11-04 does not exist



Discussion:
this is new in 1.1: it creates checkpoints even in the middle of files.

btw, rather use 1.1.latest (currently 1.1.1). there were some bug fixes (but maybe not this bug).
Is /mnt/backup/borg-data-repo on NFS?
I will get 1.1.1 when my ppa roles the update out (LocutusOfBorg PPA)

Yes, the repo is located on a NFS mount on a NAS server on the LAN.

I changed the --checkpoint-interval to be 24 hours. This solved the fail during borg create, but it seems it is doomed to fail as when the operation completes it borks again. 

Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4024, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 3952, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 148, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 528, in do_create
    create_inner(archive, cache)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 494, in create_inner
    archive.save(comment=args.comment, timestamp=args.timestamp)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 476, in save
    self.manifest.write()
  File "/usr/lib/python3/dist-packages/borg/helpers.py", line 410, in write
    self.repository.put(self.MANIFEST_ID, self.key.encrypt(data))
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1048, in put
    size = self.io.read(segment, offset, id, read_data=False)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1356, in read
    header = fd.read(self.put_header_fmt.size)
OSError: [Errno 116] Stale file handle

Can you confirm your previous borg version number and that this worked with the same repo and NFS configuration?
I am not sure of the exact previous version, but from the ppa history it should have been 1.0.11-2.
Looking into the stale handle issue from my NAS server, it did have a default timeout of 5 minutes - I changed this to be 24 hours and went back to default borg checkpoints intervals. This seems to have corrected the issue as I've passed by several check point intervals now without a stale handle error.
Otherwise the previous version had no issues with the exact same settings - I run a monthly backup using a script.
Can you add the NFS settings you have changed to fix this to this ticket?
I stand corrected, it just failed for a stale file handle at a check point write. 

I use unraid NAS and the NFS parameter is called "Tunable (fuse_remember):" 

https://forums.lime-technology.com/topic/47534-stale-file-handle/

Hmm, I am a bit unsure if that is something that borg should / must address or whether borg could expect the fs layer to be more well-behaved.

Opinions? @enkore 
https://forums.lime-technology.com/topic/25097-stale-nfs-file-handles/?do=findComment&comment=231586

more detail. perhaps an issue with the NAS - but 1.0.11 did not exhibit this issue - so it appears something changed with 1.1.1 to be sensitive to it. Doing another run with the client mount disabling caching (noac).

Ok, did some testing and managed to get it to work consistently.
unraid has a default handle timeout of 5 1/2 minutes, bumped that to 2 hours
ubuntu nfs mount also has a timeout, to get around that I used "--checkpoint-interval 900" and it appears that 15 minutes is enough to keep handles fresh, the default 30 minutes is too much.
I didn't fiddle to see what the upper boundary was before it times out.

Ideas for "fixing" this on the borg side:
- adding a time component to the repo FD cache and not using FDs that were not requested in the last 5 minutes
- adding a wrapper around the file object that catches ESTALE and closes/reopens the file transparently (guess it would also need to track seek position)

Resolving patch:
diff --git a/src/borg/constants.py b/src/borg/constants.py
--- a/src/borg/constants.py
+++ b/src/borg/constants.py
@@ -52,6 +52,8 @@
 
 DEFAULT_SEGMENTS_PER_DIR = 1000
 
+FD_MAX_AGE = 4 * 60  # 4 minutes
+
 CHUNK_MIN_EXP = 19  # 2**19 == 512kiB
 CHUNK_MAX_EXP = 23  # 2**23 == 8MiB
 HASH_WINDOW_SIZE = 0xfff  # 4095B
diff --git a/src/borg/lrucache.py b/src/borg/lrucache.py
--- a/src/borg/lrucache.py
+++ b/src/borg/lrucache.py
@@ -39,6 +39,12 @@ def get(self, key, default=None):
         self._lru.append(key)
         return value
 
+    def upd(self, key, value):
+        # special use only: update the value for an existing key without having to dispose it first
+        # this method complements __setitem__ which should be used for the normal use case.
+        assert key in self._cache, "Unexpected attempt to update a non-existing item."
+        self._cache[key] = value
+
     def clear(self):
         for value in self._cache.values():
             self._dispose(value)
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -3,6 +3,7 @@
 import os
 import shutil
 import struct
+import time
 from binascii import hexlify, unhexlify
 from collections import defaultdict
 from configparser import ConfigParser
@@ -1135,8 +1136,7 @@ class SegmentFull(Exception):
 
     def __init__(self, path, limit, segments_per_dir, capacity=90):
         self.path = path
-        self.fds = LRUCache(capacity,
-                            dispose=self.close_fd)
+        self.fds = LRUCache(capacity, dispose=self._close_fd)
         self.segment = 0
         self.limit = limit
         self.segments_per_dir = segments_per_dir
@@ -1148,7 +1148,8 @@ def close(self):
         self.fds.clear()
         self.fds = None  # Just to make sure we're disabled
 
-    def close_fd(self, fd):
+    def _close_fd(self, ts_fd):
+        ts, fd = ts_fd
         safe_fadvise(fd.fileno(), 0, 0, 'DONTNEED')
         fd.close()
 
@@ -1262,13 +1263,31 @@ def get_write_fd(self, no_new=False, raise_full=False):
         return self._write_fd
 
     def get_fd(self, segment):
-        try:
-            return self.fds[segment]
-        except KeyError:
+        # note: get_fd() returns a fd with undefined file pointer position,
+        # so callers must always seek() to desired position afterwards.
+        now = time.monotonic()
+
+        def open_fd():
             fd = open(self.segment_filename(segment), 'rb')
-            self.fds[segment] = fd
+            self.fds[segment] = (now, fd)
             return fd
 
+        try:
+            ts, fd = self.fds[segment]
+        except KeyError:
+            fd = open_fd()
+        else:
+            if now - ts > FD_MAX_AGE:
+                # we do not want to touch long-unused file handles to
+                # avoid ESTALE issues (e.g. on network filesystems).
+                del self.fds[segment]
+                fd = open_fd()
+            else:
+                # fd is fresh enough, so we use it.
+                # also, we update the timestamp of the lru cache entry.
+                self.fds.upd(segment, (now, fd))
+        return fd
+
     def close_segment(self):
         # set self._write_fd to None early to guard against reentry from error handling code paths:
         fd, self._write_fd = self._write_fd, None


Test patch: