Repository: borgbackup/borg

Issue:
deal with old unused FDs in lrucache
https://github.com/borgbackup/borg/issues/4360#issuecomment-467780888

as noted there, there might be a problem with old, open, but idle / unused FDs in the lrucache.

if we try to fetch an FD for a specific segment from the lrucache, we check if it's old and if so, we discard it (and the discard code also closes it) and rather reopen the segment file to avoid issues with stale file handles.

but if an FD just sits in the the cache for a long time, it might unnecessarily bind resources (not much within borg, but could be much in the filesystem's cache as seen in the davfs discussion).

ideas:
- borg usually works rather linearly, always appending stuff at the end when creating an archive and reading over it linearly when extracting an archive.
- exception: borg's fuse fs (borg mount): archive metadata is read once in a linear way, but we can't predict how content data is read
- also repo.list is not in on-disk order, repo.scan is.
- having a lot of old and idle FDs could be reduced by reducing the lrucache size, but considering fuse mounts and other non-linear access that is maybe not a good idea.
- we could also scan all lrucache entries and discard old stuff every time we fetch an entry from there if N seconds have elapsed since last fetch.


Discussion:
I'll be happy to test changes that may improve DavFS cache usage!

Resolving patch:
diff --git a/src/borg/lrucache.py b/src/borg/lrucache.py
--- a/src/borg/lrucache.py
+++ b/src/borg/lrucache.py
@@ -50,7 +50,6 @@ def clear(self):
             self._dispose(value)
         self._cache.clear()
 
-    # useful for testing
     def items(self):
         return self._cache.items()
 
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -1216,6 +1216,7 @@ def __init__(self, path, limit, segments_per_dir, capacity=90):
         self.segments_per_dir = segments_per_dir
         self.offset = 0
         self._write_fd = None
+        self._fds_cleaned = 0
 
     def close(self):
         self.close_segment()
@@ -1346,20 +1347,26 @@ def open_fd():
             self.fds[segment] = (now, fd)
             return fd
 
+        def clean_old():
+            # we regularly get rid of all old FDs here:
+            if now - self._fds_cleaned > FD_MAX_AGE // 8:
+                self._fds_cleaned = now
+                for k, ts_fd in list(self.fds.items()):
+                    ts, fd = ts_fd
+                    if now - ts > FD_MAX_AGE:
+                        # we do not want to touch long-unused file handles to
+                        # avoid ESTALE issues (e.g. on network filesystems).
+                        del self.fds[k]
+
+        clean_old()
         try:
             ts, fd = self.fds[segment]
         except KeyError:
             fd = open_fd()
         else:
-            if now - ts > FD_MAX_AGE:
-                # we do not want to touch long-unused file handles to
-                # avoid ESTALE issues (e.g. on network filesystems).
-                del self.fds[segment]
-                fd = open_fd()
-            else:
-                # fd is fresh enough, so we use it.
-                # also, we update the timestamp of the lru cache entry.
-                self.fds.upd(segment, (now, fd))
+            # we only have fresh enough stuff here.
+            # update the timestamp of the lru cache entry.
+            self.fds.upd(segment, (now, fd))
         return fd
 
     def close_segment(self):


Test patch: