Repository: borgbackup/borg

Issue:
caching repo uses large amounts of temp. disk space
Contexts (I think it's not super relevant, as I have another repo with another context and the same issue):
1. An attic repository, containing incremental snapshots, done on a server which a high uptime and ECC memory. Made a copy, did a conversion.
2. borg check passes without problem

Scenario: perform a "tar c" of the contents of the mount, just to read all the files.
Expected result: borg mount doesn't crash, doesn't show errors, because `borg check` succeeded.
Actual result: 

`tar c` says:

```
tar: Removing leading `/' from member names
tar: [...]/borg-mount/2015-09-10T07-50-24/[...]/filename: File shrank by 8532154 bytes; padding with zeros
tar: [...]/borg-mount/2015-09-10T07-50-24/[...]/next-filename: Cannot stat: Transport endpoint is not connected
```

`borg mount` says:

```
Local Exception.
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/borg/remote.py", line 437, in get_many
    yield self.caching_repo.get(key)
  File "/usr/lib64/python3.4/site-packages/borg/repository.py", line 450, in get
    raise self.ObjectNotFound(id_, self.path) from None
borg.repository.ObjectNotFound: (b'\xd2u\xe0\xef\xe0\xfb\xeb\xde\xdcVF-j\xf6\x1e\xcf\x81iW\xb2\xec\xe2\xec\x8arv\xae\x18\xa0\xa4\x86\x02', '/tmp/borg-tmpnrievx_4')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 2127, in main
    exit_code = archiver.run(args)
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 2077, in run
    return args.func(args)
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 82, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line 716, in do_mount
    operations.mount(args.mountpoint, args.options, args.foreground)
  File "/usr/lib64/python3.4/site-packages/borg/fuse.py", line 281, in mount
    signal = fuse_main()
  File "/usr/lib64/python3.4/site-packages/borg/fuse.py", line 24, in fuse_main
    return llfuse.main(workers=1)
  File "src/fuse_api.pxi", line 337, in llfuse.main (src/llfuse.c:35111)
  File "src/handlers.pxi", line 324, in llfuse.fuse_read (src/llfuse.c:14688)
  File "src/handlers.pxi", line 325, in llfuse.fuse_read (src/llfuse.c:14638)
  File "/usr/lib64/python3.4/site-packages/borg/fuse.py", line 245, in read
    _, data = self.key.decrypt(id, self.repository.get(id))
  File "/usr/lib64/python3.4/site-packages/borg/remote.py", line 409, in get
    return next(self.get_many([key]))
  File "/usr/lib64/python3.4/site-packages/borg/remote.py", line 441, in get_many
    self.caching_repo.put(key, data)
  File "/usr/lib64/python3.4/site-packages/borg/repository.py", line 468, in put
    segment, offset = self.io.write_put(id, data)
  File "/usr/lib64/python3.4/site-packages/borg/repository.py", line 706, in write_put
    fd.write(b''.join((crc, header, id, data)))
  File "borg/platform_linux.pyx", line 235, in borg.platform_linux.SyncFile.write (borg/platform_linux.c:5721)
OSError: [Errno 28] No space left on device

Platform: Linux Cabochon 4.5.0-Vantage-dirty #1 SMP PREEMPT Tue Mar 29 13:02:33 EDT 2016 x86_64 Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Linux: Gentoo Base System 2.2
Borg: 1.0.3.dev210+ng088bccc  Python: CPython 3.4.3
PID: 27885  CWD: /home/cJ
```

Note that here all is running on a faster consumer CPU, but I swear I have the same problem with ECC stuff. I have observed that borg mount file access is just significantly slower on the server.



Discussion:
> I have observed that borg mount file access is just significantly slower on the server.

Could be differences between the version, if the server runs 1.0 or older master branch. The fuse code in master recently received a performance fix for some use cases (depending on the application 30-100x speedup), I see that your traceback comes from a very recent version that has these improvements.

The problem at hand is the "stupidity" of Caching repo. It's not a consistency error in your repository.

Reimplementing the Cache doesn't look like something for 1.0.x.

The current implementation of #1066 is quite a bit more than what's in the PR right now (to back off early etc). diff stat +96-27 vs. +29-15 to be precise.

So I'd say we put only a workaround in 1.0.3, namely, not caching large objects. Those are relatively unlikely to appear again (at least in my archives they are) anyway.


Resolving patch:
diff --git a/borg/remote.py b/borg/remote.py
--- a/borg/remote.py
+++ b/borg/remote.py
@@ -414,6 +414,9 @@ class RepositoryCache(RepositoryNoCache):
 
     Caches Repository GET operations using a local temporary Repository.
     """
+    # maximum object size that will be cached, 64 kiB.
+    THRESHOLD = 2**16
+
     def __init__(self, repository):
         super().__init__(repository)
         tmppath = tempfile.mkdtemp(prefix='borg-tmp')
@@ -434,7 +437,8 @@ def get_many(self, keys):
             except Repository.ObjectNotFound:
                 for key_, data in repository_iterator:
                     if key_ == key:
-                        self.caching_repo.put(key, data)
+                        if len(data) <= self.THRESHOLD:
+                            self.caching_repo.put(key, data)
                         yield data
                         break
         # Consume any pending requests


Test patch: