Repository: borgbackup/borg

Issue:
unintended file cache eviction
When creating archives, the newest file will not be registered in the file cache in order to avoid data loss when working with snapshots (see FAQ). At the end of the backup borg simply removes everything from the file cache with with an mtime the same or newer than the newest mtime seen during the current run. This can result in unintended file cache eviction in the following situations:

1) when interrupting the backup process

2) when a mount point is unavailable during the backup

3) when creating multiple archives from different sources in one repository

In the latter situation the backup is still okay, but a lot of files will be re-hashed over and over again, resulting in long backup times.

The question is... do we really have to forget the newest file when working with nanosecond timestamps? And if so, would it be possible to only evict this single cache entry instead of everything with a newer mtime?



Discussion:
It is "same mtime as the newest mtime seen in the backup run" (obviously there is no newer than newest).

I don't understand your 1), 2), 3) - can you add more details / proof of the problem you see there?

In general, the problem is that we can not rely on nanosecond timestamp granularity. Granularity can vary widely, even within the same backup set, between ns and few seconds, depending on the filesystem mounted and other parameters.

Imagine you make a snapshot at 1.1.2016 23:00:00 and backup that.

The files are in this state at snapshot time:

```
name mtime             size
A    1.1.2016 22:59:00 1234
B    1.1.2016 23:00:00 4321
```

B is the file with the newest mtime and would normally not enter the files cache.
Assume we change that and store B in the files cache.

Immediately after taking the snapshot, but still at 1.1.2016 23:00:00, file B's content (but not its size) gets changed, this is how it looks after the change:

```
name mtime             size
A    1.1.2016 22:59:00 1234
B    1.1.2016 23:00:00 4321
```

You see: mtime and size of B have not changed, but its content has changed.
If B was in the files cache, borg would skip B and not backup the changed contents in the next backup.

I assumed a 1s timestamp granularity here, but same problem is there even with better granularity.

So, I don't currently see how we could do better here.

The FAQ has a workaround to keep effects of this as small as possible. E.g. if you are annoyed because always your huge VM happens to have the newest mtime, you can just touch some arbitrary small/empty file within your backup set and then snapshot.

Thanks for the explanation, but this does not exactly match what I mean. I'll try to explain my issue more carefully...

When the file cache is written, it will only contain entries where the entry time is less than the ttl and where the mtime is less than the newest mtime, anything else is discarded. However, the "newest mtime" is only calculated for files that we have actually seen while making the backup. If there are any entries with a more recent mtime in the cache that we did _not_ see while making the backup, they will be discarded, too. Example:

Imagine having two folders, _/archive/july_ and _/archive/august_, each containing files with a date within that month. Now create two distinct backup archives within the same repository with `borg create /backup::july-{now} /archive/july` and `borg create /backup::august-{now} /archive/august`. If you repeat this, for example because it's part of a daily backup, the following will happen:

After creating the july backup, `self._newest_mtime` will at max be 31-Jul 23:59 because we have not read any files from _/archive/august_, and as a result all the august files will be removed from the file cache. When we then create the august backup, all the files have to be re-hashed - every time! This is no big deal for small backups, but it is a problem when you happen to re-hash a TB-sized folder every time.

I'm now simply avoiding the issue by sticking to only one sort of archive per repository (i.e. `borg create /backup::combined-{now} /archive/july /archive/august`) - this works for me.

However, after studying the code again today I think the issue can easily be fixed with a simple change in the cache eviction logic, that is:

in cache.py:

```
if (item[0] == 0 and bigint_to_int(item[3]) < self._newest_mtime)
    or (item[0] > 0 and item[0] < ttl):
    msgpack.pack((path_hash, item), fd) 
```

The intention is: apply the mtime restriction only to files we've seen while making the backup; keep all other entries as long as they're within the ttl.

Good catch!
Also, your fix looks correct.
Thanks for finding and fixing this likely years old bug!


Resolving patch:
diff --git a/borg/cache.py b/borg/cache.py
--- a/borg/cache.py
+++ b/borg/cache.py
@@ -196,10 +196,13 @@ def commit(self):
             ttl = int(os.environ.get('BORG_FILES_CACHE_TTL', 20))
             with open(os.path.join(self.path, 'files'), 'wb') as fd:
                 for path_hash, item in self.files.items():
-                    # Discard cached files with the newest mtime to avoid
-                    # issues with filesystem snapshots and mtime precision
+                    # Only keep files seen in this backup that are older than newest mtime seen in this backup -
+                    # this is to avoid issues with filesystem snapshots and mtime granularity.
+                    # Also keep files from older backups that have not reached BORG_FILES_CACHE_TTL yet.
                     item = msgpack.unpackb(item)
-                    if item[0] < ttl and bigint_to_int(item[3]) < self._newest_mtime:
+                    age = item[0]
+                    if age == 0 and bigint_to_int(item[3]) < self._newest_mtime or \
+                       age > 0 and age < ttl:
                         msgpack.pack((path_hash, item), fd)
         self.config.set('cache', 'manifest', hexlify(self.manifest.id).decode('ascii'))
         self.config.set('cache', 'timestamp', self.manifest.timestamp)


Test patch: