Repository: borgbackup/borg

Issue:
mtime cache misbehaving on nested backups
I am seeing borg mtime cache consistently become ineffective during my backups.  I am suspecting it has something to do with how these are structured.  This is just a guess, but I am hoping somebody will recognize the pattern.

Given file hierarchy:
```
/foo
/foo/a
/foo/b/
```

I am backing it up as 3 archives:
* /foo/a 
* /foo/b
* then /foo <- here borg cache becomes ineffective and it rescans all the files

I am running with `export BORG_FILES_CACHE_TTL=10000` 
Borg Version: 1.0.7-0ubuntu1.16.04.1

This seems related to #1430 but the fix for that was already included in 1.0.7


Discussion:
Hi @gleb,

some questions:

"I am seeing borg mtime cache consistently become ineffective during my backups." - how do you see that?

Does the archive of /foo include /foo/a/\* and /foo/b/\* also?

Also, please give the full command sequence you use.

I am using the very helpful `--filter=AME`   I believe this shows the files that weren't reported as not-changed by the cache.   They show up as A.  These correspond to the extra time/coil whine/HD light of sequential IO. But the data is deduplicated, and is not backed up again.

```
# excludes that apply to all archives
BORG_EXCLUDE_OPTS="--exclude-caches --keep-tag-files --exclude */.bundle/gems"

BORG="borg create --compression=lz4 --show-rc --stats --list --filter=AME -v ${BORG_EXCLUDE_OPTS}"
```

Yes, all the backups are recursive.   Actually a more complete description of directory structure would be something like this
- /foo/{1,2,3,4,5,6,7,8}/  <- 8 subdirectories are backed up as separate archives
- /foo/{more directories and some files} <- these are not backed up independently
- /foo  <- this backs up foo including all subdirectories and shows the problem
- /bar, etc <- more subdirectores are backed up
- / <- everything is backed up including everything above

Please check with borg list if the full archived pathes of the files you suspect being chunked/hashed again matches completely.

The cache key is H(path) with H being a hash function, so we only get a hit if the path is exactly the same.

Maybe related to #909 and maybe fixed by #1842.

Yes, paths are the same.   In fact for given file the text that `borg list` outputs in /foo and /foo/1 archives is identical.

In hash function above is path a relative path passed in or true/canonicalized file path?   It doesn't affect this issue, but is good to know.

No, i just checked that: the hash function gets the full, absolute path of the file.

```
key = id_hash(os.path.join(self.cwd, path)
```

The path join is a bit strange, but it creates the full absolute path of the file, even if you invoke `borg create repo::archive relpath`.

I modified borg/archive.py so one sees the pathes used for the path_hash (ph: ...):

```
        if not is_special_file:
            print("ph:", os.path.join(self.cwd, path))  # for debugging
            path_hash = self.key.id_hash(os.path.join(self.cwd, path).encode('utf-8', 'surrogateescape'))
            ids = cache.file_known_and_unchanged(path_hash, st, ignore_inode)
```

Besides path not matching, it could be also that borg thinks the file is modified.
This is the case if mtime, inode number or size changed.
What filesystems do you back up?
Is there something "touching" your files in between (modifying mtime)?

No, nothing is touching the files.  I mean /foo/{1-8} show no changed files, and immediately archiving /foo afterwards does.   ext4, very vanilla, single file system.

It's as if using the cache in /foo/{1-8} invalidates the cache.

Is there a python oneliner I can use to dump the cache matching a particular path?

Also, I verified that BORG_FILES_CACHE_TTL=10000 makes it into /proc/(pid)/environ

To my great surprise I was able to reproduce this problem on a standalone test case on the first try:

```
set -x
export BORG_FILES_CACHE_TTL=10000 # infinite effectively
BORG_EXCLUDE_OPTS="--exclude-caches --keep-tag-files --exclude */.bundle/gems"
BORG="borg create --compression=lz4 --show-rc --stats --list --filter=AME -v ${BORG_EXCLUDE_OPTS}"

rm -r test-repo
rm -r backup-test
export BORG_PASSPHRASE='test'
export BORG_REPO="test-repo"
borg init -e keyfile test-repo
mkdir backup-test

mkdir backup-test/{1,2,3,4,5,6,7,8,9,10,11,12}
touch backup-test/{1,2,3,4,5,6,7,8,9,10,11,12}/file.txt
touch backup-test/{1,2,3,4,5,6,7,8,9,10,11,12}/file2.txt
touch backup-test/another_file.txt

# run this block more than once
$BORG ::1_{now} backup-test/1 2>&1 | egrep '^A '
$BORG ::1_{now} backup-test/2 2>&1 | egrep '^A '
$BORG ::1_{now} backup-test/3 2>&1 | egrep '^A '
$BORG ::1_{now} backup-test/4 2>&1 | egrep '^A '
$BORG ::all_{now} backup-test/ 2>&1 | egrep '^A '
```

Even weirder, this is timing related.   By sticking sleep in different places in this script I am getting different output.

Here's a simplified script:

```
set -x
export BORG_FILES_CACHE_TTL=10000 # infinite effectively
BORG="borg create --list --filter=AME -v"

rm -r test-repo
rm -r backup-test
export BORG_PASSPHRASE='test'
export BORG_REPO="test-repo"
borg init -e keyfile test-repo
mkdir backup-test

# populate data
mkdir backup-test/{1,2,3,4}
touch backup-test/{1,2,3,4}/file.txt
touch backup-test/{1,2,3,4}/file2.txt
touch backup-test/another_file.txt

echo "first run"
$BORG ::1_{now} backup-test/1 2>&1 | egrep '^A '
$BORG ::1_{now} backup-test/2 2>&1 | egrep '^A '
$BORG ::all_{now} backup-test 2>&1 | egrep '^A '

sleep 5

echo "second run"
$BORG ::1_{now} backup-test/1 2>&1 | egrep '^A '
$BORG ::1_{now} backup-test/2 2>&1 | egrep '^A '
$BORG ::all_{now} backup-test 2>&1 | egrep '^A '
```

ok, I'll try to reproduce / debug this.

Slightly cleaned up and extended script:

```
#!/bin/bash
export BORG_REPO="test-repo"
BORG="borg create --list -v"

borg delete ::
borg init -e none ::

# populate data
rm -r backup-test
mkdir backup-test
mkdir backup-test/{1,2,3,4}
touch backup-test/{1,2,3,4}/file.txt
touch backup-test/{1,2,3,4}/file2.txt
touch backup-test/another_file.txt

# this is to avoid the "latest mtime" issue, see FAQ.
# "A" status for latest-mtime.txt is EXPECTED!
sleep 2
touch backup-test/{1,2,3,4}/latest-mtime.txt

echo "first run"
$BORG ::1_{now} backup-test/1 2>&1
$BORG ::1_{now} backup-test/2 2>&1
$BORG ::all_{now} backup-test 2>&1

sleep 5

echo "second run"
$BORG ::1_{now} backup-test/1 2>&1
$BORG ::1_{now} backup-test/2 2>&1
$BORG ::all_{now} backup-test 2>&1
```

Output:

```
++ export BORG_REPO=test-repo
++ BORG_REPO=test-repo
++ BORG='borg create --list -v'
++ borg delete ::
You requested to completely DELETE the repository *including* all archives it contains:
1_2016-11-19 20:43:13.982546         Sat, 2016-11-19 20:43:13
1_2016-11-19 20:43:14.256657         Sat, 2016-11-19 20:43:14
all_2016-11-19 20:43:14.532489       Sat, 2016-11-19 20:43:14
1_2016-11-19 20:43:19.810054         Sat, 2016-11-19 20:43:19
1_2016-11-19 20:43:20.082959         Sat, 2016-11-19 20:43:20
all_2016-11-19 20:43:20.414305       Sat, 2016-11-19 20:43:20
Type 'YES' if you understand this and want to continue: YES
++ borg init -e none ::
++ rm -r backup-test
++ mkdir backup-test
++ mkdir backup-test/1 backup-test/2 backup-test/3 backup-test/4
++ touch backup-test/1/file.txt backup-test/2/file.txt backup-test/3/file.txt backup-test/4/file.txt
++ touch backup-test/1/file2.txt backup-test/2/file2.txt backup-test/3/file2.txt backup-test/4/file2.txt
++ touch backup-test/another_file.txt
++ sleep 2
++ touch backup-test/1/latest-mtime.txt backup-test/2/latest-mtime.txt backup-test/3/latest-mtime.txt backup-test/4/latest-mtime.txt

++ echo 'first run'
first run
++ borg create --list -v '::1_{now}' backup-test/1
A backup-test/1/file.txt
A backup-test/1/file2.txt
A backup-test/1/latest-mtime.txt
d backup-test/1
++ borg create --list -v '::1_{now}' backup-test/2
A backup-test/2/file.txt
A backup-test/2/file2.txt
A backup-test/2/latest-mtime.txt
d backup-test/2
++ borg create --list -v '::all_{now}' backup-test
U backup-test/1/file.txt
U backup-test/1/file2.txt
A backup-test/1/latest-mtime.txt
d backup-test/1
U backup-test/2/file.txt
U backup-test/2/file2.txt
A backup-test/2/latest-mtime.txt
d backup-test/2
A backup-test/3/file.txt
A backup-test/3/file2.txt
A backup-test/3/latest-mtime.txt
d backup-test/3
A backup-test/4/file.txt
A backup-test/4/file2.txt
A backup-test/4/latest-mtime.txt
d backup-test/4
A backup-test/another_file.txt
d backup-test
++ sleep 5

++ echo 'second run'
second run
++ borg create --list -v '::1_{now}' backup-test/1
U backup-test/1/file.txt
U backup-test/1/file2.txt
A backup-test/1/latest-mtime.txt
d backup-test/1
++ borg create --list -v '::1_{now}' backup-test/2
U backup-test/2/file.txt
U backup-test/2/file2.txt
A backup-test/2/latest-mtime.txt
d backup-test/2
++ borg create --list -v '::all_{now}' backup-test
U backup-test/1/file.txt
U backup-test/1/file2.txt
A backup-test/1/latest-mtime.txt
d backup-test/1
U backup-test/2/file.txt
U backup-test/2/file2.txt
A backup-test/2/latest-mtime.txt
d backup-test/2
U backup-test/3/file.txt
U backup-test/3/file2.txt
A backup-test/3/latest-mtime.txt
d backup-test/3
U backup-test/4/file.txt
U backup-test/4/file2.txt
A backup-test/4/latest-mtime.txt
d backup-test/4
U backup-test/another_file.txt
d backup-test
```

As you see, only latest-mtime.txt has A status and that is expected, see FAQ.

So the only remaining question is if your original problem was also just with the "latest mtime" files?

I'ld appreciate if you base your script changes on my modified script. I had a reason why I removed all the grep and filter and crypto etc.

And I don't want to repeat same work again.

Also, if some sleeps do not matter, remove them. One sleep definitely matters, it is the one before touching latest-mtime.txt.

(also remove all posts above that are superceded by your next update)

I modified your script until it showed the problem.   The key seems to be adding `touch backup-test/latest-mtime.txt`  (which also happens to model my production environment better).   I also added `set -x` and removed repo with `rm -r test-repo` to simplify running the script - I don't believe these changes to be material.

I am happy to redo this if necessary based on your feedback.

``` diff
modified   borg_cache_bug
@@ -1,2 +1,3 @@
 #!/bin/bash
+set -x
 export BORG_REPO="test-repo"
@@ -4,3 +5,3 @@ BORG="borg create --list -v"

-borg delete ::
+rm -r test-repo
 borg init -e none ::
@@ -19,2 +20,5 @@ sleep 2
 touch backup-test/{1,2,3,4}/latest-mtime.txt
+sleep 2
+touch backup-test/latest-mtime.txt  # key to changing behavior
+sleep 2  # optional doesn't change behavior

@@ -25,3 +29,3 @@ $BORG ::all_{now} backup-test 2>&1

-sleep 5
+sleep 5 # optional, doesn't change behavior
```

Script:

``` bash
#!/bin/bash
set -x
export BORG_REPO="test-repo"
BORG="borg create --list -v"

rm -r test-repo
borg init -e none ::

# populate data
rm -r backup-test
mkdir backup-test
mkdir backup-test/{1,2,3,4}
touch backup-test/{1,2,3,4}/file.txt
touch backup-test/{1,2,3,4}/file2.txt
touch backup-test/another_file.txt

# this is to avoid the "latest mtime" issue, see FAQ.
# "A" status for latest-mtime.txt is EXPECTED!
sleep 2
touch backup-test/{1,2,3,4}/latest-mtime.txt
sleep 2
touch backup-test/latest-mtime.txt  # key to changing behavior
sleep 2  # optional doesn't change behavior

echo "first run"
$BORG ::1_{now} backup-test/1 2>&1
$BORG ::1_{now} backup-test/2 2>&1
$BORG ::all_{now} backup-test 2>&1

sleep 5 # optional, doesn't change behavior

echo "second run"
$BORG ::1_{now} backup-test/1 2>&1
$BORG ::1_{now} backup-test/2 2>&1
$BORG ::all_{now} backup-test 2>&1
```

Output:

```
+ export BORG_REPO=test-repo
+ BORG_REPO=test-repo
+ BORG='borg create --list -v'
+ rm -r test-repo
+ borg init -e none ::
+ rm -r backup-test
+ mkdir backup-test
+ mkdir backup-test/1 backup-test/2 backup-test/3 backup-test/4
+ touch backup-test/1/file.txt backup-test/2/file.txt backup-test/3/file.txt backup-test/4/file.txt
+ touch backup-test/1/file2.txt backup-test/2/file2.txt backup-test/3/file2.txt backup-test/4/file2.txt
+ touch backup-test/another_file.txt
+ sleep 2
+ touch backup-test/1/latest-mtime.txt backup-test/2/latest-mtime.txt backup-test/3/latest-mtime.txt backup-test/4/latest-mtime.txt
+ sleep 2
+ touch backup-test/latest-mtime.txt
+ sleep 2
+ echo 'first run'
first run
+ borg create --list -v '::1_{now}' backup-test/1
A backup-test/1/file.txt
A backup-test/1/file2.txt
A backup-test/1/latest-mtime.txt
d backup-test/1
+ borg create --list -v '::1_{now}' backup-test/2
A backup-test/2/file.txt
A backup-test/2/file2.txt
A backup-test/2/latest-mtime.txt
d backup-test/2
+ borg create --list -v '::all_{now}' backup-test
U backup-test/1/file.txt
U backup-test/1/file2.txt
A backup-test/1/latest-mtime.txt
d backup-test/1
U backup-test/2/file.txt
U backup-test/2/file2.txt
A backup-test/2/latest-mtime.txt
d backup-test/2
A backup-test/3/file.txt
A backup-test/3/file2.txt
A backup-test/3/latest-mtime.txt
d backup-test/3
A backup-test/4/file.txt
A backup-test/4/file2.txt
A backup-test/4/latest-mtime.txt
d backup-test/4
A backup-test/another_file.txt
A backup-test/latest-mtime.txt
d backup-test
+ sleep 5
+ echo 'second run'
second run
+ borg create --list -v '::1_{now}' backup-test/1
U backup-test/1/file.txt
U backup-test/1/file2.txt
U backup-test/1/latest-mtime.txt
d backup-test/1
+ borg create --list -v '::1_{now}' backup-test/2
U backup-test/2/file.txt
U backup-test/2/file2.txt
U backup-test/2/latest-mtime.txt
d backup-test/2
+ borg create --list -v '::all_{now}' backup-test
A backup-test/1/file.txt
A backup-test/1/file2.txt
A backup-test/1/latest-mtime.txt
d backup-test/1
A backup-test/2/file.txt
A backup-test/2/file2.txt
A backup-test/2/latest-mtime.txt
d backup-test/2
U backup-test/3/file.txt
U backup-test/3/file2.txt
U backup-test/3/latest-mtime.txt
d backup-test/3
U backup-test/4/file.txt
U backup-test/4/file2.txt
U backup-test/4/latest-mtime.txt
d backup-test/4
U backup-test/another_file.txt
A backup-test/latest-mtime.txt
d backup-test
```

Also, I verified that timestamps do not change during this script by running `stat backup-test/1/file.txt` as so:

``` diff
modified   borg_cache_bug
@@ -24,2 +24,4 @@ sleep 2  # optional doesn't change behavior

+stat backup-test/1/file.txt
+
 echo "first run"
@@ -32,4 +34,8 @@ sleep 5 # optional, doesn't change behavior
 echo "second run"
+stat backup-test/1/file.txt
 $BORG ::1_{now} backup-test/1 2>&1
+stat backup-test/1/file.txt
 $BORG ::1_{now} backup-test/2 2>&1
+stat backup-test/1/file.txt
 $BORG ::all_{now} backup-test 2>&1
+stat backup-test/1/file.txt
```

the "A" for the 2 latest-mtime files is easy to explain: their cache entry was killed by the first 2 borg commands in the second run.


Resolving patch:
diff --git a/borg/cache.py b/borg/cache.py
--- a/borg/cache.py
+++ b/borg/cache.py
@@ -191,7 +191,7 @@ def close(self):
 
     def _read_files(self):
         self.files = {}
-        self._newest_mtime = 0
+        self._newest_mtime = None
         logger.debug('Reading files cache ...')
         with open(os.path.join(self.path, 'files'), 'rb') as fd:
             u = msgpack.Unpacker(use_list=True)
@@ -222,6 +222,9 @@ def commit(self):
         if not self.txn_active:
             return
         if self.files is not None:
+            if self._newest_mtime is None:
+                # was never set because no files were modified/added
+                self._newest_mtime = 2 ** 63 - 1  # nanoseconds, good until y2262
             ttl = int(os.environ.get('BORG_FILES_CACHE_TTL', 20))
             with open(os.path.join(self.path, 'files'), 'wb') as fd:
                 for path_hash, item in self.files.items():
@@ -451,4 +454,4 @@ def memorize_file(self, path_hash, st, ids):
         # Entry: Age, inode, size, mtime, chunk ids
         mtime_ns = st.st_mtime_ns
         self.files[path_hash] = msgpack.packb((0, st.st_ino, st.st_size, int_to_bigint(mtime_ns), ids))
-        self._newest_mtime = max(self._newest_mtime, mtime_ns)
+        self._newest_mtime = max(self._newest_mtime or 0, mtime_ns)


Test patch: