Repository: borgbackup/borg

Issue:
1.1 create time ~85% higher than 1.0
I have the following scenario:
- **backup targets**: some servers exporting their contents via NFS to the **borg server**;
- **borg server**: a server with borg (Linux 64 bit binary) installed, mounting via NFS the content of **backup targets**; it's a VM with 4CPU and 8GB RAM;
- **borg repo**: a remote server mounted on **borg server** through SSHFS. **borg repo** storage can be accessed at about 6-10MB/s from **borg server**.

I was using borg 1.0.11 with that scenario:
- the **borg repo** was created with 1.0;
- I daily did many separate creates on **borg server** for each of the **backup targets** to the **borg repo**. I set BORG_FILE_CACHE_TTL to 4x of every daily creates, to avoid empting the cache before every day (with 4x it should last 4 days);
- at first I had the borg cache on **borg server**, but when its local disc was full, I deleted the cache and recreated it on the **borg repo**, using the env variable; I didn't notice any significant slowdown after moving the cache to the slow **borg repo** (at most 15-30 minutes);
- cache directory size was about 32GB;
- daily backup time of all archives was about 3h.

After update from 1.0.11 to 1.1.1:
- I replaced the 1.0.11 binary with 1.1.1 binary (no other significant changes in my script);
- first backup took about 30h (the slowdown on first 1.1 run was expected);
- cache directory size decreased to 20GB;
- second and third daily backup time of all archives are now about 5:30h. It looks the **backup targets** with many small files are the ones that get slowed a lot more (2-3x slower), other **backup targets** are just a bit slower.

So after the upgrade all the creates took about ~85% more time to complete. I suspect it may be due to the cache being on a slower location, maybe the cache size reduction in 1.1 now requires as a consequence to be accessed more times than in 1.0, so amplifying the slower cache location?
I would try moving again the cache to the **borg server**, but at 20GB it's still a bit bigger than the space available on its FS. If this is a know limitation I may increase the FS size and the move the cache there again, but I wanted to first document this behaviour, maybe it can be improved in borg?

Thanks for your time for reading this and for developing borg!


Discussion:
I don't think the cache is slower.

But the `--files-cache` change from mtime to ctime (see changelog) could make it slower if ctime is changing significantly more often than mtime for your files (you see an `A` in `borg create --list` output then, not an `U`).

For further analysis, the borg command lines of the slower stuff (before and after 1.1 migration) would be interesting.
I didn't use the `--list` option in the creates, so I don't have an history of file ctime changes.

On the borg server I set:

> BORG_CACHE_DIR=/mnt/sshfs/borg-cache
> BORG_PASSPHRASE=XXXXX
> BORG_REPO=/mnt/sshfs/borg_repository

Then, for each server target I run this command from the borg server:

> cd $TARGET_MOUNT_DIRECTORY && \\
> borg create -v --stats --show-rc --show-version --noatime -C zlib ::$ARCHIVE-$TODAY ${TARGET_DIRECTORIES[@]} &>>$BORG_LOG

The `--show-version` was added after upgrade to 1.1.
So you only added --show-version and didn't change the other options?

Correct.
(The only other change in my script was that for 1.0 I forced a borg break-lock if borg was not running, because from time to time a steal lock prevented the backup, I removed it for 1.1 since this is already addressed by borg)
OK, so you can try:

- use borg create --list and check whether files are unexpectedly often flagged as "A" (added) instead of "U" (unchanged) - if that is the case, you have a problem with ctime or inode number not being suitable for borg's files cache and you need a non-default value for --files-cache. One-time "many A" after switching --files-cache mode is expected, next backup should only have the really-changed files.

- check if --nobsdflags restores full speed. 1.0 did not gather bsdflags on linux, but 1.1 does. but maybe you do not need them. on linux, this does an extra open/fcntl on the file.

OK, I added both --list and --nobsdflags to latest create:
- I have really few A files (the ones that probably effectively changed);
- the backup times were reduced a lot (3:50h), still a bit more than with 1.0 (but it may just be random fluctuation) but a lot less than without --nobsdflags.

Note that:
- the systems I backup are all Linux systems + a EMC NAS VNX appliance (no idea what OS/FS it uses internally);
- the two slower systems to backup (the EMC NAS (2.5M+ files) and a Linux server (500K+ files)) are both a lot faster with --nobsdflags;
- while lsattr properly work on the local FS, when I use lsattr on the NFS mounted backup targets I consistently get:
> lsattr: Inappropriate ioctl for device While reading flags on DIRECTORY
> lsattr: Operation not supported While reading flags on FILE

Maybe the slowdown is due to borg trying to read bsdflags on a not supported FS, NFS in my case?
Could borg automatically set --nobsdflags when the FS doesn't support it?
Guess the speed penalty does not come from your FS not supporting the call, but from borg opening a lot of otherwise-unchanged files to query the bsdflags and then to store the bsdflags into the metadata stream.

Borg is always creating full backups (== all the fs items are in the metadata stream), so (under linux) it has to do an open() even for all the unchanged files.

Options:
- just document the speed penalty
- automatically inhibit bsd flags querying for a fs once we get "not supported2
- check whether we can use ctime for this (like ctime not having changed implying bsdflags not having changed - to really exploit that we would also have to add the bsdflags to the files cache)
Another way (on top of the suggested options) would be to enable by default --nobsdflags, as it was in 1.0. We could check what most users prefer. Also consider that many users won't notice the slowdown and will likely live with it forever, even if they don't need the bsdflags. But if bsdflags are removed and some user really need them, it should be quite easy for him to find the availability of the option, for example with a google search. And benchmarks like [this](https://github.com/gilbertchen/benchmarking) (especially the linux kernel benchmark, having quite a lot of small files) will also be affected.
The same for --noatime.
This issue is related to #3039 .
Adding a quick confirmation here to my similar experience re: increase in time / bsdflags. Prior versions of borg where few files changed, ~ 3 minutes ... if I don't disable bsdflags about 2 hours. (Arch Linux ext4 ~100GB)
For now I think we'll go with the simple solution and just document that archiving bsdflags has a speed penalty on linux that's especially notable when having lots of small files, workaround is --nobsdflags.

For the metadata, borg tries to archive everything (by default) and offers `--noxxx` options to switch it off, if desired. I don't think we should fundamentally change that in a 1.1.3 patch release.


Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -2916,6 +2916,12 @@ def define_archive_filters_group(subparser, *, sort_by=True, first_last=True):
           it had before a content change happened. This can be used maliciously as well as
           well-meant, but in both cases mtime based cache modes can be problematic.
 
+        By default, borg tries to archive all metadata that it supports archiving.
+        If that is not what you want or need, there are some tuning options:
+
+        - --nobsdflags (getting bsdflags has a speed penalty under Linux)
+        - --noatime (if atime changes frequently, the metadata stream will dedup badly)
+
         The mount points of filesystems or filesystem snapshots should be the same for every
         creation of a new archive to ensure fast operation. This is because the file cache that
         is used to determine changed files quickly uses absolute filenames.


Test patch: