Repository: borgbackup/borg

Issue:
borg delete busy-waits when reading from remote
Running "borg delete" against a remote repository spends an inordinate amount of CPU time busy-waiting for input from the server.  According to strace, select immediately returns because file descriptor 5 is ready for writing, but the client then never actually writes anything.

```
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})

... busy-waiting ...

select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
select(9, [6 8], [5], [5 6 8], {1, 0})  = 2 (in [6], out [5], left {0, 999999})
mmap(NULL, 10489856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3272ecf000
read(6, "\224\1\315\16\361\300\332\0274\2\1\0\371[\317\24E\260\344\3306\305C\246chunks\221\223"..., 10485760) = 5956
mremap(0x7f3272ecf000, 10489856, 8192, MREMAP_MAYMOVE) = 0x7f3272ecf000
munmap(0x7f32724ce000, 8192)            = 0
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
write(5, "\224\1\315\17\261\246delete\221\332\0 \325/.y\370co\236\215H&E\233\302\300\361"..., 48) = 48
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
write(5, "\224\1\315\17\262\243get\221\332\0 P\343\20\267\v\237\335:M\302\212\207\36\362\375\361\345\246\266"..., 45) = 45
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})

... busy-waiting again ...

select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
```



Discussion:
16080fe fixes this for me (<10 % avg. CPU usage of the client instead of ~40 % when deleting remotely). I did some other testing, didn't seem to break anything or to run into timeouts (I set the timeout to 100 for testing).

With the logic now it will build to_send if it's empty (the while loop), send if ready for writing, and won't select() on the write_fds if to_send is empty.

Commit 16080fe is still giving me the busy-waiting behavior.  Here's the line of code where w_fds gets set unconditionally: https://github.com/borgbackup/borg/blob/16080fe33e184a384c6b4805dbf5a76f46a08867/borg/remote.py#L245  This happens well before the portion of code changed in the commit.

The issue is most apparent when I try to do the delete over a limited-bandwidth VPN connection.

That one is correct, though. When a delete is issued the command must be sent, so it select()s on the write fd once (unconditionally), and sends the command. Line 330 should then clear the write fd from the next wait. Hm, maybe this can be sped up by creating a delete_many method on RemoteR. I'll look into it.

I had a chance to dig a little bit deeper, and I'm starting to get an idea of what's happening (even with the new code.)

In a test run of `borg delete`, the busy-wait happened on a `get` command issued on a list of 22,285 IDs:
1. `call_many` starts by writing 100 calls to the server (100 `select` + 100 `write` of length 43; wouldn't it be better to combine these into a single 4300 byte write?)
2. At this point, there are 100 calls in-flight, and 22,185 remaining in the queue.
3. The loop is now stuck in a busy-waiting state until the server's response arrives: The `w_fds` list still contains `self.stdout_fd`, so that makes `select` return immediately.  No response data is available yet for reading, and the client is not yet willing to write (because it's at its limit of 100 in-flight requests) - so it keeps looping.

I think part of the issue is the function is calling `select` before it has decided what (if anything) it wants to write.

I revised the logic there a bit, so that first `to_send` is built and then the select is done - excluding the write fd if there is nothing to_send. However, that's only part of the issue. delete() is a "singular" API, meaning that (in my understanding) these requests will be issued one-by-one anyway. I guess that we'll want to queue/buffer deletes to send them in large batches.

Steps to reproduce the behavior:

```
$ mkdir testdir && for i in `seq -w 0 99999`; do echo $i > testdir/$i; done
$ borg init -e none 127.0.100.1:/tmp/repo
$ borg create 127.0.100.1:/tmp/repo::test testdir
```

Simulate 1000ms of latency to 127.0.100.1 using Linux's netem feature:

```
$ sudo tc qdisc add dev lo root handle 1: prio bands 9 priomap 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8
$ sudo tc filter add dev lo protocol ip parent 1: prio 1 u32 match ip dst 127.0.100.1/32 flowid 1:1
$ sudo tc qdisc add dev lo parent 1:1 netem delay 1000ms
```

Before the fix (notice the user/system CPU time):

```
$ time borg check 127.0.100.1:/tmp/repo::test
Remote:
real    0m46.551s
user    0m9.660s
sys     0m2.476s
```

After the fix:

```
$ time borg check 127.0.100.1:/tmp/repo::test
Remote:
real    0m47.343s
user    0m2.184s
sys     0m0.128s
```

Undo the simulated latency:

```
$ sudo tc qdisc del dev lo parent 1:1 netem delay 1000ms
$ sudo tc filter del dev lo protocol ip parent 1: prio 1 u32 match ip dst 127.0.100.1/32 flowid 1:1
$ sudo tc qdisc del dev lo root handle 1: prio bands 9 priomap 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8
```

Numbers look much better, congrats!


Resolving patch:
diff --git a/src/borg/cache.py b/src/borg/cache.py
--- a/src/borg/cache.py
+++ b/src/borg/cache.py
@@ -11,6 +11,7 @@
 logger = create_logger()
 
 from .hashindex import ChunkIndex, ChunkIndexEntry
+from .helpers import Location
 from .helpers import Error
 from .helpers import get_cache_dir
 from .helpers import decode_dict, int_to_bigint, bigint_to_int, bin_to_hex
@@ -160,10 +161,7 @@ def create(self):
         with SaveFile(os.path.join(self.path, 'files'), binary=True) as fd:
             pass  # empty file
 
-    def _do_open(self):
-        self.config = configparser.ConfigParser(interpolation=None)
-        config_path = os.path.join(self.path, 'config')
-        self.config.read(config_path)
+    def _check_upgrade(self, config_path):
         try:
             cache_version = self.config.getint('cache', 'version')
             wanted_version = 1
@@ -174,6 +172,25 @@ def _do_open(self):
         except configparser.NoSectionError:
             self.close()
             raise Exception('%s does not look like a Borg cache.' % config_path) from None
+        # borg < 1.0.8rc1 had different canonicalization for the repo location (see #1655 and #1741).
+        cache_loc = self.config.get('cache', 'previous_location', fallback=None)
+        if cache_loc:
+            repo_loc = self.repository._location.canonical_path()
+            rl = Location(repo_loc)
+            cl = Location(cache_loc)
+            if cl.proto == rl.proto and cl.user == rl.user and cl.host == rl.host and cl.port == rl.port \
+                    and \
+                    cl.path and rl.path and \
+                    cl.path.startswith('/~/') and rl.path.startswith('/./') and cl.path[3:] == rl.path[3:]:
+                # everything is same except the expected change in relative path canonicalization,
+                # update previous_location to avoid warning / user query about changed location:
+                self.config.set('cache', 'previous_location', repo_loc)
+
+    def _do_open(self):
+        self.config = configparser.ConfigParser(interpolation=None)
+        config_path = os.path.join(self.path, 'config')
+        self.config.read(config_path)
+        self._check_upgrade(config_path)
         self.id = self.config.get('cache', 'repository')
         self.manifest_id = unhexlify(self.config.get('cache', 'manifest'))
         self.timestamp = self.config.get('cache', 'timestamp', fallback=None)
diff --git a/src/borg/testsuite/__init__.py b/src/borg/testsuite/__init__.py
--- a/src/borg/testsuite/__init__.py
+++ b/src/borg/testsuite/__init__.py
@@ -146,11 +146,11 @@ def assert_creates_file(self, path):
         yield
         self.assert_true(os.path.exists(path), '{} should exist'.format(path))
 
-    def assert_dirs_equal(self, dir1, dir2):
+    def assert_dirs_equal(self, dir1, dir2, **kwargs):
         diff = filecmp.dircmp(dir1, dir2)
-        self._assert_dirs_equal_cmp(diff)
+        self._assert_dirs_equal_cmp(diff, **kwargs)
 
-    def _assert_dirs_equal_cmp(self, diff):
+    def _assert_dirs_equal_cmp(self, diff, ignore_bsdflags=False, ignore_xattrs=False):
         self.assert_equal(diff.left_only, [])
         self.assert_equal(diff.right_only, [])
         self.assert_equal(diff.diff_files, [])
@@ -168,8 +168,9 @@ def _assert_dirs_equal_cmp(self, diff):
                 attrs.append('st_nlink')
             d1 = [filename] + [getattr(s1, a) for a in attrs]
             d2 = [filename] + [getattr(s2, a) for a in attrs]
-            d1.append(get_flags(path1, s1))
-            d2.append(get_flags(path2, s2))
+            if not ignore_bsdflags:
+                d1.append(get_flags(path1, s1))
+                d2.append(get_flags(path2, s2))
             # ignore st_rdev if file is not a block/char device, fixes #203
             if not stat.S_ISCHR(d1[1]) and not stat.S_ISBLK(d1[1]):
                 d1[4] = None
@@ -185,11 +186,12 @@ def _assert_dirs_equal_cmp(self, diff):
                 else:
                     d1.append(round(s1.st_mtime_ns, st_mtime_ns_round))
                     d2.append(round(s2.st_mtime_ns, st_mtime_ns_round))
-            d1.append(no_selinux(get_all(path1, follow_symlinks=False)))
-            d2.append(no_selinux(get_all(path2, follow_symlinks=False)))
+            if not ignore_xattrs:
+                d1.append(no_selinux(get_all(path1, follow_symlinks=False)))
+                d2.append(no_selinux(get_all(path2, follow_symlinks=False)))
             self.assert_equal(d1, d2)
         for sub_diff in diff.subdirs.values():
-            self._assert_dirs_equal_cmp(sub_diff)
+            self._assert_dirs_equal_cmp(sub_diff, ignore_bsdflags=ignore_bsdflags, ignore_xattrs=ignore_xattrs)
 
     @contextmanager
     def fuse_mount(self, location, mountpoint, *options):
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -1402,11 +1402,16 @@ def has_noatime(some_file):
         mountpoint = os.path.join(self.tmpdir, 'mountpoint')
         # mount the whole repository, archive contents shall show up in archivename subdirs of mountpoint:
         with self.fuse_mount(self.repository_location, mountpoint):
-            self.assert_dirs_equal(self.input_path, os.path.join(mountpoint, 'archive', 'input'))
-            self.assert_dirs_equal(self.input_path, os.path.join(mountpoint, 'archive2', 'input'))
+            # bsdflags are not supported by the FUSE mount
+            # we also ignore xattrs here, they are tested separately
+            self.assert_dirs_equal(self.input_path, os.path.join(mountpoint, 'archive', 'input'),
+                                   ignore_bsdflags=True, ignore_xattrs=True)
+            self.assert_dirs_equal(self.input_path, os.path.join(mountpoint, 'archive2', 'input'),
+                                   ignore_bsdflags=True, ignore_xattrs=True)
         # mount only 1 archive, its contents shall show up directly in mountpoint:
         with self.fuse_mount(self.repository_location + '::archive', mountpoint):
-            self.assert_dirs_equal(self.input_path, os.path.join(mountpoint, 'input'))
+            self.assert_dirs_equal(self.input_path, os.path.join(mountpoint, 'input'),
+                                   ignore_bsdflags=True, ignore_xattrs=True)
             # regular file
             in_fn = 'input/file1'
             out_fn = os.path.join(mountpoint, 'input', 'file1')
@@ -1426,20 +1431,6 @@ def has_noatime(some_file):
             # read
             with open(in_fn, 'rb') as in_f, open(out_fn, 'rb') as out_f:
                 assert in_f.read() == out_f.read()
-            # list/read xattrs
-            in_fn = 'input/fusexattr'
-            out_fn = os.path.join(mountpoint, 'input', 'fusexattr')
-            if not xattr.XATTR_FAKEROOT and xattr.is_enabled(self.input_path):
-                assert no_selinux(xattr.listxattr(out_fn)) == ['user.foo', ]
-                assert xattr.getxattr(out_fn, 'user.foo') == b'bar'
-            else:
-                assert xattr.listxattr(out_fn) == []
-                try:
-                    xattr.getxattr(out_fn, 'user.foo')
-                except OSError as e:
-                    assert e.errno == llfuse.ENOATTR
-                else:
-                    assert False, "expected OSError(ENOATTR), but no error was raised"
             # hardlink (to 'input/file1')
             if are_hardlinks_supported():
                 in_fn = 'input/hardlink'
@@ -1462,6 +1453,27 @@ def has_noatime(some_file):
                 out_fn = os.path.join(mountpoint, 'input', 'fifo1')
                 sto = os.stat(out_fn)
                 assert stat.S_ISFIFO(sto.st_mode)
+            # list/read xattrs
+            try:
+                in_fn = 'input/fusexattr'
+                out_fn = os.path.join(mountpoint, 'input', 'fusexattr')
+                if not xattr.XATTR_FAKEROOT and xattr.is_enabled(self.input_path):
+                    assert no_selinux(xattr.listxattr(out_fn)) == ['user.foo', ]
+                    assert xattr.getxattr(out_fn, 'user.foo') == b'bar'
+                else:
+                    assert xattr.listxattr(out_fn) == []
+                    try:
+                        xattr.getxattr(out_fn, 'user.foo')
+                    except OSError as e:
+                        assert e.errno == llfuse.ENOATTR
+                    else:
+                        assert False, "expected OSError(ENOATTR), but no error was raised"
+            except OSError as err:
+                if sys.platform.startswith(('freebsd', )) and err.errno == errno.ENOTSUP:
+                    # some systems have no xattr support on FUSE
+                    pass
+                else:
+                    raise
 
     @unittest.skipUnless(has_llfuse, 'llfuse not installed')
     def test_fuse_versions_view(self):


Test patch: