Repository: borgbackup/borg

Issue:
borg delete busy-waits when reading from remote
Running "borg delete" against a remote repository spends an inordinate amount of CPU time busy-waiting for input from the server.  According to strace, select immediately returns because file descriptor 5 is ready for writing, but the client then never actually writes anything.

```
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})

... busy-waiting ...

select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
select(9, [6 8], [5], [5 6 8], {1, 0})  = 2 (in [6], out [5], left {0, 999999})
mmap(NULL, 10489856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3272ecf000
read(6, "\224\1\315\16\361\300\332\0274\2\1\0\371[\317\24E\260\344\3306\305C\246chunks\221\223"..., 10485760) = 5956
mremap(0x7f3272ecf000, 10489856, 8192, MREMAP_MAYMOVE) = 0x7f3272ecf000
munmap(0x7f32724ce000, 8192)            = 0
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
write(5, "\224\1\315\17\261\246delete\221\332\0 \325/.y\370co\236\215H&E\233\302\300\361"..., 48) = 48
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
write(5, "\224\1\315\17\262\243get\221\332\0 P\343\20\267\v\237\335:M\302\212\207\36\362\375\361\345\246\266"..., 45) = 45
select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})

... busy-waiting again ...

select(9, [6 8], [5], [5 6 8], {1, 0})  = 1 (out [5], left {0, 999999})
```



Discussion:
16080fe fixes this for me (<10 % avg. CPU usage of the client instead of ~40 % when deleting remotely). I did some other testing, didn't seem to break anything or to run into timeouts (I set the timeout to 100 for testing).

With the logic now it will build to_send if it's empty (the while loop), send if ready for writing, and won't select() on the write_fds if to_send is empty.

Commit 16080fe is still giving me the busy-waiting behavior.  Here's the line of code where w_fds gets set unconditionally: https://github.com/borgbackup/borg/blob/16080fe33e184a384c6b4805dbf5a76f46a08867/borg/remote.py#L245  This happens well before the portion of code changed in the commit.

The issue is most apparent when I try to do the delete over a limited-bandwidth VPN connection.

That one is correct, though. When a delete is issued the command must be sent, so it select()s on the write fd once (unconditionally), and sends the command. Line 330 should then clear the write fd from the next wait. Hm, maybe this can be sped up by creating a delete_many method on RemoteR. I'll look into it.

I had a chance to dig a little bit deeper, and I'm starting to get an idea of what's happening (even with the new code.)

In a test run of `borg delete`, the busy-wait happened on a `get` command issued on a list of 22,285 IDs:
1. `call_many` starts by writing 100 calls to the server (100 `select` + 100 `write` of length 43; wouldn't it be better to combine these into a single 4300 byte write?)
2. At this point, there are 100 calls in-flight, and 22,185 remaining in the queue.
3. The loop is now stuck in a busy-waiting state until the server's response arrives: The `w_fds` list still contains `self.stdout_fd`, so that makes `select` return immediately.  No response data is available yet for reading, and the client is not yet willing to write (because it's at its limit of 100 in-flight requests) - so it keeps looping.

I think part of the issue is the function is calling `select` before it has decided what (if anything) it wants to write.

I revised the logic there a bit, so that first `to_send` is built and then the select is done - excluding the write fd if there is nothing to_send. However, that's only part of the issue. delete() is a "singular" API, meaning that (in my understanding) these requests will be issued one-by-one anyway. I guess that we'll want to queue/buffer deletes to send them in large batches.


Resolving patch:
diff --git a/borg/remote.py b/borg/remote.py
--- a/borg/remote.py
+++ b/borg/remote.py
@@ -20,6 +20,8 @@
 
 BUFSIZE = 10 * 1024 * 1024
 
+MAX_INFLIGHT = 100
+
 
 class ConnectionClosed(Error):
     """Connection closed by remote host"""
@@ -316,7 +318,6 @@ def handle_error(error, res):
 
         calls = list(calls)
         waiting_for = []
-        w_fds = [self.stdin_fd]
         while wait or calls:
             while waiting_for:
                 try:
@@ -330,6 +331,10 @@ def handle_error(error, res):
                             return
                 except KeyError:
                     break
+            if self.to_send or ((calls or self.preload_ids) and len(waiting_for) < MAX_INFLIGHT):
+                w_fds = [self.stdin_fd]
+            else:
+                w_fds = []
             r, w, x = select.select(self.r_fds, w_fds, self.x_fds, 1)
             if x:
                 raise Exception('FD exception occurred')
@@ -362,7 +367,7 @@ def handle_error(error, res):
                         else:
                             sys.stderr.write("Remote: " + line)
             if w:
-                while not self.to_send and (calls or self.preload_ids) and len(waiting_for) < 100:
+                while not self.to_send and (calls or self.preload_ids) and len(waiting_for) < MAX_INFLIGHT:
                     if calls:
                         if is_preloaded:
                             if calls[0] in self.cache:
@@ -389,8 +394,6 @@ def handle_error(error, res):
                         # that the fd should be writable
                         if e.errno != errno.EAGAIN:
                             raise
-                if not self.to_send and not (calls or self.preload_ids):
-                    w_fds = []
         self.ignore_responses |= set(waiting_for)
 
     def check(self, repair=False, save_space=False):


Test patch: