Repository: borgbackup/borg

Issue:
network speed regression in 1.1.0b3
I started to use borg 1.1.0b3, and noticed that it was unusually slow
when network bound.  So I did some benchmarks comparing various versions
of borg, as well as attic 0.16.

In most cases I did a single backup of a 112MB directory into a fresh
repo.  Sometimes I followed this with a backup that included the same
directory (unchanged) and a second directory of similar size.

Client is a laptop, connected via a solid wifi connection to a home
router, and then over the internet to a remote machine.  Outbound
network can sustain ~500kB/s over ssh.  Tests were repeated multiple times
to rule out network variations, and the results were always consistent.
I can't ping the server, but interactive usage is snappy.

Remote disk is a very fast M.2 SSD raid array.  Remote cpu is very fast,
local is decent.  Local drive is an SSD, but files were in cache
anyways, as shown by the find commands below.

No encryption for any repos.  Default chunker parameters.

I sometimes varied the compression method, as shown.

All borg versions were run using the pre-compiled 64-bit binaries.

First a summary table, and then selected logs.  The table shows the
create time and the deduplicated size.  It some cases, it shows a
followup backup.

```
  borg 1.1.0b3 zlib:    4m55s  76.0MB ; 3m10s 77.7MB  [cpu usage < 10%]
  borg 1.1.0b2 zlib:    2m27s  76.0MB ; 2m30s 77.7MB  [cpu usage < 10%]
  borg 1.0.10 zlib:     2m21s  76.0MB                 [cpu usage < 10%]
  attic 0.16 (zlib):    2m26s  75.6MB ; 2m25s 74.8MB  [cpu usage < 10%]

  borg 1.1.0b3 lzma,6: 10m30s  74.9MB                 [cpu usage 20-60%]
  borg 1.1.0b3 none:    5m35s 102.4MB                 [cpu usage < 10%]
```

Note that attic and the older borg versions fully utilize the network: 76/147 = 0.5MB/s.

I'm happy to test other versions as well.

Dan


```
BORG 1.1.0b3:

$ borg init -e none bu@host:test.borg
$ find dir1 dir2 -type f -exec cat {} \; > /dev/null
$ borg create -C zlib -s bu@host:test.borg::1 dir1
------------------------------------------------------------------------------
Archive name: 1
Archive fingerprint: d27ba2f333f713f21903f02051de77e456f36b5f527e75c870ae14e982842014
Time (start): Sun, 2017-02-19 10:50:48
Time (end):   Sun, 2017-02-19 10:55:44
Duration: 4 minutes 55.17 seconds
Number of files: 4318
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              112.38 MB             80.18 MB             76.02 MB
All archives:              112.38 MB             80.18 MB             76.02 MB

                       Unique chunks         Total chunks
Chunk index:                    3643                 4319
------------------------------------------------------------------------------
$ find dir1 dir2 -type f -exec cat {} \; > /dev/null
$ borg create -C zlib -s bu@host:test.borg::2 dir1 dir2
------------------------------------------------------------------------------
Archive name: 2
Archive fingerprint: de85bc5bb47782276fbd3d4012b0d3b13aa74f9d8b8983a7b39aedd674434b1d
Time (start): Sun, 2017-02-19 10:56:25
Time (end):   Sun, 2017-02-19 10:59:35
Duration: 3 minutes 10.01 seconds
Number of files: 5219
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              212.35 MB            158.40 MB             77.69 MB
All archives:              324.73 MB            238.58 MB            153.71 MB

                       Unique chunks         Total chunks
Chunk index:                    4454                 9551
------------------------------------------------------------------------------


BORG 1.1.0b2:

$ /tmp/borg-1.1.0b2 init --remote-path /tmp/borg-1.1.0b2 -e none bu@host:test-1.1.0b2.borg
$ find dir1 -type f -exec cat {} \; > /dev/null
$ /tmp/borg-1.1.0b2 create --remote-path /tmp/borg-1.1.0b2 -C zlib -s -v bu@host:test-1.1.0b2.borg::1 dir1
------------------------------------------------------------------------------
Archive name: 1
Archive fingerprint: a2686687772f84e4a03e5f518d32765007b1d4f3b223d0e4c985e8229bb0a3b3
Time (start): Sun, 2017-02-19 11:38:44
Time (end):   Sun, 2017-02-19 11:41:11
Duration: 2 minutes 26.73 seconds
Number of files: 4318
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              112.38 MB             80.18 MB             76.02 MB
All archives:              112.38 MB             80.18 MB             76.02 MB

                       Unique chunks         Total chunks
Chunk index:                    3643                 4319
------------------------------------------------------------------------------
$ find dir1 dir2 -type f -exec cat {} \; > /dev/null
$ /tmp/borg-1.1.0b2 create --remote-path /tmp/borg-1.1.0b2 -C zlib -s -v bu@host:test-1.1.0b2.borg::2 dir1 dir2
------------------------------------------------------------------------------
Archive name: 2
Archive fingerprint: 77b8968738388f264a7be46221ef420667d9d42aaad8c416d4d6b36e349dd9f4
Time (start): Sun, 2017-02-19 12:12:45
Time (end):   Sun, 2017-02-19 12:15:14
Duration: 2 minutes 29.77 seconds
Number of files: 5219
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              212.35 MB            158.40 MB             77.69 MB
All archives:              324.73 MB            238.58 MB            153.71 MB

                       Unique chunks         Total chunks
Chunk index:                    4454                 9551
------------------------------------------------------------------------------


ATTIC 0.16:

$ attic init bu@host:test.attic
$ find dir1 dir2 -type f -exec cat {} \; > /dev/null
$ attic create -s bu@host:test.attic::1 dir1
------------------------------------------------------------------------------
Archive name: 1
Archive fingerprint: e1900edf848f2a30deb55adf8f3dc141dae5579a7ca1f4990c584418fe488f86
Start time: Sun Feb 19 10:36:58 2017
End time: Sun Feb 19 10:39:25 2017
Duration: 2 minutes 26.22 seconds
Number of files: 4318

                       Original size      Compressed size    Deduplicated size
This archive:              112.22 MB             80.46 MB             75.56 MB
All archives:              112.22 MB             80.46 MB             75.56 MB
------------------------------------------------------------------------------
$ find dir1 dir2 -type f -exec cat {} \; > /dev/null
$ attic create -s bu@host:test.attic::2 dir1 dir2
------------------------------------------------------------------------------
Archive name: 2
Archive fingerprint: cd297c727ee2fa0e7c1643ecea4a709a6c566b9f6e1853451ccc9916d4639ba8
Start time: Sun Feb 19 10:46:15 2017
End time: Sun Feb 19 10:48:40 2017
Duration: 2 minutes 25.03 seconds
Number of files: 5219

                       Original size      Compressed size    Deduplicated size
This archive:              212.20 MB            159.11 MB             74.83 MB
All archives:              324.42 MB            239.56 MB            150.39 MB
------------------------------------------------------------------------------
```


Discussion:
@jdchristensen interesting. a while ago, we did some changes in the rpc implementation, we'll have to check whether that was between b2 and b3.

as it sounds easily reproducable, i guess we can just use git bisect to find the culprit changeset.
Some changes in the RPC code were made in b3, probably the cause. They did increase overhead, but not much, so that's probably not it - rather, some unintended side effect.

Thanks for the excellent report!
Yes, I see commits by @textshell on Nov 17 and Oct 6, 2016 that touch remote.py.  1.1.0b2 was released Oct 1.

I bisected the speed issue with following results:

a) first try to bisect was unsuccessful, i tried with few relatively big files and the speed difference sometimes was not clear enough, so bisect ended at some random changeset (that for sure was incorrect).

b) second try was successful, i tried with a kernel src tree (lots of small files) and the speed difference was always clear. bisect ended with:
```
ba553ec628b84cfa2f2f92898627b711244ea1d6 is the first bad commit

Author: Martin Hostettler <textshell@uchuujin.de>
Date:   Thu Nov 10 09:56:18 2016 +0100

    remote: Introduce rpc protocol with named parameters.
```
https://github.com/borgbackup/borg/commit/ba553ec628b84cfa2f2f92898627b711244ea1d6

The bad commits took ~2 mins to backup this, the good commits ~1 min.

This is the script I used:
```
export BORG_REPO=ssh://localhost/home/tw/w/borg/repo
export BORG_DELETE_I_KNOW_WHAT_I_AM_DOING=YES
LOG=borg.log
LOG_TEMP=borg.log.temp

pip install -q -e .
borg init --encryption none
borg --version > $LOG_TEMP
/usr/bin/time -v borg create --compression none \
                             --remote-path /home/tw/w/borg-env/bin/borg \
                             ::arch /home/tw/w/linux-4.9.11 \
                             2>> $LOG_TEMP
borg delete >/dev/null 2>&1
echo "----------------" >> $LOG_TEMP
cat $LOG_TEMP >> $LOG

borg --version
grep wall.clock $LOG_TEMP
```
At each bisect step, I had the same borg version at both ends of the ssh/localhost connection - which was the bisected version, so it started somewhere in the middle between b2 and b3 and approached the problematic change by always cutting distance in half.
borg.log:
```
borg 1.1.0b2
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 0.40
        System time (seconds): 49.96
        Percent of CPU this job got: 86%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:58.00
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 78904
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 46261
        Voluntary context switches: 209856
        Involuntary context switches: 5162
        Swaps: 0
        File system inputs: 1822024
        File system outputs: 24512
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
borg 1.0.10.dev41+ng9c42a75
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 0.46
        System time (seconds): 43.73
        Percent of CPU this job got: 83%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:53.08
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 71436
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 86115
        Voluntary context switches: 208748
        Involuntary context switches: 5320
        Swaps: 0
        File system inputs: 1822000
        File system outputs: 24512
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
borg 1.1.0b3.dev288+ng53abed6
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 1.91
        System time (seconds): 72.76
        Percent of CPU this job got: 62%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 2:00.42
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 79880
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 46953
        Voluntary context switches: 268315
        Involuntary context switches: 1778
        Swaps: 0
        File system inputs: 1818208
        File system outputs: 24536
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
borg 1.1.0b3.dev167+ng01e56a8
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 0.49
        System time (seconds): 51.37
        Percent of CPU this job got: 85%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:00.72
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 81152
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 47982
        Voluntary context switches: 209673
        Involuntary context switches: 5654
        Swaps: 0
        File system inputs: 1821048
        File system outputs: 24512
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
borg 1.1.0b3.dev218+ng8955d8b
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 1.73
        System time (seconds): 72.04
        Percent of CPU this job got: 62%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:58.48
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 85072
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 47775
        Voluntary context switches: 268472
        Involuntary context switches: 1539
        Swaps: 0
        File system inputs: 1820928
        File system outputs: 24512
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
borg 1.1.0b3.dev180+ng34f529c
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 0.45
        System time (seconds): 51.31
        Percent of CPU this job got: 86%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:59.52
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 82776
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 48562
        Voluntary context switches: 209590
        Involuntary context switches: 4726
        Swaps: 0
        File system inputs: 1821280
        File system outputs: 24512
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
borg 1.1.0b3.dev209+ng58024c0
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 0.49
        System time (seconds): 53.27
        Percent of CPU this job got: 82%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:05.40
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 81760
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 42379
        Voluntary context switches: 209614
        Involuntary context switches: 5026
        Swaps: 0
        File system inputs: 1821880
        File system outputs: 24512
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
borg 1.1.0b3.dev213+ng6c1b337
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 0.50
        System time (seconds): 50.12
        Percent of CPU this job got: 86%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:58.80
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 81764
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 42733
        Voluntary context switches: 209485
        Involuntary context switches: 4566
        Swaps: 0
        File system inputs: 1821368
        File system outputs: 24512
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
borg 1.1.0b3.dev215+ng4854fce
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 1.99
        System time (seconds): 73.52
        Percent of CPU this job got: 62%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 2:00.92
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 85148
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 48741
        Voluntary context switches: 268287
        Involuntary context switches: 2035
        Swaps: 0
        File system inputs: 1821720
        File system outputs: 24512
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
borg 1.1.0b3.dev214+ngba553ec
        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 1.42
        System time (seconds): 63.31
        Percent of CPU this job got: 60%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:47.22
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 85084
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 48261
        Voluntary context switches: 268241
        Involuntary context switches: 1654
        Swaps: 0
        File system inputs: 1820672
        File system outputs: 24512
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
----------------
```

System: Linux 4.9.11, ext4, i5-4200u, 8GB RAM, SATA SSD.
@ThomasWaldmann Just out of curiosity, did you use the same version on the remote end for all tests?  If so, which version?
Seems so

> I always had same borg version at both ends of the ssh/localhost connection.

IIRC the changes in the server were fewer than in the client.
Sorry, I missed that because it wasn't in the initial version of the comment.
It's actually slightly ambiguous: local and remote the same, or an unchanging remote version?
I think he means local and remote always the same.
@jdchristensen @enkore I rephrased my comment about the versions in https://github.com/borgbackup/borg/issues/2185#issuecomment-281256452 - I hope it is clearer now.
I reviewed the changeset in question, but couldn't find something that clearly could cause such a slowdown.
Of course, the arguments processing is a bit more heavy than before and "inspect" is used frequently, but I don't think this could cause such a slowdown.
The user CPU time almost quadrupled, so perhaps that's really the cause here.

In this case we should just remove `@api` from the methods used very frequently (put, get, `__contains__`); they'll never change anyway.

---

I think we could also optimize some other things here (I noticed this a few days ago). For example, we use single-char indices for mandatory parts of the data, like "i", "m", "b", "r". But since these are strings, they are encoded in two bytes. If we just make them a char (int), then they'll be one byte, I think this might have been the intention.

We could define some methods (like put get contains) which we reasonably expect to not change to use unnamed (tuple) arguments, instead of keyword arguments. This saves 3-10 bytes or so per request. Also, we could just drop the variable encoding for the request. Thinking about it, we never had a need to change anything there, do we expect to? This would shave some bytes off as well, also simplify some code again.
The user CPU time is only about 1% of the running time, so that in itself shouldn't be a problem.  The real puzzle is why the system CPU time increases so much (50s vs 63s).  Are any additional round-trips created by the change?

I tried pairing 1.1.0b2 and 1.1.0b3 with each other, and the speed depends just on the client version.
@jdchristensen I didn't see any additional round-trips (that was also one of my first thoughts).

If the client is older (thus does not speak the dict-based protocol), the server will also speak the old tuple-based protocol.
I monitored the traffic, and 1.1.0b3 only used 0.2% more outbound data than 1.1.0b2, so that's not the cause either.
@jdchristensen ah, nice data point. one concern about the dict based protocol was that it causes more traffic (but then we used single-letter keys rather than "speaking" keys to keep the overhead low).
Just to revisit these numbers... how do they make sense?


        Command being timed: "borg create --compression none --remote-path /home/tw/w/borg-env/bin/borg ::arch /home/tw/w/linux-4.9.11"
        User time (seconds): 0.46
        System time (seconds): 43.73
        Percent of CPU this job got: 83%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:53.08

It only took 0.46 seconds (in Borg code) to create a backup of the 4.9.11 kernel tree? But 44 seconds spent in the kernel?

...

Are these numbers swapped somehow?
It's like /usr/bin/time output it.
I also wondered about the little user time (but for bisecting, I used wall clock time).
Right, something seems wrong.  In a test I just did with 1.1.0b3, I got:

```
	Command being timed: "borg create --remote-path borg -s -C none host:/tmp/test.borg::1 dir"
	User time (seconds): 2.28
	System time (seconds): 0.24
	Percent of CPU this job got: 5%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:45.38
```

And with 1.1.0b2, I got:

```
	Command being timed: "/tmp/borg-1.1.0b2 create --remote-path /tmp/borg-1.1.0b2 -s -C none host:/tmp/test.borg::1 dir"
	User time (seconds): 2.10
	System time (seconds): 0.21
	Percent of CPU this job got: 6%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:35.56
```

In both cases, the user and system CPU time are similar, so they aren't directly contributing to the slowdown.  Incidentally, the difference in speeds seems more dramatic with compression turned on.

Could it be that something is different regarding having multiple requests going over the network at once?

Could the new code have a subtle error with default arguments, like the wait=True args?  Looking over the code, I couldn't see a problem.
@ThomasWaldmann:

> If the client is older (thus does not speak the dict-based protocol), the server will also speak the old tuple-based protocol.

But if I use a 1.1.0b3 client with a 1.1.0b2 server, I *still* see the slowdown, even though they should be using the old protocol.  This seems like it should be a big hint.  I don't think the problem has to do with cpu or network overhead, but rather, that there is an unintended behaviour change with the new code, e.g. something to do with default arguments, or other arguments getting dropped, or something like that.
@jdchristensen good point / good hint. most of the code is the same, just the wire format is tuple vs. dict and the api was switched from positional args to named args.
Yes, I'm pretty sure it's all about the `wait` argument.  Suppose a caller does something like:

```
self.repository.put(id, data, wait=False)
```

The `api` decorator puts the value of `wait` into the dictionary called `named`.  That dictionary gets passed to `call` in the argument `args`, which gets passed as a 1 element list to `call_many` in the argument `calls`.  But `call_many` uses a separate `wait` argument, defaulting to `True`, to decide whether to wait.  It might be enough to look in the dictionary to see if `wait` is overridden, but it's a bit awkward, since `call_many` gets a list of dictionaries, and they might not agree on the value of `wait`.

To be honest, this code is very convoluted and hard to analyze.  Could it be overengineered?  Do we expect frequent enough protocol changes to warrant this?  If protocol changes happen only rarely, each release could support 2 or 3 versions by simply hardcoding the (few) changes.
I've got a fix that seems to work.  I'll work on a pull request.  One thing that makes it hard to tell if the problem is completely fixed is that my pip build from git runs more slowly that the precompiled binaries.  Not sure if there's another regression, or if it's something about how it's built.

Resolving patch:
diff --git a/src/borg/remote.py b/src/borg/remote.py
--- a/src/borg/remote.py
+++ b/src/borg/remote.py
@@ -425,12 +425,16 @@ def decorator(f):
         def do_rpc(self, *args, **kwargs):
             sig = inspect.signature(f)
             bound_args = sig.bind(self, *args, **kwargs)
-            named = {}
+            named = {}  # Arguments for the remote process
+            extra = {}  # Arguments for the local process
             for name, param in sig.parameters.items():
                 if name == 'self':
                     continue
                 if name in bound_args.arguments:
-                    named[name] = bound_args.arguments[name]
+                    if name == 'wait':
+                        extra[name] = bound_args.arguments[name]
+                    else:
+                        named[name] = bound_args.arguments[name]
                 else:
                     if param.default is not param.empty:
                         named[name] = param.default
@@ -447,7 +451,7 @@ def do_rpc(self, *args, **kwargs):
                 raise self.RPCServerOutdated("{0} {1}={2!s}".format(f.__name__, name, named[name]),
                                              format_version(restriction['since']))
 
-            return self.call(f.__name__, named)
+            return self.call(f.__name__, named, **extra)
         return do_rpc
     return decorator
 


Test patch: