Repository: borgbackup/borg

Issue:
borg recreate with --exclude failing with KeyError 'data'

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yup

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

client: 1.1.10
server: 0.29.0  (rsync.net)

#### Operating system (distribution) and version.

client: Ubuntu 14.04.6 LTS
server: FreeBSD 11.1-RELEASE-p6

#### Hardware / network configuration, and filesystems used.

client: ext3
server: ZFS

#### How much data is handled by borg?

In this repo:
```
                       Original size      Compressed size    Deduplicated size
All archives:                3.70 GB              1.55 GB              1.46 GB

                       Unique chunks         Total chunks
Chunk index:                   51533                72737
```

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```
echo YES | borg recreate -vp USER@SERVER.rsync.net:conf.borg
```

## Describe the problem you're observing.

This recreate command & repo is giving a reproducible KeyError:
```
root@mail:/backup# echo YES | borg recreate -vp --exclude '*/tarsnap-cache' USER@SERVER.rsync.net:conf.borg
Remote: Borg 0.29.0: exception in RPC call:
Remote: Traceback (most recent call last):
Remote:   File "/usr/home/kibab/borgbackup-0.29.0/borg/remote.py", line 96, in serve
Remote: TypeError: open() takes from 2 to 5 positional arguments but 7 were given
Remote: Platform: FreeBSD SERVER.rsync.net 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Tue Dec 19 13:52:29 PST 2017     user@11_1:/usr/src/sys/amd64/compile/kernel.11_1amd64 amd64
Remote: Python: CPython 3.4.3
Remote:
Please note:
If you see a TypeError complaining about the number of positional arguments
given to open(), you can ignore it if it comes from a borg version < 1.0.7.
This TypeError is a cosmetic side effect of the compatibility code borg
clients >= 1.0.7 have to support older borg servers.
This problem will go away as soon as the server has been upgraded to 1.0.7+.
recreate is an experimental feature.
Type 'YES' if you understand this and want to continue: Processing 20151231
Remote: Borg 0.29.0: exception in RPC call:
Remote: Traceback (most recent call last):
Remote:   File "/usr/home/kibab/borgbackup-0.29.0/borg/remote.py", line 96, in serve
Remote: TypeError: get() missing 1 required positional argument: 'id_'
Remote: Platform: FreeBSD SERVER.rsync.net 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Tue Dec 19 13:52:29 PST 2017     user@11_1:/usr/src/sys/amd64/compile/kernel.11_1amd64 amd64
Remote: Python: CPython 3.4.3
Remote:
Local Exception
Traceback (most recent call last):
  File "borg/archiver.py", line 4501, in main
  File "borg/archiver.py", line 4433, in run
  File "borg/archiver.py", line 164, in wrapper
  File "borg/archiver.py", line 1725, in do_recreate
  File "borg/archive.py", line 1700, in recreate
  File "borg/archive.py", line 1794, in save
  File "borg/archive.py", line 510, in save
  File "borg/cache.py", line 898, in add_chunk
  File "borg/remote.py", line 477, in do_rpc
  File "borg/remote.py", line 712, in call
  File "borg/remote.py", line 883, in call_many
  File "borg/remote.py", line 709, in named_to_positional
  File "borg/remote.py", line 709, in <listcomp>
KeyError: 'data'

Platform: Linux CLIENT 5.1.11-x86_64-linode127 #1 SMP PREEMPT Mon Jun 17 21:18:26 UTC 2019 x86_64
Linux: debian jessie/sid
Borg: 1.1.10  Python: CPython 3.5.7 msgpack: 0.5.6
PID: 7767  CWD: /backup
sys.argv: ['borg', 'recreate', '-vp', '--exclude', '*/tarsnap-cache', 'USER@SERVER.rsync.net:conf.borg']
SSH_ORIGINAL_COMMAND: None
```

I'm pretty sure I successfully ran recreate on this same repo yesterday. Yes, the failure is triggered by the `--exclude '*/tarsnap-cache' ` option which I added today. A dry run of this recreate showed these files scheduled for deletion:
```
x usr/local/tarsnap-cache
x usr/local/tarsnap-cache/cseq
x usr/local/tarsnap-cache/directory
x usr/local/tarsnap-cache/lockf
```



Discussion:
You are using a highly outdated version of borg on the server side.

As people frequently run into accidentally using 0.29 with rsync.net I suggest you file an issue with them, so they (finally) document the steps required to get a current borg version on their side for their customers.

https://github.com/borgbackup/borg/blob/1.1.10/src/borg/remote.py#L883

Hah! Guess there is the bug. `cmd` (variable) there should rather be `'get'` (constant).

Guess nobody noticed that yet because usually borg is using dicts nowadays for parameter passing rather than positional args, so it is usually in the other code branch (but not for a stoneage server as here).

Resolving patch:
diff --git a/src/borg/remote.py b/src/borg/remote.py
--- a/src/borg/remote.py
+++ b/src/borg/remote.py
@@ -880,7 +880,7 @@ def handle_error(unpacked):
                         if self.dictFormat:
                             self.to_send = msgpack.packb({MSGID: self.msgid, MSG: 'get', ARGS: args})
                         else:
-                            self.to_send = msgpack.packb((1, self.msgid, 'get', self.named_to_positional(cmd, args)))
+                            self.to_send = msgpack.packb((1, self.msgid, 'get', self.named_to_positional('get', args)))
 
                 if self.to_send:
                     try:


Test patch: