Repository: borgbackup/borg

Issue:
borg recreate with --exclude failing with KeyError 'data'

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yup

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

client: 1.1.10
server: 0.29.0  (rsync.net)

#### Operating system (distribution) and version.

client: Ubuntu 14.04.6 LTS
server: FreeBSD 11.1-RELEASE-p6

#### Hardware / network configuration, and filesystems used.

client: ext3
server: ZFS

#### How much data is handled by borg?

In this repo:
```
                       Original size      Compressed size    Deduplicated size
All archives:                3.70 GB              1.55 GB              1.46 GB

                       Unique chunks         Total chunks
Chunk index:                   51533                72737
```

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

```
echo YES | borg recreate -vp USER@SERVER.rsync.net:conf.borg
```

## Describe the problem you're observing.

This recreate command & repo is giving a reproducible KeyError:
```
root@mail:/backup# echo YES | borg recreate -vp --exclude '*/tarsnap-cache' USER@SERVER.rsync.net:conf.borg
Remote: Borg 0.29.0: exception in RPC call:
Remote: Traceback (most recent call last):
Remote:   File "/usr/home/kibab/borgbackup-0.29.0/borg/remote.py", line 96, in serve
Remote: TypeError: open() takes from 2 to 5 positional arguments but 7 were given
Remote: Platform: FreeBSD SERVER.rsync.net 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Tue Dec 19 13:52:29 PST 2017     user@11_1:/usr/src/sys/amd64/compile/kernel.11_1amd64 amd64
Remote: Python: CPython 3.4.3
Remote:
Please note:
If you see a TypeError complaining about the number of positional arguments
given to open(), you can ignore it if it comes from a borg version < 1.0.7.
This TypeError is a cosmetic side effect of the compatibility code borg
clients >= 1.0.7 have to support older borg servers.
This problem will go away as soon as the server has been upgraded to 1.0.7+.
recreate is an experimental feature.
Type 'YES' if you understand this and want to continue: Processing 20151231
Remote: Borg 0.29.0: exception in RPC call:
Remote: Traceback (most recent call last):
Remote:   File "/usr/home/kibab/borgbackup-0.29.0/borg/remote.py", line 96, in serve
Remote: TypeError: get() missing 1 required positional argument: 'id_'
Remote: Platform: FreeBSD SERVER.rsync.net 11.1-RELEASE-p6 FreeBSD 11.1-RELEASE-p6 #0: Tue Dec 19 13:52:29 PST 2017     user@11_1:/usr/src/sys/amd64/compile/kernel.11_1amd64 amd64
Remote: Python: CPython 3.4.3
Remote:
Local Exception
Traceback (most recent call last):
  File "borg/archiver.py", line 4501, in main
  File "borg/archiver.py", line 4433, in run
  File "borg/archiver.py", line 164, in wrapper
  File "borg/archiver.py", line 1725, in do_recreate
  File "borg/archive.py", line 1700, in recreate
  File "borg/archive.py", line 1794, in save
  File "borg/archive.py", line 510, in save
  File "borg/cache.py", line 898, in add_chunk
  File "borg/remote.py", line 477, in do_rpc
  File "borg/remote.py", line 712, in call
  File "borg/remote.py", line 883, in call_many
  File "borg/remote.py", line 709, in named_to_positional
  File "borg/remote.py", line 709, in <listcomp>
KeyError: 'data'

Platform: Linux CLIENT 5.1.11-x86_64-linode127 #1 SMP PREEMPT Mon Jun 17 21:18:26 UTC 2019 x86_64
Linux: debian jessie/sid
Borg: 1.1.10  Python: CPython 3.5.7 msgpack: 0.5.6
PID: 7767  CWD: /backup
sys.argv: ['borg', 'recreate', '-vp', '--exclude', '*/tarsnap-cache', 'USER@SERVER.rsync.net:conf.borg']
SSH_ORIGINAL_COMMAND: None
```

I'm pretty sure I successfully ran recreate on this same repo yesterday. Yes, the failure is triggered by the `--exclude '*/tarsnap-cache' ` option which I added today. A dry run of this recreate showed these files scheduled for deletion:
```
x usr/local/tarsnap-cache
x usr/local/tarsnap-cache/cseq
x usr/local/tarsnap-cache/directory
x usr/local/tarsnap-cache/lockf
```



Discussion:
You are using a highly outdated version of borg on the server side.

As people frequently run into accidentally using 0.29 with rsync.net I suggest you file an issue with them, so they (finally) document the steps required to get a current borg version on their side for their customers.

https://github.com/borgbackup/borg/blob/1.1.10/src/borg/remote.py#L883

Hah! Guess there is the bug. `cmd` (variable) there should rather be `'get'` (constant).

Guess nobody noticed that yet because usually borg is using dicts nowadays for parameter passing rather than positional args, so it is usually in the other code branch (but not for a stoneage server as here).
@simonmichael can you reproduce the issue?

See #4653. Might be difficult to reproduce or to determine if it is fixed, because it also depends on timing of RPC calls / preload being used.
Reproduce the issue ? Yes, it happens every time.

https://github.com/ThomasWaldmann/borg/commit/2574f4dac56b04c2662edfd0fcaf594fe06de58b

Can you easily test this fix? I guess it fixes both issues seen above.
Oh yes, I'll test tomorrow. Thanks.
Sorry to say, python software installation went as always for me. I now have a patched borg installed - would you have any idea how to fix this runtime error ?
```
# python3 /usr/local/bin/borg --version
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 81, in <module>
    from .selftest import selftest
  File "/usr/local/lib/python3.4/dist-packages/borg/selftest.py", line 21, in <module>
    from .testsuite.hashindex import HashIndexDataTestCase, HashIndexRefcountingTestCase, HashIndexTestCase
  File "/usr/local/lib/python3.4/dist-packages/borg/testsuite/__init__.py", line 29, in <module>
    from pytest import raises
  File "/usr/local/lib/python3.4/dist-packages/pytest.py", line 6, in <module>
    from _pytest.assertion import register_assert_rewrite
  File "/usr/local/lib/python3.4/dist-packages/_pytest/assertion/__init__.py", line 6, in <module>
    from _pytest.assertion import rewrite
  File "/usr/local/lib/python3.4/dist-packages/_pytest/assertion/rewrite.py", line 443, in <module>
    ast.MatMult: "@",
AttributeError: 'module' object has no attribute 'MatMult'
```
Looks like a problem in the `pytest` package you have installed, not in borg.

You can avoid borg importing from there by editing `borg/testsuite/__init__.py` and replacing

```
try:
    from pytest import raises
except ImportError:
    raises = None
```
by
```
raises = None
```
You could also just edit the borg you already had installed and do the same little patch as seen in my changeset. Not the cleanest way, but if that borg already had worked, that would have been simpler.
@simonmichael you installed pytest 5, it no longer supports python 3.4 and has the metadata about that 
Thanks, both. I did think of editing the installed borg but stopped when  it seemed to be some kind of binary (from the Ubuntu PPA I believe. ) I’ll try again asap. 


PS maybe something in requirements.d needs an update ?

I see.. it was the `borg-linux64` standalone binary I had installed, since the PPA was not up to date. And I suppose, in my thrashing around while installing the git version, I upgraded the system-wide pytest lib, which broke borg-PPA, borg-git, and even borg-git-inside-virtualenv.

Well this is off topic for this issue, I'll repair my system and report back if I have success.

Upgraded my pip3, then upgraded & downgraded a bunch of python libs until I could `pip3 install .` again. This version fails with a six.wraps attribute error. But after re-creating the virtual env, the borg built inside that runs successfully. (Any tips on installing this one as the system borg ?)

And yes - I confirm your patch fixes the bug.

BTW, there are multiple files in there: https://github.com/borgbackup/borg/tree/1.1-maint/requirements.d

The one with "lock" in the filename installs "proven", specific versions of borg's requirements (not necessarily the latest stuff though). The one without "lock" usually installs the latest stuff.
PS when I wasn't needing to patch borg, the pyinstaller-generated standalone binary did a great job of hiding all this pain. +1.

Resolving patch:
diff --git a/src/borg/remote.py b/src/borg/remote.py
--- a/src/borg/remote.py
+++ b/src/borg/remote.py
@@ -872,7 +872,7 @@ def handle_error(unpacked):
                         if self.dictFormat:
                             self.to_send = msgpack.packb({MSGID: self.msgid, MSG: 'get', ARGS: args})
                         else:
-                            self.to_send = msgpack.packb((1, self.msgid, 'get', self.named_to_positional(cmd, args)))
+                            self.to_send = msgpack.packb((1, self.msgid, 'get', self.named_to_positional('get', args)))
 
                 if self.to_send:
                     try:
diff --git a/src/borg/testsuite/__init__.py b/src/borg/testsuite/__init__.py
--- a/src/borg/testsuite/__init__.py
+++ b/src/borg/testsuite/__init__.py
@@ -31,7 +31,7 @@
 
 try:
     from pytest import raises
-except ImportError:
+except:  # noqa
     raises = None
 
 has_lchflags = hasattr(os, 'lchflags') or sys.platform.startswith('linux')


Test patch: