Repository: borgbackup/borg

Issue:
Msgpack exception when trying to create archive in an old repository
<!--
Thank you for reporting an issue.

*IMPORTANT* -  *before* creating a new issue please look around:
 - Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
 - FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
 and
 - open issues in Github tracker: https://github.com/borgbackup/borg/issues
  
If you cannot find a similar problem, then create a new issue.

Please fill in as much of the template as possible.
-->

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

`borg 1.1.5`

#### Operating system (distribution) and version.

Ubuntu Bionic 18.04 amd64

#### Hardware / network configuration, and filesystems used.

Backup source (local HDD): XFS, external HDD with borg directory: ext4.

#### How much data is handled by borg?

Approximately 500 GB deduplicated and compressed (zlib) with 5 archives (full original size 2.96 TB) and 398125 unique + 2228374 total chunks.

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

`sys.argv: ['/usr/bin/borg', 'create', '--verbose', '--progress', '--stats', '--compression', 'zlib', '--exclude-from', 'attic.filter', '--one-file-system', '--list', '--filter', 'AMEh', '/media/yl3dy/backup/data.borg::2018-12-18__17-04', '/mnt/data']`
(sorry for filter name)


## Describe the problem you're observing.

Cannot create a new archive in a borg repository created and used previously with borg 1.0.9-1~bpo8+1 (created in some earlier version, 1.0.5 perhaps) in Debian 8 (oldstable) installed via backports. Previously had no issues with the drive and repository, regularly did `borg check`. The repository has no encryption. It is the first time this repository is used with borg 1.1.x. The current system has a clean borg cache. `list` and `info` run fine. `borg -vp check` doesn't show any problems. Running with `--dry-run` instead of `--stats` produces no output and no errors.

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

On every `borg create` the following error arises. It fails on the same file every time (10th file in list).

#### Include any warning/errors/backtraces from the system logs

After few files:
```
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4253, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4185, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 152, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 547, in do_create
    create_inner(archive, cache)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 511, in create_inner
    read_special=args.read_special, dry_run=dry_run, st=st)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 621, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 621, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 598, in _process
    status = archive.process_file(path, st, cache)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 1021, in process_file
    self.chunk_file(item, cache, self.stats, backup_io_iter(self.chunker.chunkify(fd, fh)))
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 949, in chunk_file
    item.chunks.append(chunk_processor(data))
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 937, in chunk_processor
    chunk_entry = cache.add_chunk(self.key.id_hash(data), data, stats, wait=False)
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 898, in add_chunk
    self.repository.put(id, data, wait=wait)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 1059, in put
    self.prepare_txn(self.get_transaction_id())
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 504, in prepare_txn
    hints = msgpack.unpack(fd)
  File "/home/yl3dy/.local/lib/python3.6/site-packages/msgpack/__init__.py", line 57, in unpack
    return unpackb(data, **kwargs)
  File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
ValueError: 72459 exceeds max_map_len(32768)
```

<!--

If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html

*IMPORTANT* - Please mark logs and text output from terminal commands 
or else Github will not display them correctly. 
An example is provided below.

Example:
```
this is an example how log text should be marked (wrap it with ```)
```
-->



Discussion:
It's msgpack in standard configuration trying to unpack your repo's hints file.

IIRC, the msgpack author recently changed near-to-infinite defaults to safer-against-DoS, lower defaults.

So, for a quick workaround, use an older msgpack-python.
Hmm, I see this change in defaults happened in msgpack 0.6.0, which is a version borg 1.1.5 never supported (see setup.py, it says `msgpack-python>=0.4.6,<0.5.0`).

Even the latest borg 1.1.8 does not support 0.6.0, so how did you get that installed?
Also, there is not even a package on pypi called `msgpack-python 0.6.0` because the author did not update the old-name package, but only the new-name `msgpack` package.

So, please fix the requirements.

**TODO: What we (borg devs) have to do is fix master branch (not 1.1-maint because it never will change the required package name for compatibility reasons) and explicitly give higher limits there at misc. places for the unpacker.**
 python-msgpack | 0.5.6-1        | bionic | source, amd64, arm64, armhf, i386, ppc64el, s390x

not something that comes from repo.
https://launchpad.net/ubuntu/+source/python-msgpack
we use 0.5.6 since 18.04
@ThomasWaldmann Yes, I use both borg and python-msgpack (0.5.6-1) from official Ubuntu Bionic repositories.
```
  File "/home/yl3dy/.local/lib/python3.6/site-packages/msgpack/__init__.py", line 57, in unpack
```
this is not where we install official Ubuntu packages :)
Yeah, I haven't noticed that! Newer msgpack was installed via pip as a dependency of pynvim (sadly, this package, as well as a number of others, aren't in official Ubuntu repos, pip doesn't have any analog of apt pinning, and I can't put them into a virtualenv). Now an archive was created without errors.
reopening, see TODO in bold in https://github.com/borgbackup/borg/issues/4220#issuecomment-448302127 .
https://git.archlinux.org/svntogit/community.git/tree/trunk/0001-Fix-msgpack-version-constraints-using-proper-setupto.patch?h=packages/borg

this will break borg in archlinux due to the removal of the "< 0.6.0" requirement as soon as someone upgrades msgpack to >=0.6.0 in archlinux. the breakage is non-obvious and won't be found in a quick test, but someone will find this issue in production then (see top post) ... @eli-schwartz 

Resolving patch:
diff --git a/src/borg/helpers/msgpack.py b/src/borg/helpers/msgpack.py
--- a/src/borg/helpers/msgpack.py
+++ b/src/borg/helpers/msgpack.py
@@ -117,22 +117,46 @@ def __next__(self):
     next = __next__
 
 
-def unpackb(packed, *, raw=True, encoding=None, unicode_errors=None, **kwargs):
+def unpackb(packed, *, raw=True, encoding=None, unicode_errors=None,
+            max_str_len=2147483647,  # 2**32-1
+            max_bin_len=2147483647,
+            max_array_len=2147483647,
+            max_map_len=2147483647,
+            max_ext_len=2147483647,
+            **kwargs):
     assert raw is True
     assert encoding is None
     assert unicode_errors is None
     try:
-        return mp_unpackb(packed, raw=raw, encoding=encoding, unicode_errors=unicode_errors, **kwargs)
+        return mp_unpackb(packed, raw=raw, encoding=encoding, unicode_errors=unicode_errors,
+                          max_str_len=max_str_len,
+                          max_bin_len=max_bin_len,
+                          max_array_len=max_array_len,
+                          max_map_len=max_map_len,
+                          max_ext_len=max_ext_len,
+                          **kwargs)
     except Exception as e:
         raise UnpackException(e)
 
 
-def unpack(stream, *, raw=True, encoding=None, unicode_errors=None, **kwargs):
+def unpack(stream, *, raw=True, encoding=None, unicode_errors=None,
+           max_str_len=2147483647,  # 2**32-1
+           max_bin_len=2147483647,
+           max_array_len=2147483647,
+           max_map_len=2147483647,
+           max_ext_len=2147483647,
+           **kwargs):
     assert raw is True
     assert encoding is None
     assert unicode_errors is None
     try:
-        return mp_unpack(stream, raw=raw, encoding=encoding, unicode_errors=unicode_errors, **kwargs)
+        return mp_unpack(stream, raw=raw, encoding=encoding, unicode_errors=unicode_errors,
+                         max_str_len=max_str_len,
+                         max_bin_len=max_bin_len,
+                         max_array_len=max_array_len,
+                         max_map_len=max_map_len,
+                         max_ext_len=max_ext_len,
+                         **kwargs)
     except Exception as e:
         raise UnpackException(e)
 


Test patch: