Repository: borgbackup/borg

Issue:
Borg fails when creating an archive
When I create an archive with commit 4f507c0 borg crashes with the following error. Commit 77966c7 works as expected.

```
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 1961, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 1911, in run
    return args.func(args)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 80, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 290, in do_create
    create_inner(archive, cache)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 262, in create_inner
    read_special=args.read_special, dry_run=dry_run, st=st)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 345, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3.5/site-packages/borg/archiver.py", line 319, in _process
    status = archive.process_file(path, st, cache, self.ignore_inode)
  File "/usr/lib/python3.5/site-packages/borg/archive.py", line 602, in process_file
    self.add_item(item)
  File "/usr/lib/python3.5/site-packages/borg/archive.py", line 228, in add_item
    self.items_buffer.add(item)
  File "/usr/lib/python3.5/site-packages/borg/archive.py", line 81, in add
    self.flush()
  File "/usr/lib/python3.5/site-packages/borg/archive.py", line 98, in flush
    self.buffer.write(chunks[-1])
TypeError: a bytes-like object is required, not '_Chunk'
```



Discussion:
The command it first occurred with was very complex, but it reproduces with this simple one, too:

```
borg create ::$(date +%Y-%m-%d)-test /home
```

Thanks, looks like that was overlooked when changing from tuples to namedtuples everywhere.

i think this indicates a missing test

Indeed. It's a specific branch for the buffer handling, but it should be testable if the metadata buffer size is monkeypatched down from ~1 MB (which is a lot of files).

https://codecov.io/github/borgbackup/borg/borg/archive.py?ref=14bd4c756de76c57823d014e634d0e9905e4df16#l-66

Reopened to fix the test / coverage.


Resolving patch:
diff --git a/borg/testsuite/archive.py b/borg/testsuite/archive.py
--- a/borg/testsuite/archive.py
+++ b/borg/testsuite/archive.py
@@ -57,6 +57,27 @@ def test(self):
             unpacker.feed(cache.objects[id])
         self.assert_equal(data, list(unpacker))
 
+    def test_partial(self):
+        big = b"0123456789" * 10000
+        data = [{b'full': 1, b'data': big}, {b'partial': 2, b'data': big}]
+        cache = MockCache()
+        key = PlaintextKey(None)
+        chunks = CacheChunkBuffer(cache, key, None)
+        for d in data:
+            chunks.add(d)
+        chunks.flush(flush=False)
+        # the code is expected to leave the last partial chunk in the buffer
+        self.assert_equal(len(chunks.chunks), 3)
+        self.assert_true(chunks.buffer.tell() > 0)
+        # now really flush
+        chunks.flush(flush=True)
+        self.assert_equal(len(chunks.chunks), 4)
+        self.assert_true(chunks.buffer.tell() == 0)
+        unpacker = msgpack.Unpacker()
+        for id in chunks.chunks:
+            unpacker.feed(cache.objects[id])
+        self.assert_equal(data, list(unpacker))
+
 
 class RobustUnpackerTestCase(BaseTestCase):
 


Test patch: