Repository: borgbackup/borg

Issue:
repository problems with checkpoint archives and multiple clients
## Have you checked borgbackup docs, FAQ, and open Github issues?

yes

## Is this a BUG / ISSUE report or a QUESTION?

BUG

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

borg-1.2 1.2.0b1

#### Operating system (distribution) and version.

Linux (Client 1 is debian, Client 2 fedora and borg  server archlinux)

#### Hardware / network configuration, and filesystems used.

Notebooks with ext4 filesystem. Borg server on zfs

#### How much data is handled by borg?

60GB

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

On Client 1
 
```
borg-1.2 create --stats --show-rc -C lz4 -c 900 --lock-wait 7200 -v -p archlinux:/home/borgtest \
    /archiv::felsenpinguin.full.20210131_1612091393  /  --exclude /proc ...
```
After 20 minutes i simulated a network problem and killed the ssh process

On Client 2

```
borg-1.2 create --stats --show-rc -C lz4 -c 900  --lock-wait 7200  -v -p archlinux:/home/borgtest \
    /archiv::brillenpinguin.full.20210131_1612093028  /  --exclude /proc ...
```

## Describe the problem you're observing.

The second backup runs to the end and gives a traceback trying to calculate the statistics.

borg check of the archive finds lots of orphaned objects

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

I can reproduce it with creating a checkpoint archive and then use the server from *another* machine.

If i continue to connect from the same machine i can resume without problems after a checkpoint archive.

#### Include any warning/errors/backtraces from the system logs

```
Local Exceptionnfig                                                             
Traceback (most recent call last):
  File "borg/archiver.py", line 4736, in main
  File "borg/archiver.py", line 4667, in run
  File "borg/archiver.py", line 181, in wrapper
  File "borg/archiver.py", line 657, in do_create
  File "borg/archiver.py", line 628, in create_inner
  File "borg/cache.py", line 407, in __str__
  File "borg/cache.py", line 434, in format_tuple
  File "borg/cache.py", line 426, in stats
  File "borg/archive.py", line 607, in calc_stats
  File "borg/archive.py", line 602, in add
  File "src/borg/hashindex.pyx", line 299, in borg.hashindex.ChunkIndex.__getitem__
KeyError: b'\x12;\x99\xfe\x9f\xec\x87\xdeD\xf7\n\xb3\x10x\\\xc7\xea\x14v\xf8\xaa\xc0h\xbc\x14\xf7\xdb<\x99\xc9t\x82'

Platform: Linux brillenpinguin 5.10.10-200.fc33.x86_64 #1 SMP Sun Jan 24 19:58:54 UTC 2021 x86_64
Linux: Fedora 33 Thirty Three
Borg: 1.2.0b1  Python: CPython 3.7.9 msgpack: 1.0.0 fuse: llfuse 1.3.8 [pyfuse3,llfuse]
PID: 22014  CWD: /root
sys.argv: ['borg-1.2', 'create', '--stats', '--show-rc', '-C', 'lz4', '-c', '900', '--lock-wait', '7200', '-v', '-p', 'archlinux:/home/borgtest/archiv::brillenpinguin.full.20210131_1612093028', '/', '--exclude', '/proc', '--exclude', '/sys', '--exclude', '/dev', '--exclude', '/run', '--exclude', '/tmp', '--exclude', '/var/tmp', '--exclude', '/var/lib/libvirt', '--exclude', '/var/www/html/ftp', '--exclude', '/backup', '--exclude', '/media', '--exclude', '/zfs_usbdisk2-88411', '--exclude', '/backup-z1e721b5']
SSH_ORIGINAL_COMMAND: None
```
<!--

If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html

*IMPORTANT* - Please mark logs and text output from terminal commands 
or else Github will not display them correctly. 
An example is provided below.

Example:
```
this is an example how log text should be marked (wrap it with ```)
```
-->



Discussion:
copied from other ticket to clarify this:

> @dietmargoldbeck the easiest explanations could be:
> 
>     * the repo was made and used by borg < 1.1.11

new repo just for beta testing. 
 
>     * the repo was made and used by an older 1.2 borg with the same bug

I only used the latest beta binaries Borg: 1.2.0b1  Python: CPython 3.7.9 msgpack: 1.0.0 fuse: llfuse 1.3.8 [pyfuse3,llfuse]

>     * you have faulty RAM - can you run memtest86+?

All machines survive mprime and are running borg-1.1 without problems. 
 
I could reproduce it with a checkpoint archive and continuing from my other notebook. Since it is easy to reproduce i opend
https://github.com/borgbackup/borg/issues/5668

I did not check yet if borg 1.1.15 is also affected.

Can you reproduce it like this?:
```
client1: borg init repo
client1: borg create --stats ... repo::c1-archive1   # and interrupt by killing ssh client on client1
client2: borg create --stats ... repo::c2-archive1   # let it finish - exception?
```
The point is the fresh repo and that client2 has no state from previous backups to that repo.

@anhofmann reported (looks pretty much the same as top post, except that it is against a borg 1.1 repo):

I have a repo that works fine with borg 1.1.11, but trying to do a `borg info` with 1.2.0b1 produces a traceback:
```
Local Exception
Traceback (most recent call last):
  File "borg/archiver.py", line 4736, in main
  File "borg/archiver.py", line 4667, in run
  File "borg/archiver.py", line 179, in wrapper
  File "borg/archiver.py", line 1406, in do_info
  File "borg/archiver.py", line 1488, in _info_repository
  File "borg/cache.py", line 407, in __str__
  File "borg/cache.py", line 434, in format_tuple
  File "borg/cache.py", line 426, in stats
  File "borg/archive.py", line 607, in calc_stats
  File "borg/archive.py", line 602, in add
  File "src/borg/hashindex.pyx", line 299, in borg.hashindex.ChunkIndex.__getitem__
KeyError: b'\xf4\xa2\xd4/\xacTs\xe0\xb6\xa6S\xe8B\x959\xb3\xac\xe8~\n\n\x9fu\xb2&Xp\n2\xda\xd84'

Platform: Linux rclone-archiv-ersteller 5.4.0-62-generic #70-Ubuntu SMP Tue Jan 12 12:45:47 UTC 2021 x86_64
Linux: debian bullseye/sid 
Borg: 1.2.0b1  Python: CPython 3.7.9 msgpack: 1.0.0 fuse: llfuse 1.3.8 [pyfuse3,llfuse]
```

I don't know what additional info I should provide, please ask, I will supply it.
I also remembered that `borg info` runs remotely over ssh and on the remote side borg is running version 1.1 (minor version is unknown to me).

 @dietmargoldbeck @anhofmann do you have archives in your repo that were made with borg 1.1?

it seems that the code path in that exception (calc_stats:607) is only triggered when processing archives that do not have some borg 1.2 metadata.

Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -617,20 +617,20 @@ def create_inner(archive, cache, fso):
                     self.print_error("Got Ctrl-C / SIGINT.")
                 else:
                     archive.save(comment=args.comment, timestamp=args.timestamp, stats=archive.stats)
-                args.stats |= args.json
-                if args.stats:
-                    if args.json:
-                        json_print(basic_json_data(manifest, cache=cache, extra={
-                            'archive': archive,
-                        }))
-                    else:
-                        log_multi(DASHES,
-                                  str(archive),
-                                  DASHES,
-                                  STATS_HEADER,
-                                  str(archive.stats),
-                                  str(cache),
-                                  DASHES, logger=logging.getLogger('borg.output.stats'))
+                    args.stats |= args.json
+                    if args.stats:
+                        if args.json:
+                            json_print(basic_json_data(manifest, cache=cache, extra={
+                                'archive': archive,
+                            }))
+                        else:
+                            log_multi(DASHES,
+                                      str(archive),
+                                      DASHES,
+                                      STATS_HEADER,
+                                      str(archive.stats),
+                                      str(cache),
+                                      DASHES, logger=logging.getLogger('borg.output.stats'))
 
         self.output_filter = args.output_filter
         self.output_list = args.output_list
diff --git a/src/borg/helpers/manifest.py b/src/borg/helpers/manifest.py
--- a/src/borg/helpers/manifest.py
+++ b/src/borg/helpers/manifest.py
@@ -74,7 +74,7 @@ def __delitem__(self, name):
         name = safe_encode(name)
         del self._archives[name]
 
-    def list(self, *, glob=None, match_end=r'\Z', sort_by=(), consider_checkpoints=False, first=None, last=None, reverse=False):
+    def list(self, *, glob=None, match_end=r'\Z', sort_by=(), consider_checkpoints=True, first=None, last=None, reverse=False):
         """
         Return list of ArchiveInfo instances according to the parameters.
 
@@ -82,6 +82,10 @@ def list(self, *, glob=None, match_end=r'\Z', sort_by=(), consider_checkpoints=F
         Apply *first* and *last* filters, and then possibly *reverse* the list.
 
         *sort_by* is a list of sort keys applied in reverse order.
+
+        Note: for better robustness, all filtering / limiting parameters must default to
+              "not limit / not filter", so a FULL archive list is produced by a simple .list().
+              some callers EXPECT to iterate over all archives in a repo for correct operation.
         """
         if isinstance(sort_by, (str, bytes)):
             raise TypeError('sort_by must be a sequence of str')


Test patch: