Repository: borgbackup/borg

Issue:
Unable to extract archive created with v1.1.8 with v1.1.10
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Bug

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

1.1.8 (pyenv/pip), 1.1.10 (pacman)

#### Operating system (distribution) and version.

Arch Linux amd64

#### Hardware / network configuration, and filesystems used.

Ext4

#### How much data is handled by borg?

```
$ borg info /mnt/backup/snapshots/naboo
Repository ID: xxx
Location: /mnt/backup/snapshots/naboo
Encrypted: No
Cache: /home/aeris/.cache/borg/xxx
Security dir: /home/aeris/.config/borg/security/xxx
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:              205.68 GB            102.23 GB             25.42 GB

                       Unique chunks         Total chunks
Chunk index:                  515791              6174310
```

#### Full borg commandline that lead to the problem (leave away excludes and passwords)

`borg extract -pv /mnt/backup/snapshots/naboo::2019-08-13`

## Describe the problem you're observing.

Unable to extract with v1.1.10 an archive generated with v1.1.8
Lead to the error below.
Error occurs on many files, excluding one by one on the `extract` command leads to another file error later in the extraction.

Borg check is clean with v1.1.8 and v1.1.10

```
$ borg check -pv --verify-data /mnt/backup/snapshots/naboo
Starting repository check
Starting repository index check                                                                       
Completed repository check, no problems found.
Starting archive consistency check...
Starting cryptographic data integrity verification...
Finished cryptographic data integrity verification, verified 515792 chunks with 0 integrity errors.                  
Analyzing archive 2018-10-13 (1/21)
[snip]
Analyzing archive 2019-08-13 (21/21)
Archive consistency check complete, no problems found.
```

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Reproductible on my archive, will try to generate a publishable one to test.

#### Include any warning/errors/backtraces from the system logs

```
Local Exceptionng: usr/lib/dri/virtio_gpu_dri.so                                                                    
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433, in run
    return set_ec(func(args))
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 179, in wrapper
    return method(self, args, repository=repository, manifest=manifest, key=key, archive=archive, **kwargs)
  File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 760, in do_extract
    preload=True, hardlink_masters=hardlink_masters):
  File "/usr/lib/python3.7/site-packages/borg/archive.py", line 467, in iter_items
    filter=lambda item: self.item_filter(item, filter)):
  File "/usr/lib/python3.7/site-packages/borg/archive.py", line 237, in unpack_many
    preload(chunks)
  File "/usr/lib/python3.7/site-packages/borg/archive.py", line 213, in preload
    self.repository.preload([c.id for c in chunks])
TypeError: 'NoneType' object is not iterable
```



Discussion:
can you try if it happens:

- with a full extract
- with latest borg at both ends


On August 18, 2019 2:43:30 PM GMT+02:00, aeris <notifications@github.com> wrote:
>## Have you checked borgbackup docs, FAQ, and open Github issues?
>
>Yes
>
>## Is this a BUG / ISSUE report or a QUESTION?
>
>Bug
>
>## System information. For client/server mode post info for both
>machines.
>
>#### Your borg version (borg -V).
>
>1.1.8 (pyenv/pip), 1.1.10 (pacman)
>
>#### Operating system (distribution) and version.
>
>Arch Linux amd64
>
>#### Hardware / network configuration, and filesystems used.
>
>Ext4
>
>#### How much data is handled by borg?
>
>~100GB
>
>#### Full borg commandline that lead to the problem (leave away
>excludes and passwords)
>
>`borg extract -pv /mnt/backup/snapshots/naboo::2019-08-13`
>
>## Describe the problem you're observing.
>
>Unable to extract with v1.1.10 an archive generated with v1.1.8
>Lead to the error below.
>Error occurs on many files, excluding one by one on the `extract`
>command leads to another file error later in the extraction.
>
>Borg check is clean with v1.1.8 and v1.1.10
>
>```
>$ borg check -pv --verify-data /mnt/backup/snapshots/naboo
>Starting repository check
>Starting repository index check                                        
>                              
>Completed repository check, no problems found.
>Starting archive consistency check...
>Starting cryptographic data integrity verification...
>Finished cryptographic data integrity verification, verified 515792
>chunks with 0 integrity errors.                  
>Analyzing archive 2018-10-13 (1/21)
>[snip]
>Analyzing archive 2019-08-13 (21/21)
>Archive consistency check complete, no problems found.
>```
>
>#### Can you reproduce the problem? If so, describe how. If not,
>describe troubleshooting steps you took before opening the issue.
>
>Reproductible on my archive, will try to generate a publishable one to
>test.
>
>#### Include any warning/errors/backtraces from the system logs
>
>```
>Local Exceptionng: usr/lib/dri/virtio_gpu_dri.so                       
>                                            
>Traceback (most recent call last):
>File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4501, in
>main
>    exit_code = archiver.run(args)
>File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4433, in
>run
>    return set_ec(func(args))
>File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 166, in
>wrapper
>    return method(self, args, repository=repository, **kwargs)
>File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 179, in
>wrapper
>return method(self, args, repository=repository, manifest=manifest,
>key=key, archive=archive, **kwargs)
>File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 760, in
>do_extract
>    preload=True, hardlink_masters=hardlink_masters):
>File "/usr/lib/python3.7/site-packages/borg/archive.py", line 467, in
>iter_items
>    filter=lambda item: self.item_filter(item, filter)):
>File "/usr/lib/python3.7/site-packages/borg/archive.py", line 237, in
>unpack_many
>    preload(chunks)
>File "/usr/lib/python3.7/site-packages/borg/archive.py", line 213, in
>preload
>    self.repository.preload([c.id for c in chunks])
>TypeError: 'NoneType' object is not iterable
>```
>
>
>-- 
>You are receiving this because you are subscribed to this thread.
>Reply to this email directly or view it on GitHub:
>https://github.com/borgbackup/borg/issues/4725

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
> with a full extract

Hum, v1.1.10 passes with full extraction, but fails with an exclusion…
v1.1.8 passes in both cases

> with latest borg at both ends

Archive and extraction are local, so same version on "both" ends.
IIRC this was a regression, introduced by a fix for a different problem.
And I've already fixed it - if you like, test with 1.1-maint branch code.
Will be in 1.1.11.
Maybe related: Issue #4607 fixed by https://github.com/ThomasWaldmann/borg/commit/16f9d45209aa4dc790796db583fe4ca13bd86a0d .

Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -234,7 +234,8 @@ def preload(chunks):
                             if source not in masters_preloaded:
                                 # we only need to preload *once* (for the 1st selected slave)
                                 chunks, _ = hardlink_masters[source]
-                                preload(chunks)
+                                if chunks is not None:
+                                    preload(chunks)
                                 masters_preloaded.add(source)
                 else:
                     # easy: we do not have a filter, thus all items are selected, thus we need to preload all chunks.


Test patch: