Repository: borgbackup/borg

Issue:
Segment missing when running borg create, but borg check does not find a problem
## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

## Is this a BUG / ISSUE report or a QUESTION?

Probably a bug.

## System information. For client/server mode post info for both machines.

#### Your borg version (borg -V).

`1.1.9`, repository was initially create with version `1.0.9`.

#### Operating system (distribution) and version.

Debian stretch.

#### Hardware / network configuration, and filesystems used.

Local directory on ZFS. Hardware is a crummy multi-slot USB disk enclosure, but ZFS shows 0 I/O errors on the pool.

#### How much data is handled by borg?

2 to 2.5 TiB.

## Describe the problem you're observing.

_Backstory_: I have a repository which was originally created using Borg `1.0.9` and most data in the repository has been inserted using that version. Then, for some reason, the chunk index was lost (when running `borg create` it started rebuilding it). Knowing that I'm using an older version and that this might be an issue which has already been solved, I upgraded to version `1.1.9` (from the official Debian repository `stretch-backports`) and let it run again and rebuild the chunk index. All output shown below is from version `1.1.9`.

Now the repository is in a state where `borg create` fails with `No such file or directory: '/srv/backup/apu.borg/data/42/425105'` (see full log below). This is the full output of the first time the error appeared. It begins by finishing rebuilding the chunk index and then terminated with the error:

```
jan 12 22:49:20 apu borg-backup.sh[20261]: Creating archive at "/srv/backup/apu.borg::{now:%Y-%m-%d-%H%M%S}"             
jan 12 23:00:49 apu borg-backup.sh[20261]: Synchronizing chunks cache...                                                       
jan 12 23:00:51 apu borg-backup.sh[20261]: Archives: 783, w/ cached Idx: 66, w/ outdated Idx: 0, w/o cached Idx: 717.    
jan 12 23:00:51 apu borg-backup.sh[20261]: Reading cached archive chunk index for 2017-11-26-065635 ...                        
jan 12 23:01:03 apu borg-backup.sh[20261]: Merging into master chunks index ...                         
[...]
jan 13 07:25:26 apu borg-backup.sh[20261]: Fetching and building archive index for 2018-09-08-071309 ...                                                          
jan 13 07:25:44 apu borg-backup.sh[20261]: Merging into master chunks index ...                                                                                                            
jan 13 07:25:44 apu borg-backup.sh[20261]: Fetching and building archive index for 2019-09-05-082930 ...                                                          
jan 13 07:25:56 apu borg-backup.sh[20261]: Merging into master chunks index ...                                                                                                            
jan 13 07:25:57 apu borg-backup.sh[20261]: Reading cached archive chunk index for 2018-11-12-081615 ...                                                                                    
jan 13 07:25:57 apu borg-backup.sh[20261]: Merging into master chunks index ...                                                                                                                
jan 13 07:25:58 apu borg-backup.sh[20261]: Fetching and building archive index for 2018-11-16-072405 ...                                                          
jan 13 07:26:18 apu borg-backup.sh[20261]: Merging into master chunks index ...                                                                                                            
jan 13 07:26:18 apu borg-backup.sh[20261]: Done.                                                        
jan 13 07:29:36 apu borg-backup.sh[20261]: Local Exception                                                                     
jan 13 07:29:36 apu borg-backup.sh[20261]: Traceback (most recent call last):                                  
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4455, in main
jan 13 07:29:36 apu borg-backup.sh[20261]:     exit_code = archiver.run(args)                           
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4387, in run                                            
jan 13 07:29:36 apu borg-backup.sh[20261]:     return set_ec(func(args))                                                                                                                   
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 154, in wrapper                                         
jan 13 07:29:36 apu borg-backup.sh[20261]:     return method(self, args, repository=repository, **kwargs)                                                         
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 557, in do_create                                                                
jan 13 07:29:36 apu borg-backup.sh[20261]:     create_inner(archive, cache)                                                                                                                
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 520, in create_inner                                                                                                                                                                                                                                                                                                           
jan 13 07:29:36 apu borg-backup.sh[20261]:     read_special=args.read_special, dry_run=dry_run, st=st)                                                                                                                                                                                                                                                                                                                                   
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 634, in _process                                                                                                      
jan 13 07:29:36 apu borg-backup.sh[20261]:     read_special=read_special, dry_run=dry_run)                                                                                                 
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 634, in _process                                                                                                                                                                                                                                                                                                               
jan 13 07:29:36 apu borg-backup.sh[20261]:     read_special=read_special, dry_run=dry_run)                                                                                                                                                                                                                                                                                                                                               
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 634, in _process                                                                                                      
jan 13 07:29:36 apu borg-backup.sh[20261]:     read_special=read_special, dry_run=dry_run)                                                                        
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 634, in _process
jan 13 07:29:36 apu borg-backup.sh[20261]:     read_special=read_special, dry_run=dry_run)                                                                                                 
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 634, in _process                                                                 
jan 13 07:29:36 apu borg-backup.sh[20261]:     read_special=read_special, dry_run=dry_run)              
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 634, in _process 
jan 13 07:29:36 apu borg-backup.sh[20261]:     read_special=read_special, dry_run=dry_run)                                                                        
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 634, in _process                                                                 
jan 13 07:29:36 apu borg-backup.sh[20261]:     read_special=read_special, dry_run=dry_run)              
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archiver.py", line 608, in _process
jan 13 07:29:36 apu borg-backup.sh[20261]:     status = archive.process_file(path, st, cache)                            
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archive.py", line 1034, in process_file                                    
jan 13 07:29:36 apu borg-backup.sh[20261]:     self.chunk_file(item, cache, self.stats, backup_io_iter(self.chunker.chunkify(fd, fh)))                                                                                                                                                                                                                                                                                                   
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archive.py", line 962, in chunk_file                                                                                                                                                                                                                                                                                                              
jan 13 07:29:36 apu borg-backup.sh[20261]:     item.chunks.append(chunk_processor(data))                                                                                                                                                                                                                                                  
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/archive.py", line 950, in chunk_processor                                  
jan 13 07:29:36 apu borg-backup.sh[20261]:     chunk_entry = cache.add_chunk(self.key.id_hash(data), data, stats, wait=False)                                                              
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/cache.py", line 899, in add_chunk
jan 13 07:29:36 apu borg-backup.sh[20261]:     self.repository.put(id, data, wait=wait)                                  
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/repository.py", line 1088, in put       
jan 13 07:29:36 apu borg-backup.sh[20261]:     self.prepare_txn(self.get_transaction_id())                               
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/repository.py", line 533, in prepare_txn
jan 13 07:29:36 apu borg-backup.sh[20261]:     self._rebuild_sparse(segment)                            
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/repository.py", line 863, in _rebuild_sparse
jan 13 07:29:36 apu borg-backup.sh[20261]:     self.compact[segment] += self.io.segment_size(segment)                    
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/repository.py", line 1324, in segment_size
jan 13 07:29:36 apu borg-backup.sh[20261]:     return os.path.getsize(self.segment_filename(segment))                    
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3.5/genericpath.py", line 50, in getsize               
jan 13 07:29:36 apu borg-backup.sh[20261]:     return os.stat(filename).st_size                                          
jan 13 07:29:36 apu borg-backup.sh[20261]: FileNotFoundError: [Errno 2] No such file or directory: '/srv/backup/apu.borg/data/42/425105'
jan 13 07:29:36 apu borg-backup.sh[20261]: Platform: Linux apu 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11)x86_64
jan 13 07:29:36 apu borg-backup.sh[20261]: Linux: debian 9.11                                                            
jan 13 07:29:36 apu borg-backup.sh[20261]: Borg: 1.1.9  Python: CPython 3.5.3                                                                                                              
jan 13 07:29:36 apu borg-backup.sh[20261]: PID: 20263  CWD: /
jan 13 07:29:36 apu borg-backup.sh[20261]: sys.argv: ['/usr/bin/borg', 'create', '--verbose', '--stats', '--exclude', '/proc', '--exclude', '/sys', '--exclude', '/dev', '--exclude', '/tmp', '--exclude', '/run', '--exclude', '/mnt', '--exclude', '/media', '--exclude', '/var/swap,', '--exclude', '/var/lib/docker', '--exclude', '/srv/backup', '--exclude', '/root/.cache', '--exclude', '/home/*/.cache', '--exclude', '/home/mic
hi/transmission', '/srv/backup/apu.borg::{now:%Y-%m-%d-%H%M%S}', '/']
jan 13 07:29:36 apu borg-backup.sh[20261]: SSH_ORIGINAL_COMMAND: None
```

The missing file does indeed not exists:

```
$ ls -d /srv/backup/apu.borg/data/42/425105                           
ls: cannot access '/srv/backup/apu.borg/data/42/425105': No such file or directory
```

But I have hourly snapshots of the filesystem containing the repository and the file did exist at some point in the past but was somehow deleted, I assume by Borg itself. So I can easily restore this specific file:

```
root@apu ~$ ls /srv/backup/.zfs/snapshot/*/apu.borg/data/42/425105
/srv/backup/.zfs/snapshot/snappy-2020-01-11-140003/apu.borg/data/42/425105
/srv/backup/.zfs/snapshot/snappy-2020-01-11-150002/apu.borg/data/42/425105
```

To find out whether more files have gone missing, I ran `borg check` (without first restoring the missing file). But to my surprise, it did not find any problems:

```
$ borg --debug check --repository-only /srv/backup/apu.borg/
using builtin fallback logging configuration                                    
35 self tests completed in 0.98 seconds                                         
Starting repository check                                               
Read committed index of transaction 425109                                      
Segment transaction is    425109                                                
Determined transaction is 425109                                                
Found 399578 segments                                                       
Starting repository index check                                                 
Completed repository check, no problems found.                          
```

Running `borg create` after that still produces the same error message as above:

```
$ ./borg-backup.sh
Creating archive at "/srv/backup/apu.borg::{now:%Y-%m-%d-%H%M%S}"
Local Exception
Traceback (most recent call last):
[...]
  File "/usr/lib/python3.5/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/srv/backup/apu.borg/data/42/425105'

Platform: Linux apu 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64
Linux: debian 9.11
Borg: 1.1.9  Python: CPython 3.5.3
PID: 27195  CWD: /root
sys.argv: ['/usr/bin/borg', 'create', '--verbose', '--stats', '--exclude', '/proc', '--exclude', '/sys', '--exclude', '/dev', '--exclude', '/tmp', '--exclude','/run', '--exclude', '/mnt', '--exclude', '/media', '--exclude', '/var/swap,', '--exclude', '/var/lib/docker', '--exclude', '/srv/backup', '--exclude', '/root/.cache', '--exclude', '/home/*/.cache', '--exclude', '/home/michi/transmission','/srv/backup/apu.borg::{now:%Y-%m-%d-%H%M%S}', '/']
SSH_ORIGINAL_COMMAND: None
```

Should I not have used `--repository-only` with `borg check`? I feared that it would take even longer, the `borg check` command above ran for about 40 h.

I would like to know how to either fix the repository (e.g. know which files are missing and that the existing ones are in good shape) or know that it can't be fixed (because there are files missing I can't restore).

#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

I'm not sure how to reproduce this. The problem seems to have been caused by an older Borg version.


Discussion:
Can you try `borg check --repair`?

If you lost the chunks index, maybe you also lost that segment file?
> Can you try `borg check --repair`?

I've started the following command: `time borg --debug --progress check --repair /srv/backup/apu.borg`

I will report back when it's done.

> If you lost the chunks index, maybe you also lost that segment file?

There may very well be a connection. Nonetheless, `~/.cache/borg` and the Borg repository are on different file systems so I still think the files have to have been deleted by the Borg process. But this happened while I was still using Borg `1.0.9`, so I don't know whether investigating that issue is particularly interesting.
Ok. The `borg check` run completed. It seems to not have found a problem.:

```
$ time borg --debug --progress check --repair /srv/backup/apu.borg
using builtin fallback logging configuration           
35 self tests completed in 0.78 seconds      
'check --repair' is an experimental feature that might result in data loss.
Type 'YES' if you understand this and want to continue: YES              
Starting repository check                             
Read committed index of transaction 425109   
Cleaned up 0 uncommitted segment files (== everything after segment 425109).
Segment transaction is    425109                       
Determined transaction is 425109             
Found 399578 segments                                            
Starting repository index check              
Completed repository check, no problems found.
Starting archive consistency check...        
Verified integrity of /srv/backup/apu.borg/index.425109
TAM-verified manifest                        
Analyzing archive 2017-09-17-211548 (1/783)                            
Cleaned up 0 uncommitted segment files (== everything after segment 425109).
Verified integrity of /srv/backup/apu.borg/hints.425109
Analyzing archive 2017-09-18-082558 (2/783)                           
Analyzing archive 2017-09-19-084635 (3/783)                            
[...]
Analyzing archive 2020-01-11-030003 (781/783)           
Analyzing archive 2020-01-11-113728.checkpoint (782/783)
Analyzing archive 2020-01-11-145229.checkpoint (783/783)
Deleting 0 orphaned and 12433 superseded objects...
Finished deleting orphaned/superseded objects.
Writing Manifest.                            
Committing repo (may take a while, due to compact_segments)...
check_free_space: required bytes 716890968, free bytes 645796069376
Finished committing repo.                    
Archive consistency check complete, no problems found.
                                                                        
real    10145m28.083s                                                   
user    1671m18.300s                                                     
sys     160m36.112s                                                        
```
I've started another `borg create` run to see whether the problem with the missing segment has been resolved. But it seems that the chunks index needs to be rebuilt again:

```
$ ./borg-backup.sh
Creating archive at "/srv/backup/apu.borg::{now:%Y-%m-%d-%H%M%S}"
Synchronizing chunks cache...
Archives: 783, w/ cached Idx: 783, w/ outdated Idx: 781, w/o cached Idx: 781.
Fetching and building archive index for 2017-11-29-083337 ...
Merging into master chunks index ...
Fetching and building archive index for 2019-05-05-083729 ...
Merging into master chunks index ...
Fetching and building archive index for 2018-06-11-064717 ...
Merging into master chunks index ...
```

Is this to be expected after running `borg check`?
It seems to "think" that most of your cached indexes are outdated.
Guess I need to look at the code about this...
So another run of `borg create` now completed successfully:

```
$ ./borg-backup.sh
Creating archive at "/srv/backup/apu.borg::{now:%Y-%m-%d-%H%M%S}"               
Synchronizing chunks cache...                                                   
Archives: 783, w/ cached Idx: 783, w/ outdated Idx: 781, w/o cached Idx: 781.   
Fetching and building archive index for 2017-11-29-083337 ...                   
Merging into master chunks index ...                                            
Fetching and building archive index for 2019-05-05-083729 ...                   
Merging into master chunks index ...                         
Fetching and building archive index for 2018-06-11-064717 ...
Merging into master chunks index ...                         
[...]
Fetching and building archive index for 2019-03-21-082113 ...
Merging into master chunks index ...
Fetching and building archive index for 2018-01-03-072113 ...
Merging into master chunks index ...
Done.
------------------------------------------------------------------------------
Archive name: 2020-01-25-012830
Archive fingerprint: 9a42e716ba4f3f915ce3fcba58686c96d04761511a932c1645cd04f7c6862083
Time (start): Sat, 2020-01-25 01:28:39
Time (end):   Sun, 2020-01-26 22:48:17
Duration: 1 days 21 hours 19 minutes 37.79 seconds
Number of files: 271610
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                2.16 TB              2.16 TB             93.72 GB
All archives:              368.16 TB            368.16 TB              2.91 TB

                       Unique chunks         Total chunks
Chunk index:                 2686778            465573365
------------------------------------------------------------------------------
```

Is there a way to find out what it did about the missing segment? Was it only referenced but not used, if that makes sense?
```
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/repository.py", line 533, in prepare_txn
jan 13 07:29:36 apu borg-backup.sh[20261]:     self._rebuild_sparse(segment)                            
jan 13 07:29:36 apu borg-backup.sh[20261]:   File "/usr/lib/python3/dist-packages/borg/repository.py", line 863, in _rebuild_sparse
jan 13 07:29:36 apu borg-backup.sh[20261]:     self.compact[segment] += self.io.segment_size(segment)
```

This only runs on version upgrades, and this trace specifically suggests that the older Borg version left a reference to a deleted segment in the hints.

Resolving patch:
diff --git a/src/borg/repository.py b/src/borg/repository.py
--- a/src/borg/repository.py
+++ b/src/borg/repository.py
@@ -883,10 +883,19 @@ def _update_index(self, segment, objects, report=None):
 
     def _rebuild_sparse(self, segment):
         """Rebuild sparse bytes count for a single segment relative to the current index."""
-        self.compact[segment] = 0
+        try:
+            segment_size = self.io.segment_size(segment)
+        except FileNotFoundError:
+            # segment does not exist any more, remove it from the mappings
+            # note: no need to self.compact.pop(segment), as we start from empty mapping.
+            self.segments.pop(segment)
+            return
+
         if self.segments[segment] == 0:
-            self.compact[segment] += self.io.segment_size(segment)
+            self.compact[segment] = segment_size
             return
+
+        self.compact[segment] = 0
         for tag, key, offset, size in self.io.iter_objects(segment, read_data=False):
             if tag == TAG_PUT:
                 if self.index.get(key, (-1, -1)) != (segment, offset):


Test patch: