Repository: borgbackup/borg

Issue:
borg delete with a non existing archive name leaves a outdated local cache
When trying to delete an archive that does not exist from a repository borg leaves the local cache in an outdated state (repo manifest time stamp does not match cache manifest timestamp). This causes the next access to the repository to do a costly cache rebuild.

I think this is caused by _delete_archives forcing a manifest write in all cases but when no archive was deleted the local cache instance has no active transaction and thus does not update the cache configuration when _delete_archives triggers a commit of the cache.

Maybe LocalCache.commit should check the manifest timestamp even if there is no active transaction and if it differs trigger an update for the local cache config anyway. If that is seen as to risky we could add code in _delete_archives to force a transaction in LocalCache and this avoid leaving an seemingly outdated cache. Or maybe not write the manifest if nothing changed.

I've encountered this problem on a system using a fairly old borg version (1.10+git1+0d9e88d) but source inspection says that this is likely still true in master. I didn't look at latest 1.1.x but this is like there also.


Discussion:
the specific command used to run into this is missing.

also it would be good to reproduce with a current borg version, master or recent 1.1.x.

1.1.10 was before an important bug fix, so you likely do not want to run that anymore anyway.
Minimal reproducer is this:
```
$ ./borg-linux64 --version
borg-linux64 1.1.17
$ ./borg-linux64 init testrepo -e none
neon@neon-pc:~/borg-test$ ./borg-linux64 --debug delete testrepo::does-not-exist
using builtin fallback logging configuration
35 self tests completed in 0.30 seconds
Verified integrity of /home/neon/borg-test/testrepo/index.1
TAM-verified manifest
Verified integrity of /home/neon/.cache/borg/bd19c7c241c7028798b7f82f97f96633d341a94de340c25f8c37c9107ddd6da1/chunks
security: read previous location '/home/neon/borg-test/testrepo'
security: read manifest timestamp '2021-11-19T17:29:17.970458'
security: determined newest manifest timestamp as 2021-11-19T17:29:17.970458
security: repository checks ok, allowing access
Archive does-not-exist not found (1/1).
Cleaned up 0 uncommitted segment files (== everything after segment 1).
Verified integrity of /home/neon/borg-test/testrepo/hints.1
check_free_space: few segments, not requiring a full free segment
check_free_space: calculated working space for compact as -33 bytes
check_free_space: required bytes 169135, free bytes 7841193984
$ ./borg-linux64 --debug delete testrepo::does-not-exist
using builtin fallback logging configuration
35 self tests completed in 0.37 seconds
Verified integrity of /home/neon/borg-test/testrepo/index.5
TAM-verified manifest
Verified integrity of /home/neon/.cache/borg/bd19c7c241c7028798b7f82f97f96633d341a94de340c25f8c37c9107ddd6da1/chunks
security: read previous location '/home/neon/borg-test/testrepo'
security: read manifest timestamp '2021-11-19T17:29:17.970458'
security: determined newest manifest timestamp as 2021-11-19T17:29:17.970458
security: repository checks ok, allowing access
Synchronizing chunks cache...
Archives: 0, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 0.
Done.
security: saving state for bd19c7c241c7028798b7f82f97f96633d341a94de340c25f8c37c9107ddd6da1 to /home/neon/.config/borg/security/bd19c7c241c7028798b7f82f97f96633d341a94de340c25f8c37c9107ddd6da1
security: current location   /home/neon/borg-test/testrepo
security: key type           2
security: manifest timestamp 2021-11-19T17:29:42.364643
Archive does-not-exist not found (1/1).
Cleaned up 0 uncommitted segment files (== everything after segment 5).
Verified integrity of /home/neon/borg-test/testrepo/hints.5
check_free_space: few segments, not requiring a full free segment
check_free_space: calculated working space for compact as -33 bytes
check_free_space: required bytes 169135, free bytes 7840428032
```

The second invocation does a sync of the chunks cache.

Adding a few archives before the delete show the expected
```
Synchronizing chunks cache...
Archives: 3, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 3.
Fetching and building archive index for a1 ...
Merging into master chunks index ...
Fetching and building archive index for a2 ...
Merging into master chunks index ...
Fetching and building archive index for a3 ...
Merging into master chunks index ...
Cache sync: had to fetch 0 B (0 chunks) because no archive had a csize set for them (due to --no-cache-sync)
Cache sync: processed 1.39 kB (3 chunks) of metadata
Cache sync: compact chunks.archive.d storage saved 134.90 kB bytes
Done.
```
reproducer:
```
#!/bin/bash
borg --version
borg init -v -e none repo
# sync!
echo "*** delete"
borg delete --debug repo::does-not-exist
echo "*** 1st info"
borg info --debug repo
# allowing access -> sync!
echo "*** 2nd info"
borg info --debug repo
# ^ no sync here
echo "*** final cleanup"
BORG_DELETE_I_KNOW_WHAT_I_AM_DOING=YES borg delete repo
```

output before fix:
```
(borg-env) tw@mba2020 borg % ./reproduce-6060.sh
borg 1.1.18.dev92+gd67a1e77
Initializing repository at "repo"
Encryption NOT enabled.
Use the "--encryption=repokey|keyfile" to enable encryption.
**Synchronizing chunks cache...**    <-------- normal behaviour.
Archives: 0, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 0.
Done.
*** delete
using builtin fallback logging configuration
35 self tests completed in 0.08 seconds
Verified integrity of /Users/tw/w/borg/repo/index.1
TAM-verified manifest
Verified integrity of /Users/tw/.cache/borg/ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4/chunks
security: read previous location '/Users/tw/w/borg/repo'
security: read manifest timestamp '2022-01-30T17:09:22.084208'
security: determined newest manifest timestamp as 2022-01-30T17:09:22.084208
security: repository checks ok, allowing access
Archive does-not-exist not found (1/1).
Cleaned up 0 uncommitted segment files (== everything after segment 1).
Verified integrity of /Users/tw/w/borg/repo/hints.1
check_free_space: few segments, not requiring a full free segment
check_free_space: calculated working space for compact as -33 bytes
check_free_space: required bytes 169135, free bytes 253097877504
*** 1st info
using builtin fallback logging configuration
35 self tests completed in 0.08 seconds
Verified integrity of /Users/tw/w/borg/repo/index.5
TAM-verified manifest
security: read previous location '/Users/tw/w/borg/repo'
security: read manifest timestamp '2022-01-30T17:09:22.084208'
security: determined newest manifest timestamp as 2022-01-30T17:09:22.084208
security: repository checks ok, allowing access
Verified integrity of /Users/tw/.cache/borg/ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4/chunks
security: read previous location '/Users/tw/w/borg/repo'
security: read manifest timestamp '2022-01-30T17:09:22.084208'
security: determined newest manifest timestamp as 2022-01-30T17:09:22.084208
security: repository checks ok, allowing access
**Synchronizing chunks cache...**        <----------------------- !!!
Archives: 0, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 0.
Done.
security: saving state for ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4 to /Users/tw/.config/borg/security/ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4
security: current location   /Users/tw/w/borg/repo
security: key type           2
security: manifest timestamp 2022-01-30T17:09:22.389184
Repository ID: ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4
Location: /Users/tw/w/borg/repo
Encrypted: No
Cache: /Users/tw/.cache/borg/ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4
Security dir: /Users/tw/.config/borg/security/ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                    0 B                  0 B                  0 B

                       Unique chunks         Total chunks
Chunk index:                       0                    0
*** 2nd info
using builtin fallback logging configuration
35 self tests completed in 0.08 seconds
Verified integrity of /Users/tw/w/borg/repo/index.5
TAM-verified manifest
security: read previous location '/Users/tw/w/borg/repo'
security: read manifest timestamp '2022-01-30T17:09:22.389184'
security: determined newest manifest timestamp as 2022-01-30T17:09:22.389184
security: repository checks ok, allowing access
Verified integrity of /Users/tw/.cache/borg/ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4/chunks
security: read previous location '/Users/tw/w/borg/repo'
security: read manifest timestamp '2022-01-30T17:09:22.389184'
security: determined newest manifest timestamp as 2022-01-30T17:09:22.389184
security: repository checks ok, allowing access
Repository ID: ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4
Location: /Users/tw/w/borg/repo
Encrypted: No
Cache: /Users/tw/.cache/borg/ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4
Security dir: /Users/tw/.config/borg/security/ecee9203c12826c101515fa61e00d85f7957c282537faf8009fc4429469e2ae4
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                    0 B                  0 B                  0 B

                       Unique chunks         Total chunks
Chunk index:                       0                    0
*** final cleanup
You requested to completely DELETE the repository *including* all archives it contains:
Type 'YES' if you understand this and want to continue: YES (from BORG_DELETE_I_KNOW_WHAT_I_AM_DOING)
```


Resolving patch:
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -1352,6 +1352,7 @@ def _delete_archives(self, args, repository):
         with Cache(repository, key, manifest, progress=args.progress, lock_wait=self.lock_wait) as cache:
             msg_delete = 'Would delete archive: {} ({}/{})' if dry_run else 'Deleting archive: {} ({}/{})'
             msg_not_found = 'Archive {} not found ({}/{}).'
+            delete_count = 0
             for i, archive_name in enumerate(archive_names, 1):
                 try:
                     archive_info = manifest.archives[archive_name]
@@ -1363,7 +1364,9 @@ def _delete_archives(self, args, repository):
                         archive = Archive(repository, key, manifest, archive_name, cache=cache,
                                           consider_part_files=args.consider_part_files)
                         archive.delete(stats, progress=args.progress, forced=args.forced)
-            if not dry_run:
+                        delete_count += 1
+            if delete_count > 0:
+                # only write/commit if we actually changed something, see #6060.
                 manifest.write()
                 repository.commit(save_space=args.save_space)
                 cache.commit()


Test patch: