Repository: borgbackup/borg

Issue:
recreate: avoid recompressing same chunk twice
Q1: when I run 
$ for archive in a1 a2 a3; do borg recreate repo::$archive --recompress always --compression abc,L ; done

... will this re-compress same chunk three times - if the chunk is shared by the 3 archives?

I now learned from another thread that --recompress accepts always and if-different - but I need if-different to be split into if-different-algo and if-different-algo-and-level - to cover all possible use cases.

Q2: Is that easy?

One way to avoid recompressing same chunk over and over again is via recompressing whole repo - IF some fail-safe to avoid multi-recompress-of-same-chunk is already in place - but whole-repo-recompress currently does not support filter by archive name.... which I need as I want to recompress/recreate only one sub-set of the archives.

The other way is to add --recompress if-different-algo-and-level.





Discussion:
A1: yes, always means always.

And we do not store the compression level, so there can be no decision based on the level.

Q2: it would need additional management and we do not currently have that (that could be e.g. an additional field in the chunks index, but we can not easily change that without causing compatibility issues esp. when downgrading).

So, considering recreate is a rare operation, I don't see enough reasons to change that.
Sounds ok: to re-phrase:
- people that need to recompress from same compression ABC with level=3 to ABC level=10 will just use --recompress always and have to wait recompression to happen for 
a) XYZ.L -> ABC.10 - no room for improvement here
b) ABC.3 -> ABC.10 - again these need to be recompressed - no rrom for improvement
c) ABC.10 -> ABC.10 - for chunks already having same compression - with 'always' on they'll be re-compressed too but we need 'always' cause with 'if-different' b) and c) will be skipped.
So having to choose between
A) doing a) and skipping b and c (--recompress if-different)
and
B) doing a,b,c (--recompress always)

I'd just choose B) wait a bit extra time and be happy.
Cool.



Final Q: When recompressing whole repo with 'always' - does it iterate archive by archive and recompresses shared chunks multiple times OR the iteration is over chunks from 1st to last?
Externally it could still say 'procesing archive 1' then 2, then archive 3, but what happens internally under the hood?


...Cause to recompres a whole repo you have two options:
a) - loop through archives doing them one by one - might compress same chunks over and over again
b) - recompress whole repo directly - MIGHT be smart enough to avoid that

Is b) smart or "dumb" currently?



Resolving patch:
diff --git a/src/borg/archive.py b/src/borg/archive.py
--- a/src/borg/archive.py
+++ b/src/borg/archive.py
@@ -2080,8 +2080,9 @@ def chunk_processor(self, target, chunk):
         if self.recompress and not self.always_recompress and chunk_id in self.cache.chunks:
             # Check if this chunk is already compressed the way we want it
             old_chunk = self.key.decrypt(chunk_id, self.repository.get(chunk_id), decompress=False)
-            if Compressor.detect(old_chunk).name == self.key.compressor.decide(data).name:
-                # Stored chunk has the same compression we wanted
+            compressor_cls, level = Compressor.detect(old_chunk)
+            if compressor_cls.name == self.key.compressor.decide(data).name and level == self.key.compressor.level:
+                # Stored chunk has the same compression method and level as we wanted
                 overwrite = False
         chunk_entry = self.cache.add_chunk(chunk_id, data, target.stats, overwrite=overwrite, wait=False)
         self.cache.repository.async_response(wait=False)
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -4809,9 +4809,8 @@ def define_borg_mount(parser):
                                    help='recompress data chunks according to `MODE` and ``--compression``. '
                                         'Possible modes are '
                                         '`if-different`: recompress if current compression is with a different '
-                                        'compression algorithm (the level is not considered); '
-                                        '`always`: recompress even if current compression is with the same '
-                                        'compression algorithm (use this to change the compression level); and '
+                                        'compression algorithm or different level; '
+                                        '`always`: recompress unconditionally; and '
                                         '`never`: do not recompress (use this option to explicitly prevent '
                                         'recompression). '
                                         'If no MODE is given, `if-different` will be used. '
diff --git a/src/borg/testsuite/compress.py b/src/borg/testsuite/compress.py
--- a/src/borg/testsuite/compress.py
+++ b/src/borg/testsuite/compress.py
@@ -133,11 +133,11 @@ def test_auto():
     compressed_lz4 = compressor_lz4.compress(data)
     compressed_zlib = compressor_zlib.compress(data)
     ratio = len(compressed_zlib) / len(compressed_lz4)
-    assert Compressor.detect(compressed_auto_zlib) == ZLIB if ratio < 0.99 else LZ4
+    assert Compressor.detect(compressed_auto_zlib)[0] == ZLIB if ratio < 0.99 else LZ4
 
     data = b'\x00\xb8\xa3\xa2-O\xe1i\xb6\x12\x03\xc21\xf3\x8a\xf78\\\x01\xa5b\x07\x95\xbeE\xf8\xa3\x9ahm\xb1~'
     compressed = compressor_auto_zlib.compress(data)
-    assert Compressor.detect(compressed) == CNONE
+    assert Compressor.detect(compressed)[0] == CNONE
 
 
 def test_obfuscate():


Test patch: