Repository: borgbackup/borg

Issue:
compression test issue on S390 (1.1-maint)
From @FelixSchwarz:

Thanks Thomas, I did miss that you had additional commits on 1.1-maint. I cherry-picked some commits which seem relevant but the build still fails:
```
_____________ ArchiverTestCase.test_compression_auto_compressible ______________
self = <borg.testsuite.archiver.ArchiverTestCase testMethod=test_compression_auto_compressible>
    def test_compression_auto_compressible(self):
        size, csize = self._get_sizes('auto,zlib', compressible=True)
        assert csize < size * 0.1
>       assert csize == 35  # same as compression 'zlib'
E       AssertionError: assert 53 == 35
build/lib.linux-s390x-3.6/borg/testsuite/archiver.py:2255: AssertionError
```

Anyway, thank you for maintaining borg the way you do (long branch maintenance, using system libraries, ...) - it really makes distro packaging much easier even though it is a lot of extra work for you.


Discussion:
Looks weird, but not like a endian issue.

@FelixSchwarz Can you run `test_compression_zlib_compressible` on S390. Does it fail with the same values in the assert expression? Maybe these tests just assume something that is not true on that system.
@FelixSchwarz Yeah, guess my feeling was right, see PR.
@FelixSchwarz Looks like the linux and macOS tests are happy, but now needs another test run on S390 with that changeset cherry-picked, see #7364.
Pushed a new build to s390x, will report about the results.
better but not there yet
```
_____________ ArchiverTestCase.test_compression_auto_compressible ______________
self = <borg.testsuite.archiver.ArchiverTestCase testMethod=test_compression_auto_compressible>
    def test_compression_auto_compressible(self):
        # this is testing whether the "auto" meta-compressor chooses the "expensive" zlib compression
        # if it detects that the data is very compressible (and not "lz4" nor "none" compression).
        auto_size, auto_csize = self._get_sizes('auto,zlib', compressible=True)
        self.cmd('delete', self.repository_location)
        zlib_size, zlib_csize = self._get_sizes('zlib', compressible=True)
        assert auto_size == zlib_size
        assert auto_csize < auto_size * 0.1  # it did compress!
>       assert auto_csize == zlib_csize  # looking at the result size, it seems to be zlib compressed
E       AssertionError: assert 53 == 100
build/lib.linux-s390x-3.6/borg/testsuite/archiver.py:2275: AssertionError
```

Unsurprisingly the `RemoteArchiverTestCase` fails with the same error but other that's about it in terms of test suite on s390x.
Ehrm, that is unexpected.

Is there something wrong with lz4 (used in the "auto" meta-compressor to detect whether data is compressible)?
Thought about this:

- Looks like the `zlib` compressed result is 100 bytes here (quite a bit more than we've seen on other platforms).
- OTOH, the `auto,zlib` compressed result is 53, so it is obviously not zlib compressed.
- It is also not `none` compressed (that would be much more).
- So this can only mean that on S390, `lz4` compressed this kind of data better than `zlib` and thus is chosen by the auto compressor.


Resolving patch:
diff --git a/src/borg/testsuite/archiver.py b/src/borg/testsuite/archiver.py
--- a/src/borg/testsuite/archiver.py
+++ b/src/borg/testsuite/archiver.py
@@ -2265,14 +2265,19 @@ def test_compression_lzma_uncompressible(self):
         assert csize >= size
 
     def test_compression_auto_compressible(self):
-        # this is testing whether the "auto" meta-compressor chooses the "expensive" zlib compression
-        # if it detects that the data is very compressible (and not "lz4" nor "none" compression).
+        # this is testing whether the "auto" meta-compressor behaves as expected:
+        # - it checks whether the data is compressible (detector is the lz4 compressor)
+        # - as the data is compressible, it runs the "expensive" zlib compression on it
+        # - it returns whatever is shortest, either the lz4 compressed data or the zlib compressed data.
         auto_size, auto_csize = self._get_sizes('auto,zlib', compressible=True)
         self.cmd('delete', self.repository_location)
         zlib_size, zlib_csize = self._get_sizes('zlib', compressible=True)
-        assert auto_size == zlib_size
+        self.cmd('delete', self.repository_location)
+        lz4_size, lz4_csize = self._get_sizes('lz4', compressible=True)
+        assert auto_size == zlib_size == lz4_size
         assert auto_csize < auto_size * 0.1  # it did compress!
-        assert auto_csize == zlib_csize  # looking at the result size, it seems to be zlib compressed
+        smallest_csize = min(zlib_csize, lz4_csize)
+        assert auto_csize == smallest_csize
 
     def test_compression_auto_uncompressible(self):
         # this is testing whether the "auto" meta-compressor chooses the "none" compression (storing the


Test patch: