Repository: borgbackup/borg

Issue:
libressl's AEAD api
it seems that they don't support these ciphers via the same EVP api as openssl:
- AES-OCB
- CHACHA20-POLY1305

For borg 1.3+ it seems like a good idea if we would deprecate the crypto modes based on AES-CTR and recommend the new AEAD modes.

So, how's the situation on OpenBSD? Are there other platforms requiring LibreSSL support?

- Do we need to support LibreSSL to run on OpenBSD or could we use OpenSSL there also?
- If we can neither use OpenSSL there nor some compatible API in LibreSSL, do we want to add some code for LibreSSL to our `borg.crypto.low_level` module that uses LibreSSL's EVP_AEAD api (which does not exist in OpenSSL)? https://man.openbsd.org/EVP_AEAD_CTX_init.3
- seems like we only get AES-GCM and chacha20-poly1305 via that api. there were objections against gcm, so we went for ocb rather. so guess that would mean with LibreSSL it would only be chacha20-poly1305 (which is fine as it is quite fast).



Discussion:
@ThomasWaldmann

> So, how's the situation on OpenBSD? Are there other platforms requiring LibreSSL support?
>
> - Do we need to support LibreSSL to run on OpenBSD or could we use OpenSSL there also?

Some background:

A lot of people, including the LibreSSL people, OpenBSD's porters and upstream projects, invested a lot of effort to get LibreSSL to work with the hundreds of tools that exist. OpenBSD for example has ~11000 ports (including flavors and subpackages) of which ~800 use LibreSSL and only 6 explicitly require OpenSSL.

Concerning your questions: Strictly taken it is possible to use OpenSSL on OpenBSD, though it is not preferred. I'm not sure about requirements from other platforms.

> - If we can neither use OpenSSL there nor some compatible API in LibreSSL, do we want to add some code for LibreSSL to our borg.crypto.low_level module that uses LibreSSL's EVP_AEAD api (which does not exist in OpenSSL)? https://man.openbsd.org/EVP_AEAD_CTX_init.3

EVP_aead seems to be borrowed from BoringSSL, and the API for EVP_aead is completely incompatible with what Borg currently uses.
I'm hoping that the EVP things for some of the djb ciphers will be eventually implemented in LibreSSL. AES-OCB is rarely used and was patent encumbered, which does not not help discussion on implementation.

It would be really cool if Borg keeps on supporting LibreSSL. However, in the end it is all about effort needed, and return on effort...

> - seems like we only get AES-GCM and chacha20-poly1305 via that api. there were objections against gcm, so we went for ocb rather. so guess that would mean with LibreSSL it would only be chacha20-poly1305 (which is fine as it is quite fast).

I do not want to start a discussion on crypto as I do not understand it. I'm wondering though why OCB is better than GCM. Is it because OCB is a bit less fragile than GCM against repeated nonces? Are there other reasons? Is there a hard reason not to use GCM?
I am roughly aware about LibreSSL and other non-OpenSSL efforts (and had my own issues when working with OpenSSL).
But we need something that supports the borg legacy (== fast AES256-CTR / HMAC-SHA256) as well as a better future (AEAD ciphers: OCB and CHPO).

The OCB patents issue is long resolved for FOSS, see: https://en.wikipedia.org/wiki/OCB_mode and as a recent update also this: https://mailarchive.ietf.org/arch/msg/cfrg/qLTveWOdTJcLn4HP3ev-vrj05Vg/

I can not give you the details about the GCM concerns, but when searching for usable AEAD ciphers I got the impression that quite some cryptographers feel bad about GCM. Considering that OCB is also faster made it clear for me to choose OCB. Years ago, there was the availability issue, but this is also resolved for most platforms since openssl 1.1.0 (and all openssl versions without AES-OCB not receiving security support any more).

So, if openssl is working on openbsd, maybe we should switch to that for borg 1.3+?

Alternatively, maybe some openbsd python/cython developer who is into crypto stuff could work on `borg.crypto.low_level` for the alternative code needed for LibreSSL? As long as they don't have OCB, one can't read such repos on LibreSSL though.

good news: i have a vagrant based openbsd-current (future 7.1) test box now. all master tests passing currently.
Update: The AES-OCB and chacha20-poly1305 based crypto is now in master branch (was #6463).

I put this into 1.3.0b1 milestone (first beta release).

@bket When the alpha releases get successful testing on openbsd with openssl, I guess we can close this because we'll just use openssl then. In case libressl catches up later regarding API and available algorithms, we can reopen this.

@bket could you help fixing the `Vagrantfile` in master branch, so it works based on `openssl 1.1.1`?

Currently it tries building on libressl, which will fail the tests.
@ThomasWaldmann, the `Vagrantfile` needs a small addition:

```
diff --git Vagrantfile Vagrantfile
index eea43d8c..359ae6d6 100644
--- Vagrantfile
+++ Vagrantfile
@@ -78,6 +78,7 @@ def packages_openbsd
     pkg_add lz4
     pkg_add zstd
     pkg_add git  # no fakeroot
+    pkg_add openssl%1.1
     pkg_add py3-pip
     pkg_add py3-virtualenv
   EOF
```

Unfortunately this is not enough as we need to explicity tell the build to link against `libcrypto` from the installed package. I have the diff below sitting in my tree, which passes some flags. Setting `BORG_OPENSSL_PREFIX` is not going to cut it because of some renaming. With this diff `pkgconfig` does the right thing, and everything builds/tests OK. Still thinking about an alternative solution.

```
Index: setup.py
--- setup.py.orig
+++ setup.py
@@ -161,12 +161,13 @@ if not on_rtd:
             pc, 'BORG_OPENSSL_PREFIX', 'libcrypto', 'libcrypto', '>=1.1.1', lib_subdir='')
     else:
         crypto_ext_lib = lib_ext_kwargs(
-            pc, 'BORG_OPENSSL_PREFIX', 'crypto', 'libcrypto', '>=1.1.1')
+            pc, 'BORG_OPENSSL_PREFIX', 'crypto', 'libecrypto11', '>=1.1.1')
 
     crypto_ext_kwargs = members_appended(
         dict(sources=[crypto_ll_source, crypto_helpers]),
         crypto_ext_lib,
         dict(extra_compile_args=cflags),
+        dict(extra_link_args=['-Wl,-rpath,/usr/local/lib/eopenssl11']),
     )
 
     compress_ext_kwargs = members_appended(
```
thanks for the info!

pkg_add: ah, it needs a percent separator. I tried `-` and `@` and ..., but didn't find that.

the renaming also explains why i did not even manually find the right `crypto.pc` file...

setup.py: if there is no easier way, guess we need to add a platform check there and just switch these values based on the platform.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -23,6 +23,7 @@
 import setup_docs
 
 is_win32 = sys.platform.startswith('win32')
+is_openbsd = sys.platform.startswith('openbsd')
 
 # Number of threads to use for cythonize, not used on windows
 cpu_threads = multiprocessing.cpu_count() if multiprocessing and multiprocessing.get_start_method() != 'spawn' else None
@@ -48,7 +49,6 @@
 
 compress_source = 'src/borg/compress.pyx'
 crypto_ll_source = 'src/borg/crypto/low_level.pyx'
-crypto_helpers = 'src/borg/crypto/_crypto_helpers.c'
 chunker_source = 'src/borg/chunker.pyx'
 hashindex_source = 'src/borg/hashindex.pyx'
 item_source = 'src/borg/item.pyx'
@@ -156,17 +156,24 @@ def lib_ext_kwargs(pc, prefix_env_var, lib_name, lib_pkg_name, pc_version, lib_s
             f"or ensure {lib_pkg_name}.pc is in PKG_CONFIG_PATH."
         )
 
+    crypto_ldflags = []
     if is_win32:
         crypto_ext_lib = lib_ext_kwargs(
             pc, 'BORG_OPENSSL_PREFIX', 'libcrypto', 'libcrypto', '>=1.1.1', lib_subdir='')
+    elif is_openbsd:
+        # use openssl (not libressl) because we need AES-OCB and CHACHA20-POLY1305 via EVP api
+        crypto_ext_lib = lib_ext_kwargs(
+            pc, 'BORG_OPENSSL_PREFIX', 'crypto', 'libecrypto11', '>=1.1.1')
+        crypto_ldflags += ['-Wl,-rpath=/usr/local/lib/eopenssl11']
     else:
         crypto_ext_lib = lib_ext_kwargs(
             pc, 'BORG_OPENSSL_PREFIX', 'crypto', 'libcrypto', '>=1.1.1')
 
     crypto_ext_kwargs = members_appended(
-        dict(sources=[crypto_ll_source, crypto_helpers]),
+        dict(sources=[crypto_ll_source]),
         crypto_ext_lib,
         dict(extra_compile_args=cflags),
+        dict(extra_link_args=crypto_ldflags),
     )
 
     compress_ext_kwargs = members_appended(
diff --git a/src/borg/archiver.py b/src/borg/archiver.py
--- a/src/borg/archiver.py
+++ b/src/borg/archiver.py
@@ -602,7 +602,6 @@ def chunkit(chunker_name, *args, **kwargs):
 
         from borg.crypto.low_level import AES256_CTR_BLAKE2b, AES256_CTR_HMAC_SHA256
         from borg.crypto.low_level import AES256_OCB, CHACHA20_POLY1305
-        from borg.crypto.low_level import is_libressl
         print("Encryption =====================================================")
         size = "1GB"
 
@@ -611,14 +610,11 @@ def chunkit(chunker_name, *args, **kwargs):
                 key_256, key_256, iv=key_128, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
             ("aes-256-ctr-blake2b", lambda: AES256_CTR_BLAKE2b(
                 key_256*4, key_256, iv=key_128, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
+            ("aes-256-ocb", lambda: AES256_OCB(
+                key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
+            ("chacha20-poly1305", lambda: CHACHA20_POLY1305(
+                key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
         ]
-        if not is_libressl:
-            tests.extend([
-                ("aes-256-ocb", lambda: AES256_OCB(
-                    key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
-                ("chacha20-poly1305", lambda: CHACHA20_POLY1305(
-                    key_256, iv=key_96, header_len=1, aad_offset=1).encrypt(random_10M, header=b'X')),
-            ])
         for spec, func in tests:
             print(f"{spec:<24} {size:<10} {timeit(func, number=100):.3f}s")
 
diff --git a/src/borg/testsuite/crypto.py b/src/borg/testsuite/crypto.py
--- a/src/borg/testsuite/crypto.py
+++ b/src/borg/testsuite/crypto.py
@@ -3,8 +3,7 @@
 import unittest
 
 
-from ..crypto.low_level import AES256_CTR_HMAC_SHA256, AES256_OCB, CHACHA20_POLY1305, UNENCRYPTED, \
-                               IntegrityError, is_libressl
+from ..crypto.low_level import AES256_CTR_HMAC_SHA256, AES256_OCB, CHACHA20_POLY1305, UNENCRYPTED, IntegrityError
 from ..crypto.low_level import bytes_to_long, bytes_to_int, long_to_bytes
 from ..crypto.low_level import hkdf_hmac_sha512
 from ..crypto.low_level import AES, hmac_sha256
@@ -103,16 +102,13 @@ def test_AE(self):
         header = b'\x23' + iv_int.to_bytes(12, 'big')
         tests = [
             # (ciphersuite class, exp_mac, exp_cdata)
+            (AES256_OCB,
+             b'b6909c23c9aaebd9abbe1ff42097652d',
+             b'877ce46d2f62dee54699cebc3ba41d9ab613f7c486778c1b3636664b1493', ),
+            (CHACHA20_POLY1305,
+             b'fd08594796e0706cde1e8b461e3e0555',
+             b'a093e4b0387526f085d3c40cca84a35230a5c0dd766453b77ba38bcff775', )
         ]
-        if not is_libressl:
-            tests += [
-                (AES256_OCB,
-                 b'b6909c23c9aaebd9abbe1ff42097652d',
-                 b'877ce46d2f62dee54699cebc3ba41d9ab613f7c486778c1b3636664b1493', ),
-                (CHACHA20_POLY1305,
-                 b'fd08594796e0706cde1e8b461e3e0555',
-                 b'a093e4b0387526f085d3c40cca84a35230a5c0dd766453b77ba38bcff775', )
-            ]
         for cs_cls, exp_mac, exp_cdata in tests:
             # print(repr(cs_cls))
             # encrypt/mac
@@ -146,16 +142,13 @@ def test_AEAD(self):
         header = b'\x12\x34\x56' + iv_int.to_bytes(12, 'big')
         tests = [
             # (ciphersuite class, exp_mac, exp_cdata)
+            (AES256_OCB,
+             b'f2748c412af1c7ead81863a18c2c1893',
+             b'877ce46d2f62dee54699cebc3ba41d9ab613f7c486778c1b3636664b1493', ),
+            (CHACHA20_POLY1305,
+             b'b7e7c9a79f2404e14f9aad156bf091dd',
+             b'a093e4b0387526f085d3c40cca84a35230a5c0dd766453b77ba38bcff775', )
         ]
-        if not is_libressl:
-            tests += [
-                (AES256_OCB,
-                 b'f2748c412af1c7ead81863a18c2c1893',
-                 b'877ce46d2f62dee54699cebc3ba41d9ab613f7c486778c1b3636664b1493', ),
-                (CHACHA20_POLY1305,
-                 b'b7e7c9a79f2404e14f9aad156bf091dd',
-                 b'a093e4b0387526f085d3c40cca84a35230a5c0dd766453b77ba38bcff775', )
-            ]
         for cs_cls, exp_mac, exp_cdata in tests:
             # print(repr(cs_cls))
             # encrypt/mac
@@ -187,9 +180,7 @@ def test_AEAD_with_more_AAD(self):
         iv_int = 0
         data = b'foo' * 10
         header = b'\x12\x34'
-        tests = []
-        if not is_libressl:
-            tests += [AES256_OCB, CHACHA20_POLY1305]
+        tests = [AES256_OCB, CHACHA20_POLY1305]
         for cs_cls in tests:
             # encrypt/mac
             cs = cs_cls(key, iv_int, header_len=len(header), aad_offset=0)


Test patch: