Repository: easybuilders/easybuild-easyblocks

Issue:
make OpenSSL aware of host CA certs
While developing a PythonBundle eb I came across a situation where one of the entries in exts_lists was trying to fetch additional dependencies from pypi.org. This was failing on `[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate`. Looking at strace of what was going on I got the impression that eb installed OpenSSL is looking into its `ssl/certs` folder for CA certificates, which is empty. It appears as if it is looking for cert hashes and not even considering ca-bundle.crt and similar.

I'm not familiar enough with inner workings of OpenSSL to come up with a solution, but it looks like it will differ slightly between distributions. 


Discussion:
Additional info: we found out that this only happens when openssl easyblock doesn't find openssl headers on host and decides to build openssl from source. In this case variable OPENSSLDIR points to installdir/ssl instead of proper host location in /etc.
@lexming I think you were involved in this discussion, anything to fix here in the OpenSSL wrapper easyblock?
1 - If the wrapper easyblock finds libraries and engines of the required version but not the include files, the first option should be to error out and ask the user to get the corresponding development package installed. Of course we need an option to go ahead and install from source anyway in case sysadmins refuse.
2 - if the wrapper easyblocks finds ANY installed openssl version it should ask it for the path to the system installed ca-certs and pass it on to the build-from-source step. Easily done through a env var if there is no other easy way.

If there is NO openssl version installed then the system is almost guaranteed to be broken and we should inform the users they need to specify the path to the systems ca-certs. I.e. the ca-certs package must be installed on the OS level, always.
This is indeed only an issue whenever OpenSSL is built from source by EB. Regardless if it is built by the wrapper or a regular OpenSSL easyconfig. Certificates are not part of OpenSSL sources.

If OpenSSL in the host is successfully wrapped by EB, the path to the certificates in the system will be known by OpenSSL. The path in `OPENSSLDIR` is set at build time, so it is unaffected by the wrapper and our symlinks.

Making the build of OpenSSL aware of any certificates in the host system is not trivial. The certificates may or may not be installed and the path to the certificates is distro dependent (*eg* `/etc/pki/tls` in RHEL, `/usr/lib/ssl` in Debian). The only reliable way to locate the certificates is through a working OpenSSL installation in the host to be able to query for `OPENSSLDIR` with `openssl version -d`. 

Cases where we the wrapper will build OpenSSL from source but we could still have a working `openssl version -d`:
1. OpenSSL in host is just missing the headers
    In this case I agree with @akesandgren, if only the headers are missing the wrapper should fail the installation and notify the user. That's usually easy to correct. (maybe add an `os_dependency`? but that is complicated with the wrapper as it can also build from source)
2. OpenSSL in host is outdated
    In this case I don't see why we should trust the certificates in the system, as those will probably be outdated as well. But it would be relatively easy to get the path and use it in our builds.
3. User is manually forcing the build of OpenSSL from source
    In this case we can try to catch `OPENSSLDIR`. If that fails, the alternative is to add an option so that the path to certificates can be set by the user.
Actually /etc/ssl/certs work on both RHEL and Debian :-) the magic of symlinks
But others may be different.
2. OpenSSL is outdated, that may just be that whatever you're building requires a newer libssl, it does NOT imply that the ca-cert info is outdated. And the system installed ca-certs should still always be used. (even if you don't really trust them for other reasons)
I ran into the same issue, as I reported for the easyconfig of OpenSSL at: https://github.com/easybuilders/easybuild-easyconfigs/issues/14901
This was for an CentOS 7 system where the standard (supported) system OpenSSL is version 1.02k, whereas EasyBuild requires OpenSSL 1.1 for more recent packages. The wrapper then installs its own version of OpenSSL 1.1. 

OpenSSL does indeed not come with certificates and depends on the certificates on the host installed with the OS. The documentation on the location of the certificates in the openssl build instructions is very sketchy, as the path of the certificates obtained from `--openssldir` is or was related to the `--prefix` setting. Details can be found at:
https://wiki.openssl.org/index.php/Compilation_and_Installation#PREFIX_and_OPENSSLDIR 

Setting --openssldir (as is done in the spec file for rpms) may not be possible, as the install step seems to try to install stuff into that system path, which easybuild should not be allowed to do. In my current EasyBuild based OpenSSL/1.1 installation the path contains additional files and folders installed by the EasyBuild OpenSSL installation, so I assume it will try to do so.
Creating a symbolic link to the system certificate path works fine however, but I am not sure how robust that solution is.

And there is still the issue of how to obtain the system certificate path. This could be indeed by done using an existing openssl, which a system should have anyway. This requirement could be put in the `osdependencies`.

Resolving patch:
diff --git a/easybuild/easyblocks/o/openssl.py b/easybuild/easyblocks/o/openssl.py
--- a/easybuild/easyblocks/o/openssl.py
+++ b/easybuild/easyblocks/o/openssl.py
@@ -1,5 +1,5 @@
 ##
-# Copyright 2009-2021 Ghent University
+# Copyright 2009-2022 Ghent University
 #
 # This file is part of EasyBuild,
 # originally created by the HPC team of Ghent University (http://ugent.be/hpc/en),
@@ -28,18 +28,81 @@
 @author: Kenneth Hoste (Ghent University)
 @author: Jens Timmerman (Ghent University)
 @author: Davide Vanzo (ACCRE - Vanderbilt University)
+@author: Alex Domingo (Vrije Universiteit Brussel)
 """
 import os
+import re
+
 from distutils.version import LooseVersion
 
 from easybuild.easyblocks.generic.configuremake import ConfigureMake
-from easybuild.tools.build_log import EasyBuildError
+from easybuild.framework.easyconfig import CUSTOM
+from easybuild.tools.build_log import EasyBuildError, print_warning
+from easybuild.tools.filetools import remove_dir, symlink
 from easybuild.tools.run import run_cmd
+from easybuild.tools.systemtools import get_shared_lib_ext
+
+GENERIC_SSL_CERTS_DIR = "/etc/ssl/certs"
 
 
 class EB_OpenSSL(ConfigureMake):
     """Support for building OpenSSL"""
 
+    @staticmethod
+    def extra_options(extra_vars=None):
+        """Easyconfig parameters specific to OpenSSL"""
+        extra_vars = ConfigureMake.extra_options(extra_vars=extra_vars)
+        extra_vars.update({
+            'ssl_certificates': [None, "Absolute path to 'certs' directory with the system SSL certificates", CUSTOM],
+        })
+        return extra_vars
+
+    def __init__(self, *args, **kwargs):
+        """Initialisation of custom class variables for OpenSSL"""
+        super(EB_OpenSSL, self).__init__(*args, **kwargs)
+
+        # path to SSL certificates
+        ssl_certs_dir = self.cfg.get('ssl_certificates')
+
+        if ssl_certs_dir is not None:
+            # check option ssl_certificates
+            ssl_certs_dir = os.path.normpath(ssl_certs_dir)
+            if not os.path.isabs(ssl_certs_dir):
+                raise EasyBuildError("ssl_certificates is not an absolute path: %s", ssl_certs_dir)
+            if os.path.basename(ssl_certs_dir) != 'certs':
+                raise EasyBuildError("ssl_certificates does not point to a 'certs' directory: %s", ssl_certs_dir)
+            if not os.path.isdir(ssl_certs_dir):
+                raise EasyBuildError("ssl_certificates 'certs' directory does not exist: %s", ssl_certs_dir)
+        else:
+            # set ssl_certs_dir from system OPENSSLDIR
+            openssldir = ''
+            openssldir_regex = re.compile(r'^OPENSSLDIR: "(.*)"$')
+            openssldir_cmd = "openssl version -d"
+
+            try:
+                (out, _) = run_cmd(openssldir_cmd, log_all=True, simple=False, trace=False)
+                openssldir = openssldir_regex.search(out).group(1)
+            except EasyBuildError:
+                self.log.info("OPENSSLDIR not found in system (openssl command failed), "
+                              "continuing with generic OPENSSLDIR path...")
+            except AttributeError:
+                self.log.debug("OPENSSLDIR not found in system (openssl reported '%s'), "
+                               "continuing with generic OPENSSLDIR path...", out)
+            else:
+                self.log.info("OPENSSLDIR determined from system openssl: %s", openssldir)
+
+            if os.path.isdir(openssldir):
+                ssl_certs_dir = os.path.join(openssldir, 'certs')
+            elif os.path.isdir(GENERIC_SSL_CERTS_DIR):
+                # fallback to generic OPENSSLDIR
+                ssl_certs_dir = GENERIC_SSL_CERTS_DIR
+                self.log.info("Falling back to generic SSL certificates directory: %s", ssl_certs_dir)
+            else:
+                self.log.info("Generic SSL certificates directory not found: %s", GENERIC_SSL_CERTS_DIR)
+
+        self.ssl_certs_dir = ssl_certs_dir
+        self.log.debug("SSL certificates directory: %s", self.ssl_certs_dir)
+
     def configure_step(self, cmd_prefix=''):
         """
         Configure step
@@ -52,30 +115,48 @@ def configure_step(self, cmd_prefix=''):
 
         return out
 
-    def sanity_check_step(self):
-        """Custom sanity check"""
+    def install_step(self):
+        """Installation of OpenSSL and SSL certificates"""
+        super(EB_OpenSSL, self).install_step()
+
+        # SSL certificates
+        # OPENSSLDIR is already populated by the installation of OpenSSL
+        # try to symlink system certificates in the empty 'certs' directory
+        openssl_certs_dir = os.path.join(self.installdir, 'ssl', 'certs')
 
-        libdir = None
-        for libdir_cand in ['lib', 'lib64']:
-            if os.path.exists(os.path.join(self.installdir, libdir_cand)):
-                libdir = libdir_cand
+        if self.ssl_certs_dir:
+            remove_dir(openssl_certs_dir)
+            symlink(self.ssl_certs_dir, openssl_certs_dir)
+        else:
+            print_warning("OpenSSL successfully installed without system SSL certificates. "
+                          "Some packages might experience limited functionality.")
 
-        if libdir is None:
-            raise EasyBuildError("Failed to determine library directory.")
+    def sanity_check_step(self):
+        """Custom sanity check"""
 
+        # basic paths
         custom_paths = {
-            'files': [os.path.join(libdir, x) for x in ['libcrypto.a', 'libcrypto.so', 'libssl.a', 'libssl.so']] +
-            ['bin/openssl'],
-            'dirs': [],
+            'files': ['bin/openssl'],
+            'dirs': ['include', 'ssl'],
         }
 
-        if LooseVersion(self.version) < LooseVersion("1.1"):
-            custom_paths['files'].extend([os.path.join(libdir, 'libcrypto.so.1.0.0'),
-                                          os.path.join(libdir, 'libssl.so.1.0.0')])
-            custom_paths['dirs'].append(os.path.join(libdir, 'engines'))
-        else:
-            custom_paths['files'].extend([os.path.join(libdir, 'libcrypto.so.1.1'),
-                                          os.path.join(libdir, 'libssl.so.1.1')])
-            custom_paths['dirs'].append(os.path.join(libdir, 'engines-1.1'))
+        # add libraries
+        lib_dir = 'lib'
+        lib_sonames = ['libcrypto', 'libssl']
+        shlib_ext = get_shared_lib_ext()
+        lib_files = [os.path.join(lib_dir, '%s.%s') % (x, y) for x in lib_sonames for y in ['a', shlib_ext]]
+
+        custom_paths['files'].extend(lib_files)
+
+        # add engines
+        engines_dir = 'engines'
+        if LooseVersion(self.version) >= LooseVersion("1.1"):
+            engines_dir = 'engines-1.1'
+
+        custom_paths['dirs'].append(os.path.join(lib_dir, engines_dir))
+
+        # add SSL certificates
+        if self.ssl_certs_dir:
+            custom_paths['dirs'].append('ssl/certs')
 
         super(EB_OpenSSL, self).sanity_check_step(custom_paths=custom_paths)
diff --git a/easybuild/easyblocks/o/openssl_wrapper.py b/easybuild/easyblocks/o/openssl_wrapper.py
--- a/easybuild/easyblocks/o/openssl_wrapper.py
+++ b/easybuild/easyblocks/o/openssl_wrapper.py
@@ -222,7 +222,7 @@ def __init__(self, *args, **kwargs):
         self.log.debug("Found the following include directories in host system: %s", ', '.join(sys_include_dirs))
 
         # headers are located in 'include/openssl' by default
-        ssl_include_subdirs = [self.name.lower()]
+        ssl_include_subdirs = ['openssl']
         if self.majmin_version == '1.1':
             # but version 1.1 can be installed in 'include/openssl11/openssl' as well, for example in CentOS 7
             # prefer 'include/openssl' as long as the version of headers matches
@@ -256,8 +256,10 @@ def __init__(self, *args, **kwargs):
                 self.log.info("System OpenSSL header file %s not found", opensslv_path)
 
         if not self.system_ssl['include']:
-            self.log.info("OpenSSL headers not found in host system, falling back to OpenSSL in EasyBuild")
-            return
+            err_msg = ("OpenSSL v%s headers not found in host system, but libraries for v%s are present. "
+                       "Install the development package of OpenSSL for your system or force building OpenSSL from "
+                       "source in EasyBuild by setting 'wrap_system_openssl = False' in the OpenSSL easyconfig.")
+            raise EasyBuildError(err_msg, self.version, self.system_ssl['version'])
 
         # Check system OpenSSL binary
         if self.majmin_version == '1.1':
@@ -265,7 +267,7 @@ def __init__(self, *args, **kwargs):
             self.system_ssl['bin'] = which('openssl11')
 
         if not self.system_ssl['bin']:
-            self.system_ssl['bin'] = which(self.name.lower())
+            self.system_ssl['bin'] = which('openssl')
 
         if self.system_ssl['bin']:
             self.log.info("System OpenSSL binary found: %s", self.system_ssl['bin'])


Test patch: