Repository: easybuilders/easybuild-framework

Issue:
Templates not being resolved in exts_list
I am trying to use this : 
```
exts_defaultclass = 'ConfigureMakePythonPackage'
exts_list = [
        ('sip', '4.19.17', {
                'source_urls': 'https://www.riverbankcomputing.com/static/Downloads/sip/4.19.17/',
                'source_tmpl': '%(name)s-%(version)s.tar.gz',
                'checksums': ['12bcd8f4d5feefc105bc075d12c5090ee783f7380728563c91b8b95d0ec45df3'],
                'prefix_opt': '--sysroot=',
                'configopts': 'configure.py --sip-module PyQt5.sip --sipdir=%(installdir)s/share/python%(pyshortver)s/sip INCDIR=$EBROOTPYTHON/include/python%(pyshortver)sm'
        }),
        ('PyQt5', '5.12.3', {
                'source_urls': 'https://www.riverbankcomputing.com/static/Downloads/PyQt5/5.12.3/',
                'source_tmpl': '%(name)s_gpl-%(version)s.tar.gz',
                'checksums': ['0db0fa37debab147450f9e052286f7a530404e2aaddc438e97a7dcdf56292110'],
                'prefix_opt': '--sysroot=',
                'configopts': 'configure.py --confirm --qmake=%(installdir)s/bin/qmake --verbose --sip=%(installdir)s/bin/sip --sip-incdir=%(installdir)s/share/python%(pyshortver)s/sip INCDIR=$EBROOTPYTHON/include/python%(pyshortver)m'
        })
]
``` 

However, the templates do not get resolved, and the build fails with 
```
ERROR: Build of /home/mboisson/git/easybuild-easyconfigs/easybuild/easyconfigs/q/Qt5/Qt5-5.12.8-GCCcore-9.3.0.eb failed (err: 'build failed (first 300 chars): cmd " /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python configure.py --sip-module PyQt5.sip --sipdir=%(installdir)s/share/python%(pyshortver)s/sip INCDIR=$EBROOTPYTHON/include/python%(pyshortver)sm:$EBROOTPYTHON/include/python%(pyshortver)s" exited with exit code 1 and output:\n/bin/bash: -c: l')
``` 

I don't know if this is related to https://github.com/easybuilders/easybuild-framework/pull/3237 or if there's something I don't understand... 

@akesandgren, @boegel  ?


Discussion:
@mboisson Can you share (a snippet of) a debug log where it produces into about (failing of) resolving of templates and which template values are supported at that time?
```
== 2020-04-30 20:13:01,277 toolchain.py:733 INFO Commands symlinked to /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/ccache via /tmp/eb-qi3pu342/tmpswl0xgsm: gcc, g++, gfortran, gfortran, gfortran
== 2020-04-30 20:13:01,277 filetools.py:434 INFO Command python found at /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python
== 2020-04-30 20:13:01,277 pythonpackage.py:380 INFO Python command being used: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python
== 2020-04-30 20:13:01,277 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(plat_specific=False, prefix="/tmp/"))'
== 2020-04-30 20:13:01,346 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import distutils.sysconfig; print(distutils.sysconfig.get_python_lib(plat_specific=True, prefix="/tmp/"))'
== 2020-04-30 20:13:01,435 filetools.py:1460 INFO Creating directory /tmp/mboisson/Qt5/5.12.8/GCCcore-9.3.0/sip (parents: True, set_gid: False, sticky: False)
== 2020-04-30 20:13:01,437 run.py:222 INFO running cmd: tar xzf /home/mboisson/.local/easybuild/sources/q/Qt5/sip-4.19.17.tar.gz
== 2020-04-30 20:13:01,533 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import sys; print("%s.%s.%s" % sys.version_info[:3])'
== 2020-04-30 20:13:01,576 pythonpackage.py:512 INFO Not auto-enabling checking of $LDSHARED, Python version 3.6.9 is not recent enough
== 2020-04-30 20:13:01,577 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -V
== 2020-04-30 20:13:01,590 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import sys; print(sys.executable)'
== 2020-04-30 20:13:01,631 environment.py:97 INFO Environment variable PYTHONNOUSERSITE set to 1 (previously undefined)
== 2020-04-30 20:13:01,632 run.py:222 INFO running cmd: /cvmfs/soft.computecanada.ca/gentoo/2019/usr/bin/python -c 'import sys; print(sys.path)'
== 2020-04-30 20:13:01,682 easyconfig.py:1891 WARNING Unable to resolve template value configure.py --sip-module PyQt5.sip --sipdir=%(installdir)s/share/python%(pyshortver)s/sip INCDIR=$EBROOTPYTHON/include/python%(pyshortver)sm:         $EBROOTPYTHON/include/python%(pyshortver)s with dict {'arch': 'x86_64', 'nameletter': 's', 'toolchain_name': 'GCCcore', 'toolchain_version': '9.3.0', 'version_major': '4', 'version_minor': '19', 'version_major_minor': '4.19',             'bitbucket_account': 'qt5', 'github_account': 'qt5', 'name': 'sip', 'version': '4.19.17', 'versionsuffix': '', 'versionprefix': '', 'namelower': 'sip', 'nameletterlower': 's', 'installdir': '/home/mboisson/.local/easybuild/software/2019/ Core/qt/5.12.8', 'builddir': '/tmp/mboisson/Qt5/5.12.8/GCCcore-9.3.0'}
``` 
@boegel this bit ?
Without digging through the code, my thoughts are that, since, obviously extensions *must* have their own scope (else, `'source_tmpl': '%(name)s_gpl-%(version)s.tar.gz'` would break, as these must refer to the extension `version` and `name`), they also don't get master-config dependencies (and thus javaver, pyver etc.) exposed into the extensions scope... somehow
It appears indeed that they don't have the master-config dependencies. 
Maybe it needs something like this ? https://github.com/easybuilders/easybuild-framework/pull/3237
Or maybe it is because I'm using `multi_deps` and not `dependencies` over `Python` ? @bartoldeman 
I think @Micket is right, this is basically because dependency-related templates like `%(pyver)s` are not defined. But the `'dependencies'` easyconfig parameter does get copied down to the extension though (the whole parent easyconfig basically is, and we overwrite things like `name`/`version` afterwards to be correct for the extension, see https://github.com/easybuilders/easybuild-framework/blob/33d0073451116c13da8f6048629c62a6786da957/easybuild/framework/extension.py#L98).

So we need to figure out why `%(pyver)s` & co are not included in the dict with template values.

@mboisson Can you share the full easyconfig file you're using here, so we can reproduce the issue, or a full log file to stare at?
Mmm, the full log is 87MB big... 

The recipe is https://gist.github.com/mboisson/9032258b0c6649686daa424620f538a7

I think it must have something to do with `multi_deps`
@boegel I ran into that already and investigated while developing #3285. See my comment: https://github.com/easybuilders/easybuild-framework/pull/3285#issuecomment-615272582

TLDR: The extension handling (especially templating) needs a rework. Currently the extensions are used with templates replaced in one place, instantiated later with another round of templating which need to be kept in sync but (I guess) nobody knows what template values are available at which point. My proposal: `cfg['ext_list']['options']` needs disabled templating, extensions need to be instantiated before anything is accessed (no direct access to the above `cfg['ext_list']['options']`) and most problems are solved.

On the `pyver` issue: This happens because extension replacement and such is done before multi-deps kick in. So at that point the Python dependency is still unknown and especially the python version is unknown. Hence the template resolving fails leading to hard to track errors later on. Hence my PR #3285 but it is held up on the above issue.
I only understand all of the above partly. However, I am available to do some testing on a set of changes if needs be. 
Note that this is a regression from 4.2.0. It used to work in version 4.1.x (not this specific recipe, but other recipes are broken).
After discussing this with @mboisson and @bartoldeman: two problems were being mixed up.

The issue where `%(pyshortver)s` is not being resolved when using for an extension is a genuine bug, it seems like that never actually worked.

I was testing this with a modified `plotly.py-4.4.1-intel-2019b.eb`:

```diff
@@ -17,9 +17,11 @@ exts_default_options = {'source_urls': [PYPI_SOURCE]}
 exts_list = [
     ('retrying', '1.3.3', {
         'checksums': ['08c039560a6da2fe4f2c426d0766e284d3b736e355f8dd24b37367b0bb41973b'],
+        'preinstallopts': 'echo "pyshortver: %(pyshortver)s" && ',
     }),
     ('plotly', version, {
         'checksums': ['acc94f17452471ca3446c2ce491c4d1affb99b9ddd9eac4e05614ac4318f8780'],
+        'preinstallopts': 'echo "pyshortver: %(pyshortver)s" && ',
     }),
 ]

```

The regression that @mboisson referred to turned out to be another (self-inflicted) issue.
I spent some more time looking at this, and the problem boils down to this: for every extension a copy is made of the `EasyConfig` instance when an `Extension` instance is created.

This is done using the "copy constructor" method `EasyConfig.copy`, which effectively creates an entirely new `EasyConfig` instance, and copies over a couple of things (like `self._config`, which is the actual dictionary with all easyconfig parameters).

What does *not* get copied is the list of template values, and this explains the problem here: when `multi_deps` is involved, dependencies like `Python` are handled by iterating over a list of lists of build dependencies. And only when actually iterating are build dependencies taken into account for generating template values (see also the recent fix in #3346), which is not the case when creating the new `EasyConfig` instance.

So, the fix for this could be to basically also copy the template values over in `EasyConfig.copy`.

I'll get a PR opened for this, would be nice to somehow cover this in the tests too...

Resolving patch:
diff --git a/easybuild/framework/easyblock.py b/easybuild/framework/easyblock.py
--- a/easybuild/framework/easyblock.py
+++ b/easybuild/framework/easyblock.py
@@ -2128,7 +2128,7 @@ def install_step(self):
         """Install built software (abstract method)."""
         raise NotImplementedError
 
-    def extensions_step(self, fetch=False):
+    def extensions_step(self, fetch=False, install=True):
         """
         After make install, run this.
         - only if variable len(exts_list) > 0
@@ -2142,7 +2142,7 @@ def extensions_step(self, fetch=False):
 
         # load fake module
         fake_mod_data = None
-        if not self.dry_run:
+        if install and not self.dry_run:
 
             # load modules for build dependencies as extra modules
             build_dep_mods = [dep['short_mod_name'] for dep in self.cfg.dependencies(build_only=True)]
@@ -2255,11 +2255,12 @@ def extensions_step(self, fetch=False):
                                       rpath_filter_dirs=self.rpath_filter_dirs)
 
             # real work
-            ext.prerun()
-            txt = ext.run()
-            if txt:
-                self.module_extra_extensions += txt
-            ext.postrun()
+            if install:
+                ext.prerun()
+                txt = ext.run()
+                if txt:
+                    self.module_extra_extensions += txt
+                ext.postrun()
 
         # cleanup (unload fake module, remove fake module dir)
         if fake_mod_data:
diff --git a/easybuild/framework/easyconfig/easyconfig.py b/easybuild/framework/easyconfig/easyconfig.py
--- a/easybuild/framework/easyconfig/easyconfig.py
+++ b/easybuild/framework/easyconfig/easyconfig.py
@@ -570,6 +570,9 @@ def copy(self, validate=None):
         if self.path:
             ec.path = self.path
 
+        # also copy template values, since re-generating them may not give the same set of template values straight away
+        ec.template_values = copy.deepcopy(self.template_values)
+
         return ec
 
     def update(self, key, value, allow_duplicate=True):


Test patch:
diff --git a/test/framework/easyconfig.py b/test/framework/easyconfig.py
--- a/test/framework/easyconfig.py
+++ b/test/framework/easyconfig.py
@@ -2663,11 +2663,16 @@ def test_copy(self):
         test_easyconfigs = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'easyconfigs', 'test_ecs')
         ec1 = EasyConfig(os.path.join(test_easyconfigs, 't', 'toy', 'toy-0.0.eb'))
 
+        # inject fake template value, just to check whether they are copied over too
+        ec1.template_values['pyshortver'] = '3.7'
+
         ec2 = ec1.copy()
 
         self.assertEqual(ec1, ec2)
         self.assertEqual(ec1.rawtxt, ec2.rawtxt)
         self.assertEqual(ec1.path, ec2.path)
+        self.assertEqual(ec1.template_values, ec2.template_values)
+        self.assertFalse(ec1.template_values is ec2.template_values)
 
     def test_eq_hash(self):
         """Test comparing two EasyConfig instances."""
@@ -3529,6 +3534,14 @@ def test_iter_builddeps_templates(self):
 
         test_ec = os.path.join(self.test_prefix, 'test.eb')
         test_ec_txt = toy_ec_txt + "\nmulti_deps = {'Python': ['2.7.15', '3.6.6']}"
+
+        # inject extension that uses %(pyshortver)s, to check whether the template value is properly resolved
+        test_ec_txt += '\n'.join([
+            '',
+            "exts_defaultclass = 'Toy_Extension'",
+            "exts_list = [('bar', '0.0', {'preinstallopts': 'echo \\'py%(pyshortver)s\\' && '})]",
+        ])
+
         write_file(test_ec, test_ec_txt)
 
         ec = EasyConfig(test_ec)
@@ -3560,6 +3573,14 @@ def test_iter_builddeps_templates(self):
         self.assertTrue('pyshortver' in ec.template_values)
         self.assertEqual(ec.template_values['pyshortver'], '3.6')
 
+        # check that extensions inherit these template values too
+        # cfr. https://github.com/easybuilders/easybuild-framework/issues/3317
+        eb = EasyBlock(ec)
+        eb.silent = True
+        eb.extensions_step(fetch=True, install=False)
+        ext = eb.ext_instances[0]
+        self.assertEqual(ext.cfg['preinstallopts'], "echo 'py3.6' && ")
+
     def test_fix_deprecated_easyconfigs(self):
         """Test fix_deprecated_easyconfigs function."""
         test_ecs_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'easyconfigs', 'test_ecs')