Repository: easybuilders/easybuild-framework

Issue:
reverse substitution of %(version)s in generated easyconfig
I have hit a problem building with EasyBuild 4.0.0

The issue looks to be the reverse substitution of %(version)s in the source_urls, where the version picked up is not that of the main package but of an extension from exts_list.

`$ eb --buildpath=/ichec/home/users/brownwdl --job-cores=12 --optarch=march=skylake-avx512  --try-toolchain "foss,2019b" --dry-run R-bundle-Bioconductor-3.9-foss-2019a-R-3.6.0.eb -r`

I am trying to use an updated toolchain (because I have it already built) so a temporary easyconfig has been created on-the-fly.   This re-orders the items in the easyconfig and seems to have reverse-substituted the specified version ‘3.9’ with  %(version) .

The original easyconfig for R-bundle-Bioconductor-3.9-foss-2019a-R-3.6.0.eb has this:
```
easyblock = 'Bundle'

name = 'R-bundle-Bioconductor'
version = '3.9'
versionsuffix = '-R-%(rver)s'
.
.
exts_default_options = {
    'source_urls': [
        'https://bioconductor.org/packages/3.9/bioc/src/contrib/',
        'https://bioconductor.org/packages/3.9/bioc/src/contrib/Archive/%(name)s',
        'https://bioconductor.org/packages/3.9/data/annotation/src/contrib/',
        'https://bioconductor.org/packages/3.9/data/experiment/src/contrib/',
        'https://cran.r-project.org/src/contrib/Archive/%(name)s',  # package archive
        'https://cran.r-project.org/src/contrib/',  # current version of packages
        'https://cran.freestatistics.org/src/contrib',  # mirror alternative for current packages
    ],
    'source_tmpl': '%(name)s_%(version)s.tar.gz'
}
```
The version is explicitly written as ‘3.9’ which happens to exactly match the 'version' value.  The ‘tweaked’ easyconfig that is created for --try-toolchain "foss,2019b" has this:
```
easyblock = 'Bundle'

name = 'R-bundle-Bioconductor'
version = '3.9'
versionsuffix = '-R-%(rver)s'
.
.
exts_default_options = {
    'source_tmpl': '%(name)s_%(version)s.tar.gz',
    'source_urls': ['https://bioconductor.org/packages/%(version)s/bioc/src/contrib/', 'https://bioconductor.org/packages/%(version)s/bioc/src/contrib/Archive/%(name)s', 'https://bioconductor.org/packages/%(version)s/data/annotation/src/contrib/', 'https://bioconductor.org/packages/%(version)s/data/experiment/src/contrib/', 'https://cran.r-project.org/src/contrib/Archive/%(name)s', 'https://cran.r-project.org/src/contrib/', 'https://cran.freestatistics.org/src/contrib'],
}
```
When in the build process it gets to the BiocGenerics extension:
```
exts_list = [
    ('readr', '1.3.1', {
        'checksums': ['33f94de39bb7f2a342fbb2bd4e5afcfec08798eac39672ee18042ac0b349e4f3'],
    }),
    ('BiocGenerics', '0.30.0', {
        'checksums': ['2e6d2fa6ed389cfae6fd75713379ba7dcd164a8161f37f1bbed596b70c3207d9'],
    }),
```
It fails.  It works for the first extension ‘readr’ but I think that it had successfully downloaded that source on an earlier date (quite possibly with an earlier version of EasyBuild).
```
== 2019-11-20 14:31:14,022 filetools.py:404 INFO Command eb found at /ichec/work/rcsi01/packages/easybuild/software/EasyBuild/4.0.0/bin/eb
== 2019-11-20 14:31:14,044 easyblock.py:668 INFO Found file readr_1.3.1.tar.gz at /ichec/work/rcsi01/packages/easybuild/sources/r/R-bundle-Bioconductor/extensions/readr_1.3.1.tar.gz
== 2019-11-20 14:31:14,101 easyblock.py:528 INFO md5 checksum for /ichec/work/rcsi01/packages/easybuild/sources/r/R-bundle-Bioconductor/extensions/readr_1.3.1.tar.gz: b41b0e9251c19b1c54c032e1f55be9d0
== 2019-11-20 14:31:14,104 easyblock.py:528 INFO sha256 checksum for /ichec/work/rcsi01/packages/easybuild/sources/r/R-bundle-Bioconductor/extensions/readr_1.3.1.tar.gz: 33f94de39bb7f2a342fbb2bd4e5afcfec08798eac39672ee18042ac0b349e4f3
== 2019-11-20 14:31:14,108 easyblock.py:534 INFO Checksum for extension source readr_1.3.1.tar.gz verified
== 2019-11-20 14:31:14,108 easyblock.py:456 INFO Fetched extension patches: []
== 2019-11-20 14:31:14,109 filetools.py:404 INFO Command eb found at /ichec/work/rcsi01/packages/easybuild/software/EasyBuild/4.0.0/bin/eb
== 2019-11-20 14:31:14,555 filetools.py:559 WARNING URL https://bioconductor.org/packages/0.30.0/bioc/src/contrib/BiocGenerics_0.30.0.tar.gz was not found (HTTP response code 404), not trying again
== 2019-11-20 14:31:14,555 filetools.py:585 WARNING Download of https://bioconductor.org/packages/0.30.0/bioc/src/contrib/BiocGenerics_0.30.0.tar.gz to /ichec/work/rcsi01/packages/easybuild/sources/r/R-bundle-Bioconductor/extensions/BiocGenerics_0.30.0.tar.gz failed, done trying
```
Because of the change it has substituted the %(version) from the version of the *extension*, not the main package, and that cannot work.  Really the hard-coded '3.9' should have been left unchanged.




Discussion:
We're indeed too aggressively replacing `3.9` with `%(version)s` here, since when installing an extension we'll replace the use of `%(version)s` with the extension version rather than the parent version...

This is probably easy to fix, I'll try and look into it ASAP

Resolving patch:
diff --git a/easybuild/framework/easyconfig/format/format.py b/easybuild/framework/easyconfig/format/format.py
--- a/easybuild/framework/easyconfig/format/format.py
+++ b/easybuild/framework/easyconfig/format/format.py
@@ -52,8 +52,8 @@
 DEPENDENCY_PARAMETERS = ['builddependencies', 'dependencies', 'hiddendependencies']
 
 # values for these keys will not be templated in dump()
-EXCLUDED_KEYS_REPLACE_TEMPLATES = ['description', 'easyblock', 'exts_list', 'homepage', 'name', 'toolchain',
-                                   'version', 'multi_deps'] + DEPENDENCY_PARAMETERS
+EXCLUDED_KEYS_REPLACE_TEMPLATES = ['description', 'easyblock', 'exts_default_options', 'exts_list',
+                                   'homepage', 'multi_deps', 'name', 'toolchain', 'version'] + DEPENDENCY_PARAMETERS
 
 # ordered groups of keys to obtain a nice looking easyconfig file
 GROUPED_PARAMS = [
@@ -69,7 +69,10 @@
     ['preinstallopts', 'installopts'],
     ['parallel', 'maxparallel'],
 ]
-LAST_PARAMS = ['sanity_check_paths', 'moduleclass']
+LAST_PARAMS = ['exts_default_options', 'exts_list',
+               'sanity_check_paths', 'sanity_check_commands',
+               'modextrapaths', 'modextravars',
+               'moduleclass']
 
 SANITY_CHECK_PATHS_DIRS = 'dirs'
 SANITY_CHECK_PATHS_FILES = 'files'


Test patch:
diff --git a/test/framework/easyconfig.py b/test/framework/easyconfig.py
--- a/test/framework/easyconfig.py
+++ b/test/framework/easyconfig.py
@@ -1748,7 +1748,7 @@ def test_dump_template(self):
             "easyblock = 'EB_foo'",
             '',
             "name = 'Foo'",
-            "version = '0.0.1'",
+            "version = '1.0.0'",
             "versionsuffix = '-test'",
             '',
             "homepage = 'http://foo.com/'",
@@ -1760,7 +1760,7 @@ def test_dump_template(self):
             '}',
             '',
             "sources = [",
-            "    'foo-0.0.1.tar.gz',",
+            "    'foo-1.0.0.tar.gz',",
             ']',
             '',
             "dependencies = [",
@@ -1768,14 +1768,23 @@ def test_dump_template(self):
             ']',
             '',
             "preconfigopts = '--opt1=%s' % name",
-            "configopts = '--opt2=0.0.1'",
+            "configopts = '--opt2=1.0.0'",
             '',
+            'exts_default_options = {',
+            "    'source_urls': ['https://example.com/files/1.0.0'],"
+            '}',
+            'exts_list = [',
+            '  ("ext1", "1.0.0"),'
+            '  ("ext2", "2.1.3"),'
+            ']',
             "sanity_check_paths = {",
             "    'files': ['files/foo/foobar', 'files/x-test'],",
             "    'dirs':[],",
             '}',
             '',
-            "foo_extra1 = 'foobar'"
+            "foo_extra1 = 'foobar'",
+            '',
+            'moduleclass = "tools"',
         ])
 
         handle, testec = tempfile.mkstemp(prefix=self.test_prefix, suffix='.eb')
@@ -1790,7 +1799,7 @@ def test_dump_template(self):
         patterns = [
             r"easyblock = 'EB_foo'",
             r"name = 'Foo'",
-            r"version = '0.0.1'",
+            r"version = '1.0.0'",
             r"versionsuffix = '-test'",
             r"homepage = 'http://foo.com/'",
             r'description = "foo description"',  # no templating for description
@@ -1799,6 +1808,10 @@ def test_dump_template(self):
             # r"dependencies = \[\n    \('bar', '1.2.3', '%\(versionsuffix\)s'\),\n\]",
             r"preconfigopts = '--opt1=%\(name\)s'",
             r"configopts = '--opt2=%\(version\)s'",
+            # no %(version)s template used in exts_default_options or exts_list
+            # see https://github.com/easybuilders/easybuild-framework/issues/3091
+            r"exts_default_options = {'source_urls': \['https://example.com/files/1\.0\.0'\]}",
+            r"\('ext1', '1\.0\.0'\),",
             r"sanity_check_paths = {\n    'files': \['files/%\(namelower\)s/foobar', 'files/x-test'\]",
         ]
 
@@ -1806,6 +1819,8 @@ def test_dump_template(self):
             regex = re.compile(pattern, re.M)
             self.assertTrue(regex.search(ectxt), "Pattern '%s' found in: %s" % (regex.pattern, ectxt))
 
+        ectxt.endswith('moduleclass = "tools"')
+
         # reparsing the dumped easyconfig file should work
         EasyConfig(testec)