Repository: easybuilders/easybuild-framework

Issue:
RFF: dump a copy of --dump-env-script output in the %(installdir)/easybuild directory as an .env file for each build
I think the Subject: covers it.

But if it doesn't...... sometimes its nice to have a quick reference (especially for users) of what the environment was like when a certain package was built.  Especially if they want to use the EB version to build a custom version.  %(installdir)/easybuild already has a -devel file but its basically useless for users.  And teaching users how to use EB and run --dump-env-script is a bit much.   So lets just make a "easy build" copy for them.

More beer rewards available though given the picture boegel sent me of the beer stocks purchased for the EasyBuild user meeting, I imagine most of those at the hackathon on Friday will be drowning already.  Goodness gracious.   If y'all (those listed as being there) drink all of that, I'm going to have to work on your next trip to SC in Austin.



Discussion:
and yes I forgot the ending 's' on both uses of %(installdir) like I usually do... pep8 or eb would not be happy 
I've been meaning to get this covered in the documentation (https://github.com/hpcugent/easybuild/pull/263), but I think there's an approach already that' may satisfy your needs, see the use of the `BuildEnv` easyblock at https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/b/buildenv/buildenv-default-intel-2016b.eb, which would give you a module like `buildenv/default-intel-2016b`.

The use case here is slightly different though, this is still a valid request that should be easy to implement and would be useful in many cases...
It would be nice to hear Robert's opinion on having a native Lmod function to read such buildenv files and spit the information out accordingly. In this way the users will only have to interact with Lmod. 
The request is actually good. In my case I wanted to work on TensorFlow which is already installed but with...different settings than I want. Knowing which env-vars were defined would be great. Maybe even to the point of actual invocations. Especially for TensorFlow there are a lot of `TF_NEED_<FOO>` env vars that define how TF is going to be build. In case of an updated EB/EC this information may be lost.
...bump...
I have a user here who wants to add their own USER_MAEAM (not to be confused with USER_MEAMC) code to LAMMPS.  I'd love to just give them LAMMPS buildenv (and not the toolchain buildenv). Once again, in order to do that I'm going to have to use --force, destroy the current install and wait for it to complete.  It would be SO much nicer if that was already just in the installdir/easybuild directory (with the other useful files there).



Resolving patch:
diff --git a/easybuild/framework/easyblock.py b/easybuild/framework/easyblock.py
--- a/easybuild/framework/easyblock.py
+++ b/easybuild/framework/easyblock.py
@@ -59,7 +59,7 @@
 from easybuild.framework.easyconfig.format.format import SANITY_CHECK_PATHS_DIRS, SANITY_CHECK_PATHS_FILES
 from easybuild.framework.easyconfig.parser import fetch_parameters_from_easyconfig
 from easybuild.framework.easyconfig.style import MAX_LINE_LENGTH
-from easybuild.framework.easyconfig.tools import get_paths_for
+from easybuild.framework.easyconfig.tools import dump_env_easyblock, get_paths_for
 from easybuild.framework.easyconfig.templates import TEMPLATE_NAMES_EASYBLOCK_RUN_STEP, template_constant_dict
 from easybuild.framework.extension import Extension, resolve_exts_filter_template
 from easybuild.tools import config, run
@@ -3740,6 +3740,14 @@ def build_and_install_one(ecdict, init_env, progress_bar=None, task_id=None):
         result = app.run_all_steps(run_test_cases=run_test_cases)
 
         if not dry_run:
+            # Copy over the build environment used during the configuraton
+            reprod_spec = os.path.join(reprod_dir, app.cfg.filename())
+            try:
+                dump_env_easyblock(app, ec_path=reprod_spec, silent=True)
+                _log.debug("Created build environment dump for easyconfig %s", reprod_spec)
+            except EasyBuildError as err:
+                _log.warning("Failed to create build environment dump for easyconfig %s: %s", reprod_spec, err)
+
             # also add any extension easyblocks used during the build for reproducibility
             if app.ext_instances:
                 copy_easyblocks_for_reprod(app.ext_instances, reprod_dir)
@@ -3918,9 +3926,9 @@ def copy_easyblocks_for_reprod(easyblock_instances, reprod_dir):
     for easyblock_instance in easyblock_instances:
         for easyblock_class in inspect.getmro(type(easyblock_instance)):
             easyblock_path = inspect.getsourcefile(easyblock_class)
-            # if we reach EasyBlock or ExtensionEasyBlock class, we are done
-            # (ExtensionEasyblock is hardcoded to avoid a cyclical import)
-            if easyblock_class.__name__ in [EasyBlock.__name__, 'ExtensionEasyBlock']:
+            # if we reach EasyBlock, Extension or ExtensionEasyBlock class, we are done
+            # (Extension and ExtensionEasyblock are hardcoded to avoid a cyclical import)
+            if easyblock_class.__name__ in [EasyBlock.__name__, 'Extension', 'ExtensionEasyBlock']:
                 break
             else:
                 easyblock_paths.add(easyblock_path)
diff --git a/easybuild/framework/easyconfig/tools.py b/easybuild/framework/easyconfig/tools.py
--- a/easybuild/framework/easyconfig/tools.py
+++ b/easybuild/framework/easyconfig/tools.py
@@ -553,6 +553,45 @@ def review_pr(paths=None, pr=None, colored=True, branch='develop', testing=False
     return '\n'.join(lines)
 
 
+def dump_env_easyblock(app, orig_env=None, ec_path=None, script_path=None, silent=False):
+    if orig_env is None:
+        orig_env = copy.deepcopy(os.environ)
+    if ec_path is None:
+        raise EasyBuildError("The path to the easyconfig relevant to this environment dump is required")
+    if script_path is None:
+        # Assume we are placing it alongside the easyconfig path
+        script_path = '%s.env' % os.path.splitext(ec_path)[0]
+    # Compose script
+    ecfile = os.path.basename(ec_path)
+    script_lines = [
+        "#!/bin/bash",
+        "# script to set up build environment as defined by EasyBuild v%s for %s" % (EASYBUILD_VERSION, ecfile),
+        "# usage: source %s" % os.path.basename(script_path),
+    ]
+
+    script_lines.extend(['', "# toolchain & dependency modules"])
+    if app.toolchain.modules:
+        script_lines.extend(["module load %s" % mod for mod in app.toolchain.modules])
+    else:
+        script_lines.append("# (no modules loaded)")
+
+    script_lines.extend(['', "# build environment"])
+    if app.toolchain.vars:
+        env_vars = sorted(app.toolchain.vars.items())
+        script_lines.extend(["export %s='%s'" % (var, val.replace("'", "\\'")) for (var, val) in env_vars])
+    else:
+        script_lines.append("# (no build environment defined)")
+
+    write_file(script_path, '\n'.join(script_lines))
+    msg = "Script to set up build environment for %s dumped to %s" % (ecfile, script_path)
+    if silent:
+        _log.info(msg)
+    else:
+        print_msg(msg, prefix=False)
+
+    restore_env(orig_env)
+
+
 def dump_env_script(easyconfigs):
     """
     Dump source scripts that set up build environment for specified easyconfigs.
@@ -587,31 +626,8 @@ def dump_env_script(easyconfigs):
         app.check_readiness_step()
         app.prepare_step(start_dir=False)
 
-        # compose script
-        ecfile = os.path.basename(ec.path)
-        script_lines = [
-            "#!/bin/bash",
-            "# script to set up build environment as defined by EasyBuild v%s for %s" % (EASYBUILD_VERSION, ecfile),
-            "# usage: source %s" % os.path.basename(script_path),
-        ]
-
-        script_lines.extend(['', "# toolchain & dependency modules"])
-        if app.toolchain.modules:
-            script_lines.extend(["module load %s" % mod for mod in app.toolchain.modules])
-        else:
-            script_lines.append("# (no modules loaded)")
-
-        script_lines.extend(['', "# build environment"])
-        if app.toolchain.vars:
-            env_vars = sorted(app.toolchain.vars.items())
-            script_lines.extend(["export %s='%s'" % (var, val.replace("'", "\\'")) for (var, val) in env_vars])
-        else:
-            script_lines.append("# (no build environment defined)")
-
-        write_file(script_path, '\n'.join(script_lines))
-        print_msg("Script to set up build environment for %s dumped to %s" % (ecfile, script_path), prefix=False)
-
-        restore_env(orig_env)
+        # create the environment dump
+        dump_env_easyblock(app, orig_env=orig_env, ec_path=ec.path, script_path=script_path)
 
 
 def categorize_files_by_type(paths):


Test patch:
diff --git a/test/framework/options.py b/test/framework/options.py
--- a/test/framework/options.py
+++ b/test/framework/options.py
@@ -4815,8 +4815,8 @@ def test_prefix_option(self):
         expected = ['buildpath', 'containerpath', 'installpath', 'packagepath', 'prefix', 'repositorypath']
         self.assertEqual(sorted(regex.findall(txt)), expected)
 
-    def test_dump_env_config(self):
-        """Test for --dump-env-config."""
+    def test_dump_env_script(self):
+        """Test for --dump-env-script."""
 
         fftw = 'FFTW-3.3.7-gompic-2018a'
         gcc = 'GCC-4.9.2'
diff --git a/test/framework/toy_build.py b/test/framework/toy_build.py
--- a/test/framework/toy_build.py
+++ b/test/framework/toy_build.py
@@ -2016,6 +2016,23 @@ def test_reproducibility(self):
 
         self.assertTrue(os.path.exists(reprod_ec))
 
+        # Also check that the dumpenv script is placed alongside it
+        dumpenv_script = '%s.env' % os.path.splitext(reprod_ec)[0]
+        reprod_dumpenv = os.path.join(reprod_dir, dumpenv_script)
+        self.assertTrue(os.path.exists(reprod_dumpenv))
+
+        # Check contents of the dumpenv script
+        patterns = [
+            """#!/bin/bash""",
+            """# usage: source toy-0.0.env""",
+            # defining build env
+            """# (no modules loaded)""",
+            """# (no build environment defined)""",
+        ]
+        env_file = open(reprod_dumpenv, "r").read()
+        for pattern in patterns:
+            self.assertTrue(pattern in env_file)
+
         # Check that the toytoy easyblock is recorded in the reprod easyconfig
         ec = EasyConfig(reprod_ec)
         self.assertEqual(ec.parser.get_config_dict()['easyblock'], 'EB_toytoy')
@@ -2114,6 +2131,24 @@ def test_toy_toy(self):
         load1_regex = re.compile('load.*toy/0.0-one', re.M)
         self.assertTrue(load1_regex.search(mod2_txt), "Pattern '%s' found in: %s" % (load1_regex.pattern, mod2_txt))
 
+        # Check the contents of the dumped env in the reprod dir to ensure it contains the dependency load
+        reprod_dir = os.path.join(self.test_installpath, 'software', 'toy', '0.0-two', 'easybuild', 'reprod')
+        dumpenv_script = os.path.join(reprod_dir, 'toy-0.0-two.env')
+        reprod_dumpenv = os.path.join(reprod_dir, dumpenv_script)
+        self.assertTrue(os.path.exists(reprod_dumpenv))
+
+        # Check contents of the dumpenv script
+        patterns = [
+            """#!/bin/bash""",
+            """# usage: source toy-0.0-two.env""",
+            # defining build env
+            """module load toy/0.0-one""",
+            """# (no build environment defined)""",
+        ]
+        env_file = open(reprod_dumpenv, "r").read()
+        for pattern in patterns:
+            self.assertTrue(pattern in env_file)
+
     def test_toy_sanity_check_commands(self):
         """Test toy build with extra sanity check commands."""