Repository: easybuilders/easybuild-framework

Issue:
Dependencies of first build are still loaded for all next builds with an easystack file
I was installing a whole bunch of apps using an easystack file, and suddenly it crashed at the installation of `zlib-1.2.12.eb` because `$EBROOTZLIB` was already set. I hadn't done that myself (started with a clean environment), and after inspecting the log file, I noticed that for all installations after the very first one Easybuild was warning that a bunch of modules were loaded. This list consisted of the (build) dependencies of the first application that it installed.

I was able to reproduce it with a simple example, using this easystack file:
```
easyconfigs:
  - attr-2.5.1-GCCcore-11.2.0.eb
  - zlib-1.2.12.eb
```

Installing this with `eb -f --experimental --easystack test.yml` (note that the dependencies of `attr` are already available on my system) results in a successful installation of `attr`, but:
```
== FAILED: Installation ended unsuccessfully (build directory: 
/data/eb/build/zlib/1.2.12/system-system): build failed (first 300 chars): 
Module is already loaded (EBROOTZLIB is set), installation cannot continue. 
(took 0 secs)
```

The `attr` log has:
```
== 2023-01-25 12:45:58,939 easyblock.py:2248 WARNING Loaded modules detected: ['EasyBuild/4.7.0']
```
But the one for `zlib` has:
```
== 2023-01-25 12:46:04,854 easyblock.py:2248 WARNING Loaded modules detected: ['EasyBuild/4.7.0', 'GCCcore/11.2.0', 'zlib/1.2.11-GCCcore-11.2.0', 'binutils/2.37-GCCcore-11.2.0']
```

So it looks like it didn't unload the `attr` (build) dependencies. This is with EB 4.7.0, and based on some older logs of applications built with a similar easystack (but with the old structure) and EB 4.6.2, the latter version didn't have this issue.


Discussion:
That likely introduced with the changes in #4057, which includes a large refactor.
We're probably not properly resetting the build environment after completing an installation?

Thanks a lot for reporting this @bedroge, we'll look into catching this in a CI test, and fixing it - as it is now, easystack files aren't exactly production ready, that's clear...

Resolving patch:
diff --git a/easybuild/main.py b/easybuild/main.py
--- a/easybuild/main.py
+++ b/easybuild/main.py
@@ -62,6 +62,7 @@
 from easybuild.tools.config import find_last_log, get_repository, get_repositorypath, build_option
 from easybuild.tools.containers.common import containerize
 from easybuild.tools.docs import list_software
+from easybuild.tools.environment import restore_env
 from easybuild.tools.filetools import adjust_permissions, cleanup, copy_files, dump_index, load_index
 from easybuild.tools.filetools import locate_files, read_file, register_lock_cleanup_signal_handlers, write_file
 from easybuild.tools.github import check_github, close_pr, find_easybuild_easyconfig
@@ -232,6 +233,9 @@ def process_easystack(easystack_path, args, logfile, testing, init_session_state
     """
     easystack = parse_easystack(easystack_path)
 
+    # keep copy of original environment, so we can restore it for every easystack entry
+    init_env = copy.deepcopy(os.environ)
+
     global _log
 
     # TODO: insert fast loop that validates if all command line options are valid. If there are errors in options,
@@ -244,10 +248,14 @@ def process_easystack(easystack_path, args, logfile, testing, init_session_state
     do_cleanup = True
     for (path, ec_opts) in easystack.ec_opt_tuples:
         _log.debug("Starting build for %s" % path)
+
         # wipe easyconfig caches
         easyconfig._easyconfigs_cache.clear()
         easyconfig._easyconfig_files_cache.clear()
 
+        # restore environment
+        restore_env(init_env)
+
         # If EasyConfig specific arguments were supplied in EasyStack file
         # merge arguments with original command line args
         if ec_opts is not None:


Test patch:
diff --git a/test/framework/easystack.py b/test/framework/easystack.py
--- a/test/framework/easystack.py
+++ b/test/framework/easystack.py
@@ -29,6 +29,7 @@
 @author: Kenneth Hoste (Ghent University)
 """
 import os
+import re
 import sys
 from unittest import TextTestRunner
 
@@ -127,6 +128,25 @@ def test_easystack_invalid_key2(self):
         error_pattern += r"instead found keys: .*, invalid_key"
         self.assertErrorRegex(EasyBuildError, error_pattern, parse_easystack, test_easystack)
 
+    def test_easystack_restore_env_after_each_build(self):
+        """Test that the build environment is reset for each easystack item"""
+        test_es_txt = '\n'.join([
+            "easyconfigs:",
+            "  - toy-0.0-gompi-2018a.eb:",
+            "  - libtoy-0.0.eb:",
+        ])
+        test_es_path = os.path.join(self.test_prefix, 'test.yml')
+        write_file(test_es_path, test_es_txt)
+
+        args = [
+            '--experimental',
+            '--easystack',
+            test_es_path
+        ]
+        stdout = self.eb_main(args, do_build=True, raise_error=True)
+        regex = re.compile(r"WARNING Loaded modules detected: \[.*gompi/2018.*\]\n")
+        self.assertFalse(regex.search(stdout), "Pattern '%s' should not be found in: %s" % (regex.pattern, stdout))
+
     def test_missing_easyconfigs_key(self):
         """Test that EasyStack file that doesn't contain an EasyConfigs key will fail with sane error message"""
         topdir = os.path.dirname(os.path.abspath(__file__))