Repository: easybuilders/easybuild-framework

Issue:
Python RuntimeError during sanity check with python3
When installing easybuild with `pip install easybuild` using python3 (I tried with python3.5 and python3.6),  the built process fails during the sanity checking step with
```
RuntimeError: dictionary changed size during iteration
```
that happens in [the block line 173 to 177 of easybuild/tools/environment.py](https://github.com/easybuilders/easybuild-framework/blob/2e5d9f00f9f83e1f27b38d1aa17e1b614f2c4aad/easybuild/tools/environment.py#L173-L177). I put the [full traceback](https://github.com/easybuilders/easybuild-framework/files/3987846/traceback.txt) in attachment because maybe it's a bit too long.

I tried only with a couple of easyconfigs, for instance [Julia-1.2.0-linux-x86_64.eb](https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/j/Julia/Julia-1.2.0-linux-x86_64.eb). It can be reproduced with `eb --install-latest-eb-release` as well.


Discussion:
This is a valid bug. Reduced code sample:

```
def modify_env(old, new, verbose=True):
    oldKeys = old.keys()
    newKeys = new.keys()
    # Snip
    for key in oldKeys:
        if key not in newKeys:
            _log.debug("Key in old environment found that is not in new one: %s (%s)" % (key, old[key]))
            os.unsetenv(key)
            del os.environ[key]

modify_env(os.environ, env, verbose=False)
```

As you can see: It iterates over `os.environ.keys()` and calls `del os.environ[key]`

Solution: Copy the keys: `oldKeys = list(old.keys())` and add a test
This clearly is a bug, and we'll fix it for the next EasyBuild release, but I'm puzzled how we have missed this up until now...

@rsarm Can you try using `eb --install-latest-eb-release --debug --logtostdout`, which should tell you which environment variable was deleted, which then triggers the bug?
 in Python 3 the keys method was changed to a view, while it was a list/copy in python 2. And System Python is usually still python 2. I guess that's why it went unnoticed so far 
No, there's more going on...

This works fine for me, but it's not supposed to because of the `del os.environ` in `modify_env`:

```
python3 -c 'from easybuild.tools.environment import modify_env; import os; modify_env(os.environ, {})'
```

I don't understand why this doesn't trigger `RuntimeError: dictionary changed size during iteration`, while a trivial example like this does:

```
$ python3
Python 3.7.2 (default, Dec 27 2018, 07:35:06)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> d = {1: 1, 2: 2}
>>> for key in d:
...   del d[key]
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration
OK, apparently sufficiently recent versions of Python 3.5 and 3.6 won't raise a `RuntimeError` because a workaround was put in place, see https://github.com/python/cpython/pull/2409 .

That explains why we didn't run into this during testing, it's only an issue in practice when using a Python 3.5 or 3.6 that is not latest.
fixed in #3140

Resolving patch:
diff --git a/easybuild/tools/environment.py b/easybuild/tools/environment.py
--- a/easybuild/tools/environment.py
+++ b/easybuild/tools/environment.py
@@ -113,7 +113,7 @@ def unset_env_vars(keys, verbose=True):
     if keys and verbose and build_option('extended_dry_run'):
         dry_run_msg("Undefining environment variables:\n", silent=build_option('silent'))
 
-    for key in keys:
+    for key in list(keys):
         if key in os.environ:
             _log.info("Unsetting environment variable %s (value: %s)" % (key, os.environ[key]))
             old_environ[key] = os.environ[key]
@@ -155,24 +155,25 @@ def read_environment(env_vars, strict=False):
 
 def modify_env(old, new, verbose=True):
     """
-    Compares 2 os.environ dumps. Adapts final environment.
+    Compares two os.environ dumps. Adapts final environment.
     """
-    oldKeys = old.keys()
-    newKeys = new.keys()
-    for key in newKeys:
+    old_keys = list(old.keys())
+    new_keys = list(new.keys())
+
+    for key in new_keys:
         # set them all. no smart checking for changed/identical values
-        if key in oldKeys:
+        if key in old_keys:
             # hmm, smart checking with debug logging
             if not new[key] == old[key]:
-                _log.debug("Key in new environment found that is different from old one: %s (%s)" % (key, new[key]))
+                _log.debug("Key in new environment found that is different from old one: %s (%s)", key, new[key])
                 setvar(key, new[key], verbose=verbose)
         else:
-            _log.debug("Key in new environment found that is not in old one: %s (%s)" % (key, new[key]))
+            _log.debug("Key in new environment found that is not in old one: %s (%s)", key, new[key])
             setvar(key, new[key], verbose=verbose)
 
-    for key in oldKeys:
-        if key not in newKeys:
-            _log.debug("Key in old environment found that is not in new one: %s (%s)" % (key, old[key]))
+    for key in old_keys:
+        if key not in new_keys:
+            _log.debug("Key in old environment found that is not in new one: %s (%s)", key, old[key])
             os.unsetenv(key)
             del os.environ[key]
 


Test patch:
diff --git a/test/framework/environment.py b/test/framework/environment.py
--- a/test/framework/environment.py
+++ b/test/framework/environment.py
@@ -71,6 +71,66 @@ def test_setvar(self):
         self.assertEqual(os.environ['FOO'], 'barfoo')
         self.assertEqual(txt, '')
 
+    def test_modify_env(self):
+        """Test for modify_env function."""
+
+        old_env_vars = {
+            'TEST_ENV_VAR_TO_UNSET1': 'foobar',
+            'TEST_ENV_VAR_TO_UNSET2': 'value does not matter',
+            'TEST_COMMON_ENV_VAR_CHANGED': 'old_value',
+            'TEST_COMMON_ENV_VAR_SAME_VALUE': 'this_value_stays',
+        }
+        new_env_vars = {
+            'TEST_COMMON_ENV_VAR_CHANGED': 'new_value',
+            'TEST_NEW_ENV_VAR1': '1',
+            'TEST_NEW_ENV_VAR2': 'two 2 two',
+            'TEST_COMMON_ENV_VAR_SAME_VALUE': 'this_value_stays',
+        }
+
+        # prepare test environment first:
+        # keys in new_env should not be set yet, keys in old_env are expected to be set
+        for key in new_env_vars:
+            if key in os.environ:
+                del os.environ[key]
+        for key in old_env_vars:
+            os.environ[key] = old_env_vars[key]
+
+        env.modify_env(os.environ, new_env_vars)
+
+        self.assertEqual(os.environ.get('TEST_ENV_VAR_TO_UNSET1'), None)
+        self.assertEqual(os.environ.get('TEST_ENV_VAR_TO_UNSET2'), None)
+        self.assertEqual(os.environ.get('TEST_COMMON_ENV_VAR_CHANGED'), 'new_value')
+        self.assertEqual(os.environ.get('TEST_COMMON_ENV_VAR_SAME_VALUE'), 'this_value_stays')
+        self.assertEqual(os.environ.get('TEST_NEW_ENV_VAR1'), '1')
+        self.assertEqual(os.environ.get('TEST_NEW_ENV_VAR2'), 'two 2 two')
+
+        # extreme test case: empty entire environment (original env is restored for next tests)
+        env.modify_env(os.environ, {})
+
+    def test_unset_env_vars(self):
+        """Test unset_env_vars function."""
+
+        os.environ['TEST_ENV_VAR'] = 'test123'
+        # it's fair to assume $HOME will always be set
+        home = os.getenv('HOME')
+        self.assertTrue(home)
+
+        key_not_set = 'NO_SUCH_ENV_VAR'
+        if key_not_set in os.environ:
+            del os.environ[key_not_set]
+
+        res = env.unset_env_vars(['HOME', 'NO_SUCH_ENV_VAR', 'TEST_ENV_VAR'])
+
+        self.assertFalse('HOME' in os.environ)
+        self.assertFalse('NO_SUCH_ENV_VAR' in os.environ)
+        self.assertFalse('TEST_ENV_VAR' in os.environ)
+
+        expected = {
+            'HOME': home,
+            'TEST_ENV_VAR': 'test123',
+        }
+        self.assertEqual(res, expected)
+
 
 def suite():
     """ returns all the testcases in this module """