Repository: ESMCI/cime

Issue:
CIME error with python >= version 2.7.14
Using python 2.7.14, I get a value error in case.submit on anvil with E3SM, the traceback follows
```
Traceback (most recent call last):
  File "/var/spool/torque/mom_priv/jobs/1788221.bmgt1.lcrc.anl.gov.SC", line 87, in <module>
    _main_func(__doc__)
  File "/var/spool/torque/mom_priv/jobs/1788221.bmgt1.lcrc.anl.gov.SC", line 82, in _main_func
    success = case_run(case, skip_pnl=skip_pnl)
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/case_run.py", line 290, in case_run
    lid = run_model(case, lid, skip_pnl)
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/case_run.py", line 147, in run_model
    return run_and_log_case_status(functor, "case.run", caseroot=case.get_value("CASEROOT"))
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1359, in run_and_log_case_status
    rv = func()
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/case_run.py", line 146, in <lambda>
    functor = lambda: _run_model_impl(case, lid, skip_pnl=skip_pnl)
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/case_run.py", line 83, in _run_model_impl
    pre_run_check(case, lid, skip_pnl=skip_pnl)
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/case_run.py", line 40, in pre_run_check
    case.load_env()
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/case.py", line 1238, in load_env
    env_module.load_env(self)
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/XML/env_mach_specific.py", line 82, in load_env
    self.load_modules(modules_to_load)
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/XML/env_mach_specific.py", line 108, in load_modules
    self._load_modules_generic(modules_to_load)
  File "/blues/gpfs/home/lvanroe/acme_master/cime/scripts/Tools/../../scripts/lib/CIME/XML/env_mach_specific.py", line 309, in _load_modules_generic
    os.environ[key] = newenv[key]
  File "/lcrc/group/acme/lvanroe/conda/lib/python2.7/os.py", line 473, in __setitem__
    putenv(key, item)
ValueError: illegal environment variable name
```
case.submit appears to be failing when loading the ```LS_COLORS=.........```.  Indeed the python 2.7.14 release docs (https://raw.githubusercontent.com/python/cpython/c707893f9cee870bba8364b3a06eb9cfa3b80e58/Misc/NEWS) show that equal signs are now prohibited in environmental variables with putenv, and this is the problem here.

This should be reproducible on any platform with python 2.7.14.


Discussion:
Is this because you are trying to set LS_COLORS in the config_machines.xml file?   Not to argue the point that you have found a problem but why would you do that?
This is the default configuration for anvil in cime.  I'm not adding anything special and the LS_COLORS variable is not in config_machines.xml.  In fact I have searched in vain to see where that variable is being set.  I agree with you that setting this variable makes no sense.
@jedwards4b after a bit more digging it looks like the anvil global bashrc sets the LS_COLORS variable and it looks like line 272 of scripts/lib/CIME/XML/env_mach_specific.py loads the users environment, which causes the problem on anvil.  I think the simplest workaround for me right now is to unset LS_COLORS on anvil.

@jedwards4b, can you please comment on why this issue has been closed?  I've had a similar issue here too and concur that this is indeed a bug that cause E3SM to break on the current version (as of yesterday).  It would be great if this could be fixed.  

If it isn't a bug but user error, e.g., the issue is related to specific set up on the user end, it would be helpful to know how to avoid this conflict.  

Any advice you can provide is greatly appreciated.  Thanks!
Yes, this needs to be reworked. 
@jgfouca do you have time to look into this?  In particular we should be using the python interface to modules and avoid loading the full environment.
@jedwards4b , yes, I can look

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_mach_specific.py b/scripts/lib/CIME/XML/env_mach_specific.py
--- a/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/scripts/lib/CIME/XML/env_mach_specific.py
@@ -6,6 +6,7 @@
 from CIME.XML.env_base import EnvBase
 from CIME.utils import transform_vars, get_cime_root
 import string, resource
+from collections import OrderedDict
 
 logger = logging.getLogger(__name__)
 
@@ -66,7 +67,7 @@ def _get_envs_for_case(self, case):
 
         return envs_to_set
 
-    def load_env(self, case):
+    def load_env(self, case, force_method=None):
         """
         Should only be called by case.load_env
         """
@@ -74,11 +75,11 @@ def load_env(self, case):
         # in the environment_variables block
         modules_to_load = self._get_modules_for_case(case)
         if (modules_to_load is not None):
-            self.load_modules(modules_to_load)
+            self._load_modules(modules_to_load, force_method=force_method)
 
         envs_to_set = self._get_envs_for_case(case)
         if (envs_to_set is not None):
-            self.load_envs(envs_to_set)
+            self._load_envs(envs_to_set)
 
         self._get_resources_for_case(case)
 
@@ -93,12 +94,10 @@ def _get_resources_for_case(self, case):
                 limits = (int(val), limits[1])
                 resource.setrlimit(attr, limits)
 
-    def load_modules(self, modules_to_load):
-        module_system = self.get_module_system_type()
+    def _load_modules(self, modules_to_load, force_method=None):
+        module_system = self.get_module_system_type() if force_method is None else force_method
         if (module_system == "module"):
             self._load_module_modules(modules_to_load)
-        elif (module_system == "module_lmod"):
-            self._load_modules_generic(modules_to_load)
         elif (module_system == "soft"):
             self._load_modules_generic(modules_to_load)
         elif (module_system == "generic"):
@@ -120,7 +119,7 @@ def list_modules(self):
         else:
             source_cmd = ""
 
-        if (module_system in ["module", "module_lmod"]):
+        if (module_system in ["module"]):
             return run_cmd_no_fail("{}module list".format(source_cmd), combine_output=True)
         elif (module_system == "soft"):
             # Does soft really not provide this capability?
@@ -161,7 +160,7 @@ def make_env_mach_specific_file(self, shell, case):
         with open(filename, "w") as fd:
             fd.write("\n".join(lines))
 
-    def load_envs(self, envs_to_set):
+    def _load_envs(self, envs_to_set):
         for env_name, env_value in envs_to_set:
             os.environ[env_name] = "" if env_value is None else env_value
 
@@ -231,10 +230,25 @@ def _get_module_commands(self, modules_to_load, shell):
         # Note this is independent of module system type
         mod_cmd = self.get_module_system_cmd_path(shell)
         cmds = []
+        last_action = None
+        last_cmd    = None
+
         for action, argument in modules_to_load:
             if argument is None:
                 argument = ""
-            cmds.append("{} {} {}".format(mod_cmd, action, "" if argument is None else argument))
+
+            if action == last_action:
+                last_cmd = "{} {}".format(last_cmd, argument)
+            else:
+                if last_cmd is not None:
+                    cmds.append(last_cmd)
+
+                last_cmd = "{} {} {}".format(mod_cmd, action, "" if argument is None else argument)
+                last_action = action
+
+        if last_cmd:
+            cmds.append(last_cmd)
+
         return cmds
 
     def _load_module_modules(self, modules_to_load):
@@ -270,37 +284,31 @@ def _load_modules_generic(self, modules_to_load):
         ###################################################
         # Parse the output to set the os.environ dictionary
         ###################################################
-        newenv = {}
-        dolater = []
+        newenv = OrderedDict()
+        lastkey = None
         for line in output.splitlines():
-            if line.find('$')>0:
-                dolater.append(line)
-                continue
-
-            m=re.match(r'^(\S+)=(\S+)\s*;*\s*$',line)
-            if m:
-                key = m.groups()[0]
-                val = m.groups()[1]
+            if "=" in line:
+                key, val = line.split("=", 1)
                 newenv[key] = val
+                lastkey = key
+            else:
+                newenv[lastkey] += "\n" + line
 
-        # Now that initial newenv has been set, resolve variables
-        for line in dolater:
-            m=re.match(r'^(\S+)=(\S+)\s*;*\s*$',line)
-            if m:
-                key = m.groups()[0]
-                valunresolved = m.groups()[1]
-                val = string.Template(valunresolved).safe_substitute(newenv)
-                expect(val is not None,
-                       'string value {} unable to be resolved'.format(valunresolved))
-                newenv[key] = val
+        # resolve variables
+        for key, val in newenv.items():
+            newenv[key] = string.Template(val).safe_substitute(newenv)
 
         # Set environment with new or updated values
         for key in newenv:
-            if key in os.environ and key not in newenv:
-                del(os.environ[key])
+            if key in os.environ and os.environ[key] == newenv[key]:
+                pass
             else:
                 os.environ[key] = newenv[key]
 
+        for oldkey in list(os.environ.keys()):
+            if oldkey not in newenv:
+                del os.environ[oldkey]
+
     def _load_none_modules(self, modules_to_load):
         """
         No Action required


Test patch:
diff --git a/scripts/tests/scripts_regression_tests.py b/scripts/tests/scripts_regression_tests.py
--- a/scripts/tests/scripts_regression_tests.py
+++ b/scripts/tests/scripts_regression_tests.py
@@ -1758,6 +1758,49 @@ def test_create_test_longname(self):
     ###########################################################################
         self._create_test(["SMS.f19_g16.2000_SATM_XLND_SICE_SOCN_XROF_XGLC_SWAV", "--no-build"])
 
+    ###########################################################################
+    def test_env_loading(self):
+    ###########################################################################
+        self._create_test(["--no-build", "TESTRUNPASS.f19_g16_rx1.A"], test_id=self._baseline_name)
+
+        casedir = os.path.join(self._testroot,
+                               "%s.%s" % (CIME.utils.get_full_test_name("TESTRUNPASS.f19_g16_rx1.A", machine=self._machine, compiler=self._compiler), self._baseline_name))
+        self.assertTrue(os.path.isdir(casedir), msg="Missing casedir '%s'" % casedir)
+
+        with Case(casedir, read_only=True) as case:
+            env_mach = case.get_env("mach_specific")
+            if env_mach.get_module_system_type() == "module":
+                orig_env = dict(os.environ)
+
+                env_mach.load_env(case)
+                module_env = dict(os.environ)
+
+                os.environ.clear()
+                os.environ.update(orig_env)
+
+                env_mach.load_env(case, force_method="generic")
+                generic_env = dict(os.environ)
+
+                os.environ.clear()
+                os.environ.update(orig_env)
+
+                problems = ""
+                for mkey, mval in module_env.items():
+                    if mkey not in generic_env:
+                        if not mkey.startswith("PS") and mkey != "OLDPWD":
+                            problems += "Generic missing key: {}\n".format(mkey)
+                    elif mval != generic_env[mkey] and mkey not in ["_", "SHLVL"] and not mkey.endswith("()"):
+                        problems += "Value mismatch for key {}: {} != {}\n".format(mkey, repr(mval), repr(generic_env[mkey]))
+
+                for gkey in generic_env.keys():
+                    if gkey not in module_env:
+                        problems += "Modules missing key: {}\n".format(gkey)
+
+                self.assertEqual(problems, "", msg=problems)
+
+            else:
+                self.skipTest("Skipping env load test - Only works on machines that support env modules")
+
 ###############################################################################
 class X_TestSingleSubmit(TestCreateTestCommon):
 ###############################################################################