Repository: ESMCI/cime

Issue:
CIME assumes all output from "module" commands are errors
I realized this when I was loading a module on Summitdev at OLCF that always gives output, and ./case.setup failed. The details are below, and this is for commit 7822b6b8903feb26c4171160423b472351ddb035. ~~Also, I noticed that the cime/utils/python/CIME directory now only contains compiled python code. Am I missing something, or do we no longer have access to the CIME python source code?~~ Nevermind, I found the cime/scripts/lib/CIME directory now. Thanks.

```
[imn@summitdev-login1:~/ACME/cime/scripts/summit_test1] 8-) ./case.setup 
ERROR: module command module load DefApps failed with message:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
There are messages associated with the following module(s):
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

DefApps:
   Please report any missing or needed software versions to help@olcf.ornl.gov
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```


Just to be sure about this, I tested a module ls command, and got the following:



```
[imn@summitdev-login1:~/ACME/cime/scripts/summit_test1] 8-) ./case.setup 
ERROR: module command module ls  failed with message:
Currently Loaded Modules:
  1) xl/20161123   2) spectrum_mpi/10.1.0.2-20161221   3) hsi/5.0.2.p5   4) xalt/0.7.5   5) DefApps   6) python/3.5.2
```



Discussion:
I should also mention that the Lmod system we're using likes to give output, so this is causing errors with every configuration I've tried so far.
OK, I see the offending problem. Our module system puts all output to stderr instead of stdout, even though it returns a 0 exit code (see below). In cime/scripts/lib/CIME/XML/env_mach_specific.py, the stderr is tested, and if it contains anything, it fails with:

```
225             stat, py_module_code, errout = run_cmd(cmd)
226             expect(stat==0 and len(errout) == 0,
227                    "module command %s failed with message:\n%s"%(cmd,errout))
```

Can we change this to just say "expect(stat==0,...)"?


```
[imn@summitdev-login1:~/ACME/cime/scripts/lib/CIME] 8-) module add DefApps 2> /dev/null
[imn@summitdev-login1:~/ACME/cime/scripts/lib/CIME] 8-) module add DefApps 1> /dev/null
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
There are messages associated with the following module(s):
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

DefApps:
   Please report any missing or needed software versions to help@olcf.ornl.gov
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[imn@summitdev-login1:~/ACME/cime/scripts/lib/CIME] 8-) 
```

Also, I looked, and Titan does the same thing by putting the output into stderr by default, so this change would seem necessary in both cases. Titan's modules just don't output text as often.
So, I think I should separate these issues and handle the failed module purge in a separate issue. To fix the stderr problem, I propose the following change to both allow the scripts to keep going and print out the output from stderr from the module command.

```
index 14083b7..73fc073 100644
--- a/cime/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/cime/scripts/lib/CIME/XML/env_mach_specific.py
@@ -223,7 +223,9 @@ class EnvMachSpecific(EnvBase):
         for cmd in self._get_module_commands(modules_to_load, "python"):
             logger.debug("module command is %s"%cmd)
             stat, py_module_code, errout = run_cmd(cmd)
-            expect(stat==0 and len(errout) == 0,
+            if ( len(errout) != 0 ) :
+                print(errout)
+            expect(stat==0,
                    "module command %s failed with message:\n%s"%(cmd,errout))
             exec(py_module_code)
```

@mrnorman due to the shell function implementation of module, we often cannot rely on sane return codes which is why the code looks for stderr instead. I can't think of a good alternative at the moment.
Closing. Still no way avoid this problem due to env modules not using return codes. The lmod issues were fixed in ACME.
I ran into the same problem on the Dutch supercomputer Cartesius , which displays an informational message when the NetCDF library is loaded. 

However, contrary to the argument made above that module shouldn't route text to stderr, the message I see appears to be stdout output: 

```
$ module load netcdf-c/impi/intel/4.3.3.1 >text 2>error
$ cat text
load NETCDF C Parallel Library for Intel MPI 5 or higher (SURFSARA_NETCDFC_ROOT,SURFSARA_NETCDFC_LIB,SURFSARA_NETCDFC_INCLUDE).
```
Any thoughts? I have a workaround (as proposed by MrNorman in https://github.com/ESMCI/cime/issues/1625#issuecomment-305172013 ) but it doesn't seem right that stdout from module is seen as an error. Using CIME version https://github.com/CESM-Development/cime/tags/cime5.4.0-alpha.18
I can take a look.
@lvankampenhout , can you 'cat error' in your example above?

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_mach_specific.py b/scripts/lib/CIME/XML/env_mach_specific.py
--- a/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/scripts/lib/CIME/XML/env_mach_specific.py
@@ -273,7 +273,7 @@ def _load_module_modules(self, modules_to_load):
         for cmd in self._get_module_commands(modules_to_load, "python"):
             logger.debug("module command is {}".format(cmd))
             stat, py_module_code, errout = run_cmd(cmd)
-            expect(stat==0 and len(errout) == 0,
+            expect(stat==0 and (len(errout) == 0 or self.allow_error()),
                    "module command {} failed with message:\n{}".format(cmd, errout))
             exec(py_module_code)
 
@@ -354,6 +354,17 @@ def get_module_system_type(self):
         module_system = self.get_child("module_system")
         return self.get(module_system, "type")
 
+    def allow_error(self):
+        """
+        Return True if stderr output from module commands should be assumed
+        to be an error. Default False. This is necessary since implementations
+        of environment modules are highlty variable and some systems produce
+        stderr output even when things are working fine.
+        """
+        module_system = self.get_child("module_system")
+        value = self.get(module_system, "allow_error")
+        return value.upper() == "TRUE" if value is not None else False
+
     def get_module_system_init_path(self, lang):
         init_nodes = self.get_optional_child("init_path", attributes={"lang":lang}, root=self.get_child("module_system"))
         return self.text(init_nodes) if init_nodes is not None else None


Test patch: