Repository: ESMCI/cime

Issue:
Problem with module commands. Possibly trying to do multiple module swaps with same command?
On cori-knl, with master of feb20th, our GNU builds no longer work. It looks like something related to how CIME treats the module commands.

```Directory: /global/cscratch1/sd/ndk/wacmy/m64-feb20/cime/scripts
/global/cscratch1/sd/ndk/wacmy/m64-feb20/cime/scripts
cori03% create_test SMS_Ld5.ne4_ne4.FC5AV1C-L --compiler=gnu
Using project from .cesm_proj: acme
Creating test directory /global/cscratch1/sd/ndk/acme_scratch/cori-knl/SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu.20180220_165518_yl9w63
RUNNING TESTS:
  SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu
Starting CREATE_NEWCASE for test SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu with 1 procs
Finished CREATE_NEWCASE for test SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu in 3.592422 seconds (PASS)
Starting XML for test SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu with 1 procs
Finished XML for test SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu in 0.973876 seconds (PASS)
Starting SETUP for test SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu with 1 procs
Finished SETUP for test SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu in 2.305529 seconds (FAIL). [COMPLETED 1 of 1]
    Case dir: /global/cscratch1/sd/ndk/acme_scratch/cori-knl/SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu.20180220_165518_yl9w63
    Errors were:
        ERROR: module command /opt/modules/default/bin/modulecmd python swap cray-mpich cray-mpich/7.6.2 PrgEnv-intel PrgEnv-gnu failed with message:
        ModuleCmd_Switch.c(162):ERROR:11: Usage is 'switch oldmodule newmodule'

Due to presence of batch system, create_test will exit before tests are complete.
To force create_test to wait for full completion, use --wait
At test-scheduler close, state is:
FAIL SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu (phase SETUP)
    Case dir: /global/cscratch1/sd/ndk/acme_scratch/cori-knl/SMS_Ld5.ne4_ne4.FC5AV1C-L.cori-knl_gnu.20180220_165518_yl9w63
test-scheduler took 7.09203696251 seconds
2.187u 0.528s 0:08.71 30.9% 0pf+0w
cori03% 
```

Can repeat the error on command line like so:

```
cori12% module swap cray-mpich cray-mpich/7.6.2 
cori12% module swap PrgEnv-intel PrgEnv-gnu
cori12% module swap cray-mpich cray-mpich/7.6.2 PrgEnv-intel PrgEnv-gnu
ModuleCmd_Switch.c(162):ERROR:11: Usage is 'switch oldmodule newmodule'
cori12%
```


Discussion:
@ndkeen , this is a mistake in something I did. Fix coming.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_mach_specific.py b/scripts/lib/CIME/XML/env_mach_specific.py
--- a/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/scripts/lib/CIME/XML/env_mach_specific.py
@@ -233,11 +233,28 @@ def _get_module_commands(self, modules_to_load, shell):
         last_action = None
         last_cmd    = None
 
+        # Normally, we will try to combine or batch module commands together...
+        #
+        # module load X
+        # module load Y
+        # module load Z
+        #
+        # is the same as ...
+        #
+        # module load X Y Z
+        #
+        # ... except the latter is significatly faster due to performing 1/3 as
+        # many forks.
+        #
+        # Not all module commands support batching though and we enurmerate those
+        # here.
+        actions_that_cannot_be_batched = ["swap"]
+
         for action, argument in modules_to_load:
             if argument is None:
                 argument = ""
 
-            if action == last_action:
+            if action == last_action and action not in actions_that_cannot_be_batched:
                 last_cmd = "{} {}".format(last_cmd, argument)
             else:
                 if last_cmd is not None:


Test patch: