Repository: ESMCI/cime

Issue:
Allow default mpilib to differ based on compiler
On cheyenne, we need to use a different mpilib for gnu than for intel (openmpi for gnu, mpt for intel). However, as far as I can tell, there's no way to get the correct behavior out of the box: For gnu, you need to specify the mpilib at create_newcase or create_test time with `--mpilib openmpi`. It would be great if there were a way to specify default mpilibs and/or available mpilibs on a per-compiler basis. For example, I'd like to be able to do this for cheyenne:

```xml
    <MPILIBS compiler="intel">mpt,openmpi</MPILIBS>
    <MPILIBS compiler="gnu">openmpi</MPILIBS>
```

But currently I can't: besides violating the xml schema, it also appears that queries for the mpilib don't pass in the compiler attribute (if I'm understanding the error message correctly).


Discussion:
As an aside: I don't like that you can specify `--mpilib` in `create_test`: It took me a while to figure out why my tests were failing despite similar tests passing that @fischer-ncar had run: Eventually @jedwards4b pointed out that @fischer-ncar must be running the cheyenne gnu tests with `--mpilib openmpi`, but this doesn't show up anywhere in the test name. I feel that we should either remove the `--mpilib` option to `create_test`, or have it modify all test names to add something like `_Mopenmpi`.

The general rationale here is: If you know the code base, then everything important to reproduce a test should be in the test name.
@billsacks , I agree. The flags for create_test should either not directly influence the test behavior or should automatically update the test name.
@mvertens @jedwards4b I'm adding this to the cesm2 milestone because it seems it could trip up users trying to create a gnu case on cheyenne; feel free to remove it from the milestone if you disagree with its priority.
The clm testlist isn't working anymore because of this issue. The following tests aren't working:

ERS_D_Ld6.f10_f10_musgs.I1850Clm45BgcCrop.cheyenne_gnu.clm-clm50CMIP6frc		
ERS_Ld5.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.mosart-default		
SMS_D_Ld5.f10_f10_musgs.I2000Clm50BgcCruGs.cheyenne_gnu.mosart-default




@ekluzek the workaround is to specify `--mpilib openmpi` in your `create_test` command.
It's not because of this issue that the test isn't working anymore, since this issue is requesting a feature that never existed and the test used to work.
I get the workaround for setting mpilib in create_test. So that's probably OK, as I can run my tests that way.

The question I have is that I've never had to explicitly set mpilib on the command line for create_test before. But, the gnu tests worked in the past. So did gnu work with mpt in the past, and now requires openmpi? Or what changed so that I have to be explicit about mpilib on the create_test command line?
gnu never worked well with mpt on cheyenne.  openmpi was suppose to be a temporary work around until mpt was fixed.  It's looking like a fix for mpt isn't going to be coming.
OK, that makes sense. I realize changes for the content of this issue "allowing different defaults for different compilers" might be involved. What I'm wondering is if we can add a simpler change that will just abort, if you use a machine/compiler/mpilib combination that's known not to work. Could that be done? Should I open that as a separate issue?
How are you to make that determination and how would someone override it?  
You could add it to the XML in config_machines.xml as something like...

  <machine MACH="cheyenne">
    <DESC>NCAR SGI platform, os is Linux, 36 pes/node, batch system is PBS</DESC>
    <NODENAME_REGEX>.*.cheyenne.ucar.edu</NODENAME_REGEX>
    <OS>LINUX</OS>
    <COMPILERS>intel,gnu</COMPILERS>
    <MPILIBS>mpt,openmpi</MPILIBS>
    <INVALID_COMPILER_MPILIB_COMBINATIONS>gnu_mpt</INVALID_COMPILER_MPILIB_COMBINATIONS>

It would be an optional field that isn't required, but if it's given the compiler/mpi combination is checked  to see if it's in the list and if it is -- the build aborts. I suppose you could then have a command line option to override it, or another option in the XML. Or honestly, for an individual case, the user could just remove the contents of "INVALID_COMPILER_MPILIB_COMBINATIONS" as if there's nothing in there, there would be no restrictions. I think the last one is the best option.

Should I create a separate issue for doing that?
well you could but I think that it's a really bad idea.   I should have the ability to choose the preferred compiler/mpilib combination working soon.

I don't think we should abort based on a bad machine/compiler/mpilib combination.  You'll also need to add compsets, because some compsets ran fine with cheyenne/gnu/mpt.  Then there's stealth system updates.  One combination might work one day, then be broken the next.  I just think it would be impossible to keep up with all of the system changes.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/compilerblock.py b/scripts/lib/CIME/XML/compilerblock.py
--- a/scripts/lib/CIME/XML/compilerblock.py
+++ b/scripts/lib/CIME/XML/compilerblock.py
@@ -224,10 +224,6 @@ def _add_elem_to_lists(self, name, elem, value_lists):
         value_lists - A dictionary of PossibleValues, containing the lists
                       of all settings for each variable.
         """
-        # Skip this if the element's MPILIB is not valid.
-        if "MPILIB" in elem.keys() and \
-           not self._machobj.is_valid_MPIlib(elem.get("MPILIB")):
-            return
         setting, depends = self._elem_to_setting(elem)
         if name not in value_lists:
             value_lists[name] = PossibleValues(name, setting,
diff --git a/scripts/lib/CIME/XML/compilers.py b/scripts/lib/CIME/XML/compilers.py
--- a/scripts/lib/CIME/XML/compilers.py
+++ b/scripts/lib/CIME/XML/compilers.py
@@ -37,13 +37,17 @@ def __init__(self, machobj, infile=None, compiler=None, mpilib=None, files=None,
 
         self.machine  = machobj.get_machine_name()
         self.os = machobj.get_value("OS")
-        if mpilib is None:
-            mpilib = machobj.get_default_MPIlib()
-        self.mpilib = mpilib
         if compiler is None:
             compiler = machobj.get_default_compiler()
         self.compiler       = compiler
 
+        if mpilib is None:
+            if compiler is None:
+                mpilib = machobj.get_default_MPIlib()
+            else:
+                mpilib = machobj.get_default_MPIlib(attributes={'compiler':compiler})
+        self.mpilib = mpilib
+
         self.compiler_nodes = None # Listed from last to first
         #Append the contents of $HOME/.cime/config_compilers.xml if it exists
         #This could cause problems if node matchs are repeated when only one is expected
diff --git a/tools/load_balancing_tool/load_balancing_submit.py b/tools/load_balancing_tool/load_balancing_submit.py
--- a/tools/load_balancing_tool/load_balancing_submit.py
+++ b/tools/load_balancing_tool/load_balancing_submit.py
@@ -238,7 +238,7 @@ def load_balancing_submit(compset, res, pesfile, mpilib, compiler, project, mach
         compiler = machobj.get_default_compiler()
         print "compiler is {}".format(compiler)
     if mpilib is None:
-        mpilib = machobj.get_default_MPIlib()
+        mpilib = machobj.get_default_MPIlib({"compiler":compiler})
 
 
 


Test patch:
diff --git a/scripts/tests/scripts_regression_tests.py b/scripts/tests/scripts_regression_tests.py
--- a/scripts/tests/scripts_regression_tests.py
+++ b/scripts/tests/scripts_regression_tests.py
@@ -1797,7 +1797,7 @@ def is_valid_MPIlib(self, _):
         """Assume all MPILIB settings are valid."""
         return True
 
-    def get_default_MPIlib(self):
+    def get_default_MPIlib(self, attributes=None):
         return "mpich2"
 
     def get_default_compiler(self):
@@ -2192,8 +2192,8 @@ def test_compiler_changeable_at_build_time(self):
         xml1 = """<compiler><SUPPORTS_CXX>FALSE</SUPPORTS_CXX></compiler>"""
         xml2 = """<compiler COMPILER="gnu"><SUPPORTS_CXX>TRUE</SUPPORTS_CXX></compiler>"""
         tester = self.xml_to_tester(xml1+xml2)
-        tester.assert_variable_equals("SUPPORTS_CXX", "FALSE")
         tester.assert_variable_equals("SUPPORTS_CXX", "TRUE", env={"COMPILER": "gnu"})
+        tester.assert_variable_equals("SUPPORTS_CXX", "FALSE")
 
     def test_base_flags(self):
         """Test that we get "base" compiler flags."""