Repository: ESMCI/cime

Issue:
missing mpi-serial mpirun case on hobart
hobart seems to be missing an option for mpirun for a mpi-serial build. This is with: cime5.3.0-alpha.02

It has

    <mpirun mpilib="default">
      <executable>mpiexec</executable>
      <arguments>
        <arg name="machine_file">--machinefile $ENV{PBS_NODEFILE}</arg>
        <arg name="num_tasks"> -n $TOTALPES </arg>
      </arguments>
    </mpirun>


and needs

    <mpirun mpilib="mpi-serial">
      <executable></executable>
    </mpirun>

Really can't the above be for every machine?

Sean Swenson ran into this problem trying to run with a hobart_intel build with mpi-serial on burnt, which is configured the same as hobart, but doesn't have a batch system.

Here's his details...
I created a case on cgd machine 'burnt'.  I used --mach hobart.  When building with mpi-serial, and running case.submit -- no-batch, I got this error:

ERROR: Undefined env var 'PBS_NODEFILE'

My workaround was to remove mpiexec and arguments from env_mach_specific.


Discussion:
If burnt is a different machine, why doesn't it just have its own entry?
> If burnt is a different machine, why doesn't it just have its own entry?

Well it's not a machine that we want to support per say. So we don't really want to duplicate all the hobart settings for it. But, it's one that a few people might want to use for restricted cases. It's really in the realm of a user-defined machine that we want people to be able to use. And I'm going to show Sean how to set it up as a personal machine they can use.

But, for hobart when mpi-serial is on -- you don't need to run thorough mpiexec, and I would say that you shouldn't because it only adds complexity. So this is an appropriate change for hobart. And I think we should make sure that there is a mpi-serial configuration for mpirun setup for almost all machines that doesn't go through any mpi executable.
Does your proposed solution prevent running mpi-serial jobs on Hobart's head node?
We have removed the requirement that mpi-serial be named as a supported MPILIB in config_machines.xml so it seems to me reasonable that we should also remove the requirement
to have ```<mpirun mpilib="mpi-serial">
  <executable></executable>
</mpirun>``` in that file.   I will take this on.
@gold2718 it'll still be submitted to batch, so won't run on the head node. It does make it a little easier for a user to get around submitting to batch -- but they still have to know what they are doing. And they would know they are maneuvering around the batch system, so presumably would know that's frowned upon.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_mach_specific.py b/scripts/lib/CIME/XML/env_mach_specific.py
--- a/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/scripts/lib/CIME/XML/env_mach_specific.py
@@ -355,7 +355,7 @@ def get_mpirun(self, case, attribs, check_members=None, job="case.run", exe_only
 
                     if xml_attrib == value:
                         matches += 1
-                    elif key == "mpilib" and xml_attrib == "default":
+                    elif key == "mpilib" and value != "mpi-serial" and xml_attrib == "default":
                         is_default = True
                     else:
                         all_match = False
@@ -374,6 +374,10 @@ def get_mpirun(self, case, attribs, check_members=None, job="case.run", exe_only
                         best_match = mpirun_node
                         best_num_matched = matches
 
+        # if there are no special arguments required for mpi-serial it need not have an entry in config_machines.xml
+        if "mpilib" in attribs and attribs["mpilib"] == "mpi-serial" and best_match is None:
+            return "",[]
+
         expect(best_match is not None or default_match is not None,
                "Could not find a matching MPI for attributes: %s" % attribs)
 
diff --git a/scripts/lib/CIME/XML/machines.py b/scripts/lib/CIME/XML/machines.py
--- a/scripts/lib/CIME/XML/machines.py
+++ b/scripts/lib/CIME/XML/machines.py
@@ -256,7 +256,8 @@ def is_valid_MPIlib(self, mpilib, attributes=None):
         >>> machobj.is_valid_MPIlib("fake-mpi")
         False
         """
-        return mpilib == "mpi-serial" or self.get_field_from_list("MPILIBS", reqval=mpilib, attributes=attributes) is not None
+        return mpilib == "mpi-serial" or \
+            self.get_field_from_list("MPILIBS", reqval=mpilib, attributes=attributes) is not None
 
     def has_batch_system(self):
         """


Test patch: