Repository: ESMCI/cime

Issue:
mpi-serial builds rebuild a lot even when reusing a sharedlib build
When you run a test suite, tests with the same build configuration reuse the sharedlib build space for some libraries. The intent is that these libraries shouldn't need to be rebuilt each time, but instead the later tests can reuse the builds from the earlier tests.

But it seems this isn't working as intended for mpi-serial builds. (See #690 for the original reporting of this issue; but I'm copying the relevant information below.)

I looked at how long the SHAREDLIB_BUILD takes for tests that are sharing an already-built sharedlib build, for CLM (yellowstone_intel, with cime at d321e4b).

For mpich2 cases, an initial SHAREDLIB_BUILD took 185 sec (or maybe longer - it's possible that the case I looked at wasn't really the first), and a subsequent build (for a later test) took 37 sec. So the sharedlib build is saving substantial time. (Though I'm surprised that it still takes 37 sec, when it should have nothing to do. Some of this time must be spent in generating namelists, etc. - but 37 sec still seems like a lot.)

However: For mpi-serial cases, an initial SHAREDLIB_BUILD took 286 sec, and subsequent builds (for later tests) took 175 sec. So mpi-serial cases seem to be doing a lot of rebuilds.

I also looked at the modification times on various files in the sharedlib build area. For the mpi-serial builds, quite a few .mod and .a files are recreated in each build. It looks to me like the root cause of the problem is that there are some mpi header files (.h) copied into the include/ directory. It appears that these mpi-serial header files are re-copied with each build; I'm thinking they should be copied with a makefile rule, so that they are only re-copied if the version in the source tree has changed. It looks like this re-copying of the .h files has a cascading effect, presumably because make sees updated prerequisites which force other things to be rebuilt: lib/libmpi-serial.a is rebuilt, as are about half of the .mod files in csm_share, csm_share.a, and essentially all CLM files. I think that this constant re-creation of libraries causes problems when tests in the model_build phase are trying to use them: they're trying to read files that are changing out from under them.

**Brief summary of what I think the issue is: It appears that these mpi-serial header files are re-copied with each build; I'm thinking they should be copied with a makefile rule, so that they are only re-copied if the version in the source tree has changed.**



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/XML/env_mach_specific.py b/utils/python/CIME/XML/env_mach_specific.py
--- a/utils/python/CIME/XML/env_mach_specific.py
+++ b/utils/python/CIME/XML/env_mach_specific.py
@@ -219,9 +219,12 @@ def _match_attribs(self, attribs, compiler, debug, mpilib):
 
     def _match(self, my_value, xml_value):
         if (xml_value.startswith("!")):
-            return my_value != xml_value[1:]
+            result = my_value != xml_value[1:]
         else:
-            return my_value == xml_value
+            result = my_value == xml_value
+        logger.debug("(env_mach_specific) _match %s %s %s"%(my_value, xml_value, result))
+        return result
+
 
     def _get_module_commands(self, modules_to_load, shell):
         # Note this is independent of module system type
diff --git a/utils/python/CIME/build.py b/utils/python/CIME/build.py
--- a/utils/python/CIME/build.py
+++ b/utils/python/CIME/build.py
@@ -275,15 +275,15 @@ def case_build(caseroot, case, sharedlib_only=False, model_only=False):
     # Load modules
     case.load_env()
 
-    build_checks(case, build_threaded, comp_interface, use_esmf_lib,
-                                        debug, compiler, mpilib, sharedlibroot, complist,
-                                        ninst_build, smp_value, model_only)
+    sharedpath = build_checks(case, build_threaded, comp_interface,
+                              use_esmf_lib, debug, compiler, mpilib,
+                              sharedlibroot, complist, ninst_build, smp_value, model_only)
 
     t2 = time.time()
     logs = []
 
     if not model_only:
-        logs = build_libraries(case, exeroot, caseroot, cimeroot, libroot, mpilib, lid,
+        logs = build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, mpilib, lid,
                                machines_file)
 
     if not sharedlib_only:
@@ -304,7 +304,10 @@ def case_build(caseroot, case, sharedlib_only=False, model_only=False):
 def build_checks(case, build_threaded, comp_interface, use_esmf_lib, debug, compiler, mpilib,
                  sharedlibroot, complist, ninst_build, smp_value, model_only):
 ###############################################################################
-
+    """
+    check if a build needs to be done and warn if a clean is warrented first
+    returns the relative sharedpath directory for sharedlibraries
+    """
     ninst_value  = case.get_value("NINST_VALUE")
     smp_build    = case.get_value("SMP_BUILD")
     build_status = case.get_value("BUILD_STATUS")
@@ -332,10 +335,10 @@ def build_checks(case, build_threaded, comp_interface, use_esmf_lib, debug, comp
 
     debugdir = "debug" if debug else "nodebug"
     threaddir = "threads" if (os.environ["SMP"] == "TRUE" or build_threaded) else "nothreads"
-    sharedpath = os.path.join(sharedlibroot, compiler, mpilib, debugdir, threaddir)
+    sharedpath = os.path.join(compiler, mpilib, debugdir, threaddir)
+
     logger.debug("compiler=%s mpilib=%s debugdir=%s threaddir=%s"%
                  (compiler,mpilib,debugdir,threaddir))
-    os.environ["SHAREDPATH"] = sharedpath
 
     expect( ninst_build == ninst_value or ninst_build == "0",
             """
@@ -400,40 +403,39 @@ def build_checks(case, build_threaded, comp_interface, use_esmf_lib, debug, comp
     if not model_only:
         create_namelists(case)
 
-    return
+    return sharedpath
 
 ###############################################################################
-def build_libraries(case, exeroot, caseroot, cimeroot, libroot, mpilib, lid, machines_file):
+def build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, mpilib, lid, machines_file):
 ###############################################################################
 
-    if (mpilib == "mpi-serial"):
-        for header_to_copy in glob.glob(os.path.join(cimeroot, "externals/mct/mpi-serial/*.h")):
-            shutil.copy(header_to_copy, os.path.join(libroot, "include"))
-
-    sharedpath = os.environ["SHAREDPATH"]
-    shared_lib = os.path.join(sharedpath, "lib")
-    shared_inc = os.path.join(sharedpath, "include")
+    shared_lib = os.path.join(exeroot, sharedpath, "lib")
+    shared_inc = os.path.join(exeroot, sharedpath, "include")
     for shared_item in [shared_lib, shared_inc]:
         if (not os.path.exists(shared_item)):
             os.makedirs(shared_item)
 
     libs = ["mct", "gptl", "pio", "csm_share"]
     logs = []
-
+    sharedlibroot = case.get_value("SHAREDLIBROOT")
     for lib in libs:
-        full_lib_path = os.path.join(sharedpath, lib)
+        if lib == "csm_share":
+            # csm_share adds its own dir name
+            full_lib_path = os.path.join(sharedlibroot, sharedpath)
+        else:
+            full_lib_path = os.path.join(sharedlibroot, sharedpath, lib)
         # pio build creates its own directory
         if (lib != "pio" and not os.path.exists(full_lib_path)):
             os.makedirs(full_lib_path)
 
-        file_build = os.path.join(sharedpath, "%s.bldlog.%s" % (lib, lid))
-        my_file = os.path.join(os.path.dirname(machines_file), "buildlib.%s" % lib)
+        file_build = os.path.join(exeroot, "%s.bldlog.%s" % (lib, lid))
+        my_file = os.path.join(cimeroot, "cime_config", "buildlib.%s" % lib)
         if lib == "pio":
             my_file = "PYTHONPATH=%s:%s:$PYTHONPATH %s"%(os.path.join(cimeroot,"scripts"),
                                                           os.path.join(cimeroot,"utils","python"), my_file)
         with open(file_build, "w") as fd:
-            stat = run_cmd("%s %s %s" %
-                           (my_file, sharedpath, caseroot),
+            stat = run_cmd("%s %s %s %s" %
+                           (my_file, full_lib_path, os.path.join(exeroot,sharedpath), caseroot),
                            from_dir=exeroot,
                            verbose=True, arg_stdout=fd,
                            arg_stderr=subprocess.STDOUT)[0]
@@ -451,14 +453,13 @@ def build_libraries(case, exeroot, caseroot, cimeroot, libroot, mpilib, lid, mac
     if comp_lnd == "clm" and not "clm4_0" in clm_config_opts:
         logging.info("         - Building clm4_5/clm5_0 Library ")
         esmfdir = "esmf" if case.get_value("USE_ESMF_LIB") else "noesmf"
-        sharedpath = os.environ["SHAREDPATH"]
-        bldroot = os.path.join(sharedpath, case.get_value("COMP_INTERFACE"), esmfdir, "clm","obj" )
-        libroot = os.path.join(sharedpath, case.get_value("COMP_INTERFACE"), esmfdir, "lib")
-        incroot = os.path.join(sharedpath, case.get_value("COMP_INTERFACE"), esmfdir, "include")
+        bldroot = os.path.join(sharedlibroot, sharedpath, case.get_value("COMP_INTERFACE"), esmfdir, "clm","obj" )
+        libroot = os.path.join(exeroot, sharedpath, case.get_value("COMP_INTERFACE"), esmfdir, "lib")
+        incroot = os.path.join(exeroot, sharedpath, case.get_value("COMP_INTERFACE"), esmfdir, "include")
         file_build = os.path.join(exeroot, "lnd.bldlog.%s" %  lid)
         config_lnd_dir = os.path.dirname(case.get_value("CONFIG_LND_FILE"))
 
-        for ndir in [bldroot, libroot]:
+        for ndir in [bldroot, libroot, incroot]:
             if (not os.path.isdir(ndir)):
                 os.makedirs(ndir)
 


Test patch: