Repository: ESMCI/cime

Issue:
CLM is rebuilt for every build in the test suite, rather than reusing the sharedlibbuild
### Summary of issue

In recent versions of cime, the sharedlib build functionality for CLM has stopped working: For each build in a given test suite, CLM is being completely rebuilt, rather than reusing the existing build.

### CIME versions where this works and doesn't work

This worked in clm4518/n04/cime5.4.0-alpha.03 (a CLM-specific tag basically the same as cime5.4.0-alpha.03), but is broken in cime5.6.6. You can reproduce this problem by cloning https://github.com/escomp/ctsm and checking out either clm5.0.dev009 or clm5.0.dev010: the main difference between those two tags is the cime version. I have also verified that it's broken in a more recent cime version: 58a5e43. I have not figured out the exact cime version where this stopped working, but my best guess is that this is related to the details of building the csm_share library; I see that this build (Makefile and buildlib.csm_share) have been extensively reworked between these two cime versions.

### How to observe this problem

You can see this problem by either:

1. Building a single ERP test (I used `ERP_Ld5.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-default`): In clm5.0.dev009, the clm build time for case2 (which is written to TestStatus.log) is < 1 sec; in clm5.0.dev010, the clm build time for case2 is about the same as for case1.

2. Building a set of SMS tests. To test this, I created a suite of 36 tests like `SMS_D_Ld1.f10_f10_musgs.I2000Clm50SpGs.cheyenne_intel`, differing only in their length of time (Ld1, Ld2, etc.). (In retrospect, even a test suite of, say, 4 tests would probably have been sufficient.) I built this test suite on cheyenne, using a single compute node to do the builds. In clm5.0.dev009, the clm build times were again < 1 sec for all but one of the tests. In clm5.0.dev010, the clm build times were about the same (> 30 sec) for all tests. The total test suite build time increased from 804 sec to 1393 sec – illustrating the importance of solving this problem.

In both cases, you can also see the problem by looking at the total SHAREDLIB_BUILD times (comparing those times in dev009 and dev010). The differences in SHAREDLIB_BUILD times can be explained by the differences in clm build times, suggesting that the full rebuild of clm is the culprit here.

### What I have figured out

Although the rebuild of CLM is the direct problem, it seems like this arises from an updating of time stamps for csm_share .mod/.h files and/or libcsm_share.a. Note that both the csm_share .mod/.h files and libcsm_share.a appear in a case-specific location (the INSTALL_SHAREDPATH) – not just in the sharedlib area. In clm5.0.dev009, the time stamps on the csm_share .mod/.h files and libcsm_share.a remained at the time of the initial sharedlib build. In clm5.0.dev010, these time stamps are updated for each case. I believe that this causes the rebuild: some CLM build rules depend on csm_share header files, and it looks like all objects depend on libcsm_share.a due to this rule in the makefile:

https://github.com/ESMCI/cime/blob/b78593b3aaeeafc56ec8239905b41820e3ab00c2/scripts/Tools/Makefile#L852-L854

### Preserving times of the .mod and .h files

I tried preserving times when copying .mod and .h files to the install location, with this change:

```diff
diff --git a/scripts/Tools/Makefile b/scripts/Tools/Makefile
index 2c165bc..468e305 100644
--- a/scripts/Tools/Makefile
+++ b/scripts/Tools/Makefile
@@ -859,7 +859,7 @@ $(EXEC_SE): $(OBJS) $(ULIBDEP) $(CSMSHARELIB) $(MCTLIBS) $(PIOLIB) $(GPTLLIB)
 ifdef INCLUDE_DIR
   $(COMPLIB): $(OBJS)
        $(AR) -r $(COMPLIB) $(OBJS)
-       $(CP) *.$(MOD_SUFFIX) *.h $(INCLUDE_DIR)
+       $(CP) -p *.$(MOD_SUFFIX) *.h $(INCLUDE_DIR)
 else
   $(COMPLIB): $(OBJS)
        $(AR) -r $(COMPLIB) $(OBJS)
```

That worked to preserve the time stamps of these files, but didn't solve the problem - probably because the time stamp of libcsm_share.a is still updated. It's possible that this diff is necessary but not sufficient.

### Preserving times of libcsm_share.a

**I haven't figured out exactly what's needed to preserve times of libcsm_share.a, and I think that's the big missing piece at this point.** In the older cime, libcsm_share.a was built in the sharedlib location, and then copied to the install location using `cp -p`. In the new cime, it looks like libcsm_share.a is built directly in the install location. This results in libcsm_share.a being rebuilt for every case, which implies that its time stamp is updated for each case – causing the rebuild of CLM.

Would it be possible to go back to the old logic – building libcsm_share.a in the sharedlib location then copying it to the install location in a way that preserves time stamps? It's not clear to me if the new method (building directly in the install location) was put in place for an important reason, or whether this change was more accidental.

### Something else that may be needed

In trying to track down the problem, I also noticed that there is no `flush` call after setting case2's SHAREDLIBROOT in `system_tests_compare_two`; I'm not sure if this is needed?:

```diff
diff --git a/scripts/lib/CIME/SystemTests/system_tests_compare_two.py b/scripts/lib/CIME/SystemTests/system_tests_compare_two.py
index f8ec9e8..a6d2c89 100644
--- a/scripts/lib/CIME/SystemTests/system_tests_compare_two.py
+++ b/scripts/lib/CIME/SystemTests/system_tests_compare_two.py
@@ -190,6 +190,7 @@ class SystemTestsCompareTwo(SystemTestsCommon):
                 # the MPAS build system
                 self._case2.set_value("SHAREDLIBROOT",
                                       self._case1.get_value("SHAREDLIBROOT"))
+                self._case2.flush()

             self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
         else:
```

(This alone didn't seem to help anything, but again I'm wondering if this is another necessary piece?)

### How to test that a proposed fix indeed solves the problem

In testing a proposed fix, I'd like to see that the CLM build times for everything but the first case are < 1 sec for:

1. `ERP_Ld5.f10_f10_musgs.I1850Clm50BgcCrop.cheyenne_intel.clm-default` (looking at the build times for case2)

2. A test suite with 4 or more tests like `SMS_D_Ld1.f10_f10_musgs.I2000Clm50SpGs.cheyenne_intel`

### Some notifications

@ekluzek this can explain the longer build times you noticed when updating cime

@jedwards4b @jgfouca - do either of you have any thoughts on how to stop updating the time stamp of libcsm_share.a?


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -553,7 +553,7 @@ def _build_run_args_str(self, job, no_batch, **run_args):
         if not batch_env_flag:
             return run_args_str
         else:
-            return "{} ARGS_FOR_SCRIPT='{}'".format(batch_env_flag, run_args_str)
+            return "{} ARGS_FOR_SCRIPT=\'{}\'".format(batch_env_flag, run_args_str)
 
     def _submit_single_job(self, case, job, dep_jobs=None, allow_fail=False,
                            no_batch=False, skip_pnl=False, mail_user=None, mail_type=None,


Test patch: