Repository: ESMCI/cime

Issue:
Build cprnc as part of create_test
It seems like the python implementation of cprnc (#1031 ) has stalled out (due to no fault of @mfdeakin-sandia 's -- we just hit some roadblocks with this, particularly related to netcdf dependencies). It occurred to me: The primary motivation for that python implementation was to avoid needing to maintain a build of cprnc on all of the systems we test on (users need to pre-build this on their system, we need to keep this up-to-date... which we're often not very good about... etc.).

Now that the cprnc build is more robust, can we simply build cprnc as part of each `create_test` instantiation? I feel that would make the Fortran version of cprnc a sustainable solution.


Discussion:
@billsacks , that's a pretty good idea. What about users who want to use cprnc manually?
I'm not sure if it is, I think that we have some machines/compiler combinations which cprnc doesn't work with and we have a default build of cprnc on those systems that work for all compilers.   I also don't like the added compiication of users having to debug cprnc every time they get a test failure.  How about doing this in create_test iff a default cprnc is not defined in config_machines.xml?
The bigger problem was you needed a build of cprnc for each fortran-netcdf combo on a system.  That's solved by having cprnc be a script that sets the environment for and calls just one combo.
I agree with Jim. The standard build of cprnc should be used unless there isn't one available and then cprnc should be built. Note, that initial versions of CAM testing had as part of it a build of cprnc. This added complexity to the system, if the cprnc build failed, and people didn't know what to do.. And that happened often enough that it was an issue. So there was a purposeful choice to move to a system that didn't rebuild cprnc every time for standard systems. For systems that don't have cprnc built -- I do see adding the capability of building it as part of the system to make it easier to test on new systems. But, for standard systems I think it's better to have a version that's already built for everyone.
I don't feel strongly about this, so I don't want to spend too much energy arguing my point, but I do want to take one more stab at explaining myself.

First, the motivation: I realized that cprnc on hobart hadn't been updated in a long time, and so was marking tests as passed that should have failed (due to a change in dimension sizes of variables). This is definitely not the first time I have had something like this happen. I thought it would be nice if you could rely on the cprnc behavior being stable across machines.

Regarding @jgfouca 's point

> What about users who want to use cprnc manually?

I think this is a very good and important point: I do this a ton myself. I'd still imagine providing a cprnc build somewhere on the system, but automated tests wouldn't use that one.

Regarding @jedwards4b 's points:

> I think that we have some machines/compiler combinations which cprnc doesn't work with and we have a default build of cprnc on those systems that work for all compilers

I imagined we would only build cprnc once for a given create_test invocation (i.e., once per test suite, not once per test), using the default compiler on that machine.

> I also don't like the added compiication of users having to debug cprnc every time they get a test failure. 

Sorry, I don't understand this point.

@rljacob - Sorry, I can't figure out if you're supporting my suggestion or saying it's unnecessary.

@ekluzek I think we can deal with this problem much better now:

> if the cprnc build failed, and people didn't know what to do.. And that happened often enough that it was an issue

We can put a build of cprnc in scripts_regression_tests (it only takes about 5 seconds) to ensure that it will always work on our standard test machines.
@billsacks yeah, I would add a build of cprnc to scripts_regression_tests as it should have already been in there. Then it would be good to hear that a lot of people are running it and not finding any problems. That would be a good test case to show that adding it to create_test would be fine.

There is a valid point that if cprnc isn't built as part of create_test that the version being used will be out of date. And if there is an important feature missing in a new version of cprnc, that can be a problem with create_test. Essentially the current mechanism means that if you update cprnc -- you should create an issue that cprnc should be updated on standard machines. I don't think that step is happening now. And the builds of cprnc are probably happening sporadically. But, better tracking of when the builds are being done could show how off they are and indicate if a build as part of create_test would be better.

The last part of this though is how often does cprnc change in such a way that create_test needs the newer version? I suspect not so often...
> The last part of this though is how often does cprnc change in such a way that create_test needs the newer version? I suspect not so often...

But which is the cause and which is the effect? There are a few changes I've been wanting to make to cprnc for a while, but haven't because of (1) uncertainty about whether we are sticking with the fortran or moving to python, and (2) the pain involved in making any update, particularly one that requires coordination between cprnc code and scripts (as one of my changes does) -- since this involves synchronization across all cesm and e3sm test platforms.
From today's cime call: Feeling is that it could make sense to tackle this. I'll plan to take it at some point.

The main reason not to do this is if we'd be moving ahead with the python version of cprnc. @mfdeakin-sandia was close to having this working, though still needed to do more testing. But this would require having an installation of python's netcdf4 on the system. (See also #1031.) If we started building cprnc as part of the test system, as suggested in the current issue, then I wouldn't see much need for having a python version myself, and there are downsides to the python version, like performance and requiring a netcdf4-python installation. The main upside I see to the python version is that it allows tighter integration between cprnc and the rest of the test system. (I originally thought it would be easier to add new functionality in the python version, but I'm not sure that's really true: you have to be very careful about how you write things in order to maintain decent performance, so it can actually be harder in some ways.) We may wait to make a final call until we've had more discussions about what netcdf python library (if any) we'll allow as a cime dependency (we want something for ctsm, so cime can write netcdf parameter files, but the scipy netcdf would probably be sufficient there: we probably don't need netcdf4 support).
Incorrect closing due to automatic github detection of something from e3sm.

Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -247,6 +247,12 @@ def _build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, lid
     if uses_kokkos(case):
         libs.append("kokkos")
 
+    # Check if we need to build our own cprnc
+    if case.get_value("TEST"):
+        cprnc_loc = case.get_value("CCSM_CPRNC")
+        if not cprnc_loc or not os.path.exists(cprnc_loc):
+            libs.append("cprnc")
+
     logs = []
     sharedlibroot = os.path.abspath(case.get_value("SHAREDLIBROOT"))
     for lib in libs:
@@ -258,6 +264,9 @@ def _build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, lid
             full_lib_path = os.path.join(sharedlibroot, sharedpath)
         elif lib == "mpi-serial":
             full_lib_path = os.path.join(sharedlibroot, sharedpath, "mct", lib)
+        elif lib == "cprnc":
+            full_lib_path = os.path.join(sharedlibroot, compiler, "cprnc")
+            case.set_value("CCSM_CPRNC", os.path.join(full_lib_path, "cprnc"))
         else:
             full_lib_path = os.path.join(sharedlibroot, sharedpath, lib)
         # pio build creates its own directory


Test patch:
diff --git a/scripts/tests/scripts_regression_tests.py b/scripts/tests/scripts_regression_tests.py
--- a/scripts/tests/scripts_regression_tests.py
+++ b/scripts/tests/scripts_regression_tests.py
@@ -1157,9 +1157,9 @@ def _create_test(self, extra_args, test_id=None, pre_run_errors=False, run_error
             self._wait_for_tests(test_id, expect_works=(not pre_run_errors and not run_errors))
 
     ###########################################################################
-    def _wait_for_tests(self, test_id, expect_works=True):
+    def _wait_for_tests(self, test_id, expect_works=True, always_wait=False):
     ###########################################################################
-        if self._hasbatch:
+        if self._hasbatch or always_wait:
             timeout_arg = "--timeout={}".format(GLOBAL_TIMEOUT) if GLOBAL_TIMEOUT is not None else ""
             expected_stat = 0 if expect_works else CIME.utils.TESTS_FAILED_ERR_CODE
             run_cmd_assert_result(self, "{}/wait_for_tests {} *{}/TestStatus".format(TOOLS_DIR, timeout_arg, test_id),
@@ -2408,6 +2408,21 @@ def test_configure(self):
 
         self.assertEqual(case_env_contents, man_env_contents)
 
+    ###########################################################################
+    def test_self_build_cprnc(self):
+    ###########################################################################
+        self._create_test(["ERS.f19_g16_rx1.A", "--no-build"], test_id=self._baseline_name)
+
+        casedir = os.path.join(self._testroot,
+                               "{}.{}".format(CIME.utils.get_full_test_name("ERS.f19_g16_rx1.A", machine=self._machine, compiler=self._compiler), self._baseline_name))
+
+        run_cmd_assert_result(self, "./xmlchange CCSM_CPRNC=this_is_a_broken_cprnc",
+                              from_dir=casedir)
+        run_cmd_assert_result(self, "./case.build", from_dir=casedir)
+        run_cmd_assert_result(self, "./case.submit", from_dir=casedir)
+
+        self._wait_for_tests(self._baseline_name, always_wait=True)
+
 ###############################################################################
 class X_TestSingleSubmit(TestCreateTestCommon):
 ###############################################################################