Repository: ESMCI/cime

Issue:
Some fake tests fail on machines where ccsm_cprnc isn't defined
I tried doing #3419 (undefined ccsm_cprnc on CESM's main testing machines), but encountered failures in two tests run from scripts_regression_tests:

<details>
  <summary>Click to see scripts_regression_tests output</summary>

```
======================================================================
FAIL: test_bless_test_results (__main__.Q_TestBlessTestResults)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./scripts_regression_tests.py", line 1779, in test_bless_test_results
    self._create_test(compargs)
  File "./scripts_regression_tests.py", line 1269, in _create_test
    expected_stat=expected_stat)
  File "./scripts_regression_tests.py", line 71, in run_cmd_assert_result
    test_obj.assertEqual(stat, expected_stat, msg=msg)
AssertionError:
COMMAND:  /glade/work/sacks/cime/scripts/create_test -c fake_testing_only_20210208_131340 TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A --baseline-root  /glade/scratch/sacks/scripts_regression_test.20210208_124239/baselines -t fake_testing_only_20210208_131340-20210208_131400 --
baseline-root /glade/scratch/sacks/scripts_regression_test.20210208_124239/baselines --test-root=/glade/scratch/sacks/scripts_regression_test.20210208_124239 --output-root=/glade/scratch/sacks/scripts_regression_test.20210208_124239 --wait
FROM_DIR: /glade/work/sacks/cime/scripts/tests
SHOULD HAVE WORKED, INSTEAD GOT STAT 100
OUTPUT: Testnames: ['TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel']
Using project from env PROJECT: P93300606
create_test will do up to 1 tasks simultaneously
create_test will use up to 45 cores simultaneously
Creating test directory /glade/scratch/sacks/scripts_regression_test.20210208_124239/TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel.C.fake_testing_only_20210208_131340-20210208_131400
RUNNING TESTS:
  TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel
Starting CREATE_NEWCASE for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel with 1 procs
Finished CREATE_NEWCASE for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel in 1.402558 seconds (PASS)
Starting XML for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel with 1 procs
Finished XML for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel in 0.280539 seconds (PASS)
Starting SETUP for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel with 1 procs
Finished SETUP for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel in 2.669576 seconds (PASS)
Starting SHAREDLIB_BUILD for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel with 1 procs
Finished SHAREDLIB_BUILD for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel in 0.266993 seconds (PASS)
Starting MODEL_BUILD for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel with 4 procs
Finished MODEL_BUILD for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel in 0.281374 seconds (PASS)
Starting RUN for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel with 1 proc on interactive node and 1 procs on compute nodes
Finished RUN for test TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel in 1.538014 seconds (PEND). [COMPLETED 1 of 1]
Waiting for tests to finish
DIFF TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel (phase BASELINE)
    Case dir: /glade/scratch/sacks/scripts_regression_test.20210208_124239/TESTRUNDIFFRESUBMIT_P1.f19_g16_rx1.A.cheyenne_intel.C.fake_testing_only_20210208_131340-20210208_131400
test-scheduler took 19.2259368896 seconds
ERRPUT:




======================================================================
FAIL: test_rebless_namelist (__main__.Q_TestBlessTestResults)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./scripts_regression_tests.py", line 1824, in test_rebless_namelist
    cases = self._create_test(compargs, test_id=test_id)
  File "./scripts_regression_tests.py", line 1269, in _create_test
    expected_stat=expected_stat)
  File "./scripts_regression_tests.py", line 71, in run_cmd_assert_result
    test_obj.assertEqual(stat, expected_stat, msg=msg)
AssertionError:
COMMAND:  /glade/work/sacks/cime/scripts/create_test -c fake_testing_only_20210208_131419 cime_test_only_pass -t fake_testing_only_20210208_131419-20210208_131433 --baseline-root /glade/scratch/sacks/scripts_regression_test.20210208_124239/baselines --test-root=/glade/s
cratch/sacks/scripts_regression_test.20210208_124239 --output-root=/glade/scratch/sacks/scripts_regression_test.20210208_124239 --wait
FROM_DIR: /glade/work/sacks/cime/scripts/tests
SHOULD HAVE WORKED, INSTEAD GOT STAT 100
OUTPUT: Testnames: ['TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel', 'TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel', 'TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel']
Using project from env PROJECT: P93300606
create_test will do up to 3 tasks simultaneously
create_test will use up to 45 cores simultaneously
Creating test directory /glade/scratch/sacks/scripts_regression_test.20210208_124239/TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel.C.fake_testing_only_20210208_131419-20210208_131433
Creating test directory /glade/scratch/sacks/scripts_regression_test.20210208_124239/TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel.C.fake_testing_only_20210208_131419-20210208_131433
Creating test directory /glade/scratch/sacks/scripts_regression_test.20210208_124239/TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel.C.fake_testing_only_20210208_131419-20210208_131433
RUNNING TESTS:
  TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel
  TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel
  TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel
Starting CREATE_NEWCASE for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel with 1 procs
Starting CREATE_NEWCASE for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel with 1 procs
Starting CREATE_NEWCASE for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel with 1 procs
Finished CREATE_NEWCASE for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel in 1.603687 seconds (PASS)
Finished CREATE_NEWCASE for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel in 1.608365 seconds (PASS)
Finished CREATE_NEWCASE for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel in 1.608246 seconds (PASS)
Starting XML for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel with 1 procs
Starting XML for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel with 1 procs
Starting XML for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel with 1 procs
Finished XML for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel in 0.612801 seconds (PASS)
Finished XML for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel in 0.637148 seconds (PASS)
Finished XML for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel in 0.640477 seconds (PASS)
Starting SETUP for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel with 1 procs
Starting SETUP for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel with 1 procs
Starting SETUP for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel with 1 procs
Finished SETUP for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel in 2.773067 seconds (PASS)
Starting SHAREDLIB_BUILD for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel with 1 procs
Finished SETUP for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel in 2.805390 seconds (PASS)
Finished SETUP for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel in 2.822763 seconds (PASS)
Finished SHAREDLIB_BUILD for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel in 0.278168 seconds (PASS)
Starting MODEL_BUILD for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel with 4 procs
Starting SHAREDLIB_BUILD for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel with 1 procs
Finished SHAREDLIB_BUILD for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel in 0.285211 seconds (PASS)
Finished MODEL_BUILD for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel in 0.285831 seconds (PASS)
Starting RUN for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel with 1 proc on interactive node and 1 procs on compute nodes
Starting MODEL_BUILD for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel with 4 procs
Starting SHAREDLIB_BUILD for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel with 1 procs
Finished SHAREDLIB_BUILD for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel in 0.295563 seconds (PASS)
Finished MODEL_BUILD for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel in 0.296318 seconds (PASS)
Starting RUN for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel with 1 proc on interactive node and 1 procs on compute nodes
Starting MODEL_BUILD for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel with 4 procs
Finished MODEL_BUILD for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel in 0.295311 seconds (PASS)
Starting RUN for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel with 1 proc on interactive node and 1 procs on compute nodes
Finished RUN for test TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel in 1.582058 seconds (PEND). [COMPLETED 1 of 3]
Finished RUN for test TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel in 4.679715 seconds (PEND). [COMPLETED 2 of 3]
Finished RUN for test TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel in 5.125933 seconds (PEND). [COMPLETED 3 of 3]
Waiting for tests to finish
DIFF TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel (phase BASELINE)
    Case dir: /glade/scratch/sacks/scripts_regression_test.20210208_124239/TESTRUNPASS_P1.f19_g16_rx1.A.cheyenne_intel.C.fake_testing_only_20210208_131419-20210208_131433
DIFF TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel (phase BASELINE)
    Case dir: /glade/scratch/sacks/scripts_regression_test.20210208_124239/TESTRUNPASS_P1.f45_g37_rx1.A.cheyenne_intel.C.fake_testing_only_20210208_131419-20210208_131433
DIFF TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel (phase BASELINE)
    Case dir: /glade/scratch/sacks/scripts_regression_test.20210208_124239/TESTRUNPASS_P1.ne30_g16_rx1.A.cheyenne_intel.C.fake_testing_only_20210208_131419-20210208_131433
test-scheduler took 18.0713598728 seconds
ERRPUT:

```
</details>

Digging into the cprnc output files, it looks like the problem is that CCSM_CPRNC doesn't get defined for these tests, leading to the scripts trying to run an empty command for cprnc, giving this output:

```
/bin/sh: -m: command not found
```

(because I think it tries to run cprnc -m, but cprnc isn't defined).

It looks like the problem is that these fake tests, and possibly others, override the build method, so that the build doesn't do anything. But the logic for building cprnc on the fly and changing CCSM_CPRNC to point to that build is in build.py: `_build_libraries`. So it seems like, if we want to support running scripts_regression_tests on machines without CCSM_CPRNC defined, then these fake tests need some extra logic to handle the cprnc build.


Discussion:
But is the cprnc built individually for each test in scripts_regression_tests or, for that matter, each test in a test list?
I think that we need to figure out how to build it once and use it for all the tests. 
@jedwards4b I have addressed this question (the best I can determine) in #3854 . Let's move broader discussion on how to proceed in CESM to that issue.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
I don't have strong opinions on whether this should be addressed; do others?

Resolving patch:
diff --git a/CIME/hist_utils.py b/CIME/hist_utils.py
--- a/CIME/hist_utils.py
+++ b/CIME/hist_utils.py
@@ -326,20 +326,30 @@ def _compare_hists(
             if not ".nc" in hist1:
                 logger.info("Ignoring non-netcdf file {}".format(hist1))
                 continue
-            success, cprnc_log_file, cprnc_comment = cprnc(
-                model,
-                os.path.join(from_dir1, hist1),
-                os.path.join(from_dir2, hist2),
-                case,
-                from_dir1,
-                multiinst_driver_compare=multiinst_driver_compare,
-                outfile_suffix=outfile_suffix,
-                ignore_fieldlist_diffs=ignore_fieldlist_diffs,
-            )
+            try:
+                success, cprnc_log_file, cprnc_comment = cprnc(
+                    model,
+                    os.path.join(from_dir1, hist1),
+                    os.path.join(from_dir2, hist2),
+                    case,
+                    from_dir1,
+                    multiinst_driver_compare=multiinst_driver_compare,
+                    outfile_suffix=outfile_suffix,
+                    ignore_fieldlist_diffs=ignore_fieldlist_diffs,
+                )
+            except:
+                cprnc_comment = "CPRNC executable not found"
+                cprnc_log_file = None
+                success = False
+
             if success:
                 comments += "    {} matched {}\n".format(hist1, hist2)
             else:
-                if cprnc_comment == CPRNC_FIELDLISTS_DIFFER:
+                if not cprnc_log_file:
+                    comments += cprnc_comment
+                    all_success = False
+                    return all_success, comments, 0
+                elif cprnc_comment == CPRNC_FIELDLISTS_DIFFER:
                     comments += "    {} {} {}\n".format(hist1, FIELDLISTS_DIFFER, hist2)
                 else:
                     comments += "    {} {} {}\n".format(hist1, DIFF_COMMENT, hist2)
@@ -428,6 +438,11 @@ def cprnc(
     """
     if not cprnc_exe:
         cprnc_exe = case.get_value("CCSM_CPRNC")
+    expect(
+        os.path.isfile(cprnc_exe) and os.access(cprnc_exe, os.X_OK),
+        f"cprnc {cprnc_exe} does not exist or is not executable",
+    )
+
     basename = os.path.basename(file1)
     multiinst_regex = re.compile(r".*%s[^_]*(_[0-9]{4})[.]h.?[.][^.]+?[.]nc" % model)
     mstr = ""


Test patch:
diff --git a/CIME/tests/base.py b/CIME/tests/base.py
--- a/CIME/tests/base.py
+++ b/CIME/tests/base.py
@@ -61,7 +61,7 @@ def setUp(self):
         self._hasbatch = self.MACHINE.has_batch_system() and not self.NO_BATCH
         self._do_teardown = not self.NO_TEARDOWN
         self._root_dir = os.getcwd()
-
+        self._cprnc = self.MACHINE.get_value("CCSM_CPRNC")
         customize_path = os.path.join(utils.get_src_root(), "cime_config", "customize")
         self._config = Config.load(customize_path)
 
diff --git a/CIME/tests/test_sys_bless_tests_results.py b/CIME/tests/test_sys_bless_tests_results.py
--- a/CIME/tests/test_sys_bless_tests_results.py
+++ b/CIME/tests/test_sys_bless_tests_results.py
@@ -17,6 +17,10 @@ def setUp(self):
         # recording baselines are working
         restrictive_mask = 0o027
         self._orig_umask = os.umask(restrictive_mask)
+        if not self._cprnc:
+            self.skipTest(
+                "Test cannot run without cprnc program defined in config_machines.xml"
+            )
 
     def tearDown(self):
         super().tearDown()
diff --git a/doc/source/users_guide/testing.rst b/doc/source/users_guide/testing.rst
--- a/doc/source/users_guide/testing.rst
+++ b/doc/source/users_guide/testing.rst
@@ -459,6 +459,8 @@ CIME's scripts regression tests
 for the stand-alone CIME testing. With no arguments, it will run the full suite. You can limit testing to a specific
 test class or even a specific test within a test class.
 
+**CCSM_CPRNC should be defined in config_machines.xml for your system prior to running scripts_regression_tests.py**
+
 Run full suite::
 
   python scripts/lib/CIME/tests/scripts_regression_tests.py