Repository: ESMCI/cime

Issue:
Run Fortran unit tests automatically as part of test suites
This issue came up today in a discussion with @mvertens @jedwards4b @jgfouca @rljacob @agsalin @mfdeakin-sandia 

We'd like to run the Fortran unit tests automatically when you run a test suite. In my mind, this applies to both the cime Fortran unit tests and any component's unit tests (currently mainly used in CLM).

At the time, we thought of having a system test that just runs the unit tests, and then you could add this system test to the test suite. But upon further reflection, I don't think that makes sense, because the current system test infrastructure assumes you have a case, with a compset, grid, etc., and that makes no sense for the unit tests.

I'm thinking it would make more sense to run these via some higher-level wrapper, which runs these unit tests along with various other tests. For cime, this could fit into scripts_regression_tests, although I'm not sure if that's the right home for it, because this has nothing to do with testing the scripts. For component models, we'd presumably have some other wrapper script specific to that component model. (e.g., for CLM, we could have a script that fires off the CLM system test suite and unit tests all at once.)


Discussion:
I agree that it's not a system test and it doesn't belong in scripts_regression_tests.   What steps currently need to be taken to run the cime fortran unit tests?  Can we update the wiki with that information?  

> I agree that it's not a system test and it doesn't belong in scripts_regression_tests. What steps currently need to be taken to run the cime fortran unit tests? Can we update the wiki with that information?

This is documented in cime/README.unit_testing. The wiki currently points to that file.

There are some outstanding issues related to making this more generic - i.e., not yellowstone-specific. e.g., https://github.com/ESMCI/cime/issues/872 and https://github.com/CESM-Development/cime/issues/12

Jim Edwards addressed this for the Fortran unit tests within cime by adding them to scripts_regression_tests (we felt this wasn't their ideal home, but would do well enough).

What remains is coming up with a general solution for running components' unit tests as part of their test suites.
Maybe take the "fake" system test approach used for HOMME?
Looking back at my original comment: At this point we have introduced a CLM-specific wrapper script for running the CLM system tests, so in principle we could run the unit tests from there, too. However, at the moment I prefer @jgfouca 's idea of using the "fake" system test approach, mainly because it leads to more consistent reporting of the test results.

This amounts to having a "system" test that runs `cime/scripts/fortran_unit_testing/run_tests.py` with appropriate settings for `--test-spec-dir` (for cime this is simply the top-level cime directory; for CLM this is the src subdirectory, etc.) and `--build-dir`. The full output from this should be captured in something like TestStatus.log.

Resolving patch:
diff --git a/scripts/lib/CIME/SystemTests/funit.py b/scripts/lib/CIME/SystemTests/funit.py
new file mode 100644
--- /dev/null
+++ b/scripts/lib/CIME/SystemTests/funit.py
@@ -0,0 +1,65 @@
+"""
+CIME FUNIT test. This class inherits from SystemTestsCommon. It runs
+the fortran unit tests; grid and compset are ignored.
+"""
+from CIME.XML.standard_module_setup import *
+from CIME.SystemTests.system_tests_common import SystemTestsCommon
+from CIME.build import post_build
+from CIME.utils import append_testlog, get_cime_root
+from CIME.test_status import *
+
+logger = logging.getLogger(__name__)
+
+class FUNIT(SystemTestsCommon):
+
+    def __init__(self, case):
+        """
+        initialize an object interface to the FUNIT system test
+        """
+        SystemTestsCommon.__init__(self, case)
+        case.load_env()
+
+    def build_phase(self, sharedlib_only=False, model_only=False):
+        if not sharedlib_only:
+            exeroot  = self._case.get_value("EXEROOT")
+            logfile = os.path.join(exeroot, "funit.bldlog")
+            with open(logfile, "w") as fd:
+                fd.write("No-op\n")
+
+            post_build(self._case, [logfile], build_complete=True)
+
+    def get_test_spec_dir(self):
+        """
+        Override this to change what gets tested.
+        """
+        return get_cime_root()
+
+    def run_phase(self):
+
+        rundir   = self._case.get_value("RUNDIR")
+        exeroot  = self._case.get_value("EXEROOT")
+        mach     = self._case.get_value("MACH")
+
+        log = os.path.join(rundir, "funit.log")
+        if os.path.exists(log):
+            os.remove(log)
+
+        test_spec_dir = self.get_test_spec_dir()
+        unit_test_tool = os.path.abspath(os.path.join(test_spec_dir,"scripts","fortran_unit_testing","run_tests.py"))
+        args = "--build-dir {} --test-spec-dir {} --machine {}".format(exeroot, test_spec_dir, mach)
+        stat = run_cmd("{} {} >& funit.log".format(unit_test_tool, args), from_dir=rundir)[0]
+
+        append_testlog(open(os.path.join(rundir, "funit.log"), "r").read())
+
+        expect(stat == 0, "RUN FAIL for FUNIT")
+
+    # Funit is a bit of an oddball test since it's not really running the E3SM model
+    # We need to override some methods to make the core infrastructure work.
+
+    def _generate_baseline(self):
+        with self._test_status:
+            self._test_status.set_status(GENERATE_PHASE, TEST_PASS_STATUS)
+
+    def _compare_baseline(self):
+        with self._test_status:
+            self._test_status.set_status(BASELINE_PHASE, TEST_PASS_STATUS)


Test patch:
diff --git a/config/config_tests.xml b/config/config_tests.xml
--- a/config/config_tests.xml
+++ b/config/config_tests.xml
@@ -352,6 +352,15 @@ NODEFAIL          Tests restart upon detected node failure. Generates fake failu
     <DOUT_S>FALSE</DOUT_S>
   </test>
 
+  <test NAME="FUNIT">
+    <DESC>Run fortran unit tests. Grid and compset (and most case settings) are ignored.</DESC>
+    <INFO_DBUG>1</INFO_DBUG>
+    <STOP_OPTION>ndays</STOP_OPTION>
+    <STOP_N>11</STOP_N>
+    <CHECK_TIMING>FALSE</CHECK_TIMING>
+    <DOUT_S>FALSE</DOUT_S>
+  </test>
+
   <test NAME="LDSTA">
     <DESC>Tests the short term archiver's last date functionality.</DESC>
     <INFO_DBUG>1</INFO_DBUG>