Repository: ESMCI/cime

Issue:
Build pFUnit on some other machines
We should build pFUnit on some other machines so we can run unit tests there:

- cheyenne
- hobart

Instructions for building pFUnit are:

1. Download pFUnit from
   http://sourceforge.net/projects/pfunit/files/latest/download

2. Set the PFUNIT environment variable. For example::

     ```
     export PFUNIT=/glade/p/cesmdata/cseg/tools/pFUnit/pFUnit3.2.8_cheyenne_Intel17.0.1_MPI_openMP
     ```

3. Build pFUnit:

```
     mkdir build
     cd build
     cmake -DMPI=YES -DOPENMP=YES ..
     make -j 4
```

4. Run pFUnit's own unit tests:

```
     make tests
```

   Note: this didn't work on yellowstone. Instead, I had to do it by hand:

```
     DAV_CORES=4 execca
     mpirun.lsf tests/tests.x
```

5. Install pFUnit on your system:

```
     make install INSTALL_DIR=$PFUNIT
```


Discussion:
There are already some pFUnit builds on hobart, in Sean Santos's home directory. I could initially test with these, but eventually want to rebuild this using our standard build arguments (as given above), put in some more central location.
On cheyenne, I built pFUnit3.2.8 with:

```
module load intel/17.0.1
module load cmake

export PFUNIT=/glade/p/cesmdata/cseg/tools/pFUnit/pFUnit3.2.8_cheyenne_Intel17.0.1_MPI_openMP
mkdir build
cd build
cmake -DMPI=YES -DOPENMP=YES -DCMAKE_INSTALL_PREFIX=$PFUNIT ..
make -j 4
make install
```

I'm currently unable to get the self-tests to run: I'm getting messages like this:

```
ctrl_connect/connect: Connection refused
```

when I try to run the self-tests.
FYI.  There is some work going on in CMDV to containerize pFUnit.
@rljacob - that sounds great, thanks for the heads-up. For now we at least need this on cheyenne (the yellowstone replacement), but that could be useful in the future.
I'm running into trouble on cheyenne. The fundamental problem is that (1) On the login nodes, I can build the unit tests, but cannot run them, but (2) On the compute nodes, I cannot build the unit tests. The unit test workflow requires building and running in the same place. I'm working with CISL to try to resolve this.
I got the pFUnit build and self-tests working on cheyenne with this procedure:

```
export PFUNIT=/glade/p/cesmdata/cseg/tools/pFUnit/pFUnit3.2.8_cheyenne_Intel17.0.1_MPI_openMP
mkdir build
cd build
cmake -DMPI=YES -DOPENMP=YES -DCMAKE_INSTALL_PREFIX=$PFUNIT -DROBUST=NO ..
make -j 4
make install
```

Then from an interactive session (with qinteractive) I was able to run `make tests`.

I'm still waiting for a resolution to the problem of how to build and run unit tests from the same place.

Resolving patch:
diff --git a/scripts/lib/CIME/BuildTools/cmakemacroswriter.py b/scripts/lib/CIME/BuildTools/cmakemacroswriter.py
--- a/scripts/lib/CIME/BuildTools/cmakemacroswriter.py
+++ b/scripts/lib/CIME/BuildTools/cmakemacroswriter.py
@@ -89,7 +89,8 @@ def set_variable(self, name, value):
         >>> s.getvalue()
         u'set(foo "bar")\\n'
         """
-        self.write_line("set(" + name + ' "' + value + '")')
+        value_transformed = self._transform_value(value)
+        self.write_line("set(" + name + ' "' + value_transformed + '")')
 
     def start_ifeq(self, left, right):
         """Write out a statement to start a conditional block.
@@ -117,3 +118,22 @@ def end_ifeq(self):
         """
         self.indent_left()
         self.write_line("endif()")
+
+    def _transform_value(self, value):
+        """Some elements need their values transformed in some way for CMake to handle them properly.
+        This method does those transformations.
+
+        Args:
+        - value (str): value of element
+
+        Returns transformed value
+        """
+
+        # Not all variables need leading & trailing whitespace removed, but some
+        # do. In particular, compiler variables (MPICC, MPICXX, MPIFC, SCC,
+        # SCXX, SFC) are only handled correctly if leading & trailing whitespace
+        # is removed. It doesn't seem to hurt to remove whitespace from all
+        # variables.
+        value_transformed = value.strip()
+
+        return value_transformed


Test patch:
diff --git a/README.unit_testing b/README.unit_testing
--- a/README.unit_testing
+++ b/README.unit_testing
@@ -1,6 +1,5 @@
-# To run all CIME unit tests on caldera, run the following command:
-# (Note that this must be done from an interactive caldera session, not from yellowstone)
-# Note also that this requires module load all-python-libs
+# To run all the CIME Fortran unit tests, run the following command:
+# On yellowstone, this requires module load all-python-libs
 #
 # The creation of a temporary directory ensures that you are doing a completely
 # clean build of the unit tests. (The use of the --clean flag to run_tests.py
diff --git a/scripts/tests/scripts_regression_tests.py b/scripts/tests/scripts_regression_tests.py
--- a/scripts/tests/scripts_regression_tests.py
+++ b/scripts/tests/scripts_regression_tests.py
@@ -181,7 +181,9 @@ def setUpClass(cls):
     def _has_unit_test_support(self):
         default_compiler = MACHINE.get_default_compiler()
         compiler = Compilers(MACHINE, compiler=default_compiler)
-        pfunit_path = compiler.get_optional_compiler_node("PFUNIT_PATH")
+        attrs = {'MPILIB': 'mpi-serial', 'compile_threaded': 'false'}
+        pfunit_path = compiler.get_optional_compiler_node("PFUNIT_PATH",
+                                                          attributes=attrs)
         if pfunit_path is None:
             return False
         else:
diff --git a/src/drivers/mct/unit_test/CMakeLists.txt b/src/drivers/mct/unit_test/CMakeLists.txt
--- a/src/drivers/mct/unit_test/CMakeLists.txt
+++ b/src/drivers/mct/unit_test/CMakeLists.txt
@@ -11,6 +11,11 @@ add_definitions(
   -DNUM_COMP_INST_ESP=1
   )
 
+# The following definitions are needed when building with the mpi-serial library
+if (USE_MPI_SERIAL)
+  add_definitions(-DNO_MPI2 -DNO_MPIMOD)
+endif()
+
 # Add source directories from stubs. This should be done first, so that in the
 # case of name collisions, the drv versions take precedence (when there are two
 # files with the same name, the one added later wins).
@@ -46,6 +51,9 @@ include_directories(${CMAKE_CURRENT_BINARY_DIR})
 # tests need all of these libraries, but it's easiest just to set the same list
 # for everyone.
 set(DRV_UNIT_TEST_LIBS drv;csm_share;esmf_wrf_timemgr;mct;mpeu)
+if (USE_MPI_SERIAL)
+  list(APPEND DRV_UNIT_TEST_LIBS mpi-serial)
+endif()
 list(APPEND DRV_UNIT_TEST_LIBS ${NETCDF_LIBRARIES})
 
 # Add the test directories
diff --git a/tools/unit_testing/Examples/circle_area/tests/CMakeLists.txt b/tools/unit_testing/Examples/circle_area/tests/CMakeLists.txt
--- a/tools/unit_testing/Examples/circle_area/tests/CMakeLists.txt
+++ b/tools/unit_testing/Examples/circle_area/tests/CMakeLists.txt
@@ -1,7 +1,9 @@
 cmake_minimum_required(VERSION 2.8)
-project(circle_area_tests Fortran C)
 
 list(APPEND CMAKE_MODULE_PATH ${CIME_CMAKE_MODULE_DIRECTORY})
+include(CIME_initial_setup)
+
+project(circle_area_tests Fortran C)
 
 include(CIME_utils)
 
diff --git a/tools/unit_testing/Examples/interpolate_1d/tests/CMakeLists.txt b/tools/unit_testing/Examples/interpolate_1d/tests/CMakeLists.txt
--- a/tools/unit_testing/Examples/interpolate_1d/tests/CMakeLists.txt
+++ b/tools/unit_testing/Examples/interpolate_1d/tests/CMakeLists.txt
@@ -1,7 +1,10 @@
 cmake_minimum_required(VERSION 2.8)
-project(interpolate_1d_tests Fortran C)
 
 list(APPEND CMAKE_MODULE_PATH ${CIME_CMAKE_MODULE_DIRECTORY})
+include(CIME_initial_setup)
+
+project(interpolate_1d_tests Fortran C)
+
 include(CIME_utils)
 
 add_subdirectory(../src interpolate_1d)
diff --git a/tools/unit_testing/run_tests.py b/tools/unit_testing/run_tests.py
--- a/tools/unit_testing/run_tests.py
+++ b/tools/unit_testing/run_tests.py
@@ -8,8 +8,9 @@
 
 from standard_script_setup import *
 from CIME.BuildTools.configure import configure
-from CIME.utils import run_cmd_no_fail, stringify_bool
+from CIME.utils import run_cmd_no_fail, stringify_bool, expect
 from CIME.XML.machines import Machines
+from CIME.XML.compilers import Compilers
 from CIME.XML.env_mach_specific import EnvMachSpecific
 from xml_test_list import TestSuiteSpec, suites_from_xml
 import subprocess
@@ -82,21 +83,24 @@ def parse_command_line(args):
                         help="""Number of processes to use for build."""
                         )
 
+    parser.add_argument("--use-mpi", action="store_true",
+                        help="""If specified, run unit tests with an mpi-enabled version
+                        of pFUnit, via mpirun. (Default is to use a serial build without
+                        mpirun.) This requires a pFUnit build with MPI support.""")
+
     parser.add_argument("--mpilib",
-    help="""MPI Library to use in build.
-    If not specified, use the default for this machine/compiler.
-    Must match an MPILIB option in config_compilers.xml.
-    e.g., for yellowstone, can use 'mpich2'."""
+                        help="""MPI Library to use in build.
+                        If not specified, use the default for this machine/compiler.
+                        Must match an MPILIB option in config_compilers.xml.
+                        e.g., for yellowstone, can use 'mpich2'.
+                        Only relevant if --use-mpi is specified."""
     )
 
-    parser.add_argument("--no-mpirun", action="store_true",
-                        help="""If specified, then run executables without an mpirun command""")
-
     parser.add_argument("--mpirun-command",
-        help="""Command to use to run an MPI executable.
-        If not specified, uses the default for this machine.
-        Ignored if --no-mpirun is specified."""
-        )
+                        help="""Command to use to run an MPI executable.
+                        If not specified, uses the default for this machine.
+                        Only relevant if --use-mpi is specified."""
+    )
     parser.add_argument(
         "--test-spec-dir", default=".",
         help="""Location where tests are specified.
@@ -115,8 +119,10 @@ def parse_command_line(args):
 override the command provided by Machines."""
     )
     parser.add_argument(
-        "--no-openmp",  action="store_true",
-        help="""Default is to include OPENMP support for tests; this option excludes openmp"""
+        "--use-openmp",  action="store_true",
+        help="""If specified, include OpenMP support for tests.
+        (Default is to run without OpenMP support.) This requires a pFUnit build with
+        OpenMP support."""
     )
     parser.add_argument(
         "--xml-test-list",
@@ -137,16 +143,19 @@ def parse_command_line(args):
 
     return output, args.build_dir, args.build_optimized, args.clean,\
         args.cmake_args, args.compiler, args.enable_genf90, args.machine, args.machines_dir,\
-        args.make_j, args.mpilib, args.no_mpirun, args.mpirun_command, args.test_spec_dir, args.ctest_args,\
-        args.no_openmp, args.xml_test_list, args.verbose
+        args.make_j, args.use_mpi, args.mpilib, args.mpirun_command, args.test_spec_dir, args.ctest_args,\
+        args.use_openmp, args.xml_test_list, args.verbose
 
 
-def cmake_stage(name, test_spec_dir, build_optimized, mpirun_command, output, cmake_args=None, clean=False, verbose=False, enable_genf90=True, color=True):
+def cmake_stage(name, test_spec_dir, build_optimized, use_mpiserial, mpirun_command, output,
+                cmake_args=None, clean=False, verbose=False, enable_genf90=True, color=True):
     """Run cmake in the current working directory.
 
     Arguments:
     name - Name for output messages.
     test_spec_dir - Test specification directory to run CMake on.
+    use_mpiserial (logical) - If True, we'll tell CMake to include mpi-serial for tests
+                              that need it
     build_optimized (logical) - If True, we'll build in optimized rather than debug mode
     """
     # Clear CMake cache.
@@ -175,6 +184,8 @@ def cmake_stage(name, test_spec_dir, build_optimized, mpirun_command, output, cm
             "-DCMAKE_BUILD_TYPE="+build_type,
             "-DPFUNIT_MPIRUN='"+mpirun_command+"'",
             ]
+        if use_mpiserial:
+            cmake_command.append("-DUSE_MPI_SERIAL=ON")
         if verbose:
             cmake_command.append("-Wdev")
 
@@ -212,6 +223,26 @@ def make_stage(name, output, make_j, clean=False, verbose=True):
 
     run_cmd_no_fail(" ".join(make_command), arg_stdout=None, arg_stderr=subprocess.STDOUT)
 
+def find_pfunit(compilerobj, mpilib, use_openmp):
+    """Find the pfunit installation we'll be using, and print its path
+
+    Aborts if necessary information cannot be found.
+
+    Args:
+    - compilerobj: Object of type Compilers
+    - mpilib: String giving the mpi library we're using
+    - use_openmp: Boolean
+    """
+    attrs = {"MPILIB": mpilib,
+             "compile_threaded": "true" if use_openmp else "false"
+             }
+
+    pfunit_path = compilerobj.get_optional_compiler_node("PFUNIT_PATH", attributes=attrs)
+    expect(pfunit_path is not None,
+           """PFUNIT_PATH not found for this machine and compiler, with MPILIB=%s and compile_threaded=%s.
+You must specify PFUNIT_PATH in config_compilers.xml, with attributes MPILIB and compile_threaded."""%(mpilib, attrs['compile_threaded']))
+    logger.info("Using PFUNIT_PATH: %s"%pfunit_path.text)
+
 #=================================================
 # Iterate over input suite specs, building the tests.
 #=================================================
@@ -220,12 +251,10 @@ def make_stage(name, output, make_j, clean=False, verbose=True):
 def _main():
     output, build_dir, build_optimized, clean,\
         cmake_args, compiler, enable_genf90, machine, machines_dir,\
-        make_j, mpilib, no_mpirun, mpirun_command, test_spec_dir, ctest_args,\
-        no_openmp, xml_test_list, verbose \
+        make_j, use_mpi, mpilib, mpirun_command, test_spec_dir, ctest_args,\
+        use_openmp, xml_test_list, verbose \
         = parse_command_line(sys.argv)
 
-    use_openmp = not no_openmp
-
 #=================================================
 # Find directory and file paths.
 #=================================================
@@ -253,7 +282,7 @@ def _main():
     else:
         machobj = Machines(machine=machine)
 
-# Create build directory if necessary.
+    # Create build directory if necessary.
     build_dir = os.path.abspath(build_dir)
 
     if not os.path.isdir(build_dir):
@@ -262,15 +291,23 @@ def _main():
     # Switch to the build directory.
     os.chdir(build_dir)
 
-#=================================================
-# Functions to perform various stages of build.
-#=================================================
-    if mpilib is None:
+    #=================================================
+    # Functions to perform various stages of build.
+    #=================================================
+
+    if not use_mpi:
+        mpilib = "mpi-serial"
+    elif mpilib is None:
         mpilib = machobj.get_default_MPIlib()
-        logger.warn("Using mpilib: %s"%mpilib)
+        logger.info("Using mpilib: %s"%mpilib)
+
     if compiler is None:
         compiler = machobj.get_default_compiler()
-        logger.warn("Compiler is %s"%compiler)
+        logger.info("Compiler is %s"%compiler)
+
+    compilerobj = Compilers(machobj, compiler=compiler, mpilib=mpilib)
+
+    find_pfunit(compilerobj, mpilib=mpilib, use_openmp=use_openmp)
 
     debug = not build_optimized
     os_ = machobj.get_value("OS")
@@ -289,11 +326,17 @@ def _main():
     os.environ["MPILIB"] = mpilib
     if use_openmp:
         os.environ["compile_threaded"] = "true"
-    os.environ["CC"] = find_executable("mpicc")
-    os.environ["FC"] = find_executable("mpif90")
+    else:
+        os.environ["compile_threaded"] = "false"
+
     os.environ["UNIT_TEST_HOST"] = socket.gethostname()
+    if "NETCDF_PATH" in os.environ and not "NETCDF" in os.environ:
+        # The CMake Netcdf find utility that we use (from pio2) seems to key off
+        # of the environment variable NETCDF, but not NETCDF_PATH
+        logger.info("Setting NETCDF environment variable: %s"%os.environ["NETCDF_PATH"])
+        os.environ["NETCDF"] = os.environ["NETCDF_PATH"]
 
-    if no_mpirun:
+    if not use_mpi:
         mpirun_command = ""
     elif mpirun_command is None:
         mpi_attribs = {
@@ -306,7 +349,7 @@ def _main():
         # We can get away with specifying case=None since we're using exe_only=True
         mpirun_command, _ = machspecific.get_mpirun(case=None, attribs=mpi_attribs, exe_only=True)
         mpirun_command = machspecific.get_resolved_value(mpirun_command)
-        logger.warn("mpirun command is '%s'"%mpirun_command)
+        logger.info("mpirun command is '%s'"%mpirun_command)
 
 #=================================================
 # Run tests.
@@ -332,7 +375,8 @@ def _main():
 
             if not os.path.islink("Macros.cmake"):
                 os.symlink(os.path.join(build_dir,"Macros.cmake"), "Macros.cmake")
-            cmake_stage(name, directory, build_optimized, mpirun_command, output, verbose=verbose,
+            use_mpiserial = not use_mpi
+            cmake_stage(name, directory, build_optimized, use_mpiserial, mpirun_command, output, verbose=verbose,
                         enable_genf90=enable_genf90, cmake_args=cmake_args)
             make_stage(name, output, make_j, clean=clean, verbose=verbose)