Repository: ESMCI/cime

Issue:
compile_threaded issue on anlworkstation
In config/e3sm/machines/config_compilers.xml, -fopenmp is appended only when  compile_threaded
is set to true:
```
<compiler COMPILER="gnu">
  <CFLAGS>
    ...
    <append compile_threaded="true"> -fopenmp </append>
    ...
  </CFLAGS>
  ...
  <FFLAGS>
    ...
    <append compile_threaded="true"> -fopenmp </append>
    ...
  </FFLAGS>
  ...
  <LDFLAGS>
    <append compile_threaded="true"> -fopenmp </append>
  </LDFLAGS>
  ...
</compiler>
```
When cime_developer test suite is run on anlworkstation,
compile_threaded seems to be false by default. This has
caused two tests to fail during MODEL_BUILD: 
ERP.f45_g37_rx1.A
PET_P4.f19_f19.A

Errors were:
```
        perf_mod.F90:(.text+0x7): undefined reference to `omp_get_max_threads_'
    
        ...
        
        gptl.c:(.text+0x455): undefined reference to `omp_get_thread_num'

        ...
```
Should CIME set compile_threaded to true by default?
CIME should also fix the link errors even when compile_threaded is set to false


Discussion:
This seems to indicate that the flags to the gptl build are incorrect - can you post the gptl build log?

Here is the gptl build log:
```
make -f /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/Makefile install -C /scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/gnu/mpich/nodebug/threads/mct/gptl MACFILE=/scratch/autotest/cime_pio1_scratch/test_cime_pio1/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/case2/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/Macros.make MODEL=gptl GPTL_DIR=/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing GPTL_LIBDIR=/scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/gnu/mpich/nodebug/threads/mct/gptl SHAREDPATH=/scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/gnu/mpich/nodebug/threads/mct CIME_MODEL=e3sm CASEROOT="/scratch/autotest/cime_pio1_scratch/test_cime_pio1/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/case2/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz" CASETOOLS="/scratch/autotest/cime_pio1_scratch/test_cime_pio1/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/case2/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/Tools" CIMEROOT="/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test" COMP_INTERFACE="mct" COMPILER="gnu" DEBUG="FALSE" EXEROOT="/scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld" INCROOT="/scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/lib/include" LIBROOT="/scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/lib" MACH="anlworkstation" MPILIB="mpich" NINST_VALUE="c1a1l1i1o1r1g1w1e1" OS="LINUX" PIO_VERSION="1" SHAREDLIBROOT="/scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld" SMP_PRESENT="TRUE" USE_ESMF_LIB="FALSE" USE_MOAB="FALSE" COMPARE_TO_NUOPC="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE"   compile_threaded=true 
make: Entering directory `/scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/gnu/mpich/nodebug/threads/mct/gptl'
mpicc   -c -I/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing  -mcmodel=medium   -fopenmp   -O   -O2     -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DMCT_INTERFACE -DHAVE_MPI /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/gptl.c
mpicc   -c -I/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing  -mcmodel=medium   -fopenmp   -O   -O2     -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DMCT_INTERFACE -DHAVE_MPI /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/GPTLutil.c
mpicc   -c -I/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing  -mcmodel=medium   -fopenmp   -O   -O2     -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DMCT_INTERFACE -DHAVE_MPI /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/GPTLget_memusage.c
mpicc   -c -I/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing  -mcmodel=medium   -fopenmp   -O   -O2     -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DMCT_INTERFACE -DHAVE_MPI /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/GPTLprint_memusage.c
mpicc   -c -I/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing  -mcmodel=medium   -fopenmp   -O   -O2     -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DMCT_INTERFACE -DHAVE_MPI /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/gptl_papi.c
mpicc   -c -I/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing  -mcmodel=medium   -fopenmp   -O   -O2     -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DMCT_INTERFACE -DHAVE_MPI /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/f_wrappers.c
mpif90  -c -I/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing  -mcmodel=medium -fconvert=big-endian -ffree-line-length-none -ffixed-line-length-none   -fopenmp   -O   -O2     -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DMCT_INTERFACE -DHAVE_MPI -ffree-form   /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/perf_utils.F90
mpif90  -c -I/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing  -mcmodel=medium -fconvert=big-endian -ffree-line-length-none -ffixed-line-length-none   -fopenmp   -O   -O2     -DFORTRANUNDERSCORE -DNO_R16 -DCPRGNU -DMCT_INTERFACE -DHAVE_MPI -ffree-form   /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/perf_mod.F90
ar rv libgptl.a gptl.o GPTLutil.o GPTLget_memusage.o GPTLprint_memusage.o gptl_papi.o f_wrappers.o perf_mod.o perf_utils.o
a - gptl.o
a - GPTLutil.o
a - GPTLget_memusage.o
a - GPTLprint_memusage.o
a - gptl_papi.o
a - f_wrappers.o
a - perf_mod.o
a - perf_utils.o
cp -p /scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/gptl.h /scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/gnu/mpich/nodebug/threads/mct/include
cp -p *.mod /scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/gnu/mpich/nodebug/threads/mct/include
cp -p libgptl.a /scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/gnu/mpich/nodebug/threads/mct/lib
make: Leaving directory `/scratch/autotest/cime_pio1_scratch/acme/scratch/ERP.f45_g37_rx1.A.anlworkstation_gnu.20190312_090614_lzdltz/bld/case2bld/gnu/mpich/nodebug/threads/mct/gptl'
/scratch/autotest/cime_pio1_scratch/CIME_PIO1_Test/src/share/timing/perf_utils.F90:21:7:

    save
       1
Warning: Legacy Extension: Blanket SAVE statement at (1) follows previous SAVE statement
ar: creating libgptl.a
```
One more question - what cime revision is this? (head of master?)
Assuming it is head of master, I've reproduced the problem and will fix.
Yes, it is head of master. Please go ahead to provide a fix, many thanks.

Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -18,6 +18,7 @@ def get_standard_makefile_args(case):
                  "OCN_SUBMODEL", "CISM_USE_TRILINOS", "USE_ALBANY", "USE_PETSC"]
 
     make_args = "CIME_MODEL={} ".format(case.get_value("MODEL"))
+    make_args += " compile_threaded={} ".format(case.get_build_threaded())
     for var in variables:
         make_args+=xml_to_make_variable(case, var)
 
diff --git a/scripts/lib/CIME/case/case.py b/scripts/lib/CIME/case/case.py
--- a/scripts/lib/CIME/case/case.py
+++ b/scripts/lib/CIME/case/case.py
@@ -1328,7 +1328,7 @@ def get_build_threaded(self):
         Returns True if current settings require a threaded build/run.
         """
         force_threaded = self.get_value("FORCE_BUILD_SMP")
-        smp_present = bool(force_threaded) or self.thread_count > 1
+        smp_present = force_threaded or self.thread_count > 1
         return smp_present
 
     def _check_testlists(self, compset_alias, grid_name, files):


Test patch: