Repository: ESMCI/cime

Issue:
ERP test is broken 
When I try to create an ERP test for CESM with the latest master I get the following error in the build:

```
2017-04-29 17:13:15: Exception during build:
ERROR: 
ERROR, SMP VALUES HAVE CHANGED
  SMP_BUILD = c1a1l1i1o1r1g1w1e1
  SMP_VALUE = c1a1l1i1o1r1g1w1e1
  smpstr = c0a0l0i0o0r0g0w0e0
  A manual clean of your obj directories is strongly recommended
  You should execute the following:
    ./case.build --clean
  Then rerun the build script interactively
  ---- OR ----
  You can override this error message at your own risk by executing:
    ./xmlchange -file env_build.xml -id SMP_BUILD -val 0
  Then rerun the build script interactively

```
I have narrowed down the problem to the following hash: 

`670cf97 Merge remote-tracking branch 'origin/master' into jgfouca/new_query_tools 

I believe the problem is related to the fact that when we are halving the threads - (which in the case of yellowstone goes from 2->1) the build-threaded option should NOT change. That was always the intent of this test. You want to keep the threading compilation the same. 

@jedwards noticed yesterday that this test was working on cheyenne - but this is a red herring since on cheyenne the threads are set to 1 in the initial pe-layout and so halving them keeps the threads at 1. On yellowstone the threads start at 2- so halving them gives the bad behavior we are seeing.

This is now a critical issue in our testing for the CESM2 release.


Discussion:
@jedwards - please look at the above - I think I have isolated at least the cause of the problem. 
It appears that now the ERP test is failing in the scripts_regression_tests.

FAIL ERP.f45_g37_rx1.A.yellowstone_intel (phase MODEL_BUILD)
 2017-04-30 19:50:17: Exception during build:
ERROR: ERROR: ocn.buildlib failed, cat /glade/scratch/mvertens/scripts_regression_test.20170430_190847/ERP.f45_g37_rx1.A.yellowstone_intel.fake_testing_only_20170430_194610/bld/ocn.bldlog.170430-195015
ERROR: ice.buildlib failed, cat /glade/scratch/mvertens/scripts_regression_test.20170430_190847/ERP.f45_g37_rx1.A.yellowstone_intel.fake_testing_only_20170430_194610/bld/ice.bldlog.170430-195015
ERROR: rof.buildlib failed, cat /glade/scratch/mvertens/scripts_regression_test.20170430_190847/ERP.f45_g37_rx1.A.yellowstone_intel.fake_testing_only_20170430_194610/bld/rof.bldlog.170430-195015
ERROR: lnd.buildlib failed, cat /glade/scratch/mvertens/scripts_regression_test.20170430_190847/ERP.f45_g37_rx1.A.yellowstone_intel.fake_testing_only_20170430_194610/bld/lnd.bldlog.170430-195015
ERROR: wav.buildlib failed, cat /glade/scratch/mvertens/scripts_regression_test.20170430_190847/ERP.f45_g37_rx1.A.yellowstone_intel.fake_testing_only_20170430_194610/bld/wav.bldlog.170430-195015
ERROR: atm.buildlib failed, cat /glade/scratch/mvertens/scripts_regression_test.20170430_190847/ERP.f45_g37_rx1.A.yellowstone_intel.fake_testing_only_20170430_194610/bld/atm.bldlog.170430-195015
ERROR: esp.buildlib failed, cat /glade/scratch/mvertens/scripts_regression_test.20170430_190847/ERP.f45_g37_rx1.A.yellowstone_intel.fake_testing_only_20170430_194610/bld/esp.bldlog.170430-195015
ERROR: glc.buildlib failed, cat /glade/scratch/mvertens/scripts_regression_test.20170430_190847/ERP.f45_g37_rx1.A.yellowstone_intel.fake_testing_only_20170430_194610/bld/glc.bldlog.170430-195015
Traceback (most recent call last):
  File "/glade/p/work/mvertens/cesm2_0_alpha06j.ndep/cime/scripts/Tools/../../scripts/lib/CIME/SystemTests/system_tests_common.py", line 87, in build
    model_only=(phase_name==MODEL_BUILD_PHASE))
  File "/glade/p/work/mvertens/cesm2_0_alpha06j.ndep/cime/scripts/Tools/../../scripts/lib/CIME/SystemTests/erp.py", line 83, in build_phase
    self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
  File "/glade/p/work/mvertens/cesm2_0_alpha06j.ndep/cime/scripts/Tools/../../scripts/lib/CIME/SystemTests/system_tests_common.py", line 118, in build_indv
    sharedlib_only=sharedlib_only, model_only=model_only)
  File "/glade/p/work/mvertens/cesm2_0_alpha06j.ndep/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 572, in case_build
    return run_and_log_case_status(functor, "case.build", caseroot=caseroot)
  File "/glade/p/work/mvertens/cesm2_0_alpha06j.ndep/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 1222, in run_and_log_case_status
    rv = func()
  File "/glade/p/work/mvertens/cesm2_0_alpha06j.ndep/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 571, in <lambda>
    functor = lambda: _case_build_impl(caseroot, case, sharedlib_only, model_only)
  File "/glade/p/work/mvertens/cesm2_0_alpha06j.ndep/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 521, in _case_build_impl
    lid, caseroot, cimeroot, compiler))
  File "/glade/p/work/mvertens/cesm2_0_alpha06j.ndep/cime/scripts/Tools/../../scripts/lib/CIME/build.py", line 84, in _build_model
    expect(not thread_bad_results, "\n".join(thread_bad_results))
  File "/glade/p/work/mvertens/cesm2_0_alpha06j.ndep/cime/scripts/Tools/../../scripts/lib/CIME/utils.py", line 29, in expect
    raise exc_type("%s %s"%(error_prefix,error_msg))


Resolving patch:
diff --git a/scripts/lib/CIME/SystemTests/erp.py b/scripts/lib/CIME/SystemTests/erp.py
--- a/scripts/lib/CIME/SystemTests/erp.py
+++ b/scripts/lib/CIME/SystemTests/erp.py
@@ -31,6 +31,7 @@ def build_phase(self, sharedlib_only=False, model_only=False):
         and tasks. This test will fail for components (e.g. pop) that do not reproduce exactly
         with different numbers of mpi tasks.
         """
+        self._case.set_value("BUILD_THREADED",True)
         if sharedlib_only:
             return self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
 
@@ -58,7 +59,6 @@ def build_phase(self, sharedlib_only=False, model_only=False):
         # affect the executable that is used
         for bld in range(1,3):
             logging.warn("Starting bld %s"%bld)
-            self._case.set_value("BUILD_THREADED",True)
 
             if (bld == 2):
                 # halve the number of tasks and threads
@@ -79,7 +79,6 @@ def build_phase(self, sharedlib_only=False, model_only=False):
                 case_setup(self._case, test_mode=True, reset=True)
 
             # Now rebuild the system, given updated information in env_build.xml
-
             self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
             shutil.move("%s/%s.exe"%(exeroot,cime_model),
                         "%s/%s.ERP%s.exe"%(exeroot,cime_model,bld))
diff --git a/scripts/lib/CIME/case_setup.py b/scripts/lib/CIME/case_setup.py
--- a/scripts/lib/CIME/case_setup.py
+++ b/scripts/lib/CIME/case_setup.py
@@ -156,8 +156,6 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False,
             case.set_value("TOTALPES", pestot)
             thread_count = env_mach_pes.get_max_thread_count(models)
             build_threaded = case.get_build_threaded()
-            expect(not (build_threaded and compiler == "nag"),
-                   "it is not possible to run with OpenMP if using the NAG Fortran compiler")
             cost_pes = env_mach_pes.get_cost_pes(pestot, thread_count, machine=case.get_value("MACH"))
             case.set_value("COST_PES", cost_pes)
 
diff --git a/scripts/lib/CIME/namelist.py b/scripts/lib/CIME/namelist.py
--- a/scripts/lib/CIME/namelist.py
+++ b/scripts/lib/CIME/namelist.py
@@ -1006,7 +1006,6 @@ def set_variable_value(self, group_name, variable_name, value, var_size=1):
         group_name = group_name.lower()
 
         minindex, maxindex, step = get_fortran_variable_indices(variable_name, var_size)
-        original_var = variable_name
         variable_name = get_fortran_name_only(variable_name.lower())
 
         expect(minindex > 0, "Indices < 1 not supported in CIME interface to fortran namelists... lower bound=%s"%minindex)


Test patch: