Repository: ESMCI/cime

Issue:
create_test --force-procs and --force-threads options should change test name
I see the convenience of the `--force-procs` and `--force-threads` options added to create_test in #1016 (addressing #897 ). However, for the sake of baseline generation and comparison, as well as tracking the history of passing / failing tests, I feel that it is a Bad Idea to allow changing test configuration without noting this in the test name.

Currently, you could run:

```
./create_test --force-procs 64 SMS.f45_g37.A.yellowstone_intel --generate foo
```

then later run

```
./create_test SMS.f45_g37.A.yellowstone_intel --compare foo
```

If there were differences due to the different PE layout (which is expected for some compsets, unexpected but possible due to bugs for others), then it would be very confusing to try to track down the cause of these differences.

I would suggest that specifying `--force-procs` or `--force-threads` modifies all of the test names to note this modification, by adding `_PNxM` to the test name - unless the test already has a `_PNxM` modifier, in which case it appears that that modifier currently takes precedence over the `--force` option. (I don't think we have a current naming convention for `--force-threads` without `--force-procs`, but we could use something like `PxM`.)

Tagging @jgfouca @jedwards4b 


Discussion:
Perhaps you should think of these as an extension for acme, not intended for cesm use.   Because of our use of testlist (acme does not partake) we can easily change pe-layout in a single test.   You are correct that using these in the cesm testing system will lead to namelist differences and test failures that may be difficult to track down - so my recommendation is don't use them.
In that case, would it make sense to add a note in the documentation that these options should NOT be used in CESM, at least not if you're doing --compare or --generate? Or should these be put in the section of ACME-specific options in create_test, to avoid confusing CESM users?
I think that modifying the MxN in the test name is a possibility @jgfouca ?
@jedwards4b maybe, we'd have to ask @amametjanov more about the use case that led him to file the original request for this feature.
@billsacks one thing we could do is to prevent baseline operations when these --force flags are used.
Another thing we could do is to dynamically rename the tests based on these options.
Sorry for the spam; yet another thing that may be good is restricting these options to the "acme" model.
@jgfouca, that was needed for running all tests on Cetus with 64x16 PEs. Cetus does not allow more than 1024 node-hours per person at a time. Acme_developer has 38 tests: 38 tests x 16 nodes/test = 608 nodes to concurrently run all 38 tests. 
@jgfouca - I am happy with any of the options you mention: preventing baseline operations, dynamically renaming the tests, or restricting these options to the acme model.
@amametjanov would you be opposed to dynamically re-naming the tests? For example:
./create_test --force-procs=16 SMS.ne4_ne4.A
... * renames test to SMS_P16.ne4_ne4.A *
No, I guess we'll just have to create baselines with the new test names, when needed. Any comments @jayeshkrishna?

If you're using the results for baseline testing, then I strongly recommend we go in this direction due to the issues that @billsacks raised.
Yes, agreed.

Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -119,22 +119,22 @@ def initialize_derived_attributes(self):
         """
         env_mach_pes = self.get_env("mach_pes")
         comp_classes = self.get_values("COMP_CLASSES")
-        total_tasks = env_mach_pes.get_total_tasks(comp_classes)
+        total_tasks  = env_mach_pes.get_total_tasks(comp_classes)
+        pes_per_node = self.get_value("PES_PER_NODE")
 
         self.thread_count = env_mach_pes.get_max_thread_count(comp_classes)
         self.tasks_per_node = env_mach_pes.get_tasks_per_node(total_tasks, self.thread_count)
         logger.debug("total_tasks %s thread_count %s"%(total_tasks, self.thread_count))
         self.num_nodes = env_mach_pes.get_total_nodes(total_tasks, self.thread_count)
         self.tasks_per_numa = int(math.ceil(self.tasks_per_node / 2.0))
-        smt_factor = max(1,int(self.get_value("MAX_TASKS_PER_NODE")/self.get_value("PES_PER_NODE")))
+        smt_factor = max(1,int(self.get_value("MAX_TASKS_PER_NODE") / pes_per_node))
 
-        self.cores_per_task = ((self.get_value("MAX_TASKS_PER_NODE")/smt_factor) \
-                               / self.tasks_per_node) * 2
+        threads_per_node = self.tasks_per_node * self.thread_count
+        threads_per_core = 1 if (threads_per_node <= pes_per_node) else smt_factor
+        self.cores_per_task = self.thread_count / threads_per_core
 
         return total_tasks
 
-
-
     # Define __enter__ and __exit__ so that we can use this as a context manager
     # and force a flush on exit.
     def __enter__(self):
@@ -774,7 +774,9 @@ def configure(self, compset_name, grid_name, machine_name=None,
             self.set_value("TIMER_LEVEL", 4)
         if test:
             self.set_value("TEST",True)
+
         total_tasks = self.initialize_derived_attributes()
+
         # Make sure that parallel IO is not specified if total_tasks==1
         if total_tasks == 1:
             for compclass in self._component_classes:
@@ -783,6 +785,8 @@ def configure(self, compset_name, grid_name, machine_name=None,
                 if pio_typename in ("pnetcdf", "netcdf4p"):
                     self.set_value(key, "netcdf")
 
+        # Set TOTAL_CORES
+        self.set_value("TOTAL_CORES", total_tasks * self.cores_per_task )
 
     def get_compset_var_settings(self):
         compset_obj = Compsets(infile=self.get_value("COMPSETS_SPEC_FILE"))
diff --git a/utils/python/CIME/case_setup.py b/utils/python/CIME/case_setup.py
--- a/utils/python/CIME/case_setup.py
+++ b/utils/python/CIME/case_setup.py
@@ -207,6 +207,7 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False):
                 elif job != "case.test":
                     logger.info("Writing %s script from input template %s" % (job, input_batch_script))
                     env_batch.make_batch_script(input_batch_script, job, case, pestot, tasks_per_node, num_nodes, thread_count)
+
             # Make sure pio settings are consistant
             for comp in models:
                 pio_stride = case.get_value("PIO_STRIDE_%s"%comp)
diff --git a/utils/python/CIME/utils.py b/utils/python/CIME/utils.py
--- a/utils/python/CIME/utils.py
+++ b/utils/python/CIME/utils.py
@@ -289,13 +289,15 @@ def parse_test_name(test_name):
 
     return rv
 
-def get_full_test_name(partial_test, grid=None, compset=None, machine=None, compiler=None, testmod=None):
+def get_full_test_name(partial_test, caseopts=None, grid=None, compset=None, machine=None, compiler=None, testmod=None):
     """
     Given a partial CIME test name, return in form TESTCASE.GRID.COMPSET.MACHINE_COMPILER[.TESTMODS]
     Use the additional args to fill out the name if needed
 
     >>> get_full_test_name("ERS", grid="ne16_fe16", compset="JGF", machine="melvin", compiler="gnu")
     'ERS.ne16_fe16.JGF.melvin_gnu'
+    >>> get_full_test_name("ERS", caseopts=["D", "P16"], grid="ne16_fe16", compset="JGF", machine="melvin", compiler="gnu")
+    'ERS_D_P16.ne16_fe16.JGF.melvin_gnu'
     >>> get_full_test_name("ERS.ne16_fe16", compset="JGF", machine="melvin", compiler="gnu")
     'ERS.ne16_fe16.JGF.melvin_gnu'
     >>> get_full_test_name("ERS.ne16_fe16.JGF", machine="melvin", compiler="gnu")
@@ -305,7 +307,7 @@ def get_full_test_name(partial_test, grid=None, compset=None, machine=None, comp
     >>> get_full_test_name("ERS.ne16_fe16.JGF", machine="melvin", compiler="gnu", testmod="mods/test")
     'ERS.ne16_fe16.JGF.melvin_gnu.mods-test'
     """
-    _, _, partial_grid, partial_compset, partial_machine, partial_compiler, partial_testmod = parse_test_name(partial_test)
+    partial_testcase, partial_caseopts, partial_grid, partial_compset, partial_machine, partial_compiler, partial_testmod = parse_test_name(partial_test)
 
     required_fields = [
         (partial_grid, grid, "grid"),
@@ -333,9 +335,19 @@ def get_full_test_name(partial_test, grid=None, compset=None, machine=None, comp
         else:
             result += ".%s" % testmod.replace("/", "-")
     elif (testmod is not None):
-        expect(arg_val == partial_val, # pylint: disable=undefined-loop-variable
+        expect(testmod == partial_testmod,
                "Mismatch in field testmod, partial string '%s' indicated it should be '%s' but you provided '%s'" % (partial_test, partial_testmod, testmod))
 
+    if (partial_caseopts is None):
+        if caseopts is None:
+            # No casemods for this test and that's OK
+            pass
+        else:
+            result = result.replace(partial_testcase, "%s_%s" % (partial_testcase, "_".join(caseopts)), 1)
+    elif caseopts is not None:
+        expect(caseopts == partial_caseopts,
+               "Mismatch in field caseopts, partial string '%s' indicated it should be '%s' but you provided '%s'" % (partial_test, partial_caseopts, caseopts))
+
     return result
 
 def get_current_branch(repo=None):


Test patch:
diff --git a/utils/python/CIME/test_scheduler.py b/utils/python/CIME/test_scheduler.py
--- a/utils/python/CIME/test_scheduler.py
+++ b/utils/python/CIME/test_scheduler.py
@@ -12,7 +12,7 @@
 from CIME.XML.standard_module_setup import *
 import CIME.compare_namelists
 import CIME.utils
-from CIME.utils import append_status, TESTS_FAILED_ERR_CODE
+from CIME.utils import append_status, TESTS_FAILED_ERR_CODE, parse_test_name, get_full_test_name
 from CIME.test_status import *
 from CIME.XML.machines import Machines
 from CIME.XML.env_test import EnvTest
@@ -30,6 +30,46 @@
 PHASES = [TEST_START, CREATE_NEWCASE_PHASE, XML_PHASE, SETUP_PHASE,
           SHAREDLIB_BUILD_PHASE, MODEL_BUILD_PHASE, RUN_PHASE] # Order matters
 
+###############################################################################
+def _translate_test_names_for_new_pecount(test_names, force_procs, force_threads):
+###############################################################################
+    new_test_names = []
+    for test_name in test_names:
+        testcase, caseopts, grid, compset, machine, compiler, testmod = parse_test_name(test_name)
+        rewrote_caseopt = False
+        if caseopts is not None:
+            for idx, caseopt in enumerate(caseopts):
+                if caseopt.startswith("P"):
+                    caseopt = caseopt[1:]
+                    if "x" in caseopt:
+                        old_procs, old_thrds = caseopt.split("x")
+                    else:
+                        old_procs, old_thrds = caseopt, None
+
+                    new_procs = force_procs if force_procs is not None else old_procs
+                    new_thrds = force_threads if force_threads is not None else old_thrds
+
+                    newcaseopt = ("P%s" % new_procs) if new_thrds is None else ("P%sx%s" % (new_procs, new_thrds))
+
+                    # No idea why pylint thinks this is unsubscriptable
+                    caseopts[idx] = newcaseopt # pylint: disable=unsubscriptable-object
+
+                    rewrote_caseopt = True
+                    break
+
+        if not rewrote_caseopt:
+            force_procs = "M" if force_procs is None else force_procs
+            newcaseopt = ("P%s" % force_procs) if force_threads is None else ("P%sx%s" % (force_procs, force_threads))
+            if caseopts is None:
+                caseopts = [newcaseopt]
+            else:
+                caseopts.append(newcaseopt)
+
+        new_test_name = get_full_test_name(testcase, caseopts=caseopts, grid=grid, compset=compset, machine=machine, compiler=compiler, testmod=testmod)
+        new_test_names.append(new_test_name)
+
+    return new_test_names
+
 ###############################################################################
 class TestScheduler(object):
 ###############################################################################
@@ -52,13 +92,15 @@ def __init__(self, test_names, test_data=None,
         self._save_timing   = save_timing
         self._queue         = queue
         self._test_data     = {} if test_data is None else test_data # Format:  {test_name -> {data_name -> data}}
-        self._force_procs   = force_procs
-        self._force_threads = force_threads
 
         self._allow_baseline_overwrite  = allow_baseline_overwrite
 
         self._machobj = Machines(machine=machine_name)
 
+        # If user is forcing procs or threads, re-write test names to reflect this.
+        if force_procs or force_threads:
+            test_names = _translate_test_names_for_new_pecount(test_names, force_procs, force_threads)
+
         self._no_setup = no_setup
         self._no_build = no_build or no_setup or namelists_only
         self._no_run   = no_run or self._no_build
@@ -316,13 +358,6 @@ def _create_newcase_phase(self, test):
         if self._output_root is not None:
             create_newcase_cmd += " --output-root %s " % self._output_root
 
-        if self._force_procs or self._force_threads:
-            pecount_str = "M" if self._force_procs is None else str(self._force_procs)
-            if self._force_threads is not None:
-                pecount_str += "x%d" % self._force_threads
-
-            create_newcase_cmd += " --pecount %s" % pecount_str
-
         if test_mods is not None:
             files = Files()
             (component,modspath) = test_mods.split('/',1)
@@ -535,7 +570,7 @@ def _get_procs_needed(self, test, phase, threads_in_flight=None, no_batch=False)
     ###########################################################################
         if phase == RUN_PHASE and (self._no_batch or no_batch):
             test_dir = self._get_test_dir(test)
-            out = run_cmd_no_fail("./xmlquery TOTALPES -value", from_dir=test_dir)
+            out = run_cmd_no_fail("./xmlquery TOTAL_CORES -value", from_dir=test_dir)
             return int(out)
         elif (phase == SHAREDLIB_BUILD_PHASE):
             # Will force serialization of sharedlib builds
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -1199,7 +1199,7 @@ def test_cime_case_force_pecount(self):
                               % (SCRIPT_DIR, self._baseline_name, self._testroot, self._testroot))
 
         casedir = os.path.join(self._testroot,
-                               "%s.%s" % (CIME.utils.get_full_test_name("TESTRUNPASS_Mmpi-serial.f19_g16_rx1.A", machine=self._machine, compiler=self._compiler), self._baseline_name))
+                               "%s.%s" % (CIME.utils.get_full_test_name("TESTRUNPASS_Mmpi-serial_P16x8.f19_g16_rx1.A", machine=self._machine, compiler=self._compiler), self._baseline_name))
         self.assertTrue(os.path.isdir(casedir), msg="Missing casedir '%s'" % casedir)
 
         with Case(casedir, read_only=True) as case:
@@ -1207,6 +1207,9 @@ def test_cime_case_force_pecount(self):
 
             self.assertEqual(case.get_value("NTHRDS_CPL"), 8)
 
+            expected_cores = 16 * case.cores_per_task
+            self.assertEqual(case.get_value("TOTAL_CORES"), expected_cores)
+
 ###############################################################################
 class X_TestSingleSubmit(TestCreateTestCommon):
 ###############################################################################