Repository: ESMCI/cime

Issue:
Request for derived variable: NTASKS_PER_INST_COMPNAME
Particularly with the new multi-driver code, it would be helpful if we had a set of derived variables (not to be set directly) in env_mach_pes.xml like `NTASKS_PER_INST_COMPNAME` for each component.

Reasons why this would be useful:

1. With the old multi-instance code, it was often confusing to users that the number of tasks per instance was _not_ NTASKS_COMPNAME but rather was (I think) floor(NTASKS_COMPNAME/NINST_COMPNAME). With the new multi-driver capability, this relationship is apparently more confusing: the number of tasks per instance is floor(NTASKS_COMPNAME/NINST_COMPNAME) if running without multi-driver, but is simply NTASKS_COMPNAME if running with multi-driver. Number of tasks per instance is something that users want to know, and it is asking a lot for them to all know this subtlety, so it would be great if this were listed explicitly for users.

2. Some components need to know the number of tasks per instance at build time. Before multi-instance, this was as simple as querying NTASKS_COMPNAME. With multi-instance, this needed to be expanded to also consider NINST_COMPNAME. With multi-driver, this gets more complicated, requiring all of these components to duplicate the logic where ntasks per instance depends on whether multi-driver is set. And what if the logic for number of tasks per instance changes again in the future? I am a firm believer in having any rule be coded in one and only one place, arguing for a central place in the cime scripts where the rule "number of tasks per instance" is encoded.

cc @gold2718 @dabail10 (for cice) @alperaltuntas (for pop, where this might also be an issue)


Discussion:
I always wanted the behavior from the current multi-driver implementation. That is, you specify NTASKS_COMPNAME for one instance and the total NTASKS is computed from that.
I would suggest implementing that as a solution to the issues you raise but I worry about the number of multi-instance run scripts that will break and have to be rewritten. Does anyone have any idea about this?

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_mach_pes.py b/scripts/lib/CIME/XML/env_mach_pes.py
--- a/scripts/lib/CIME/XML/env_mach_pes.py
+++ b/scripts/lib/CIME/XML/env_mach_pes.py
@@ -15,7 +15,7 @@ def __init__(self, case_root=None, infile="env_mach_pes.xml", components=None):
         """
         self._components = components
         self._component_value_list = ["NTASKS", "NTHRDS", "NINST",
-                                      "ROOTPE", "PSTRID", "NINST_LAYOUT"]
+                                      "ROOTPE", "PSTRID", "NINST_LAYOUT", "NTASKS_PER_INST"]
         schema = os.path.join(get_cime_root(), "config", "xml_schemas", "env_mach_pes.xsd")
         EnvBase.__init__(self, case_root, infile, schema=schema)
 
diff --git a/scripts/lib/CIME/case_setup.py b/scripts/lib/CIME/case_setup.py
--- a/scripts/lib/CIME/case_setup.py
+++ b/scripts/lib/CIME/case_setup.py
@@ -129,19 +129,24 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False):
         # In CIME there can be multiple instances of each component model (an ensemble) NINST is the instance of that component.
         multi_driver = case.get_value("MULTI_DRIVER")
         for comp in models:
+            ntasks = case.get_value("NTASKS_{}".format(comp))
             if comp == "CPL":
+                case.set_value("NTASKS_PER_INST_{}".format(comp), ntasks )
                 continue
             ninst  = case.get_value("NINST_{}".format(comp))
-            ntasks = case.get_value("NTASKS_{}".format(comp))
             # But the NINST_LAYOUT may only be concurrent in multi_driver mode
             if multi_driver:
                 expect(case.get_value("NINST_LAYOUT_{}".format(comp)) == "concurrent",
                        "If multi_driver is TRUE, NINST_LAYOUT_{} must be concurrent".format(comp))
-            elif ninst > ntasks:
-                if ntasks == 1:
-                    case.set_value("NTASKS_{}".format(comp), ninst)
-                else:
-                    expect(False, "NINST_{} value {:d} greater than NTASKS_{} {:d}".format(comp, ninst, comp, ntasks))
+                case.set_value("NTASKS_PER_INST_{}".format(comp), ntasks)
+            else:
+                if ninst > ntasks:
+                    if ntasks == 1:
+                        case.set_value("NTASKS_{}".format(comp), ninst)
+                        ntasks = ninst
+                    else:
+                        expect(False, "NINST_{} value {:d} greater than NTASKS_{} {:d}".format(comp, ninst, comp, ntasks))
+                case.set_value("NTASKS_PER_INST_{}".format(comp), ntasks / ninst)
 
         if os.path.exists("case.run"):
             logger.info("Machine/Decomp/Pes configuration has already been done ...skipping")


Test patch:
diff --git a/scripts/tests/scripts_regression_tests.py b/scripts/tests/scripts_regression_tests.py
--- a/scripts/tests/scripts_regression_tests.py
+++ b/scripts/tests/scripts_regression_tests.py
@@ -321,7 +321,6 @@ def test_c_create_clone_keepexe(self):
         if os.path.exists(testdir):
             shutil.rmtree(testdir)
         prevtestdir = cls._testdirs[0]
-        cls._testdirs.append(testdir)
         user_mods_dir = os.path.join(CIME.utils.get_python_libs_root(), "..", "tests", "user_mods_test3")
 
         cmd = "%s/create_clone --clone %s --case %s --keepexe --user-mods-dir %s" \