Repository: ESMCI/cime

Issue:
testnames in new NUOPC driver
Currently, in a separate fork (that will be brought into CIME in the next few months) we are working to bring in a separate NUOPC driver into CIME. 
Note that this new driver will have its own cime_config directory. However, create_newcase and create_test will apply to both.
The way we are currently doing this, based on discussions in a previous CIME weekly meeting, is to have added
a separate --driver argument to both create_newcase and create_test.

For create_newcase we have the following:
```
   ./create_newcase .... --driver nuopc (will give you the nuopc driver)
   ./create_newcase      --driver mct (will give you the mct driver - this is the default)

```
For create_test, the tests currently have the SAME NAME but there is a new --driver option to create_test
```
   ./create_test .... --driver nuopc (will give you the nuopc driver)
   ./create_test      --driver mct (will give you the mct driver - this is the default)

```
When the testing was first implemented by @apcraig, an alternative
approach was taken which was to include _EN to a test name in order to
differentiate between the nuopc and mct interfaces.

So in the origina scheme developed by @apcraig - the following was
`(1) ./create_test SMS_EN.T31_g37_rx1.A.yellowston_intel`

The new scheme does the following
`(2) ./create_test SMS.T31_g37_rx1.A.yellowstone_intel --driver nuopc
`
I feel that there are definitely advantages to using (2). 
- it follows the recommended path forwards from our CIME discussion
- we can potentially share test lists between the two drivers and it is consistent with the way create_newcase is also called.
- we don't have to have a plethora of new qualifiers in test names. 

@apcraig feels that the new approach is a step backwards.Some of the reasons from @apcraig follow:
- we have _E, _D, etc is to be able to uniquely identify the configuration from the testname and to make it easy to create different tests.  
- it might make sense to be able to build and run both mct and nuopc tests from a given sandbox.  
- there are lots of reasons why implementing this as a separate stand-alone flag is not such a good idea.  At the very least, it should add maybe an _EN to the test name. 

I'd like to have wider input as to the best approach moving forwards. @jedwards4b, @rjacob, @jgfouca, @billsacks, @gold2718 - and other - it would be great if you could provide feedback.





Discussion:
I do not see NUOPC as a test option but see it as a different model driver.
Current test options are **D**ebug, **E**SMF (deprecated, can we take it out yet?), **CG** (Gregorian Calendar) and **L** (test run length). Disregarding the ESMF option which we have removed from the models (with good cause IMHO), the remaining options are very different from the underlying driver running the system. 
As we move forward, the selection of a driver will be orthogonal to the other options specified in a case (experiment) or test.
If we decide to support a shorthand for the driver, I vote for putting it in the machine/compiler section (e.g., `./create_test SMS.T31_g37_rx1.A.yellowstone_intel_nuopc`) although I am fine with just the `--driver nuopc` option.
Are we expecting bfb results when we compare the two drivers?  If we were than usning the --driver option allows the generate and compare functions of create_test to work for interdriver-comparison.
I think if the nuopc driver is substantially different than the MCT driver and/or if it might give different answers (for any cases either now or in the future) -- the testname needs to have the driver in the name. Otherwise, you'll compare nuopc driver test results to MCT driver results and not see why they are different. I'm pretty sure the answer to my starting question is a yes on both counts. And the namelists are definitely going to be different.

Overall I'm fine with however that shows up in the testname as either a test option or after the machine/compiler. I just think it's critical to be there. In terms of being similar to something else, it's most similar to the ESMF test option. And yes it is a driver, but it's an alternative driver and as such is something different to test with. So you can argue it makes sense as a test option.
In many ways, I agree with @gold2718.  The nuopc and mct driver are different driver/coupler models.  In that sense, they really should be associated with different compsets.  We use compsets to differentiate model combinations, pop vs docn, mosart vs rtm, and likewise, mct coupler vs nuopc coupler.  If cime is really going to be a community tool for assembling different coupled systems, CESM being just one of them, then there has to be enough flexibility in the naming convention, design, and implementation to support many couplers and many prognostic model components within the compset naming convention.  And in fact when I first started the nuopc implementation I started by adding new compsets for the nuopc configurations.

I switched to the "test option" mode, _EN, because it was a lot easier to do and was good enough for prototyping and initial testing.  And because working with the compsets is still difficult.  But again, if cime is a community tool now, the ability to significantly extend compsets has to be easier.

For now, at the very least, I think we do need a way to differentiate mct vs nuopc coupled test results, whether that is with compset or test option or something else. 
@jedwards4b.  While it would be great to have mct and nuopc coupled system produce bfb results, I do not believe that will be possible and it's not something we're shooting for now.
Along the line of @apcraig's comments about CIME as a community tool I would like to note that there are other proposed uses of CIME (e.g., incorporating case management of uncoupled models) that will lead to more drivers being added to the system.
@ekluzek's comment about namelists got me thinking about what happens to testlist XML files. We will almost certainly want different testmods for different drivers and in the case where the compset is the same, having the driver as a new attribute would be the natural way to distinguish the testlist entries.
To me, this makes having the driver as a first-class item in `create_test` pretty important so I now feel more strongly that we need the `--driver` option.
Either we need a way to distinguish mct vs nuopc tests via the test name or we need to stop relying on the test name as a test identifier and find another approach.
We could still have a --driver nuopc argument to ./create_test - but add a
nuopc or mct suffix to the created test automatically as part of the test
creation. That way the final test name would have that info - but
everything else could stay the way it has been implemented now.

On Tue, Aug 1, 2017 at 2:29 PM, jedwards4b <notifications@github.com> wrote:

> Either we need a way to distinguish mct vs nuopc tests via the test name
> or we need to stop relying on the test name as a test identifier and find
> another approach.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/1781#issuecomment-319456649>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AHlxE7Pth64rdZXcg378jB-zNo11_GQmks5sT26UgaJpZM4Op9hi>
> .
>

What I am proposing is effectively adding a prefix to the generated testid.

On Tue, Aug 1, 2017 at 2:32 PM, Mariana Vertenstein <mvertens@ucar.edu>
wrote:

> We could still have a --driver nuopc argument to ./create_test - but add a
> nuopc or mct suffix to the created test automatically as part of the test
> creation. That way the final test name would have that info - but
> everything else could stay the way it has been implemented now.
>
> On Tue, Aug 1, 2017 at 2:29 PM, jedwards4b <notifications@github.com>
> wrote:
>
>> Either we need a way to distinguish mct vs nuopc tests via the test name
>> or we need to stop relying on the test name as a test identifier and find
>> another approach.
>>
>> —
>> You are receiving this because you authored the thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/ESMCI/cime/issues/1781#issuecomment-319456649>, or mute
>> the thread
>> <https://github.com/notifications/unsubscribe-auth/AHlxE7Pth64rdZXcg378jB-zNo11_GQmks5sT26UgaJpZM4Op9hi>
>> .
>>
>
>

I think what @mvertens proposes is better than what we have now, which is that you can't tell which driver was used when you generate a test.  It might be able to differentiate the different drivers in the testname.  (Actually, if the driver is part of the testid, then the test results will not show that even though the testname will.  Remember that the testid is removed when we post test results.  So maybe using testid is not going to work).   But even if we use that approach, I think it's not a good long term solution.  I don't think we should be creating new approaches to deal with these issues.  I think there are a set of rules governing how the cime scripts work, even if they are not documented.  Prepending the testid with the driver type, adding a test option (_EN), and adding the driver to the machine name are also inconsistent with the current design.  The current design uses compsets to define different combinations of model components and the driver/coupler is one of those components.  The more I think about it, the more I feel that using compsets is the only consistent way to handle this long term.  If we cannot deal with a new coupler in the current compset implementation, then how is cime going to be a community tool?  If cime is being advertised as a community tool, then maybe some time should be spent making sure it's extensible in all the ways it has to be to serve the community.
The traditional way to do this is to extend the longname:
TIME_ATM_LND_ICE_OCN_ROF_GLC_WAV[_ESP][_BGC][_DRV]  (11 items total, 3 optional)

The driver is mandatory so really it should be:
TIME_DRV_ATM_LND_ICE_OCN_ROF_GLC_WAV[_ESP][_BGC] (11 items total, 2 optional)

Then you would have to put the driver in the compset alias and the testname would be unique that way, not by a test extension.  Testnames are distinquished by compset alias, the test type and an extension.

But there's no enforced rules for making compset aliases and its getting harder to both keep it brief and include all the info you want in the alias.   I want to get away from the compset longname and alias as the only way to specify a model for cime to build.  The longname is to cumbersome to extend. So I like the "--driver" extension.   Its also much easier to implement (its already been done!)

If the problem is making unique subdir names in a directory full of tests, the create_test script can add a string anywhere based on the --driver value as Mariana suggests. 
> Either we need a way to distinguish mct vs nuopc tests via the test name or we need to stop relying on the test name as a test identifier and find another approach.

Agreed. How about adding it to the machine_compiler section?
As for @apcraig's latest post, do we need to distinguish the driver from the mediator? I'm tending to agree with @apcraig that now might be a good time to rethink the compset given all the complexity we are proposing to introduce (different drivers, different mediators, optional components replacing the current stub system).
I have long wanted compsets to look more like grid longnames (with a different separator like `!`) and I think that syntax would work with compsets and allow for a driver option (d!), a mediator option (m!? md!? me!?), a bgc option (b! or bgc!), forcing (f!), and initial conditions (init!?).
Missing component types would be stubs (MCT) or simply not present (NUOPC). The driver and mediator could default based on the model (CESM, ACME, etc).
Trying to convince myself that `--driver` is better, I think of the compset as the definition of the experiment. The choice of mediator surely affects the science that is run but so does the driver.
Picking a random compset, `ADSOMAQP` would be:
```
f!2000_a!DATM%NYF_o!DOCN%SOMAQP_m!MCT
```
Adopting a new compset syntax permits extensibility, flexibility, and minimalism (e.g., no stubs) while maintaining backwards compatibility with the old compset definition.
I'm late to the party here (just got back from vacation). I just skimmed through these comments and haven't fully digested them. I'll loudly echo the comment made by a few others that it is very important that the driver appear in the test name somehow. I don't have strong feelings as to how it appears.

I'll also add: Once we have come up with the way for the driver to appear in the test name, doesn't that make the `--driver` option to `create_test` redundant? It makes the most sense to me to just specify the desired test list, with the chosen driver specified (in some way) by the test names (possibly using different drivers for different tests). I understand the argument that, in the short-term, there's some benefit in saying, "run our existing test list but use nuopc rather than mct". But I don't think that justifies putting in a mechanism for doing so, any more than it makes sense to, e.g., have an option to create_test saying "run this test list for CLM45 but change all CLM45 tests to CLM50" (which would be equally useful for CLM developers).
Support _V$driver test option. Example: SMS_Vnuopc.grid.compset

Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -12,7 +12,7 @@
 
 ###############################################################################
 def _build_model(build_threaded, exeroot, clm_config_opts, incroot, complist,
-                 lid, caseroot, cimeroot, compiler, buildlist):
+                 lid, caseroot, cimeroot, compiler, buildlist, comp_interface):
 ###############################################################################
     logs = []
 
@@ -79,7 +79,7 @@ def _build_model(build_threaded, exeroot, clm_config_opts, incroot, complist,
         cime_model = get_model()
         file_build = os.path.join(exeroot, "{}.bldlog.{}".format(cime_model, lid))
 
-        config_dir = os.path.join(cimeroot, "src", "drivers", "mct", "cime_config")
+        config_dir = os.path.join(cimeroot, "src", "drivers", comp_interface, "cime_config")
         f = open(file_build, "w")
         bldroot = os.path.join(exeroot, "cpl", "obj")
         if not os.path.isdir(bldroot):
@@ -206,7 +206,7 @@ def _build_checks(case, build_threaded, comp_interface, use_esmf_lib,
     return sharedpath
 
 ###############################################################################
-def _build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, lid, compiler, buildlist):
+def _build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, lid, compiler, buildlist, comp_interface):
 ###############################################################################
 
     shared_lib = os.path.join(exeroot, sharedpath, "lib")
@@ -215,7 +215,7 @@ def _build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, lid
         if (not os.path.exists(shared_item)):
             os.makedirs(shared_item)
     mpilib = case.get_value("MPILIB")
-    libs = ["gptl", "mct", "pio", "csm_share"]
+    libs = ["gptl", comp_interface, "pio", "csm_share"]
     if mpilib == "mpi-serial":
         libs.insert(0, mpilib)
     logs = []
@@ -228,7 +228,7 @@ def _build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, lid
             # csm_share adds its own dir name
             full_lib_path = os.path.join(sharedlibroot, sharedpath)
         elif lib == "mpi-serial":
-            full_lib_path = os.path.join(sharedlibroot, sharedpath, "mct", lib)
+            full_lib_path = os.path.join(sharedlibroot, sharedpath, comp_interface, lib)
         else:
             full_lib_path = os.path.join(sharedlibroot, sharedpath, lib)
         # pio build creates its own directory
@@ -261,9 +261,9 @@ def _build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, lid
         if comp_lnd == "clm" and "clm4_0" not in clm_config_opts:
             logging.info("         - Building clm4_5/clm5_0 Library ")
             esmfdir = "esmf" if case.get_value("USE_ESMF_LIB") else "noesmf"
-            bldroot = os.path.join(sharedlibroot, sharedpath, case.get_value("COMP_INTERFACE"), esmfdir, "clm","obj" )
-            libroot = os.path.join(exeroot, sharedpath, case.get_value("COMP_INTERFACE"), esmfdir, "lib")
-            incroot = os.path.join(exeroot, sharedpath, case.get_value("COMP_INTERFACE"), esmfdir, "include")
+            bldroot = os.path.join(sharedlibroot, sharedpath, comp_interface, esmfdir, "clm","obj" )
+            libroot = os.path.join(exeroot, sharedpath, comp_interface, esmfdir, "lib")
+            incroot = os.path.join(exeroot, sharedpath, comp_interface, esmfdir, "include")
             file_build = os.path.join(exeroot, "lnd.bldlog.{}".format( lid))
             config_lnd_dir = os.path.dirname(case.get_value("CONFIG_LND_FILE"))
 
@@ -513,12 +513,12 @@ def _case_build_impl(caseroot, case, sharedlib_only, model_only, buildlist):
 
     if not model_only:
         logs = _build_libraries(case, exeroot, sharedpath, caseroot,
-                               cimeroot, libroot, lid, compiler, buildlist)
+                                cimeroot, libroot, lid, compiler, buildlist, comp_interface)
 
     if not sharedlib_only:
         os.environ["INSTALL_SHAREDPATH"] = os.path.join(exeroot, sharedpath) # for MPAS makefile generators
         logs.extend(_build_model(build_threaded, exeroot, clm_config_opts, incroot, complist,
-                                lid, caseroot, cimeroot, compiler, buildlist))
+                                lid, caseroot, cimeroot, compiler, buildlist, comp_interface))
 
         if not buildlist:
             # in case component build scripts updated the xml files, update the case object
diff --git a/scripts/lib/CIME/buildlib.py b/scripts/lib/CIME/buildlib.py
--- a/scripts/lib/CIME/buildlib.py
+++ b/scripts/lib/CIME/buildlib.py
@@ -43,13 +43,14 @@ def parse_input(argv):
 
 def build_cime_component_lib(case, compname, libroot, bldroot):
     cimeroot  = case.get_value("CIMEROOT")
+    comp_interface = case.get_value("COMP_INTERFACE")
     compclass = compname[1:]
 
     with open(os.path.join(bldroot,'Filepath'), 'w') as out:
         out.write(os.path.join(case.get_value('CASEROOT'), "SourceMods",
                                "src.{}\n".format(compname)) + "\n")
         if compname.startswith('d'):
-            out.write(os.path.join(cimeroot, "src", "components", "data_comps", compname, "mct") + "\n")
+            out.write(os.path.join(cimeroot, "src", "components", "data_comps", compname, comp_interface) + "\n")
             out.write(os.path.join(cimeroot, "src", "components", "data_comps", compname) + "\n")
         elif compname.startswith('x'):
             out.write(os.path.join(cimeroot, "src", "components", "xcpl_comps", "xshare") + "\n")
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -761,9 +761,13 @@ def configure(self, compset_name, grid_name, machine_name=None,
                   multi_driver=False, ninst=1, test=False,
                   walltime=None, queue=None, output_root=None,
                   run_unsupported=False, answer=None,
-                  input_dir=None):
+                  input_dir=None, driver=None):
 
         expect(check_name(compset_name, additional_chars='.'), "Invalid compset name {}".format(compset_name))
+
+        if driver:
+            self.set_lookup_value("COMP_INTERFACE", driver)
+
         #--------------------------------------------
         # compset, pesfile, and compset components
         #--------------------------------------------
@@ -1403,7 +1407,7 @@ def create(self, casename, srcroot, compset_name, grid_name,
                multi_driver=False, ninst=1, test=False,
                walltime=None, queue=None, output_root=None,
                run_unsupported=False, answer=None,
-               input_dir=None):
+               input_dir=None, driver=None):
         try:
             # Set values for env_case.xml
             self.set_lookup_value("CASE", os.path.basename(casename))
@@ -1419,7 +1423,7 @@ def create(self, casename, srcroot, compset_name, grid_name,
                            walltime=walltime, queue=queue,
                            output_root=output_root,
                            run_unsupported=run_unsupported, answer=answer,
-                           input_dir=input_dir)
+                           input_dir=input_dir, driver=driver)
 
             self.create_caseroot()
 


Test patch:
diff --git a/scripts/lib/CIME/test_scheduler.py b/scripts/lib/CIME/test_scheduler.py
--- a/scripts/lib/CIME/test_scheduler.py
+++ b/scripts/lib/CIME/test_scheduler.py
@@ -411,19 +411,22 @@ def _create_newcase_phase(self, test):
                     mpilib = case_opt[1:]
                     create_newcase_cmd += " --mpilib {}".format(mpilib)
                     logger.debug (" MPILIB set to {}".format(mpilib))
-                if case_opt.startswith('N'):
+                elif case_opt.startswith('N'):
                     expect(ncpl == 1,"Cannot combine _C and _N options")
                     ninst = case_opt[1:]
                     create_newcase_cmd += " --ninst {}".format(ninst)
                     logger.debug (" NINST set to {}".format(ninst))
-                if case_opt.startswith('C'):
+                elif case_opt.startswith('C'):
                     expect(ninst == 1,"Cannot combine _C and _N options")
                     ncpl = case_opt[1:]
                     create_newcase_cmd += " --ninst {} --multi-driver" .format(ncpl)
                     logger.debug (" NCPL set to {}" .format(ncpl))
-                if case_opt.startswith('P'):
+                elif case_opt.startswith('P'):
                     pesize = case_opt[1:]
                     create_newcase_cmd += " --pecount {}".format(pesize)
+                elif case_opt.startswith('V'):
+                    driver = case_opt[1:]
+                    create_newcase_cmd += " --driver {}".format(driver)
 
         # create_test mpilib option overrides default but not explicitly set case_opt mpilib
         if mpilib is None and self._mpilib is not None:
@@ -547,24 +550,13 @@ def _xml_phase(self, test):
                     #     envtest.set_test_parameter("PTS_LAT", "36.6")
                     #     envtest.set_test_parameter("PTS_LON", "262.5")
 
-                elif opt.startswith('I'):
-                    # Marker to distinguish tests with same name - ignored
-                    continue
-
-                elif opt.startswith('M'):
-                    # M option handled by create newcase
-                    continue
-
-                elif opt.startswith('P'):
-                    # P option handled by create newcase
-                    continue
-
-                elif opt.startswith('N'):
-                    # handled in create_newcase
-                    continue
-                elif opt.startswith('C'):
-                    # handled in create_newcase
-                    continue
+                elif (opt.startswith('I') or # Marker to distinguish tests with same name - ignored
+                      opt.startswith('M') or # handled in create_newcase
+                      opt.startswith('P') or # handled in create_newcase
+                      opt.startswith('N') or # handled in create_newcase
+                      opt.startswith('C') or # handled in create_newcase
+                      opt.startswith('V')):  # handled in create_newcase
+                    pass
                 elif opt.startswith('IOP'):
                     logger.warning("IOP test option not yet implemented")
                 else: