Repository: ESMCI/cime

Issue:
Need for a new compset naming convention
I would like to add a new component to cesm and thus to cime for the cmeps driver.   This component would be optional and only initially needed for a few compsets.  I do not want to repeat the experience of adding the IAC and ESP components which have created a bit of spaghetti code in this repository and in cmeps.    See also https://github.com/ESCOMP/CMEPS/issues/254


Discussion:
Instead of strict positional notation, we could add another key letter and let things be in any order.
so for example this long name:
1850_EAM%CMIP6_ELM%SPBC_CICE%PRES_DOCN%DOM_SROF_SGLC_SWAV
would be:
I%1850,L%ELM%SPBC,S%CICE%PRES,O%DOCN%DOM,A%EAM%CMIP6

using commas instead of underscore.   The order doesn't matter.  Anything not included but expected is assumed to be stub.

Still seems like there's a need for a minimum number of things to put on that line.   At least an initial condition "I".

Not sure what to do with things like "_BGC" which was not a separate model but a modifier for other models (right?).  If you want to turn on BGC, you instead have to modify each component setting:  CAM%FOOBGC,  MPASO%BARBGC.

This is pretty much what we are considering, but I don't like overloading the '%' delimiter 

Sure some other character can be the delimiter.
Here is what I am considering (using the compset @rljacob chose):

1850_A-EAM%CMIP6_L-ELM%SPBC_I-CICE%PRES_O-DOCN%DOM

I don't see any reason for a modifier on the date field, but it must always be first, others can be in any order.  
@jedwards4b - thanks. I would suggest replacing `A-` with` a: `. Dashes are used in the test name - but colons never are.
I would also suggest every entry have a prefix. So 1850 would be `t:` for time. If there are bgc entries across all components it should have a driver modifier.

t:1850_a:EAM%CMIP6_l:ELM%SPBC_i:CICE%PRES_o:DOCN%DOM_d:BGC

Alternatively, we could have a 3 letter modifier for clarity (which I am starting to think might be helpful)

tim:1850_atm:EAM%CMIP6_lnd:ELM%SPBC_ice:CICE%PRES_onc:DOCN%DOM_drv:BGC

Thoughts?

I'd still like to get rid of special case-wide modifiers like "BGC".  If you want to turn on bgc in several components, you need to modify what is after the "%" in each component.  That makes it more clear which ones actually have BGC.
@rljacob - I see your point. But what if its a driver level setting that needs to be recognized by each component and that there would be problems if components do not set this consistently.
So the BGC setting currently dictates a whole set of fields and co2 settings that are transferred between components. I don't see how that could be done by each component specifying this separately.

Oh I didn't notice the "d:" which does help.  But the pattern is broken.  Maybe it should be "d:CPL7%BGC" or "d:CMEPS%BGC"?
Yes - that sounds good. @jedwards4b and I are leaning towards a 3 letter prefix for clarity. As we get more components 1 letter will start being ambiguous. 
I agree we don't want to limit ourselves to 26 components.
I like the direction this is going.

In terms of a specific separator character: before settling on one, would it be worth trying to build and run a short test case with the proposed character in the case name? – since we have had problems with some characters in the past, e.g., in the build process. (Relevant given that people are now sometimes creating tests with compset long names, so the compset long name appears in the case name.)
I will test using both alias and long name.  I am referring to [this page ](https://en.wikipedia.org/wiki/Filename) for allowed characters.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/component.py b/scripts/lib/CIME/XML/component.py
--- a/scripts/lib/CIME/XML/component.py
+++ b/scripts/lib/CIME/XML/component.py
@@ -207,6 +207,7 @@ def _get_description_v3(self, compsetname, comp_class):
                 parts = opt_parts.pop(0).split("%")
                 reqset = set(parts)
                 fullset = set(parts + opt_parts)
+
                 match, complist = self._get_description_match(
                     compsetname, reqset, fullset, modifier_mode
                 )
@@ -258,14 +259,18 @@ def _get_description_match(self, compsetname, reqset, fullset, modifier_mode):
         (False, None)
         >>> obj._get_description_match("1850_CAM50%WCCM_Barn",set(["CAM50", "WCCM"]), set(["CAM50","WCCM","RCO2"]), "+")
         (True, ['CAM50', 'WCCM'])
+        >>> obj._get_description_match("scn:1850_atm:CAM50%WCCM_Barn",set(["CAM50", "WCCM"]), set(["CAM50","WCCM","RCO2"]), "+")
+        (True, ['CAM50', 'WCCM'])
         """
         match = False
         comparts = compsetname.split("_")
         matchcomplist = None
-
         for comp in comparts:
+            if ":" in comp:
+                comp = comp.split(":")[1]
             complist = comp.split("%")
             cset = set(complist)
+
             if cset == reqset or (cset > reqset and cset <= fullset):
                 if modifier_mode == "1":
                     expect(
diff --git a/scripts/lib/CIME/case/case.py b/scripts/lib/CIME/case/case.py
--- a/scripts/lib/CIME/case/case.py
+++ b/scripts/lib/CIME/case/case.py
@@ -738,6 +738,10 @@ def _valid_compset_impl(self, compset_name, compset_alias, comp_classes, comp_ha
         >>> caseroot = os.path.join(workdir, 'caseroot')  # use non-existent caseroot to avoid error about not being a valid case directory in Case __init__ method
         >>> Case(caseroot, read_only=False)._valid_compset_impl('2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV', None, ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF', 'GLC', 'WAV', 'ESP'], {'datm':1,'satm':1,'dlnd':2,'slnd':2,'dice':3,'sice':3,'docn':4,'socn':4,'drof':5,'srof':5,'sglc':6,'swav':7,'ww3':7,'sesp':8})
         ('2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV_SESP', ['2000', 'DATM%NYF', 'SLND', 'DICE%SSMI', 'DOCN%DOM', 'DROF%NYF', 'SGLC', 'SWAV', 'SESP'])
+        >>> Case(caseroot, read_only=False)._valid_compset_impl('2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV', None, ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF', 'GLC', 'WAV', 'ESP'], {'datm':1,'satm':1,'dlnd':2,'slnd':2,'dice':3,'sice':3,'docn':4,'socn':4,'drof':5,'srof':5,'sglc':6,'swav':7,'ww3':7,'sesp':8})
+        ('2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV_SESP', ['2000', 'DATM%NYF', 'SLND', 'DICE%SSMI', 'DOCN%DOM', 'DROF%NYF', 'SGLC', 'SWAV', 'SESP'])
+        >>> Case(caseroot, read_only=False)._valid_compset_impl('atm:DATM%NYF_rof:DROF%NYF_scn:2000_ice:DICE%SSMI_ocn:DOCN%DOM', None, ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF', 'GLC', 'WAV', 'ESP'], {'datm':1,'satm':1,'dlnd':2,'slnd':2,'dice':3,'sice':3,'docn':4,'socn':4,'drof':5,'srof':5,'sglc':6,'swav':7,'ww3':7,'sesp':8})
+        ('2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV_SESP', ['2000', 'DATM%NYF', 'SLND', 'DICE%SSMI', 'DOCN%DOM', 'DROF%NYF', 'SGLC', 'SWAV', 'SESP'])
         >>> Case(caseroot, read_only=False)._valid_compset_impl('2000_DATM%NYF_DICE%SSMI_DOCN%DOM_DROF%NYF', None, ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF', 'GLC', 'WAV', 'ESP'], {'datm':1,'satm':1,'dlnd':2,'slnd':2,'dice':3,'sice':3,'docn':4,'socn':4,'drof':5,'srof':5,'sglc':6,'swav':7,'ww3':7,'sesp':8})
         ('2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV_SESP', ['2000', 'DATM%NYF', 'SLND', 'DICE%SSMI', 'DOCN%DOM', 'DROF%NYF', 'SGLC', 'SWAV', 'SESP'])
         >>> Case(caseroot, read_only=False)._valid_compset_impl('2000_DICE%SSMI_DOCN%DOM_DATM%NYF_DROF%NYF', None, ['CPL', 'ATM', 'LND', 'ICE', 'OCN', 'ROF', 'GLC', 'WAV', 'ESP'], {'datm':1,'satm':1,'dlnd':2,'slnd':2,'dice':3,'sice':3,'docn':4,'socn':4,'drof':5,'srof':5,'sglc':6,'swav':7,'ww3':7,'sesp':8})
@@ -759,9 +763,22 @@ def _valid_compset_impl(self, compset_name, compset_alias, comp_classes, comp_ha
         # Find the models declared in the compset
         model_set = [None] * len(comp_classes)
         components = compset_name.split("_")
-        model_set[0] = components[0]
         noncomps = []
         allstubs = True
+        colonformat = ":" in compset_name
+        if colonformat:
+            # make sure that scn: is component[0] as expected
+            for i in range(1, len(components)):
+                if components[i].startswith("scn:"):
+                    tmp = components[0]
+                    components[0] = components[i]
+                    components[i] = tmp
+                    break
+
+                model_set[0] = components[0][4:]
+        else:
+            model_set[0] = components[0]
+
         for model in components[1:]:
             match = Case.__mod_match_re__.match(model.lower())
             expect(match is not None, "No model match for {}".format(model))
@@ -769,6 +786,9 @@ def _valid_compset_impl(self, compset_name, compset_alias, comp_classes, comp_ha
             # Check for noncomponent appends (BGC & TEST)
             if mod_match in ("bgc", "test"):
                 noncomps.append(model)
+            elif ":" in mod_match:
+                comp_ind = comp_hash[mod_match[4:]]
+                model_set[comp_ind] = model
             else:
                 expect(mod_match in comp_hash, "Unknown model type, {}".format(model))
                 comp_ind = comp_hash[mod_match]
@@ -781,7 +801,10 @@ def _valid_compset_impl(self, compset_name, compset_alias, comp_classes, comp_ha
                 stub = "S" + comp_class
                 logger.info("Automatically adding {} to compset".format(stub))
                 model_set[comp_ind] = stub
-            elif model_set[comp_ind][0] != "S":
+            elif ":" in model_set[comp_ind]:
+                model_set[comp_ind] = model_set[comp_ind][4:]
+
+            if model_set[comp_ind][0] != "S":
                 allstubs = False
 
         expect(
@@ -794,7 +817,6 @@ def _valid_compset_impl(self, compset_name, compset_alias, comp_classes, comp_ha
         compsetname = "_".join(model_set)
         for noncomp in noncomps:
             compsetname = compsetname + "_" + noncomp
-
         return compsetname, model_set
 
     # RE to match component type name without optional piece (stuff after %).
@@ -894,6 +916,8 @@ def get_compset_components(self):
         # the first element is always the date operator - skip it
         elements = compset.split("_")[1:]  # pylint: disable=maybe-no-member
         for element in elements:
+            if ":" in element:
+                element = element[4:]
             # ignore the possible BGC or TEST modifier
             if element.startswith("BGC%") or element.startswith("TEST"):
                 continue
@@ -985,6 +1009,8 @@ def _get_component_config_data(self, files):
         for i in range(1, len(self._component_classes)):
             comp_class = self._component_classes[i]
             comp_name = self._components[i - 1]
+            if ":" in comp_name:
+                comp_name = comp_name[4:]
             root_dir_node_name = "COMP_ROOT_DIR_" + comp_class
             node_name = "CONFIG_" + comp_class + "_FILE"
             compatt = {"component": comp_name}


Test patch: