Repository: ESMCI/cime

Issue:
Matches appear to differ depending on whether in config_compsets or config_component
I get different matches for the following dependent on whether it is in config_compsets or config_component.  (I've indicated the two lines which MATCH).  When this is in config_compsets, I get the first match.  When it is in config_components, I get the last.

    <entry id="SSTICE_DATA_FILENAME">
      <values>
        <value compset="DOCN%DOM" grid="a%T31.*_oi%T31"         >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_48x96_clim_c050526.nc</value>
        <value compset="DOCN%DOM" grid="a%1.9x2.5.*_oi%1.9x2.5" >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_1.9x2.5_clim_c061031.nc</value>
        **MATCH** <value compset="DOCN%DOM" grid="a%0.9x1.25.*_oi%0.9x1.25">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.9x1.25_clim_c040926.nc</value>
      <value compset="DOCN%DOM" grid="a%0.47x0.63.*_oi%0.47x0.63">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.47x0.63_clim_c061106.nc</value>
      <value compset="DOCN%DOM" grid="a%0.23x0.31.*_oi%0.23x0.31">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.23x0.31_clim_c110526.nc</value>
      <value compset="(AMIP_|HIST_|20TR_|5505_|PIPD_|%TSCH)" grid=".+"  >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_1x1_1850_2012_c130411.nc</value>
      <value compset="(AMIP_|HIST_|20TR_|5505_|PIPD_|%TSCH)" grid="a%T31.*_oi%T31">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_48x96_1850_2012_c130411.nc</value>
      <value compset="(AMIP_|HIST_|20TR_|5505_|PIPD_|%TSCH)" grid="a%1.9x2.5.*_oi%1.9x2.5">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_1.9x2.5_1850_2012_c130411.nc</value>
      **MATCH**<value compset="(AMIP_|HIST_|20TR_|5505_|PIPD_|%TSCH)" grid="a%0.9x1.25.*_oi%0.9x1.25">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.9x1.25_1850_2012_c130411.nc</value>
      <value compset="(AMIP_|HIST_|20TR_|5505_|PIPD_|%TSCH)" grid="a%0.47x0.63.*_oi%0.47x0.63">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.47x0.63_1850_2012_c130411.nc</value>
      <value compset="(AMIP_|HIST_|20TR_|5505_|PIPD_|%TSCH)" grid="a%0.23x0.31.*_oi%0.23x0.31">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.23x0.31_1850_2012_c130411.nc</value>
      <value compset="1850_" grid=".+"                          >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_1x1_clim_pi_c101029.nc</value>
      <value compset="1850_" grid="a%T31.*_oi%T31"              >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_48x96_clim_pi_c101028.nc</value>
      <value compset="1850_" grid="a%1.9x2.5.*_oi%1.9x2.5"      >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_1.9x2.5_clim_pi_c101028.nc</value>
      <value compset="1850_" grid="a%0.9x1.25.*_oi%0.9x1.25"    >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.9x1.25_clim_pi_c101028.nc</value>
      <value compset="1850_" grid="a%0.47x0.63.*_oi%0.47x0.63"  >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.47x0.63_clim_pi_c101028.nc</value>
      <value compset="1850_" grid="a%0.23x0.31.*_oi%0.23x0.31"  >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.23x0.31_clim_pi_c101028.nc</value>
        <value compset="FRC1_" grid="a%1.9x2.5.*_oi%1.9x2.5"    >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_1.9x2.5_1850_2015_c160510.nc</value>
        <value compset="SDYN_" grid="a%1.9x2.5.*_oi%1.9x2.5"    >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_1.9x2.5_1850_2015_c160510.nc</value>
        <value compset="SDYN_" grid="a%0.9x1.25.*_oi%0.9x1.25"  >$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.9x1.25_1850_2015_c160510.nc</value>
        <value compset="SDYN_" grid="a%0.47x0.63.*_oi%0.47x0.63">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.45x0.63_1850_2015_c160510.nc</value>
        <value compset="SDYN_" grid="a%0.23x0.31.*_oi%0.23x0.31">$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.23x0.31_1850_2015_c160510.nc</value>
      </values>
    </entry>



Discussion:
There are two instances of function _get_value_match: one in class entry_id.py and another in component.py so the instance in config_component.xml is processed using component.py while the
instance in config_compset.py is processed using entry_id.py  - I think that the one in entry_id.py is needed by namelist stuff, and the solution might be to have the config_compset code use the one in component.py.    One way to do that would be to introduce a new class which both compsets.py and component.py inherit from and which contains only this function.   
The issue is in get_compset_var_settings in compsets.py - it uses a generic entry_id object in order
to access the _get_value_match function, it needs to use the one in component.py - i'll leave it as an exercise to figure out how.  
We cannot do that for backwards compatibility purposes. Unfortunately,
namelist matches that parse the new python based namelist_definition_xxx.py
file pick the first match that is found when multiple matches are found
(this is done to be consistent with all of the perl based functionality) -
whereas the component.py entry-id match picks the latest (again for
backwards compatibility). This is unfortunate and confusing and should be
addressed post-release. But having just one new class that picks only one
way will potentially cause a lot of refactoring to occur and will need
extensive testing - neither of which we should take on before the release.
So a shorter term fix needs to be implemented.

On Fri, May 26, 2017 at 6:09 PM, jedwards4b <notifications@github.com>
wrote:

> There are two instances of function _get_value_match: one in class
> entry_id.py and another in component.py so the instance in
> config_component.xml is processed using component.py while the
> instance in config_compset.py is processed using entry_id.py - I think
> that the one in entry_id.py is needed by namelist stuff, and the solution
> might be to have the config_compset code use the one in component.py. One
> way to do that would be to introduce a new class which both compsets.py and
> component.py inherit from and which contains only this function.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/1617#issuecomment-304412107>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AHlxE9om0f9aBiFFsAh6SWgjAm6WuXceks5r92nFgaJpZM4NoJf_>
> .
>

So in thinking about this some more I think we do have a critical issue on our hands that will require us to do the following:

The namelist_definition.xml match inherits from entry_id  and the entry_id match - use the first match in case case of a tie. 

However, the component.py also inherits from entry_id - but has its own _get_value_match that picks (unfortunately) the ***last*** best match. 

I think we need to proceed forwards in two steps.

First, **We need to change component.py to have the first best match - otherwise we have a complete nightmare on our hands.**  That could easily be done my making the following change in component.py - change

```
                # take the *last* best match
                elif match_count >= match_max:

```
to 
```
                # take the *first* best match
                elif match_count > match_max:
```  

This might break things in both ACME and CESM prognostic config_components.py - so when this is fixed we need to carefully do testing.

I agree with Jim that we should only be inheriting from one function. That would be the next step.


In looking at CIME - there are only a few places that a careful analysis and restructuring needs to be done:

```
in the driver  config_component_cesm.xml and  config_component_acme.xml
   CCSM_BGC, ATM_NCPL, ROF_NCPL, CPL_EPBAL, CPLSEQ_OPTION, BUDGETS

in the docn config_component.xml
    SSTICE_DATA_FILENAME, SSTICE_GRID_FILENAME
```


@gold2718 had a great idea for a better approach:

Add a new, optional, attribute to the config_component.xml <values> tag as follows

&lt;values match="last"> will pick the last best match
&lt;values match="first"> will pick the first best match

In component.py, the parsing can be done to have a  default value as  "last" to be backwards compatible - but the order of matching is now clear - and can be changed if desired. Therefore there is no ambiguity and everything is backwards compatible.

I solved the compsets problem creating a Component object in compsets.py rather than a EntryId object to parse the EntryId fields. Since the Component object now has the ability to parse the "first" and "last" attribute - this can also be changed - but the default behavior will be maintained as before. Let me stress that this is a short term fix - not a proposal for a permanent solution.


So this will require no changes to the config_component.xml files. However, I am putting in <values match="last"> at this point where there is ambiguity as part of what will be my PR.

Having this attribute will permit moving forwards with a general solution that involves code reuse.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/component.py b/scripts/lib/CIME/XML/component.py
--- a/scripts/lib/CIME/XML/component.py
+++ b/scripts/lib/CIME/XML/component.py
@@ -21,7 +21,6 @@ def __init__(self, infile):
         cimeroot = get_cime_root()
         if  cimeroot in os.path.abspath(infile):
             schema = files.get_schema("CONFIG_CPL_FILE")
-
         EntryID.__init__(self, infile, schema=schema)
 
     #pylint: disable=arguments-differ
@@ -41,6 +40,11 @@ def get_valid_model_components(self):
         return components
 
     def _get_value_match(self, node, attributes=None, exact_match=False):
+        """
+        return the best match for the node <values> entries
+        Note that a component object uses a different matching algorithm than an entryid object
+        For a component object the _get_value_match used is below  and is not the one in entry_id.py
+        """
         match_value = None
         match_max = 0
         match_count = 0
@@ -50,6 +54,11 @@ def _get_value_match(self, node, attributes=None, exact_match=False):
         values = self.get_optional_node("values", root=node)
         if values is None:
             return
+
+        # determine match_type if there is a tie 
+        # ASSUME a default of "last" if "match" attribute is not there
+        match_type = values.get("match", default="last")
+
         # use the default_value if present
         val_node = self.get_optional_node("default_value", root=node)
         if val_node is None:
@@ -58,6 +67,7 @@ def _get_value_match(self, node, attributes=None, exact_match=False):
         value = val_node.text
         if value is not None and len(value) > 0 and value != "UNSET":
             match_values.append(value)
+
         for valnode in self.get_nodes("value", root=node):
             # loop through all the keys in valnode (value nodes) attributes
             for key,value in valnode.attrib.iteritems():
@@ -70,6 +80,8 @@ def _get_value_match(self, node, attributes=None, exact_match=False):
                     else:
                         match_count = 0
                         break
+
+            # a match is found
             if match_count > 0:
                 # append the current result
                 if values.get("modifier") == "additive":
@@ -82,11 +94,21 @@ def _get_value_match(self, node, attributes=None, exact_match=False):
                         del match_values[:]
                     match_values.append(valnode.text)
 
-                # take the *last* best match
-                elif match_count >= match_max:
-                    del match_values[:]
-                    match_max = match_count
-                    match_value = valnode.text
+                else: 
+                    if match_type == "last":
+                        # take the *last* best match
+                        if match_count >= match_max:
+                            del match_values[:]
+                            match_max = match_count
+                            match_value = valnode.text
+                    elif match_type == "first":
+                        # take the *first* best match
+                        if match_count > match_max:
+                            del match_values[:]
+                            match_max = match_count
+                            match_value = valnode.text
+                    else:
+                        expect(False, "match attribute can only have a value of 'last' or 'first'")
 
         if len(match_values) > 0:
             match_value = " ".join(match_values)
diff --git a/scripts/lib/CIME/XML/entry_id.py b/scripts/lib/CIME/XML/entry_id.py
--- a/scripts/lib/CIME/XML/entry_id.py
+++ b/scripts/lib/CIME/XML/entry_id.py
@@ -67,6 +67,15 @@ def _get_value_match(self, node, attributes=None, exact_match=False):
         '''
         Note that the component class has a specific version of this function
         '''
+        # if there is a <values> element - check to see if there is a match attribute
+        # if there is NOT a match attribute, then set the default to "first"
+        # this is different than the component class _get_value_match where the default is "last"
+        values_node = self.get_optional_node("values", root=node)
+        if values_node is not None:
+            match_type = values_node.get("match", default="first")
+        else:
+            match_type = "first"
+
         # Store nodes that match the attributes and their scores.
         matches = []
         nodes = self.get_nodes("value", root=node)
@@ -97,8 +106,23 @@ def _get_value_match(self, node, attributes=None, exact_match=False):
         if not matches:
             return None
 
-        # Get maximum score using custom `key` function, extract the node.
-        _, mnode = max(matches, key=lambda x: x[0])
+        # Get maximum score using either a "last" or "first" match in case of a tie
+        max_score = -1
+        mnode = None
+        for score,node in matches:
+            if match_type == "last":
+                # take the *last* best match
+                if score >= max_score:
+                    max_score = score
+                    mnode = node
+            elif match_type == "first":
+                # take the *first* best match
+                if score > max_score:
+                    max_score = score
+                    mnode = node
+            else:
+                expect(False, 
+                       "match attribute can only have a value of 'last' or 'first', value is %s" %match_type)
 
         return mnode.text
 


Test patch: