Repository: ESMCI/cime

Issue:
namelist comparison test doesn't recognize a new variable in marbl_in
I don't know if this is an issue with the namelist comparison test, or if this is a result of the fact that `marbl_in` is not a Fortran namelist (it contains `foo = bar` lines but does not have the `&[namelist name]` line at the top or the `/` at the bottom), but when we add `particulate_flux_ref_depth` to the file, instead of noting the additional line it produces a whole flurry of inequivalent line messages:

```
Inequivalent lines PON_bury_coeff = 0.5 != particulate_flux_ref_depth = 100
  NORMALIZED: PON_bury_coeff = 0.5 != particulate_flux_ref_depth = 100
Inequivalent lines parm_init_POC_bury_coeff = 1.1 != PON_bury_coeff = 0.5
  NORMALIZED: parm_init_POC_bury_coeff = 1.1 != PON_bury_coeff = 0.5
Inequivalent lines parm_init_POP_bury_coeff = 1.1 != parm_init_POC_bury_coeff = 1.1
  NORMALIZED: parm_init_POP_bury_coeff = 1.1 != parm_init_POC_bury_coeff = 1.1
Inequivalent lines parm_init_bSi_bury_coeff = 1.0 != parm_init_POP_bury_coeff = 1.1
  NORMALIZED: parm_init_bSi_bury_coeff = 1.0 != parm_init_POP_bury_coeff = 1.1
Inequivalent lines parm_CaCO3_diss =   5.0000000000000000e+04 != parm_init_bSi_bury_coeff = 1.0
  NORMALIZED: parm_CaCO3_diss =   5.0000000000000000e+04 != parm_init_bSi_bury_coeff = 1.0
[cut for brevity]
Inequivalent lines tracer_restore_vars(44) = '' != tracer_restore_vars(43) = ''
  NORMALIZED: tracer_restore_vars(44) = '' != tracer_restore_vars(43) = ''
Inequivalent lines tracer_restore_vars(45) = '' != tracer_restore_vars(44) = ''
  NORMALIZED: tracer_restore_vars(45) = '' != tracer_restore_vars(44) = ''
Inequivalent lines tracer_restore_vars(46) = '' != tracer_restore_vars(45) = ''
  NORMALIZED: tracer_restore_vars(46) = '' != tracer_restore_vars(45) = ''
Found extra lines
tracer_restore_vars(46) = ""
```


Discussion:
This is because marbl is not a fortran namelist.
Is this going to be marked "won't fix" or is it possible to update the namelist comparison to treat non-namelist files as if the contents are all a single namelist?
I think it can be fixed but I'll need to look into it.  I know that you have some novel data structures in this namelist and the example you've provided is a pretty simple case.   Do you want to provide a more extreme example?
I think from a comparison point of view, the file is fairly simple so I'm not sure what a more extreme example would look like. Variables are listed alphabetically (case-sensitive sorting, so variables starting with an upper case letter precede vars beginning with lower case letters) by group, which are marked via Fortran-esque comments:

```
! config PFTs
PFT_defaults = "CESM2"
autotroph_cnt = 3
max_grazer_prey_cnt = 3
zooplankton_cnt = 1

! config flags
ciso_lecovars_full_depth_tavg = .false.
ciso_lsource_sink = .true.
ciso_on = .true.
ladjust_bury_coeff = .false.
lcompute_nhx_surface_emis = .true.
lecovars_full_depth_tavg = .false.
lflux_gas_co2 = .true.
lflux_gas_o2 = .true.
lsource_sink = .true.
lvariable_PtoC = .true.
```

Every array element is explicitly declared (such as `tracer_restore_vars(46)`), and if we add additional tracers that specific array will get longer. We also have some MARBL-specific data-structures leading to file entries such as `grazing(1,1)%auto_ind(1) = 1` (which may get moved to `zooplankton(1)%grazing(1)%auto_ind(1)` down the line); for that I'd be happy with reporting the former as no longer being in the file while the latter are reported as new lines... not need to recognize the variable name change.

Resolving patch:
diff --git a/scripts/lib/CIME/case/case_cmpgen_namelists.py b/scripts/lib/CIME/case/case_cmpgen_namelists.py
--- a/scripts/lib/CIME/case/case_cmpgen_namelists.py
+++ b/scripts/lib/CIME/case/case_cmpgen_namelists.py
@@ -29,8 +29,7 @@ def _do_full_nl_comp(case, test, compare_name, baseline_root=None):
                             if "README" not in os.path.basename(item)\
                             and not item.endswith("doc")\
                             and not item.endswith("prescribed")\
-                            and not os.path.basename(item).startswith(".")] + \
-                            glob.glob("{}/*user_nl*".format(test_dir))
+                            and not os.path.basename(item).startswith(".")]
 
     comments = "NLCOMP\n"
     for item in all_items_to_compare:
diff --git a/scripts/lib/CIME/simple_compare.py b/scripts/lib/CIME/simple_compare.py
--- a/scripts/lib/CIME/simple_compare.py
+++ b/scripts/lib/CIME/simple_compare.py
@@ -43,7 +43,7 @@ def _skip_comments_and_whitespace(lines, idx):
     return idx
 
 ###############################################################################
-def _compare_data(gold_lines, comp_lines, case):
+def _compare_data(gold_lines, comp_lines, case, offset_method=False):
 ###############################################################################
     """
     >>> teststr = '''
@@ -55,7 +55,7 @@ def _compare_data(gold_lines, comp_lines, case):
     ... data7 data8 data9 data10
     ... '''
     >>> _compare_data(teststr.splitlines(), teststr.splitlines(), None)
-    ''
+    ('', 0)
 
     >>> teststr2 = '''
     ... data1
@@ -64,15 +64,27 @@ def _compare_data(gold_lines, comp_lines, case):
     ... data7 data8 data9 data10
     ... data00
     ... '''
-    >>> results = _compare_data(teststr.splitlines(), teststr2.splitlines(), None)
+    >>> results,_ = _compare_data(teststr.splitlines(), teststr2.splitlines(), None)
     >>> print(results)
     Inequivalent lines data2 data3 != data2 data30
       NORMALIZED: data2 data3 != data2 data30
     Found extra lines
     data00
     <BLANKLINE>
+    >>> teststr3 = '''
+    ... data1
+    ... data4 data5 data6
+    ... data7 data8 data9 data10
+    ... data00
+    ... '''
+    >>> results,_ = _compare_data(teststr3.splitlines(), teststr2.splitlines(), None, offset_method=True)
+    >>> print(results)
+    Inequivalent lines data4 data5 data6 != data2 data30
+      NORMALIZED: data4 data5 data6 != data2 data30
+    <BLANKLINE>
     """
     comments = ""
+    cnt = 0
     gidx, cidx = 0, 0
     gnum, cnum = len(gold_lines), len(comp_lines)
     while (gidx < gnum or cidx < cnum):
@@ -81,15 +93,15 @@ def _compare_data(gold_lines, comp_lines, case):
 
         if (gidx == gnum):
             if (cidx == cnum):
-                return comments
+                return comments, cnt
             else:
                 comments += "Found extra lines\n"
                 comments += "\n".join(comp_lines[cidx:]) + "\n"
-                return comments
+                return comments, cnt
         elif (cidx == cnum):
             comments += "Missing lines\n"
             comments += "\n".join(gold_lines[gidx:1]) + "\n"
-            return comments
+            return comments, cnt
 
         gold_value = gold_lines[gidx].strip()
         gold_value = gold_value.replace('"',"'")
@@ -102,11 +114,17 @@ def _compare_data(gold_lines, comp_lines, case):
         if (norm_gold_value != norm_comp_value):
             comments += "Inequivalent lines {} != {}\n".format(gold_value, comp_value)
             comments += "  NORMALIZED: {} != {}\n".format(norm_gold_value, norm_comp_value)
+            cnt += 1
+        if offset_method and (norm_gold_value != norm_comp_value):
+            if gnum > cnum:
+                gidx += 1
+            else:
+                cidx += 1
+        else:
+            gidx += 1
+            cidx += 1
 
-        gidx += 1
-        cidx += 1
-
-    return comments
+    return comments, cnt
 
 ###############################################################################
 def compare_files(gold_file, compare_file, case=None):
@@ -118,7 +136,14 @@ def compare_files(gold_file, compare_file, case=None):
     expect(os.path.exists(gold_file), "File not found: {}".format(gold_file))
     expect(os.path.exists(compare_file), "File not found: {}".format(compare_file))
 
-    comments = _compare_data(open(gold_file, "r").readlines(),
-                             open(compare_file, "r").readlines(),
-                             case)
+    comments, cnt = _compare_data(open(gold_file, "r").readlines(),
+                             open(compare_file, "r").readlines(), case)
+
+    if cnt > 0:
+        comments2, cnt2 = _compare_data(open(gold_file, "r").readlines(),
+                                        open(compare_file, "r").readlines(),
+                                        case, offset_method=True)
+        if cnt2 < cnt:
+            comments = comments2
+
     return comments == "", comments


Test patch: