internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ESMCI/cime

Issue:
Archive error for pop.dd files
The following tests are failing at the end of the runs.  Looks like this error was introduced with #3326.

    FAIL SMS_D.T62_g37.G1850ECO.cheyenne_intel.pop-cice_ecosys RUN time=233
    FAIL SMS_Ld20.T62_g17.G.cheyenne_intel.pop-performance_eval RUN time=93
    FAIL SMS_Ld2.f09_g17.B1850.cheyenne_intel.allactive-defaultio RUN time=160
    FAIL SMS_Ld2.f09_g17.BHIST.cheyenne_gnu.MS_Ld2.f09_g17.B1850.cheyenne_intelallactive-defaultio RUN time=499
    FAIL SMS.T62_g17.C1850ECO.cheyenne_intel.pop-ecosys RUN time=183
    FAIL SMS.T62_g37.C.cheyenne_intel.pop-default RUN time=20

This is the error message for the pop.dd files.  It's the same for all tests, just the test names
are different. 
ERROR: Failed to get extension for file 'SMS_Ld2.f09_g17.B1850.cheyenne_intel.allactive-defaultio.GC.01082020.pop.dd.0001-01-01-03600'



Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/archive_base.py b/scripts/lib/CIME/XML/archive_base.py
--- a/scripts/lib/CIME/XML/archive_base.py
+++ b/scripts/lib/CIME/XML/archive_base.py
@@ -144,6 +144,8 @@ def _get_extension(model, filepath):
     'hm'
     >>> _get_extension("mom", "bixmc5.mom6.hmz._0001_01_03_42300.nc")
     'hmz'
+    >>> _get_extension("pop", "casename.pop.dd.0001-01-02-00000")
+    'dd'
     """
     # Remove with component namechange
     if model == "fv3gfs":
@@ -166,6 +168,11 @@ def _get_extension(model, filepath):
         full_regex_str = model+r'\d?_?(\d{4})?\.('+ext_regex+r')[-\w\.]*\.nc\.?'
         full_regex = re.compile(full_regex_str)
         m = full_regex.search(basename)
+        if m is None:
+            # try without .nc part
+            full_regex_str = model+r'\d?_?(\d{4})?\.('+ext_regex+r')[-\w\.]*'
+            full_regex = re.compile(full_regex_str)
+            m = full_regex.search(basename)
         if m is not None:
             if m.group(1) is not None:
                 result = m.group(1)+'.'+m.group(2)


Test patch:
Repository: ESMCI/cime

Issue:
resubmit not working on systems where submit not allowed on compute nodes
The ssh command used on TACC systems for resubmit is not working, it currently results in a failure after ssh timeout. 



Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -690,9 +690,13 @@ def _submit_single_job(self, case, job, dep_jobs=None, allow_fail=False,
             sequence = (batchsubmit, submitargs, batchredirect, get_batch_script_for_job(job), run_args)
 
         submitcmd = " ".join(s.strip() for s in sequence if s is not None)
+        if submitcmd.startswith("ssh"):
+            # add ` before cd $CASEROOT and at end of command
+            submitcmd = submitcmd.replace("cd $CASEROOT","\'cd $CASEROOT") + "\'"
         if dry_run:
             return submitcmd
         else:
+            submitcmd = case.get_resolved_value(submitcmd)
             logger.info("Submitting job script {}".format(submitcmd))
             output = run_cmd_no_fail(submitcmd, combine_output=True)
             jobid = self.get_job_id(output)


Test patch:
Repository: ESMCI/cime

Issue:
Domain filename being stripped from input_data_list
In doing release testing, I ran into an issue where the domain filename was being stripped from the full path of the domain file.  For example

    domain1 = /glade/scratch/fischer/inputdata/atm/datm7/
Should be 
   domain1 = /glade/scratch/fischer/inputdata/atm/datm7/domain.T62.050609.nc

It looks like scripts/lib/CIME/nmlgen.py at line 496 should be
                    filepath = os.path.join(domain_filepath, filename.strip())
Instead of 
                    filepath = os.path.join(domain_filepath, os.path.dirname(filename.strip()))

I'm testing this fix now.  This was happening for several tests, but I'm concentrating on 
T62_g16.GIAF.cheyenne_intel for now.





Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/nmlgen.py b/scripts/lib/CIME/nmlgen.py
--- a/scripts/lib/CIME/nmlgen.py
+++ b/scripts/lib/CIME/nmlgen.py
@@ -493,7 +493,7 @@ def create_stream_file_and_update_shr_strdata_nml(self, config, caseroot, #pylin
                     continue
                 filepath, filename = os.path.split(filename)
                 if not filepath:
-                    filepath = os.path.join(domain_filepath, os.path.dirname(filename.strip()))
+                    filepath = os.path.join(domain_filepath, filename.strip())
                 string = "domain{:d} = {}\n".format(i+1, filepath)
                 hashValue = hashlib.md5(string.rstrip().encode('utf-8')).hexdigest()
                 if hashValue not in lines_hash:


Test patch:
Repository: ESMCI/cime

Issue:
non-netcdf history files cause a fail in testing baselines
For example pop saves a binary file pop.dd.  - currently testing will call cprnc with this file which causes a fail since it is not netcdf.  


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/hist_utils.py b/scripts/lib/CIME/hist_utils.py
--- a/scripts/lib/CIME/hist_utils.py
+++ b/scripts/lib/CIME/hist_utils.py
@@ -62,6 +62,9 @@ def copy_histfiles(case, suffix):
         num_copied += len(test_hists)
         for test_hist in test_hists:
             test_hist = os.path.join(rundir,test_hist)
+            if not test_hist.endswith('.nc'):
+                logger.info("Will not compare non-netcdf file {}".format(test_hist))
+                continue
             new_file = "{}.{}".format(test_hist, suffix)
             if os.path.exists(new_file):
                 os.remove(new_file)
@@ -246,6 +249,9 @@ def _compare_hists(case, from_dir1, from_dir2, suffix1="", suffix2="", outfile_s
         num_compared += len(match_ups)
 
         for hist1, hist2 in match_ups:
+            if not '.nc' in hist1:
+                logger.info("Ignoring non-netcdf file {}".format(hist1))
+                continue
             success, cprnc_log_file, cprnc_comment = cprnc(model, os.path.join(from_dir1,hist1),
                                                            os.path.join(from_dir2,hist2), case, from_dir1,
                                                            multiinst_driver_compare=multiinst_driver_compare,


Test patch:
Repository: ESMCI/cime

Issue:
Global name 'extra_machines_dir' is not defined
Unable to setup any cases.

./create_newcase --case test --compset X --res f19_g16
cd test
./case.setup

Error message is
  File "/glade/u/home/fischer/code/cesm2_nightly_tests/cime/scripts/Tools/../../scripts/lib/CIME/case/case_setup.py", line 122, in _case_setup_impl
    configure(Machines(machine=mach, extra_machines_dir=extra_machines_dir),
NameError: global name 'extra_machines_dir' is not defined



Discussion:
Ah, there was a conflict resolution problem when I merged my last PR. Fix coming.

Resolving patch:
diff --git a/scripts/lib/CIME/case/case_setup.py b/scripts/lib/CIME/case/case_setup.py
--- a/scripts/lib/CIME/case/case_setup.py
+++ b/scripts/lib/CIME/case/case_setup.py
@@ -80,6 +80,7 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False,
     mpilib = case.get_value("MPILIB")
     sysos = case.get_value("OS")
     comp_interface = case.get_value("COMP_INTERFACE")
+    extra_machines_dir = case.get_value("EXTRA_MACHDIR")
     expect(mach is not None, "xml variable MACH is not set")
 
     # Check that $DIN_LOC_ROOT exists - and abort if not a namelist compare tests
@@ -106,7 +107,9 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False,
         # Cannot leave case in bad state (missing env_mach_specific.xml)
         if clean and not os.path.isfile("env_mach_specific.xml"):
             case.flush()
-            configure(Machines(machine=mach), caseroot, ["Makefile"], compiler, mpilib, debug, comp_interface, sysos, noenv=True)
+            configure(Machines(machine=mach, extra_machines_dir=extra_machines_dir),
+                      caseroot, ["Makefile"], compiler, mpilib, debug, comp_interface, sysos, noenv=True,
+                      extra_machines_dir=extra_machines_dir)
             case.read_xml()
 
     if not clean:


Test patch:
Repository: ESMCI/cime

Issue:
Add a new r05_amazon grid for ROF components
We'd like to add a new grid for ROF components to run over the same region as the 5x5_amazon regional grid that is used for CTSM. We want this primarily for testing simple regional grids for mizuRoute. It'll also allow us to do CTSM/ROF configurations with very small PE counts and even a single processor with mpi-serial.

I propose this will be called 

r05_amazon

It's on the half-degree grid, but only over the 5x5_amazon region that is used by CTSM. This grid would only be allowed for either stand-alone ROF configurations or ROF with CTSM and DATM.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/grids.py b/scripts/lib/CIME/XML/grids.py
--- a/scripts/lib/CIME/XML/grids.py
+++ b/scripts/lib/CIME/XML/grids.py
@@ -111,7 +111,7 @@ def _read_config_grids(self, name, compset, atmnlev, lndnlev):
                 if compset_attrib and not_compset_attrib:
                     compset_match = re.search(compset_attrib, compset)
                     not_compset_match = re.search(not_compset_attrib, compset)
-                    if compset_match is not None and not_compset_match is not None:
+                    if compset_match is not None and not_compset_match is None:
                         foundcompset = True
                         model_gridnode = node
                         logger.debug("Found match for {} with compset_match {} and not_compset_match {}"


Test patch:
Repository: ESMCI/cime

Issue:
Add ability to pick up config_grids.xml in a user-mods-directory and append it to other grids
A group of us discussed this idea last Friday, in the context of making CESM easier to support variable resolution mesh grids. create_newcase has an option --gridfile to use a different config_grids.xml file than the standard one. This can be done to support non-standard resolutions in the system. It would be helpful if the user-mods directory option "--user-mods-dir" was enhanced to look for a config_grids.xml file in that directory, and if found append it to the standard config_grids.xml files that are used (using the include syntax that is used for these files).

I open this up for discussion. If on discussion we think this is a good idea, someone could implement it.

@mvertens @jedwards4b @fischer-ncar


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/grids.py b/scripts/lib/CIME/XML/grids.py
--- a/scripts/lib/CIME/XML/grids.py
+++ b/scripts/lib/CIME/XML/grids.py
@@ -18,8 +18,11 @@ def __init__(self, infile=None, files=None):
             infile = files.get_value("GRIDS_SPEC_FILE")
         logger.debug(" Grid specification file is {}".format(infile))
         schema = files.get_schema("GRIDS_SPEC_FILE")
+        try:
+            GenericXML.__init__(self, infile, schema)
+        except:
+            expect(False, "Could not initialize Grids")
 
-        GenericXML.__init__(self, infile, schema)
         self._version = self.get_version()
 
         self._comp_gridnames = self._get_grid_names()
diff --git a/scripts/lib/CIME/case/case.py b/scripts/lib/CIME/case/case.py
--- a/scripts/lib/CIME/case/case.py
+++ b/scripts/lib/CIME/case/case.py
@@ -1611,6 +1611,16 @@ def create(self, casename, srcroot, compset_name, grid_name,
             self.set_lookup_value("CASE", os.path.basename(casename))
             self.set_lookup_value("CASEROOT", self._caseroot)
             self.set_lookup_value("SRCROOT", srcroot)
+            # if the top level user_mods_dir contains a config_grids.xml file and
+            # gridfile was not set on the command line, use it.
+            if user_mods_dir:
+                um_config_grids = os.path.join(user_mods_dir,"config_grids.xml")
+                if os.path.exists(um_config_grids):
+                    if gridfile:
+                        logger.WARNING("A config_grids file was found in {} but also provided on the command line {}, command line takes precident".format(um_config_grids, gridfile))
+                    else:
+                        gridfile = um_config_grids
+
 
             # Configure the Case
             self.configure(compset_name, grid_name, machine_name=machine_name,


Test patch:
Repository: ESMCI/cime

Issue:
NLCOMP fails with python3 because dictionaries no longer support `has_key`
When using python3, I get:

```
$ ./case.cmpgen_namelists
Comparing namelists with baselines 'lilac_0703a'
Generating namelists to baselines 'lilac_0703b'
Exception during namelist operations:
'dict' object has no attribute 'has_key'
Traceback (most recent call last):
  File "/Users/sacks/ctsm/ctsm3/cime/scripts/Tools/../../scripts/lib/CIME/case/case_cmpgen_namelists.py", line 123, in case_cmpgen_namelists
    success, output = _do_full_nl_comp(self, test_name, compare_name, baseline_root)
  File "/Users/sacks/ctsm/ctsm3/cime/scripts/Tools/../../scripts/lib/CIME/case/case_cmpgen_namelists.py", line 45, in _do_full_nl_comp
    success, current_comments = compare_runconfigfiles(baseline_counterpart, item, test)
  File "/Users/sacks/ctsm/ctsm3/cime/scripts/Tools/../../scripts/lib/CIME/simple_compare.py", line 171, in compare_runconfigfiles
    comments = findDiff(gold_dict, compare_dict, case=case)
  File "/Users/sacks/ctsm/ctsm3/cime/scripts/Tools/../../scripts/lib/CIME/simple_compare.py", line 215, in findDiff
    if not d2.has_key(k):
AttributeError: 'dict' object has no attribute 'has_key'
```

I have a fix incoming.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/simple_compare.py b/scripts/lib/CIME/simple_compare.py
--- a/scripts/lib/CIME/simple_compare.py
+++ b/scripts/lib/CIME/simple_compare.py
@@ -212,7 +212,7 @@ def _parse_runconfig(filename):
 def findDiff(d1, d2, path="", case=None):
     comment = ""
     for k in d1.keys():
-        if not d2.has_key(k):
+        if not k in d2:
             comment += path + ":\n"
             comment +=  k + " as key not in d2\n"
         else:


Test patch:
Repository: ESMCI/cime

Issue:
Unexpected error when creating a new case
When trying to create a new case with `cime5.8.28`, using a compset that I know is tested in `aux_pop`, I got the following error:

```
STOP:
This compset and grid combination is untested in CESM.  Override this warning with the --run-unsupported option to create_newcase.
```

I poked around a bit, and think I tracked the issue down:

```
$ git diff
diff --git a/scripts/lib/CIME/case/case.py b/scripts/lib/CIME/case/case.py
index ac8fe6316..c37d0a3f4 100644
--- a/scripts/lib/CIME/case/case.py
+++ b/scripts/lib/CIME/case/case.py
@@ -1507,9 +1507,9 @@ def _check_testlists(self, compset_alias, grid_name, files):
             #   called if run_unsupported is False.
             tests = Testlist(tests_spec_file, files)
             testlist = tests.get_tests(compset=compset_alias, grid=grid_name, supported_only=True)
-            test_categories = ["prealpha", "prebeta", "test_release", "aux_"]
+            test_categories = ["prealpha", "prebeta", "test_release"]
             for test in testlist:
-                if test["category"] in test_categories \
+                if test["category"] in test_categories or test["category"].startswith("aux_") \
                    or get_cime_default_driver() in test["category"]:
                     testcnt += 1
         if testcnt > 0:
```

With the fix above, I get

```
*********************************************************************************************************************************
This compset and grid combination is not scientifically supported, however it is used in 3 tests.
*********************************************************************************************************************************
```

This seems in line with what existed [before the `test_categories`was used for the check](https://github.com/ESMCI/cime/commit/36901fe#diff-94e5f4e6427bf3d7a6f9349c2ab8c225L1466-L1470)



Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case/case.py b/scripts/lib/CIME/case/case.py
--- a/scripts/lib/CIME/case/case.py
+++ b/scripts/lib/CIME/case/case.py
@@ -1507,9 +1507,9 @@ def _check_testlists(self, compset_alias, grid_name, files):
             #   called if run_unsupported is False.
             tests = Testlist(tests_spec_file, files)
             testlist = tests.get_tests(compset=compset_alias, grid=grid_name, supported_only=True)
-            test_categories = ["prealpha", "prebeta", "test_release", "aux_"]
+            test_categories = ["prealpha", "prebeta", "test_release"]
             for test in testlist:
-                if test["category"] in test_categories \
+                if test["category"] in test_categories or "aux_" in test["category"] \
                    or get_cime_default_driver() in test["category"]:
                     testcnt += 1
         if testcnt > 0:


Test patch:
Repository: ESMCI/cime

Issue:
Missing chem_mech files in E3SM CaseDocs after renaming CAM to EAM
After [renaming CAM to EAM in E3SM](https://github.com/E3SM-Project/E3SM/pull/3845), the following two files are not copied to CaseDocs
```
chem_mech.doc
chem_mech.in
```
Need to change the 'cam' substring in 'camconf' near the end of cime/scripts/lib/CIME/case/preview_namelists.py. The piece of codes are copied below
```
# Copy over chemistry mechanism docs if they exist
    if (os.path.isdir(os.path.join(casebuild, "camconf"))):
        for file_to_copy in glob.glob(os.path.join(casebuild, "camconf", "*chem_mech*")):
            safe_copy(file_to_copy, docdir)
```
To make it work for both cam and eam, need help to replace the substring 'cam' with the atm COMP_NAME.  Thanks.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case/preview_namelists.py b/scripts/lib/CIME/case/preview_namelists.py
--- a/scripts/lib/CIME/case/preview_namelists.py
+++ b/scripts/lib/CIME/case/preview_namelists.py
@@ -107,6 +107,7 @@ def create_namelists(self, component=None):
             safe_copy(file_to_copy, docdir)
 
     # Copy over chemistry mechanism docs if they exist
-    if (os.path.isdir(os.path.join(casebuild, "camconf"))):
-        for file_to_copy in glob.glob(os.path.join(casebuild, "camconf", "*chem_mech*")):
+    atmconf = self.get_value("COMP_ATM") + "conf"
+    if (os.path.isdir(os.path.join(casebuild, atmconf))):
+        for file_to_copy in glob.glob(os.path.join(casebuild, atmconf, "*chem_mech*")):
             safe_copy(file_to_copy, docdir)


Test patch:
Repository: ESMCI/cime

Issue:
Change name of CCSM_cppdefs
At some point, we should change "CCSM_cppdefs" to something like "CIME_cppdefs".  

Maybe we could keep both for a little while (one would be a copy of the other) to give models time to adjust.


Discussion:
This should be pretty easy. @WesCoomber , you should note that this will impact the E3SM build system too.

Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -493,7 +493,7 @@ def _build_model_thread(config_dir, compclass, compname, caseroot, libroot, bldr
 def _create_build_metadata_for_component(config_dir, libroot, bldroot, case):
 ###############################################################################
     """
-    Ensure that crucial Filepath and CCSM_CPPDEFS files exist for this component.
+    Ensure that crucial Filepath and CIME_CPPDEFS files exist for this component.
     In many cases, the bld/configure script will have already created these.
     """
     bc_path = os.path.join(config_dir, "buildlib_cmake")
diff --git a/scripts/lib/CIME/buildlib.py b/scripts/lib/CIME/buildlib.py
--- a/scripts/lib/CIME/buildlib.py
+++ b/scripts/lib/CIME/buildlib.py
@@ -66,13 +66,16 @@ def build_cime_component_lib(case, compname, libroot, bldroot, use_old=True):
         elif compname.startswith('s'):
             out.write(os.path.join(cimeroot, "src", "components", "stub_comps_"+comp_interface, compname, "src") + "\n")
 
-    with open(os.path.join(confdir, "CCSM_cppdefs"), "w") as out:
+    with open(os.path.join(confdir, "CIME_cppdefs"), "w") as out:
         out.write("")
 
     # Build the component
     if get_model() != "e3sm" or use_old:
         safe_copy(os.path.join(confdir, "Filepath"), bldroot)
-        safe_copy(os.path.join(confdir, "CCSM_cppdefs"), bldroot)
+        if os.path.exists(os.path.join(confdir, "CIME_cppdefs")):
+            safe_copy(os.path.join(confdir, "CIME_cppdefs"), bldroot)
+        elif os.path.exists(os.path.join(confdir, "CCSM_cppdefs")):
+            safe_copy(os.path.join(confdir, "CCSM_cppdefs"), bldroot)
         run_gmake(case, compclass, compname, libroot, bldroot)
 
 ###############################################################################


Test patch:
Repository: ESMCI/cime

Issue:
Avoid building CDEPS with CTSM's LILAC
For now, we want to avoid building CDEPS with CTSM's LILAC because it's not needed in this configuration and CDEPS relies on unreleased ESMF code. I will submit a PR that avoids building CDEPS when using LILAC, similarly to what's done for ufs.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -370,6 +370,7 @@ def _build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, lid
             os.makedirs(shared_item)
 
     mpilib = case.get_value("MPILIB")
+    lilac_mode = case.get_value("LILAC_MODE")
     ufs_driver = os.environ.get("UFS_DRIVER")
     cpl_in_complist = False
     for l in complist:
@@ -390,7 +391,13 @@ def _build_libraries(case, exeroot, sharedpath, caseroot, cimeroot, libroot, lid
 
     # Build shared code of CDEPS nuopc data models
     build_script = {}
-    if comp_interface == "nuopc" and (not ufs_driver or ufs_driver != 'nems'):
+    if (comp_interface == "nuopc" and (not ufs_driver or ufs_driver != 'nems')
+        and lilac_mode != 'on'):
+        # For now, we avoid building CDEPS with CTSM's LILAC because it's not needed in
+        # this configuration and CDEPS relies on unreleased ESMF code. Eventually we will
+        # require CDEPS (for its streams functionality), at which point CDEPS should only
+        # require released ESMF code; then we should remove the 'lilac_mode' part of this
+        # conditional.
         libs.append("CDEPS")
 
     ocn_model = case.get_value("COMP_OCN")


Test patch:
Repository: ESMCI/cime

Issue:
test IRT_N2_Ln9.f19_g16_rx1.A failing
With pio2 the mask is read incorrectly from the ice stream data file.
This causes the test to fail since #3800 


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case/case.py b/scripts/lib/CIME/case/case.py
--- a/scripts/lib/CIME/case/case.py
+++ b/scripts/lib/CIME/case/case.py
@@ -892,9 +892,6 @@ def _setup_mach_pes(self, pecount, multi_driver, ninst, machine_name, mpilib):
             mach_pes_obj.set_value(rootpe_str, rootpe)
             mach_pes_obj.set_value(pstrid_str, pstrid)
 
-        if multi_driver:
-            mach_pes_obj.set_value("MULTI_DRIVER", True)
-
         # Make sure that every component has been accounted for
         # set, nthrds and ntasks to 1 otherwise. Also set the ninst values here.
         for compclass in self._component_classes:
@@ -909,6 +906,9 @@ def _setup_mach_pes(self, pecount, multi_driver, ninst, machine_name, mpilib):
             key = "NTHRDS_{}".format(compclass)
             if compclass not in pes_nthrds:
                 mach_pes_obj.set_value(compclass,1)
+        if multi_driver:
+            mach_pes_obj.set_value("MULTI_DRIVER", True)
+
 
     def configure(self, compset_name, grid_name, machine_name=None,
                   project=None, pecount=None, compiler=None, mpilib=None,


Test patch: