internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ESMCI/cime

Issue:
get_timing needs update for ESMF 8.3.0
The format of the ESMF_Profile.summary file has changed in ESMF 8.3.0
and the get_timing routine is no longer parsing it correctly leading to 
incorrect results in the cesm timing file. 


Discussion:

Resolving patch:
diff --git a/CIME/get_timing.py b/CIME/get_timing.py
--- a/CIME/get_timing.py
+++ b/CIME/get_timing.py
@@ -108,8 +108,12 @@ def _gettime2_mct(self, heading_padded):
     def _gettime2_nuopc(self):
         self.nprocs = 0
         self.ncount = 0
-        #        expression = re.compile(r'\s*\MED:\s*\(med_phases_profile\)\s+(\d+)\s+(\d+)')
-        expression = re.compile(r"\s*\[ATM]\s*RunPhase1\s+(\d+)\s+(\d+)")
+        if self.version < 0:
+            self._get_esmf_profile_version()
+        if self.version == 0:
+            expression = re.compile(r"\s*\[ATM]\s*RunPhase1\s+(\d+)\s+(\d+)")
+        else:
+            expression = re.compile(r"\s*\[ATM]\s*RunPhase1\s+\d+\s+(\d+)\s+(\d+)")
 
         for line in self.finlines:
             match = expression.match(line)
@@ -450,7 +454,6 @@ def _getTiming(self, inst=0):
             nsteps = ncount / nprocs
         elif self._driver == "nuopc":
             nprocs, nsteps = self.gettime2("")
-
         adays = nsteps * tlen / ncpl
         odays = nsteps * tlen / ncpl
         if ocn_ncpl and inittype == "TRUE":


Test patch:
Repository: ESMCI/cime

Issue:
machine id fix #4179 breaks nersc systems 
The PR https://github.com/ESMCI/cime/pull/4179 solved the machine id problem
at NCAR but creates a new problem at NERSC.  Since both cori-knl and cori-haswell
share the same front end, this change makes it impossible to submit to cori-knl.


Discussion:

Resolving patch:
diff --git a/CIME/case/case.py b/CIME/case/case.py
--- a/CIME/case/case.py
+++ b/CIME/case/case.py
@@ -184,12 +184,13 @@ def __init__(self, case_root=None, read_only=True, record=False, non_local=False
                     machobj = Machines(machine=mach, extra_machines_dir=extra_machdir)
                 else:
                     machobj = Machines(machine=mach)
-                probed_machine = machobj.probe_machine_name()
-                if probed_machine:
-                    expect(
-                        mach == probed_machine,
-                        f"Current machine {probed_machine} does not match case machine {mach}.",
-                    )
+                if "NCAR_HOST" in os.environ:
+                    probed_machine = machobj.probe_machine_name()
+                    if probed_machine:
+                        expect(
+                            mach == probed_machine,
+                            f"Current machine {probed_machine} does not match case machine {mach}.",
+                        )
             self.initialize_derived_attributes()
 
     def check_if_comp_var(self, vid):


Test patch:
Repository: ESMCI/cime

Issue:
case.submit --prereq should disable rpointer checks
When submitting Run B that depends on Run A, it's possible (probable?) that Run A will be generating the restart files needed by Run B. Thus, the CONTINUE_RUN=TRUE [checks](https://github.com/ESMCI/cime/blob/a00a6ed2fb2e8a8dceedc3973585d7cc3d867707/CIME/case/case_submit.py#L52-L82) for existence and case-name matching of the `rpointer` file should be disabled.

Adding `and not prereq` to this conditional seems to work:
https://github.com/ESMCI/cime/blob/a00a6ed2fb2e8a8dceedc3973585d7cc3d867707/CIME/case/case_submit.py#L53


Discussion:

Resolving patch:
diff --git a/CIME/case/case_submit.py b/CIME/case/case_submit.py
--- a/CIME/case/case_submit.py
+++ b/CIME/case/case_submit.py
@@ -50,7 +50,13 @@ def _submit(
         hasMediator = False
 
     # Check if CONTINUE_RUN value makes sense
-    if job != "case.test" and case.get_value("CONTINUE_RUN") and hasMediator:
+    # if submitted with a prereq don't do this check
+    if (
+        job != "case.test"
+        and case.get_value("CONTINUE_RUN")
+        and hasMediator
+        and not prereq
+    ):
         rundir = case.get_value("RUNDIR")
         expect(
             os.path.isdir(rundir),


Test patch:
Repository: ESMCI/cime

Issue:
multi driver restart failing in cime 
ERROR: CONTINUE_RUN is true but this case does not appear to have restart files staged in /glade/scratch/joemci/f.c63074.FXHIST.f09_f09_mg17.Heelis_Multi_NoMag_LowKp.001/run rpointer.cpl



Discussion:

Resolving patch:
diff --git a/CIME/case/case_submit.py b/CIME/case/case_submit.py
--- a/CIME/case/case_submit.py
+++ b/CIME/case/case_submit.py
@@ -65,10 +65,10 @@ def _submit(
         # only checks for the first instance in a multidriver case
         if case.get_value("COMP_INTERFACE") == "nuopc":
             rpointer = "rpointer.cpl"
-        elif case.get_value("MULTI_DRIVER"):
-            rpointer = "rpointer.drv_0001"
         else:
             rpointer = "rpointer.drv"
+        if case.get_value("MULTI_DRIVER"):
+            rpointer = rpointer + "_0001"
         expect(
             os.path.exists(os.path.join(rundir, rpointer)),
             "CONTINUE_RUN is true but this case does not appear to have restart files staged in {} {}".format(


Test patch:
Repository: ESMCI/cime

Issue:
GitHub actions not triggering doc build
The GitHub actions workflow `docs.yml` didn't trigger after updating the `doc` directory.


Discussion:

Resolving patch:
diff --git a/doc/source/conf.py b/doc/source/conf.py
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -45,6 +45,7 @@
     "sphinx.ext.viewcode",
     "sphinx.ext.todo",
     "sphinxcontrib.programoutput",
+    "sphinx_rtd_theme",
 ]
 todo_include_todos = True
 
@@ -124,12 +125,13 @@
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
 
+html_css_files = ["custom.css"]
+
 # -- Options for HTMLHelp output ------------------------------------------
 
 # Output file base name for HTML help builder.
 htmlhelp_basename = "ondoc"
 
-
 # -- Options for LaTeX output ---------------------------------------------
 
 latex_elements = {


Test patch:
Repository: ESMCI/cime

Issue:
Provided project string is altered in job submission command
We use project codes of the form `<int>_<int>`. This is altered by cime, which seemingly treats the project string as a number. For instance, a project ID such as `2022_123` gets converted to `2022123`.

All job submission commands then fail due to the wrong project ID.

This happens by setting the project ID through the option `--project` or environment variable `$PROJECT` and then passed down to the batch system as a submit argument in `config_batch.xml` as follows:
```
<batch_system MACH="hydra" type="slurm">
    <submit_args>
        <arg flag="-A" name="$PROJECT"/>
    </submit_args>
</batch_system>
```

Steps to reproduce:
1. Create new case with `--project 2022_123`
    ``` 
    $ ./create_newcase --case test --res f19_g17 --compset I2000Clm50BgcCrop --project 2022_123
    ```
3. Execute `./case.setup`
4. Execute `./preview_run` to check the resulting `SUBMIT CMD`:
    ```
    [...]
        SUBMIT CMD:
          sbatch -t 120:00:00 -p skylake_mpi -A 2022123 .case.run --resubmit
    ```


Discussion:

Resolving patch:
diff --git a/CIME/XML/env_batch.py b/CIME/XML/env_batch.py
--- a/CIME/XML/env_batch.py
+++ b/CIME/XML/env_batch.py
@@ -675,11 +675,9 @@ def _resolve_argument(self, case, flag, name, job):
             else:
                 rval = val
 
-            # We don't want floating-point data
-            try:
+            # We don't want floating-point data (ignore anything else)
+            if str(rval).replace(".", "", 1).isdigit():
                 rval = int(round(float(rval)))
-            except ValueError:
-                pass
 
             # need a correction for tasks per node
             if flag == "-n" and rval <= 0:


Test patch:
Repository: ESMCI/cime

Issue:
case.build --clean-build will remove the CASEDIR contents if EXEDIR points to the CASEDIR
I'm reporting this from @dmleung. He setup some cases where EXEDIR was set to be the same as CASEDIR (so NOT a bld subdirectory of the CASEDIR).  If he then runs "./case.build --clean-build" it then erases the contents of the CASEDIR. Outside of that it actually functions fine, but if you clean your build it will cause problems. Note, that he did this with older cime versions and it seemed to work OK, but with cime6.0.45 it will remove the case directory contents.

The important thing to note is that this is something you should NOT do! So we can either close this as a WONTFIX as a degenerate case that you shouldn't do. Or a simple thing to do would be to add a simple check that EXEDIR should not be the same as CASEDIR, or there should be an error reported.


Discussion:
The feature you mean is ```./case.build --clean-all``` and yes it will delete the EXEDIR.   It's been there a long time, since sometime early in cime5 I think.    
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Resolving patch:
diff --git a/CIME/XML/env_build.py b/CIME/XML/env_build.py
--- a/CIME/XML/env_build.py
+++ b/CIME/XML/env_build.py
@@ -18,4 +18,19 @@ def __init__(
         initialize an object interface to file env_build.xml in the case directory
         """
         schema = os.path.join(utils.get_schema_path(), "env_entry_id.xsd")
+        self._caseroot = case_root
         EnvBase.__init__(self, case_root, infile, schema=schema, read_only=read_only)
+
+    def set_value(self, vid, value, subgroup=None, ignore_type=False):
+        """
+        Set the value of an entry-id field to value
+        Returns the value or None if not found
+        subgroup is ignored in the general routine and applied in specific methods
+        """
+        # Do not allow any of these to be the same as CASEROOT
+        if vid in ("EXEROOT", "OBJDIR", "LIBROOT"):
+            utils.expect(value != self._caseroot, f"Cannot set {vid} to CASEROOT")
+
+        return super(EnvBuild, self).set_value(
+            vid, value, subgroup=subgroup, ignore_type=ignore_type
+        )


Test patch:
Repository: ESMCI/cime

Issue:
better error messaging for xmllint not available
improve the error message when xmllint is not available


Discussion:
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Resolving patch:
diff --git a/CIME/XML/generic_xml.py b/CIME/XML/generic_xml.py
--- a/CIME/XML/generic_xml.py
+++ b/CIME/XML/generic_xml.py
@@ -474,7 +474,7 @@ def write(self, outfile=None, force_write=False):
         # xmllint provides a better format option for the output file
         xmllint = find_executable("xmllint")
 
-        if xmllint is not None:
+        if xmllint:
             if isinstance(outfile, str):
                 run_cmd_no_fail(
                     "{} --format --output {} -".format(xmllint, outfile),
@@ -691,8 +691,9 @@ def validate_xml_file(self, filename, schema):
         expect(os.path.isfile(filename), "xml file not found {}".format(filename))
         expect(os.path.isfile(schema), "schema file not found {}".format(schema))
         xmllint = find_executable("xmllint")
+
         expect(
-            os.path.isfile(xmllint),
+            xmllint and os.path.isfile(xmllint),
             " xmllint not found in PATH, xmllint is required for cime.  PATH={}".format(
                 os.environ["PATH"]
             ),


Test patch:
Repository: ESMCI/cime

Issue:
create_test --srcroot problem
I have encountered an issue with the create_test script while running PEM and PET tests:

1) The following worked for these tests:
${CODE_ROOT}/cime/scripts/create_test  --output-root "${CASE_ROOT}"  --machine "${MACHINE}" --project "${PROJECT}" --wait "e3sm_ice_developer"

2) This did not work:
${CODE_ROOT}/cime/scripts/create_test  --output-root "${CASE_ROOT}"  --srcroot "${CODE_ROOT}" --machine "${MACHINE}" --project "${PROJECT}" --wait "e3sm_ice_developer"

Am I using --srcroot incorrectly?

Both of the above examples worked for ERS and SMS tests, but not PEM and PET tests, for which only the first example worked.  The tests failed to build.


Discussion:
 What was the error message at the end of the failed test?
Trying to set flags on non-existent source:
  ../../../../../../gpfs/fs1/home/ac.afroberts/E3SM-Polar/code/newjontest/components/data_comps/datm/src/atm_comp_mct.F90,
  looked for
  /home/ac.afroberts/E3SM-Polar/code/newjontest/components/../../../../../../gpfs/fs1/home/ac.afroberts/E3SM-Polar/code/newjontest/components/data_comps/datm/src/atm_comp_mct.F90
@rljacob, @jasonb5: If you would like to look over an example of the failure on Anvil, see /lcrc/group/e3sm/ac.afroberts/E3SM-Polar/Test.ice.newjontest.master.E3SM-Project.anvil
@proteanplanet Thanks for the example, I'll take a look and have an update soon.
I was able to reproduce this using `/home/ac.boutte3/E3SM` for `--srcroot` on chyrsalis. CIME is passing the option to cmake correctly but cmake tries to use this as a search path generating invalid paths `/home/ac.boutte3/E3SM/components/../../../../gpfs/fs1/home/ac.boutte3/E3SM/components/stub_comps/satm/src/atm_comp_mct.F90` though `/gpfs/fs1/home/ac.boutte3/E3SM/components/stub_comps/satm/src/atm_comp_mct.F90` does exist.

Jim is going to take a look as he has more cmake experience.

Resolving patch:
diff --git a/CIME/case/case_clone.py b/CIME/case/case_clone.py
--- a/CIME/case/case_clone.py
+++ b/CIME/case/case_clone.py
@@ -54,7 +54,10 @@ def create_clone(
     if os.path.isdir(os.path.join(newcase_cimeroot, "share")) and get_model() == "cesm":
         srcroot = newcase_cimeroot
     else:
-        srcroot = os.path.join(newcase_cimeroot, "..")
+        srcroot = self.get_value("SRCROOT")
+        if not srcroot:
+            srcroot = os.path.join(newcase_cimeroot, "..")
+
     newcase = self.copy(newcasename, newcaseroot, newsrcroot=srcroot)
     with newcase:
         newcase.set_value("CIMEROOT", newcase_cimeroot)


Test patch:
Repository: ESMCI/cime

Issue:
Issue running short-term archiver when 'ncdump' is not in the PATH
This is a minor one, and an easy fix, I think, but if people more familiar with CIME want to offer guidance on whether a warning is adequate here, or an error is preferred, I'm all ears.

In short, if you run the short-term archiver and 'ncdump' isn't found, it gives a warning, but proceeds ... but incorrectly in cases where restarts list their needed history files in the 'nhfil' variable (in the 'get_histfiles_for_restarts' function).  As a result, these files get moved to the archive directory, and resubmits fail since the dependent hist files are no longer in the run directory.

The plan is to simply add 2-3 lines of python to check for 'ncdump' in the path.  I personally tend to think a failure should occur if it isn't found (unlikely, but I didn't have the 'bin' directory for NetCDF in my path), since it can lead to incorrect behavior, whereas currently it's just a warning.


Discussion:
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Resolving patch:
diff --git a/CIME/case/case_st_archive.py b/CIME/case/case_st_archive.py
--- a/CIME/case/case_st_archive.py
+++ b/CIME/case/case_st_archive.py
@@ -364,8 +364,10 @@ def get_histfiles_for_restarts(
     histfiles = set()
     rest_hist_varname = archive.get_entry_value("rest_history_varname", archive_entry)
     if rest_hist_varname != "unset":
-        cmd = "ncdump -v {} {} ".format(
-            rest_hist_varname, os.path.join(rundir, restfile)
+        ncdump = shutil.which("ncdump")
+        expect(ncdump, "ncdump not found in path")
+        cmd = "{} -v {} {} ".format(
+            ncdump, rest_hist_varname, os.path.join(rundir, restfile)
         )
         if testonly:
             out = "{} =".format(rest_hist_varname)


Test patch:
Repository: ESMCI/cime

Issue:
Test cases failing on izumi with latest cime
I see this in cime6.0.132. Running for example: SMS_D_Ld5_PS_Vmct.f10_f10_mg37.I2000Clm50Sp.izumi_nag.clm-default, with ctsm5.1.dev130 on izumi I get the following error.

```
    Case dir: /fs/cgd/data0/erik/ctsm_worktree/ctsm_maindev/cime/scripts/SMS_D_Ld5_PS_Vmct.f10_f10_mg37.I2000Clm50Sp.izumi_nag.clm-default.20230716_014111_wujjde
    Errors were:
        env_batch.xml appears to have changed, regenerating batch scripts
        manual edits to these file will be lost!
        
        submit_jobs case.test
        Submit job case.test
        ERROR: Command: 'qsub -q medium -l walltime=08:00:00 -A -v ARGS_FOR_SCRIPT='--skip-preview-namelist' .case.test' failed with error 'qsub: script file 'ARGS_FOR_SCRIPT=--skip-preview-namelist' cannot be loaded - No such file or directory' from dir '/fs/cgd/data0/erik/ctsm_worktree/ctsm_maindev/cime/scripts/SMS_D_Ld5_PS_Vmct.f10_f10_mg37.I2000Clm50Sp.izumi_nag.clm-default.20230716_014111_wujjde'

Waiting for tests to finish
```


Discussion:
git bisect shows that the offending commit is e105b94305b64697c61fe84f0a63c7150440bc47.
I can get it to work, if I set resolved=True for the place where resolved set to False above. The difference seems to be in the setting of BATCH_COMMAND_FLAGS which uses env variables as follows...


` xml
!     <entry id="BATCH_COMMAND_FLAGS" value="  -q $JOB_QUEUE -l walltime=$JOB_WALLCLOCK_TIME -A $PROJECT">
`
so it requires expansion of env variables for it to work.
Actually the issue is that there is no $PROJECT defined on izumi and so the -A option should be skipped.  

Resolving patch:
diff --git a/CIME/XML/env_batch.py b/CIME/XML/env_batch.py
--- a/CIME/XML/env_batch.py
+++ b/CIME/XML/env_batch.py
@@ -607,7 +607,6 @@ def _process_args(self, case, submit_arg_nodes, job, resolve=True):
                 flag, name = self._get_argument(case, arg)
             except ValueError:
                 continue
-
             if self._batchtype == "cobalt" and job == "case.st_archive":
                 if flag == "-n":
                     name = "task_count"
@@ -964,7 +963,17 @@ def _submit_single_job(
 
             return
 
-        submitargs = case.get_value("BATCH_COMMAND_FLAGS", subgroup=job)
+        submitargs = case.get_value("BATCH_COMMAND_FLAGS", subgroup=job, resolved=False)
+
+        project = case.get_value("PROJECT", subgroup=job)
+
+        if not project:
+            # If there is no project then we need to remove the project flag
+            # slurm defines --account only on machines that require it, so this strip isn't required
+            if batch_system == "pbs" or batch_system == "cobalt":
+                submitargs = submitargs.replace("-A", "")
+            elif batch_system == "lsf":
+                submitargs = submitargs.replace("-P", "")
 
         if dep_jobs is not None and len(dep_jobs) > 0:
             logger.debug("dependencies: {}".format(dep_jobs))


Test patch:
Repository: ESMCI/cime

Issue:
Better error message when missing ccs_config?
If you checkout CESM and then use `manage_externals` to checkout `cime` and a few components but forget to also checkout `ccs_config`, then `create_newcase` returns a weird XML error `AttributeError: 'NoneType' object has no attribute 'xml_element'` (@dabail10 was running into this issue after checking out CIME, MOM6, and CICE but not https://github.com/ESMCI/ccs_config_cesm). I think it would be great if CIME could check for `ccs_config` at the beginning of `create_newcase` and raise a more informative `FileNotFoundError` rather than waiting for the first subroutine looking for `ccs_config` to fail.


Discussion:

Resolving patch:
diff --git a/CIME/XML/grids.py b/CIME/XML/grids.py
--- a/CIME/XML/grids.py
+++ b/CIME/XML/grids.py
@@ -25,6 +25,10 @@ def __init__(self, infile=None, files=None, comp_interface=None):
             infile = files.get_value("GRIDS_SPEC_FILE")
         logger.debug(" Grid specification file is {}".format(infile))
         schema = files.get_schema("GRIDS_SPEC_FILE")
+        expect(
+            os.path.isfile(infile) and os.access(infile, os.R_OK),
+            f" grid file not found {infile}",
+        )
         try:
             GenericXML.__init__(self, infile, schema)
         except:
diff --git a/CIME/XML/machines.py b/CIME/XML/machines.py
--- a/CIME/XML/machines.py
+++ b/CIME/XML/machines.py
@@ -41,9 +41,12 @@ def __init__(self, infile=None, files=None, machine=None, extra_machines_dir=Non
         logger.debug("Verifying using schema {}".format(schema))
 
         self.machines_dir = os.path.dirname(infile)
+        if os.path.exists(infile):
+            checked_files.append(infile)
+        else:
+            expect(False, f"file not found {infile}")
 
         GenericXML.__init__(self, infile, schema)
-        checked_files.append(infile)
 
         # Append the contents of $HOME/.cime/config_machines.xml if it exists.
         #


Test patch:
diff --git a/CIME/tests/test_unit_xml_machines.py b/CIME/tests/test_unit_xml_machines.py
--- a/CIME/tests/test_unit_xml_machines.py
+++ b/CIME/tests/test_unit_xml_machines.py
@@ -140,8 +140,8 @@
 class TestUnitXMLMachines(unittest.TestCase):
     def setUp(self):
         Machines._FILEMAP = {}
-
-        self.machine = Machines()
+        # read_only=False for github testing
+        self.machine = Machines(machine="centos7-linux")
 
         self.machine.read_fd(io.StringIO(MACHINE_TEST_XML))
Repository: ESMCI/cime

Issue:
gpu_enabled flag
This issue appears on derecho after cime6.0.151.

It appears that the gpu_enabled flag is always true and thus cpu jobs are getting  ncpus set to half the value it should be. 

-lselect=8:ncpus=64:mpiprocs=128:ompthreads=1:mem=480GB:ngpus=0:mps=1 


Discussion:

Resolving patch:
diff --git a/CIME/case/case.py b/CIME/case/case.py
--- a/CIME/case/case.py
+++ b/CIME/case/case.py
@@ -458,7 +458,10 @@ def get_values(self, item, attribute=None, resolved=True, subgroup=None):
     def get_value(self, item, attribute=None, resolved=True, subgroup=None):
         if item == "GPU_ENABLED":
             if not self.gpu_enabled:
-                if self.get_value("GPU_TYPE") != "none":
+                if (
+                    self.get_value("GPU_TYPE") != "none"
+                    and self.get_value("NGPUS_PER_NODE") > 0
+                ):
                     self.gpu_enabled = True
             return "true" if self.gpu_enabled else "false"
 


Test patch:
Repository: ESMCI/cime

Issue:
ParamGen is using a deprecated package
It appears that the `distutils` package has been deprecated in Python 3.12, which is used in one location in ParamGen.  This is causing problems with some of the unit tests in CAM-SIMA, so this needs to be replaced with the still-supported `shutil` package.  A PR for this fix is incoming.


Discussion:

Resolving patch:
diff --git a/CIME/ParamGen/paramgen.py b/CIME/ParamGen/paramgen.py
--- a/CIME/ParamGen/paramgen.py
+++ b/CIME/ParamGen/paramgen.py
@@ -4,6 +4,7 @@
 from copy import deepcopy
 import logging
 import subprocess
+import shutil
 
 try:
     from paramgen_utils import is_logical_expr, is_formula, has_unexpanded_var
@@ -136,9 +137,7 @@ def from_xml_nml(cls, input_path, match="last", no_duplicates=False):
         """
 
         # First check whether the given xml file conforms to the entry_id_pg.xsd schema
-        from distutils.spawn import find_executable
-
-        xmllint = find_executable("xmllint")
+        xmllint = shutil.which("xmllint")
         if xmllint is None:
             logger.warning("Couldn't find xmllint. Skipping schema check")
         else:


Test patch: