internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ESMCI/cime

Issue:
code_checker return code is wrong
Needs to return non-zero error code if files had problems.



Discussion:
this works

PYTHONPATH=/glade/u/home/jedwards/sandboxes/cesm2_0_alpha/cime/scripts/Tools:/glade/u/home/jedwards/sandboxes/cesm2_0_alpha/cime/utils/python:$PYTHONPATH /glade/u/home/jedwards/sandboxes/cesm2_0_alpha/cime/scripts/Tools/code_checker --dir /glade/u/home/jedwards/sandboxes/cesm2_0_alpha/cime/utils/python/ CIME/case.py
File ../CIME/case.py has pylint problems, please fix
   Use command: /glade/apps/opt/pylint/1.6.4/bin/pylint --disable=I,C,R,logging-not-lazy,wildcard-import,unused-wildcard-import,fixme,broad-except,bare-except,eval-used,exec-used,global-statement ../CIME/case.py
5:02
but this doesn’t

PYTHONPATH=/glade/u/home/jedwards/sandboxes/cesm2_0_alpha/cime/scripts/Tools:/glade/u/home/jedwards/sandboxes/cesm2_0_alpha/cime/utils/python:$PYTHONPATH /glade/u/home/jedwards/sandboxes/cesm2_0_alpha/cime/scripts/Tools/code_checker --dir /glade/u/home/jedwards/sandboxes/cesm2_0_alpha/cime/utils/ python/CIME/case.py


Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -335,8 +335,6 @@ def set_valid_values(self, item, valid_values):
         """
         Update or create a valid_values entry for item and populate it
         """
-        if item == "CASEROOT":
-            self._caseroot = value
         result = None
         for env_file in self._env_entryid_files:
             result = env_file.set_valid_values(item, valid_values)


Test patch:
diff --git a/utils/python/tests/list_tests b/utils/python/tests/list_tests
--- a/utils/python/tests/list_tests
+++ b/utils/python/tests/list_tests
@@ -3,6 +3,7 @@
 import unittest
 DEBUG = False
 
+#pylint: disable=protected-access
 def list_tests_from():
     loader = unittest.TestLoader()
     suite = loader.discover(".", pattern="scripts_regression_tests.py")
@@ -17,5 +18,5 @@ def list_tests_from():
                     print btestname[1][1:-1] + "." + btestname[0]
 
 if __name__ == "__main__":
-  # Include the directories
-  list_tests_from()
+    # Include the directories
+    list_tests_from()
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -1791,7 +1791,7 @@ def make_pylint_test(pyfile, cimeroot):
     def test(self):
         code_checker = os.path.join(TOOLS_DIR, "code_checker")
         run_cmd_assert_result(self, "PYTHONPATH=%s:%s:$PYTHONPATH %s --dir %s %s"\
-                                  %(TOOLS_DIR, LIB_DIR, code_checker, cimeroot, pyfile))
+                                  %(TOOLS_DIR, LIB_DIR, code_checker, cimeroot, pyfile), from_dir=cimeroot)
     return test
Repository: ESMCI/cime

Issue:
ERP test doesn't halve the number of ESP tasks
The ERP test has

```
            # halve the number of tasks and threads
            for comp in ['ATM','CPL','OCN','WAV','GLC','ICE','ROF','LND']:
```

I think ESP should be included here, too. Without it, we are sometimes getting PE layouts where most components are running on N tasks, but ESP is running on 2N tasks.



Discussion:
There is now a method to get the components_class list and that should be used.
comp_classes = case.get_values("COMP_CLASSES")

I'll make this change along with the fix for #687 


Resolving patch:
diff --git a/utils/python/CIME/SystemTests/erp.py b/utils/python/CIME/SystemTests/erp.py
--- a/utils/python/CIME/SystemTests/erp.py
+++ b/utils/python/CIME/SystemTests/erp.py
@@ -61,7 +61,9 @@ def build_phase(self, sharedlib_only=False, model_only=False):
 
             if (bld == 2):
                 # halve the number of tasks and threads
-                for comp in ['ATM','CPL','OCN','WAV','GLC','ICE','ROF','LND']:
+                for comp in self._case.get_values("COMP_CLASSES"):
+                    if comp == "DRV":
+                        comp = "CPL"
                     ntasks    = self._case.get_value("NTASKS_%s"%comp)
                     nthreads  = self._case.get_value("NTHRDS_%s"%comp)
                     rootpe    = self._case.get_value("ROOTPE_%s"%comp)


Test patch:
diff --git a/cime_config/config_tests.xml b/cime_config/config_tests.xml
--- a/cime_config/config_tests.xml
+++ b/cime_config/config_tests.xml
@@ -177,7 +177,6 @@ LII    CLM initial condition interpolation test
     <HIST_OPTION>ndays</HIST_OPTION>
     <HIST_N>1</HIST_N>
     <DOUT_S>FALSE</DOUT_S>
-    <BUILD_THREADED>TRUE</BUILD_THREADED>
   </test>
 
   <test NAME="ERI">
@@ -186,7 +185,6 @@ LII    CLM initial condition interpolation test
     <STOP_OPTION>ndays</STOP_OPTION>
     <STOP_N>22</STOP_N>
     <DOUT_S>FALSE</DOUT_S>
-    <BUILD_THREADED>TRUE</BUILD_THREADED>
   </test>
 
   <test NAME="ERP">
Repository: ESMCI/cime

Issue:
More issues with user_mods and test dependencies
Similar to #719 (and the issue it raised with how #712 was fixed), if a testmod_dir depends on another testmod_dir and they want to change the same XML variable to different values, the test higher up the chain should take precendent. Using the example from #719, if our test is

```
          ciso_abio
          /       \
        abio     ciso
        /          \
    default     default
```

Then `xmlchange` commands are being called in the following order:

```
ciso_abio
abio
ciso
default
```

whereas we want

```
default
ciso
abio
ciso_abio
```

Note that this is an issue even if we are only dependent on a single test. If our test is

```
ecosys_r4
    |
 ecosys
    |
 default
```

Then we expect `xmlchange` calls from `ecosys_r4` to take precedence over those in `default` but in practice the order is reversed. I have a branch that fixes this (pull request coming shortly) but it needs a critical set of eyes on it.


Discussion:

Resolving patch:
diff --git a/utils/python/CIME/user_mod_support.py b/utils/python/CIME/user_mod_support.py
--- a/utils/python/CIME/user_mod_support.py
+++ b/utils/python/CIME/user_mod_support.py
@@ -56,11 +56,25 @@ def apply_user_mods(caseroot, user_mods_path, ninst=None):
                             expect(False, "Could not write file %s in caseroot %s"
                                    %(case_source_mods,caseroot))
 
+    # Reverse include_dirs to make sure xmlchange commands are called in the
+    # correct order; it may be desireable to reverse include_dirs above the
+    # previous loop and then append user_nl changes rather than prepend them.
+    include_dirs.reverse()
+    for include_dir in include_dirs:
         # create xmlchange_cmnds and shell_commands in caseroot
         shell_command_files = glob.glob(os.path.join(include_dir,"shell_commands")) +\
                               glob.glob(os.path.join(include_dir,"xmlchange_cmnds"))
         for shell_commands_file in shell_command_files:
             case_shell_commands = shell_commands_file.replace(include_dir, caseroot)
+            # add commands from both shell_commands and xmlchange_cmnds to
+            # the same file (caseroot/shell_commands)
+            case_shell_commands = case_shell_commands.replace("xmlchange_cmnds","shell_commands")
+            # Note that use of xmlchange_cmnds has been deprecated and will soon
+            # be removed altogether, so new tests should rely on shell_commands
+            if shell_commands_file.endswith("xmlchange_cmnds"):
+                logger.warn("xmlchange_cmnds is deprecated and will be removed " +\
+                            "in a future release; please rename %s shell_commands" %\
+                            shell_commands_file)
             with open(shell_commands_file,"r") as fd:
                 new_shell_commands = fd.read().replace("xmlchange","xmlchange --force")
             with open(case_shell_commands, "a") as fd:


Test patch:
Repository: ESMCI/cime

Issue:
 <!--  <GMAKE>make</GMAKE> <- this doesn't actually work! -->
This line is present in several acme/machines and for melvin in cesm/machines. This tag works, the comment is confusing, is there a reason it is still there?



Discussion:
@jedwards4b do you know?

 I think that at least in the past (and possibly presently) there are some places in the code where make or gmake is hard coded instead of using the GMAKE xml variable.    Maybe someone could take on confirming that these have all been removed and then remove the comment.

Maybe change the comment to "Warning: not all scripts use this variable"


Resolving patch:
diff --git a/utils/python/CIME/SystemTests/homme.py b/utils/python/CIME/SystemTests/homme.py
--- a/utils/python/CIME/SystemTests/homme.py
+++ b/utils/python/CIME/SystemTests/homme.py
@@ -28,12 +28,13 @@ def build_phase(self, sharedlib_only=False, model_only=False):
             basegen  = self._case.get_value("BASEGEN_CASE")
             basecmp  = self._case.get_value("BASECMP_CASE")
             generate = self._case.get_value("GENERATE_BASELINE")
+            gmake    = self._case.get_value("GMAKE")
 
             basename = basegen if generate else basecmp
             cmake_cmd = "cmake -C %s/components/homme/cmake/machineFiles/%s.cmake -DUSE_NUM_PROCS=%s %s/components/homme -DHOMME_BASELINE_DIR=%s/%s >& homme.bldlog" % (srcroot, mach, procs, srcroot, baseline, basename)
 
             run_cmd_no_fail(cmake_cmd, from_dir=exeroot)
-            run_cmd_no_fail("make -j8 >> homme.bldlog 2>&1", from_dir=exeroot)
+            run_cmd_no_fail("%s -j8 >> homme.bldlog 2>&1" % gmake, from_dir=exeroot)
 
             post_build(self._case, [os.path.join(exeroot, "homme.bldlog")])
 
@@ -45,6 +46,7 @@ def run_phase(self):
         compare  = self._case.get_value("COMPARE_BASELINE")
         generate = self._case.get_value("GENERATE_BASELINE")
         basegen  = self._case.get_value("BASEGEN_CASE")
+        gmake    = self._case.get_value("GMAKE")
 
         log = os.path.join(rundir, "homme.log")
         if os.path.exists(log):
@@ -52,11 +54,11 @@ def run_phase(self):
 
         if generate:
             full_baseline_dir = os.path.join(baseline, basegen, "tests", "baseline")
-            run_cmd_no_fail("make -j 4 baseline >& %s" % log, from_dir=exeroot)
+            run_cmd_no_fail("%s -j 4 baseline >& %s" % (gmake, log), from_dir=exeroot)
             if os.path.isdir(full_baseline_dir):
                 shutil.rmtree(full_baseline_dir)
             shutil.copytree(os.path.join(exeroot, "tests", "baseline"), full_baseline_dir)
         elif compare:
-            run_cmd_no_fail("make -j 4 check >& %s" % log, from_dir=exeroot)
+            run_cmd_no_fail("%s -j 4 check >& %s" % (gmake, log), from_dir=exeroot)
         else:
-            run_cmd_no_fail("make -j 4 baseline >& %s" % log, from_dir=exeroot)
+            run_cmd_no_fail("%s -j 4 baseline >& %s" % (gmake, log), from_dir=exeroot)


Test patch:
Repository: ESMCI/cime

Issue:
environment variables in env_mach_specific.xml
Are being set in spite of qualifiers.  In particular debug="true" does not seem to be honored in the following:
```
  <environment_variables debug="true">
    <env name="MP_EAGER_LIMIT">0</env>
  </environment_variables>
```


Discussion:
The problem turned out to be an errant comma in case.py in load_env. 

Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -1054,7 +1054,7 @@ def set_model_version(self, model):
     def load_env(self):
         if not self._is_env_loaded:
             compiler = self.get_value("COMPILER")
-            debug=self.get_value("DEBUG"),
+            debug=self.get_value("DEBUG")
             mpilib=self.get_value("MPILIB")
             env_module = self.get_env("mach_specific")
             env_module.load_env(compiler=compiler,debug=debug, mpilib=mpilib)


Test patch:
Repository: ESMCI/cime

Issue:
module load errors
Module load errors are not echoed to the screen and do not cause the build to abort.


Discussion:
This seems to be system dependent.  I found the problem on cori and tried to reproduce on yellowstone, but yellowstone seems to correctly error and abort if a module error occurs.  
The module load call is issued from env_mach_specific function _load_module_modules which
calls run_cmd_no_fail from utils.py.   This function is looking for a non-zero return code but the module system is returning a zero return code even though the err code is written to stderr.

Resolving patch:
diff --git a/utils/python/CIME/XML/env_mach_specific.py b/utils/python/CIME/XML/env_mach_specific.py
--- a/utils/python/CIME/XML/env_mach_specific.py
+++ b/utils/python/CIME/XML/env_mach_specific.py
@@ -244,7 +244,9 @@ def _get_module_commands(self, modules_to_load, shell):
     def _load_module_modules(self, modules_to_load):
         for cmd in self._get_module_commands(modules_to_load, "python"):
             logger.debug("module command is %s"%cmd)
-            py_module_code = run_cmd_no_fail(cmd)
+            stat, py_module_code, errout = run_cmd(cmd)
+            expect(stat==0 and len(errout) == 0, 
+                   "module command %s failed with message:\n%s"%(cmd,errout))
             exec(py_module_code)
 
     def _load_soft_modules(self, modules_to_load):


Test patch:
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -1290,7 +1290,7 @@ class MakefileTester(object):
     _makefile_template = """
 include Macros
 query:
-/techo '$({})' > query.out
+\techo '$({})' > query.out
 """
 
     def __init__(self, parent, make_string):
Repository: ESMCI/cime

Issue:
Divide by zero error in xmlchange
A relatively simple set of valid commands results in divide by zero errors from xmlchange.

```
./create_newcase -case <caseroot> -mach yellowstone -res ne30_ne30 -project P93300642 -compset FC5
cd <caseroot>
./xmlchange MAX_TASKS_PER_NODE=15
./xmlchange CALENDAR=NO_LEAP

```
```
Traceback (most recent call last):
  File "./xmlchange", line 148, in <module>
    _main_func(__doc__)
  File "./xmlchange", line 145, in _main_func
    xmlchange(caseroot, listofsettings, xmlid, xmlval, subgroup, append, noecho, force , dry)
  File "./xmlchange", line 97, in xmlchange
    with Case(caseroot, read_only=False) as case:
  File "/glade/u/home/goldy/Projects/camTrunk/cime/scripts/Tools/../../utils/python/CIME/case.py", line 104, in __init__
    self.initialize_derived_attributes()
  File "/glade/u/home/goldy/Projects/camTrunk/cime/scripts/Tools/../../utils/python/CIME/case.py", line 121, in initialize_derived_attributes
    self.cores_per_task = ((self.get_value("MAX_TASKS_PER_NODE")/smt_factor) \
ZeroDivisionError: integer division or modulo by zero

```
Also, other xmlchange commands trigger the same error message, e.g., 
`./xmlchange NTASKS_ATM=15
`
Oddly enough, I can not seem to reproduce this in a CIME-only configuration (A case). Is something one of sync between CIME and CAM?

Both the FC5 case and the A case were from the head of master (3c8ceaa3ddb88f7d653c0acafc9f693323f42452).


Discussion:
Note this bug was first found by Jerry Olson, I just narrowed it down a bit.

Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -116,7 +116,7 @@ def initialize_derived_attributes(self):
         logger.debug("total_tasks %s thread_count %s"%(total_tasks, self.thread_count))
         self.num_nodes = env_mach_pes.get_total_nodes(total_tasks, self.thread_count)
         self.tasks_per_numa = int(math.ceil(self.tasks_per_node / 2.0))
-        smt_factor = self.get_value("MAX_TASKS_PER_NODE")/self.get_value("PES_PER_NODE")
+        smt_factor = max(1,int(self.get_value("MAX_TASKS_PER_NODE")/self.get_value("PES_PER_NODE")))
 
         self.cores_per_task = ((self.get_value("MAX_TASKS_PER_NODE")/smt_factor) \
                                / self.tasks_per_node) * 2
@@ -974,7 +974,7 @@ def create_clone(self, newcase, keepexe=False, mach_dir=None, project=None):
         # copy user_ files
         cloneroot = self._caseroot
         files = glob.glob(cloneroot + '/user_*')
-        
+
         for item in files:
             shutil.copy(item, newcaseroot)
 


Test patch:
Repository: ESMCI/cime

Issue:
Remove PESetupHist from cime.setup
When running cime.setup --clean, we don't need to save files to PESetupHist



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/case_setup.py b/utils/python/CIME/case_setup.py
--- a/utils/python/CIME/case_setup.py
+++ b/utils/python/CIME/case_setup.py
@@ -105,16 +105,7 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False):
 
     # Create batch script
     if reset or clean:
-        # Clean batch script
-
-        backup_dir = "PESetupHist/b.%s" % time.strftime("%y%m%d-%H%M%S")
-        if not os.path.isdir(backup_dir):
-            os.makedirs(backup_dir)
-
         # back up relevant files
-        for fileglob in ["case.run", "env_build.xml", "env_mach_pes.xml", "Macros*"]:
-            for filename in glob.glob(fileglob):
-                shutil.copy(filename, backup_dir)
         if os.path.exists("case.run"):
             os.remove("case.run")
 
@@ -125,20 +116,15 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False):
 
             # backup and then clean test script
             if os.path.exists("case.test"):
-                shutil.copy("case.test", backup_dir)
                 os.remove("case.test")
                 logger.info("Successfully cleaned test script case.test")
 
             if os.path.exists("case.testdriver"):
-                shutil.copy("case.testdriver", backup_dir)
                 os.remove("case.testdriver")
                 logger.info("Successfully cleaned test script case.testdriver")
 
         logger.info("Successfully cleaned batch script case.run")
 
-        logger.info("Successfully cleaned batch script case.run")
-        logger.info("Some files have been saved to %s" % backup_dir)
-
         msg = "case.setup clean complete"
         append_status(msg, caseroot=caseroot, sfile="CaseStatus")
 


Test patch:
Repository: ESMCI/cime

Issue:
pelayout tool 
A tool to display and update the NTASKS, NTHRDS and ROOTPE fields from xml is desired.
This tool should have customizable io formats and should warn the user if an update is questionable. 
For example if ROOTPE_OCN < NTASKS_ATM in a B compset then a warning should be delivered.


Discussion:
The warning sounds cool but also hard.  Lets get the printing working first.  @gold2718 do you still want to do this?
I'd love to have a go (I've been playing with this for a while and have a script that I use to do this sort of thing). I'll try to put some ideas together this weekend . . .

Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -811,7 +811,8 @@ def _create_caseroot_tools(self):
                     os.path.join(toolsdir, "check_case"),
                     os.path.join(toolsdir, "archive_metadata.sh"),
                     os.path.join(toolsdir, "xmlchange"),
-                    os.path.join(toolsdir, "xmlquery"))
+                    os.path.join(toolsdir, "xmlquery"),
+                    os.path.join(toolsdir, "pelayout"))
         try:
             for exefile in exefiles:
                 destfile = os.path.join(self._caseroot,os.path.basename(exefile))


Test patch:
Repository: ESMCI/cime

Issue:
driver config_component.xml with compset attributes need model specific attribute
The config_component.xml has compset attributes that are not always the same for cesm and acme. This became particularly true in the recent PR where cesm coupling frequencies were changed and this effected the acme defaults incorrectly. 
To mitigate this in the future, all value elements in config_component.xml  also need a cime_model attribute so that cesm developments can be independent of acme developments. 



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -90,6 +90,7 @@ def __init__(self, case_root=None, read_only=True):
         self._components = []
         self._component_classes = []
         self._is_env_loaded = False
+        self._cime_model = get_model()
 
         self.thread_count = None
         self.tasks_per_node = None
@@ -485,12 +486,14 @@ def _set_comp_classes(self, comp_classes):
             env_file.set_components(comp_classes)
 
     def _get_component_config_data(self):
-        # attributes used for multi valued defaults ($attlist is a hash reference)
-        attlist = {"compset":self._compsetname, "grid":self._gridname}
+        # attributes used for multi valued defaults 
+        # attlist is a dictionary used to determine the value element that has the most matches
+        attlist = {"compset":self._compsetname, "grid":self._gridname, "cime_model":self._cime_model}
 
         # Determine list of component classes that this coupler/driver knows how
         # to deal with. This list follows the same order as compset longnames follow.
         files = Files()
+
         # Add the group and elements for the config_files.xml
         for env_file in self._env_entryid_files:
             env_file.add_elements_by_group(files, attlist)
@@ -498,6 +501,7 @@ def _get_component_config_data(self):
         drv_comp = Component(drv_config_file)
         for env_file in self._env_entryid_files:
             env_file.add_elements_by_group(drv_comp, attributes=attlist)
+
         # Add the group and elements for env_batch
         env_batch = self.get_env("batch")
         env_batch.add_elements_by_group(drv_comp, attributes=attlist)


Test patch:
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -1141,6 +1141,12 @@ def test_cime_case(self):
                             msg="Build complete had wrong value '%s'" %
                             build_complete)
 
+            model_specific_val = case.get_value("CPL_SEQ_OPTION")
+            if CIME.utils.get_model() == 'cesm':
+                self.assertEqual(model_specific_val, "RASM_OPTION1")
+            else:
+                self.assertEqual(model_specific_val, "CESM1_MOD")
+
             # Test some test properties
             self.assertEqual(case.get_value("TESTCASE"), "TESTRUNPASS")