internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ESMCI/cime

Issue:
redirect_logger is loosing stdout
Although we are capturing stderr and logger messages nicely now, we are loosing stdout.
You can see this by looking at the gptl or csm_share bldlog


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -3,7 +3,7 @@
 Warning: you cannot use CIME Classes in this module as it causes circular dependencies
 """
 import io, logging, gzip, sys, os, time, re, shutil, glob, string, random, imp
-import errno, signal, traceback, warnings, filecmp
+import errno, signal, warnings, filecmp
 import stat as statlib
 import six
 from contextlib import contextmanager
@@ -249,15 +249,6 @@ def run_sub_or_cmd(cmd, cmdargs, subname, subargs, logfile=None, case=None,
     except AttributeError:
         do_run_cmd = True
 
-    except BaseException as e:
-        if logfile:
-            msg = e.__str__()
-            excmsg = "Exception during build:\n{}\n{}".format(msg, traceback.format_exc())
-            with open(logfile, "a") as fd:
-                fd.write(excmsg)
-
-        raise
-
     if do_run_cmd:
         logger.info("   Running {} ".format(cmd))
         if case is not None:
@@ -275,6 +266,7 @@ def run_sub_or_cmd(cmd, cmdargs, subname, subargs, logfile=None, case=None,
         # refresh case xml object from file
         if case is not None:
             case.read_xml()
+
     return stat, output, errput
 
 def run_cmd(cmd, input_str=None, from_dir=None, verbose=None,
@@ -1502,3 +1494,16 @@ def __enter__(self):
     def __exit__(self, *_):
         if self._seconds is not None:
             signal.alarm(0)
+
+def filter_unicode(unistr):
+    """
+    Sometimes unicode chars can cause problems
+    """
+    return "".join([i if ord(i) < 128 else ' ' for i in unistr])
+
+def run_bld_cmd_ensure_logging(cmd, arg_logger, from_dir=None):
+    arg_logger.info(cmd)
+    stat, output, errput = run_cmd(cmd, from_dir=from_dir)
+    arg_logger.info(output)
+    arg_logger.info(errput)
+    expect(stat == 0, filter_unicode(errput))


Test patch:
Repository: ESMCI/cime

Issue:
template for log file name format
We should have a template for the log file name format so that users can add jobid or other 
information to the log file names.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -1354,9 +1354,23 @@ def get_lids(case):
 
 def new_lid():
     lid = time.strftime("%y%m%d-%H%M%S")
+    jobid = batch_jobid()
+    if jobid is not None:
+        lid = jobid+'.'+lid
     os.environ["LID"] = lid
     return lid
 
+def batch_jobid():
+    jobid = os.environ.get("PBS_JOBID")
+    if jobid is None:
+        jobid = os.environ.get("SLURM_JOB_ID")
+    if jobid is None:
+        jobid = os.environ.get("LSB_JOBID")
+    if jobid is None:
+        jobid = os.environ.get("COBALT_JOBID")
+    return jobid
+
+
 def analyze_build_log(comp, log, compiler):
     """
     Capture and report warning count,


Test patch:
Repository: ESMCI/cime

Issue:
CLM specific tests aren't working with cime5.4.0.alpha.15
The CLM specific tests, LII and SSP aren't working in cime5.4.0.alpha.15. Here's an example of what it does...

Setting resource.RLIMIT_STACK to -1 from (307200000, -1)
Machine/Decomp/Pes configuration has already been done ...skipping
If an old case build already exists, might want to run 'case.build --clean' before building
scripts/LII_D_Ld3.f19_g17_gl4.I2000Clm50BgcCrop.cheyenne_intel.clm-glcMEC_spunup_1way.GC.beta08n01_r269chintel> ./case.build 
Building test for LII in directory /glade/p/work/erik/clm_cesm20beta08externals/cime/scripts/LII_D_Ld3.f19_g17_gl4.I2000Clm50BgcCrop.cheyenne_intel.clm-glcMEC_spunup_1way.GC.beta08n01_r269chintel
ERROR: Test LII not found, aborting


Discussion:
The directory for CLM tests seems to be set correctly.

./xmlquery COMP_ROOT_DIR_LND,SYSTEM_TESTS_DIR                       
Results in group case_comps
	COMP_ROOT_DIR_LND: /glade/p/work/erik/clm_cesm20beta08externals/components/clm/

Results in group test
	SYSTEM_TESTS_DIR: /glade/p/work/erik/clm_cesm20beta08externals/cime/scripts/lib/CIME/SystemTests


Resolving patch:
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -1298,12 +1298,10 @@ def find_system_test(testname, case):
     else:
         components = ["any"]
         components.extend( case.get_compset_components())
-        env_test = case.get_env("test")
         fdir = []
         for component in components:
-            tdir = env_test.get_value("SYSTEM_TESTS_DIR",
+            tdir = case.get_value("SYSTEM_TESTS_DIR",
                                       attribute={"component":component})
-
             if tdir is not None:
                 tdir = os.path.abspath(tdir)
                 system_test_file = os.path.join(tdir  ,"{}.py".format(testname.lower()))


Test patch:
Repository: ESMCI/cime

Issue:
use MAX_MPI_TASKS_PER_NODE for core limit in test_schedular
The current scheme fails to run tests on small systems that should be okay even though they overload the node. 


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/SystemTests/pet.py b/scripts/lib/CIME/SystemTests/pet.py
--- a/scripts/lib/CIME/SystemTests/pet.py
+++ b/scripts/lib/CIME/SystemTests/pet.py
@@ -46,6 +46,7 @@ def _case_two_setup(self):
         # to the batch submission, things break. Setting MAX_TASKS_PER_NODE to half of
         # it original value prevents this.
         self._case.set_value("MAX_TASKS_PER_NODE", int(self._case.get_value("MAX_TASKS_PER_NODE") / 2))
+        self._case.set_value("MAX_MPITASKS_PER_NODE", int(self._case.get_value("MAX_MPITASKS_PER_NODE") / 2))
 
         # Need to redo case_setup because we may have changed the number of threads
         case_setup(self._case, reset=True)


Test patch:
diff --git a/scripts/lib/CIME/test_scheduler.py b/scripts/lib/CIME/test_scheduler.py
--- a/scripts/lib/CIME/test_scheduler.py
+++ b/scripts/lib/CIME/test_scheduler.py
@@ -157,7 +157,7 @@ def __init__(self, test_names, test_data=None,
 
         if parallel_jobs is None:
             self._parallel_jobs = min(len(test_names),
-                                      self._machobj.get_value("MAX_TASKS_PER_NODE"))
+                                      self._machobj.get_value("MAX_MPITASKS_PER_NODE"))
         else:
             self._parallel_jobs = parallel_jobs
 
@@ -203,7 +203,7 @@ def __init__(self, test_names, test_data=None,
 
         # Oversubscribe by 1/4
         if proc_pool is None:
-            pes = int(self._machobj.get_value("MAX_MPITASKS_PER_NODE"))
+            pes = int(self._machobj.get_value("MAX_TASKS_PER_NODE"))
             self._proc_pool = int(pes * 1.25)
         else:
             self._proc_pool = int(proc_pool)
Repository: ESMCI/cime

Issue:
scripts_regression_tests: A_RunUnitTests fail from detached head
If I have checked out master (currently at f44bca3), `./scripts_regression_tests.py A_RunUnitTests` passes. However, if I have a detached head checked out at f44bca3, I get this failure:

```
======================================================================
FAIL: test_CIME_doctests (__main__.A_RunUnitTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./scripts_regression_tests.py", line 112, in test_CIME_doctests
    run_cmd_assert_result(self, "PYTHONPATH=%s:$PYTHONPATH python -m doctest *.py 2>&1" % LIB_DIR, from_dir=os.path.join(LIB_DIR,"CIME"))
  File "./scripts_regression_tests.py", line 60, in run_cmd_assert_result
    test_obj.assertEqual(stat, expected_stat, msg=msg)
AssertionError:
COMMAND: PYTHONPATH=/glade/p/work/sacks/cime/scripts/lib:$PYTHONPATH python -m doctest *.py 2>&1
FROM_DIR: /glade/p/work/sacks/cime/scripts/lib/CIME
SHOULD HAVE WORKED, INSTEAD GOT STAT 1
OUTPUT: No handlers could be found for logger "utils"
**********************************************************************
File "utils.py", line 537, in utils.get_current_branch
Failed example:
    get_current_branch() is not None
Expected:
    True
Got:
    False
**********************************************************************
1 items had failures:
   1 of   1 in utils.get_current_branch
***Test Failed*** 1 failures.
ERRPUT:


----------------------------------------------------------------------
Ran 4 tests in 1.208s

FAILED (failures=1)
```


Discussion:
I will fix.

Resolving patch:
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -534,7 +534,11 @@ def get_current_branch(repo=None):
     """
     Return the name of the current branch for a repository
 
-    >>> get_current_branch() is not None
+    >>> if "GIT_BRANCH" in os.environ:
+    ...     get_current_branch() is not None
+    ... else:
+    ...     os.environ["GIT_BRANCH"] = "foo"
+    ...     get_current_branch() == "foo"
     True
     """
     if ("GIT_BRANCH" in os.environ):


Test patch:
Repository: ESMCI/cime

Issue:
Add option for PGI compiler on cheyenne
I didn't think CISL was going to do this. But, they've added the PGI compiler as an option to modules for both the newer 17.4 and 17.9 versions of the compiler. Should we add the ability to compile with PGI on cheyenne since it's now available (especially since we lost the yellowstone_pgi option)?

This would be the newest version of PGI that would be used in config_machines.xml. The closest is hobart which is 17.04.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/buildlib.py b/scripts/lib/CIME/buildlib.py
--- a/scripts/lib/CIME/buildlib.py
+++ b/scripts/lib/CIME/buildlib.py
@@ -82,4 +82,4 @@ def run_gmake(case, compclass, libroot, bldroot, libname="", user_cppdefs=""):
         cmd = cmd + "USER_CPPDEFS='{}'".format(user_cppdefs )
 
     _, out, _ = run_cmd(cmd, combine_output=True)
-    print(out)
+    print(out.encode('utf-8'))


Test patch:
Repository: ESMCI/cime

Issue:
ERR test not setting DOUT_S
It should be TRUE but its currently false on master, git blame doesn't make any sense to me this must have just happened because ERR tests will fail when DOUT_S is FALSE. 


Discussion:
DOUT_S should be set in err.py in _case_one_setup but this routine does not appear to be called.
Seems to be a race condition?
What I'm seeing is that at case.build time the DOUT_S in the first case is set correctly to TRUE 
but then in _case_two_setup the case1 value is being set back to false - there should be two case objects in these compare_two style tests - but I think maybe there is only one?
Yes I think that the recent caching changes somehow cause these objects not to be distinct.
I've traced this to commit 263c909

Resolving patch:
diff --git a/scripts/lib/CIME/XML/generic_xml.py b/scripts/lib/CIME/XML/generic_xml.py
--- a/scripts/lib/CIME/XML/generic_xml.py
+++ b/scripts/lib/CIME/XML/generic_xml.py
@@ -33,7 +33,7 @@ def __deepcopy__(self, _):
 class GenericXML(object):
 
     _FILEMAP = {}
-    DISABLE_CACHING = False
+    DISABLE_CACHING = True
 
     @classmethod
     def invalidate_file(cls, filepath):


Test patch:
Repository: ESMCI/cime

Issue:
PR #2260 has caused some cime_developer tests to fail on anlworkstation during phase SETUP
The errors can be reproduced on anlworkstation by running a single cime_developer test:
`./create_test SMS.T42_T42.S --machine=anlworkstation --test-root=****`

Errors were:
```
        Traceback (most recent call last):
          File "./case.setup", line 58, in <module>
            _main_func(__doc__)
          File "./case.setup", line 55, in _main_func
            case_setup(case, clean=clean, test_mode=test_mode, reset=reset)
          File "scripts/lib/CIME/case_setup.py", line 233, in case_setup
            run_and_log_case_status(functor, phase, caseroot=caseroot)
          File "scripts/lib/CIME/utils.py", line 1447, in run_and_log_case_status
            rv = func()
          File "scripts/lib/CIME/case_setup.py", line 227, in <lambda>
            functor = lambda: _case_setup_impl(case, caseroot, clean, test_mode, reset)
          File "scripts/lib/CIME/case_setup.py", line 97, in _case_setup_impl
            case.load_env()
          File "scripts/lib/CIME/case.py", line 1300, in load_env
            env_module.load_env(self)
          File "scripts/lib/CIME/XML/env_mach_specific.py", line 78, in load_env
            self._load_modules(modules_to_load, force_method=force_method)
          File "scripts/lib/CIME/XML/env_mach_specific.py", line 102, in _load_modules
            self._load_modules_generic(modules_to_load)
          File "scripts/lib/CIME/XML/env_mach_specific.py", line 295, in _load_modules_generic
            newenv[lastkey] += "\n" + line
        KeyError: None
```
Do we need some settings on anlworkstation to make it work?


Discussion:
THis is a duplicate of #2262 
@jedwards4b , I don't believe this is a duplicate. This is a failure in the main code path, not a test. I will investigate.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_mach_specific.py b/scripts/lib/CIME/XML/env_mach_specific.py
--- a/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/scripts/lib/CIME/XML/env_mach_specific.py
@@ -291,7 +291,7 @@ def _load_modules_generic(self, modules_to_load):
                 key, val = line.split("=", 1)
                 newenv[key] = val
                 lastkey = key
-            else:
+            elif lastkey is not None:
                 newenv[lastkey] += "\n" + line
 
         # resolve variables


Test patch:
Repository: ESMCI/cime

Issue:
userdefined_laptop_template files need to be changed to work
It appears that the required format for `config_machines.xml` and maybe `config_compilers.xml` has been changed; this should be updated in the templates in `userdefined_laptop_template`. These should no longer have the outer `<config_machines>` (and maybe `<config_compilers>` tags).

See the last couple of comments in #2215 for discussion.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/generic_xml.py b/scripts/lib/CIME/XML/generic_xml.py
--- a/scripts/lib/CIME/XML/generic_xml.py
+++ b/scripts/lib/CIME/XML/generic_xml.py
@@ -96,7 +96,12 @@ def read(self, infile, schema=None):
 
     def read_fd(self, fd):
         if self.tree:
-            self.add_child(_Element(ET.parse(fd).getroot()))
+            addroot = _Element(ET.parse(fd).getroot())
+            if addroot.xml_element.tag == self.name(self.root):
+                for child in self.get_children(root=addroot):
+                    self.add_child(child)
+            else:
+                self.add_child(addroot)
         else:
             self.tree = ET.parse(fd)
             self.root = _Element(self.tree.getroot())


Test patch:
Repository: ESMCI/cime

Issue:
"ERROR: locked" when .config_files.xml is present in SRCROOT
If a `.config_files.xml` file is present in `SRCROOT` (as is the case for a standalone CTSM checkout), I'm getting `ERROR: locked` when trying to create a case from cime master:

```
git clone git@github.com:ESCOMP/ctsm.git
cd ctsm
./manage_externals/checkout_externals
cd cime
git remote add esmci git@github.com:ESMCI/cime.git
git fetch esmci
git checkout esmci/master
cd scripts
./create_newcase --case test_0207a --compset A --res f19_g16_rx1
```

This gives:

```
ERROR: locked
```

If I remove `.config_files.xml` from the top-level ctsm directory, this works.

@jedwards4b @jgfouca @ekluzek @bandre-ucar 



Discussion:
Please try the current cime master, I believe I've already fixed this.
Also in future issues please indicate what version of cime you are using.
Sorry for that omission. I'm using 877250bd2, which is the latest version of master on esmci.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/entry_id.py b/scripts/lib/CIME/XML/entry_id.py
--- a/scripts/lib/CIME/XML/entry_id.py
+++ b/scripts/lib/CIME/XML/entry_id.py
@@ -435,7 +435,11 @@ def overwrite_existing_entries(self):
             if len(samenodes) > 1:
                 expect(len(samenodes) == 2, "Too many matchs for id {} in file {}".format(vid, self.filename))
                 logger.debug("Overwriting node {}".format(vid))
+                read_only = self.read_only
+                if read_only:
+                    self.read_only = False
                 self.remove_child(samenodes[0])
+                self.read_only = read_only
 
     def __iter__(self):
         for node in self.scan_children("entry"):


Test patch:
Repository: ESMCI/cime

Issue:
query_config --component not working for E3SM
output from scripts_regression_tests:
```
======================================================================
ERROR: test_query_components (__main__.X_TestQueryConfig)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./scripts_regression_tests.py", line 1452, in test_query_components
    run_cmd_no_fail("{}/query_config --components".format(SCRIPT_DIR))
  File "/lcrc/group/earthscience/jacob/cimeESMCI2/scripts/tests/../lib/CIME/utils.py", line 424, in run_cmd_no_fail
    expect(False, "Command: '{}' failed with error '{}' from dir '{}'".format(cmd, errput.encode('utf-8'), os.getcwd() if from_dir is None else from_dir))
  File "/lcrc/group/earthscience/jacob/cimeESMCI2/scripts/tests/../lib/CIME/utils.py", line 112, in expect
    raise exc_type(msg)
SystemExit: ERROR: Command: '/lcrc/group/earthscience/jacob/cimeESMCI2/scripts/query_config --components' failed with error 'ERROR: Expected one child' from dir '/lcrc/group/earthscience/jacob/cimeESMCI2/scripts/tests'
```




Discussion:
Just running it at the command line:
```
blogin1[99]: ./query_config --component dwav
ERROR: Expected one child
Exit 1
```

Resolving patch:
diff --git a/scripts/lib/CIME/XML/files.py b/scripts/lib/CIME/XML/files.py
--- a/scripts/lib/CIME/XML/files.py
+++ b/scripts/lib/CIME/XML/files.py
@@ -67,7 +67,7 @@ def set_value(self, vid, value,subgroup=None,ignore_type=False):
                 self.COMP_ROOT_DIR[vid+subgroup["component"]] = value
             else:
                 self.COMP_ROOT_DIR[vid] = value
-                
+
         else:
             expect(False, "Attempt to set a nonmutable variable {}".format(vid))
         return value
@@ -82,10 +82,13 @@ def get_schema(self, nodename, attributes=None):
         return None
 
     def get_components(self, nodename):
-        node = self.get_child("entry", {"id":nodename})
-        valnodes = self.get_children("value", root=self.get_child("values", root=node))
-        values = []
-        for valnode in valnodes:
-            value = self.get(valnode, "component")
-            values.append(value)
-        return values
+        node = self.get_optional_child("entry", {"id":nodename})
+        if node is not None:
+            valnodes = self.get_children("value", root=self.get_child("values", root=node))
+            values = []
+            for valnode in valnodes:
+                value = self.get(valnode, "component")
+                values.append(value)
+            return values
+
+        return None


Test patch:
Repository: ESMCI/cime

Issue:
xs:boolean types read as string using XML get
When using `self.get(node, attribute)`, an xs:boolean attribute is returned as a string. Python treats this as True when used as a boolean which is not what is meant by `attribute="false"`

An example of this error is line 69 of namelist_definition.py but there may be other cases where xs:boolean XML attributes are being treated as boolean in Python (line 79 of testlist.py was recently introduced). These codes should be found and fixed (maybe a more general fix for reading xs:boolean using get?).


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/namelist_definition.py b/scripts/lib/CIME/XML/namelist_definition.py
--- a/scripts/lib/CIME/XML/namelist_definition.py
+++ b/scripts/lib/CIME/XML/namelist_definition.py
@@ -65,8 +65,8 @@ def set_nodes(self, skip_groups=None):
         default_nodes = []
         for node in self.get_children("entry"):
             name = self.get(node, "id")
-            skip_default_entry = self.get(node, "skip_default_entry")
-            per_stream_entry = self.get(node, "per_stream_entry")
+            skip_default_entry = self.get(node, "skip_default_entry") == "true"
+            per_stream_entry = self.get(node, "per_stream_entry") == "true"
             set_node_values = False
             if skip_groups:
                 group_name = self._get_group_name(node)
@@ -131,7 +131,7 @@ def get_per_stream_entries(self):
         entries = []
         nodes = self.get_children("entry")
         for node in nodes:
-            per_stream_entry = self.get(node, "per_stream_entry")
+            per_stream_entry = self.get(node, "per_stream_entry") == "true"
             if per_stream_entry:
                 entries.append(self.get(node, "id"))
         return entries


Test patch:
Repository: ESMCI/cime

Issue:
Documentation string for PSTRID is incorrect
At least is seems so to me.
The `desc` string for `PSTRID` in src/drivers/mct/cime_config/config_component.xml is:
```
<desc>Number of instances for each component</desc>
```
whereas I think it should be something like:
```
<desc>The mpi global processors stride associated with the mpi tasks for the a component.</desc>
```



Discussion:
Agreed

Resolving patch:
diff --git a/doc/source/conf.py b/doc/source/conf.py
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -37,8 +37,10 @@
     'sphinx.ext.autodoc',
     'sphinx.ext.autosummary',
     'sphinx.ext.viewcode',
+    'sphinx.ext.todo',
     'sphinxcontrib.programoutput'
 ]
+todo_include_todos=True
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -165,9 +167,7 @@
 
 pdf_documents = [
     (master_doc,
-     u'CIME_Users_Guide', 
-     u'CIME Users Guide (PDF)', 
+     u'CIME_Users_Guide',
+     u'CIME Users Guide (PDF)',
      u'Staff of the NSF/CESM and DOE/E3SM projects'),
 ]
-
-


Test patch:
Repository: ESMCI/cime

Issue:
Allow referencing case envs inside of shell commands
e.g.

```
$SHELL{echo $NUM_NODES * 6 | bc}
```


Discussion:
@sarats 

Resolving patch:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -99,8 +99,11 @@ def __init__(self, case_root=None, read_only=True):
         self._component_classes = []
         self._component_description = {}
         self._is_env_loaded = False
+
         # these are user_mods as defined in the compset
         # Command Line user_mods are handled seperately
+
+        # Derived attributes
         self.thread_count = None
         self.total_tasks = None
         self.tasks_per_node = None
@@ -108,6 +111,7 @@ def __init__(self, case_root=None, read_only=True):
         self.spare_nodes = None
         self.tasks_per_numa = None
         self.cores_per_task = None
+
         # check if case has been configured and if so initialize derived
         if self.get_value("CASEROOT") is not None:
             self.initialize_derived_attributes()


Test patch: