internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ESMCI/cime

Issue:
Remove space before batch directives

In config_batch.xml we have the option to control the name of the "batch directive".
If I need to mix commands in the batch file, I don't want a single string.  It would be fine if I could set this value to be empty string and then I can do what I need.

For example:

```
   <!--batch_directive>#SBATCH</batch_directive-->
    <batch_directive></batch_directive>


...

    <directives>
      <directive>#SBATCH --job-name={{ job_id }}</directive>
      <directive>#SBATCH --nodes={{ num_nodes }}</directive>
      <directive>#SBATCH --output={{ job_id }}.%j </directive>
      <directive>#SOME_OTHER_BATCH_COMMAND </directive>
   </directives>
```

However, this approach add a space BEFORE the directives which will not work:

```
cori10% head .case.run 
#!/usr/bin/env python

# Batch system directives
 #SBATCH --job-name=SMS.ne4_ne4.m71-mar12.intel.impi.n001p0048t48x1b.BB.run
 #SBATCH --nodes=1
 #SBATCH --output=SMS.ne4_ne4.m71-mar12.intel.impi.n001p0048t48x1b.BB.run.%j 
```



Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -284,7 +284,10 @@ def get_batch_directives(self, case, job, overrides=None):
                         else:
                             directive = transform_vars(directive, default=default)
 
-                        result.append("{} {}".format("" if directive_prefix is None else directive_prefix, directive))
+                        custom_prefix = self.get(node, "prefix")
+                        prefix = directive_prefix if custom_prefix is None else custom_prefix
+
+                        result.append("{}{}".format("" if not prefix else (prefix + " "), directive))
 
         return "\n".join(result)
 


Test patch:
Repository: ESMCI/cime

Issue:
Construct not working in py3
This line 
```
case.locals()[function_name](skip_pnl=skip_pnl)
``` 
https://github.com/ESMCI/cime/blob/master/scripts/lib/CIME/XML/env_batch.py#L462

Works in py2 but in py3 generates:

 AttributeError: 'Case' object has no attribute 'locals'



Discussion:
I changed from 
```
locals()[function_name](case, skip_pnl=skip_pnl)
```
but it turns out that this also doesn't work in py3.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -455,13 +455,12 @@ def _submit_single_job(self, case, job, dep_jobs=None, no_batch=False,
         batch_system = self.get_value("BATCH_SYSTEM", subgroup=None)
         if batch_system is None or batch_system == "none" or no_batch:
             logger.info("Starting job script {}".format(job))
-
             function_name = job.replace(".", "_")
             if not dry_run:
                 if "archive" not in function_name:
-                    case.locals()[function_name](skip_pnl=skip_pnl)
+                    getattr(case,function_name)(skip_pnl=skip_pnl)
                 else:
-                    case.locals()[function_name]()
+                    getattr(case,function_name)()
 
             return
 


Test patch:
Repository: ESMCI/cime

Issue:
st_archiving of DART restart history files
DART's 'inflation' files are history restart files.
DART doesn't need a .r. restart file, but
$DATA_ASSIMILATION_SCRIPT must make one at the end
of cycle number $DATA_ASSIMILATION_CYCLES
just to contain the names of the inflation files 
needed for the next cycle, so that st_archive will handle them correctly.  
This required changes to only config/cesm/config_archive.xml.

But related to that, there's a proposed change to the
Python regex that's used to find the history restart files
in case_st_archive.py:get_histfiles_for_restarts.
It's simpler to parse and understand and seems to identify
everything that the original did, and everything that
could be a history restart file name.






Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case/case_st_archive.py b/scripts/lib/CIME/case/case_st_archive.py
--- a/scripts/lib/CIME/case/case_st_archive.py
+++ b/scripts/lib/CIME/case/case_st_archive.py
@@ -280,24 +280,27 @@ def get_histfiles_for_restarts(rundir, archive, archive_entry, restfile):
         cmd = "ncdump -v {} {} ".format(rest_hist_varname, os.path.join(rundir, restfile))
         rc, out, error = run_cmd(cmd)
         if rc != 0:
-            logger.debug(" WARNING: {} failed rc={:d}\nout={}\nerr={}".format(cmd, rc, out, error))
+            logger.info(" WARNING: {} failed rc={:d}\n    out={}\n    err={}".format(cmd, rc, out, error))
+        logger.debug(" get_histfiles_for_restarts: \n    out={}".format(out))
 
         searchname = "{} =".format(rest_hist_varname)
         if searchname in out:
             offset = out.index(searchname)
             items = out[offset:].split(",")
             for item in items:
-                # the following match has an option of having a './' at the beginning of
-                # the history filename
-                matchobj = re.search(r"\"(\.*\/*\w.*)\s?\"", item)
+                # the following match has an option of having any number of '.'s and '/'s 
+                # at the beginning of the history filename
+                matchobj = re.search(r"\"\S+\s*\"", item)
                 if matchobj:
-                    histfile = matchobj.group(1).strip()
+                    histfile = matchobj.group(0).strip('" ')
                     histfile = os.path.basename(histfile)
                     # append histfile to the list ONLY if it exists in rundir before the archiving
                     if histfile in histfiles:
                         logger.warning("WARNING, tried to add a duplicate file to histfiles")
                     if os.path.isfile(os.path.join(rundir,histfile)):
                         histfiles.add(histfile)
+                    else:
+                        logger.debug(" get_histfiles_for_restarts: histfile {} does not exist ".format(histfile))
     return histfiles
 
 ###############################################################################


Test patch:
Repository: ESMCI/cime

Issue:
CESM provenance fails if you don't have write permission in SRCROOT
If you don't have write permission in SRCROOT, the CESM provenance function fails, and thus the build ends in an error. This is because manage_externals tries to write to the log file in SRCROOT and fails to do so.

I have a fix incoming.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/provenance.py b/scripts/lib/CIME/provenance.py
--- a/scripts/lib/CIME/provenance.py
+++ b/scripts/lib/CIME/provenance.py
@@ -83,7 +83,7 @@ def _save_build_provenance_cesm(case, lid): # pylint: disable=unused-argument
     manic = os.path.join(srcroot, "manage_externals","checkout_externals")
     out = None
     if os.path.exists(manic):
-        out = run_cmd_no_fail(manic + " --status --verbose", from_dir=srcroot)
+        out = run_cmd_no_fail(manic + " --status --verbose --no-logging", from_dir=srcroot)
     caseroot = case.get_value("CASEROOT")
     with open(os.path.join(caseroot, "CaseStatus"), "a") as fd:
         if version is not None and version != "unknown":


Test patch:
Repository: ESMCI/cime

Issue:
Running tools/configure on my mac gives a `source None` line
When I run `CIMEROOT=. tools/configure --macros-format=Makefile --mpilib=mpi-serial` on my mac, the resulting `.env_mach_specific.sh` has, at the top: `source None`. Then if I run `source ./.env_mach_specific.sh` I get:

```
-bash: None: No such file or directory
```

Comparing this with a version on cheyenne, it looks like this is where it would normally source something needed for modules, and of course I don't have a module system on my Mac.

This doesn't seem to cause any problems (the other commands in env_mach_specific still get executed), but it could be good to clean up at some point.


Discussion:
This is my fault, sry. Fix coming.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_mach_specific.py b/scripts/lib/CIME/XML/env_mach_specific.py
--- a/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/scripts/lib/CIME/XML/env_mach_specific.py
@@ -49,6 +49,7 @@ def populate(self, machobj):
             else:
                 for node in nodes:
                     self.add_child(node)
+
     def _get_modules_for_case(self, case, job=None):
         module_nodes = self.get_children("modules", root=self.get_child("module_system"))
         modules_to_load = None
@@ -82,7 +83,6 @@ def load_env(self, case, force_method=None, job=None, verbose=False):
 
         self._get_resources_for_case(case)
 
-
     def _get_resources_for_case(self, case):
         resource_nodes = self.get_children("resource_limits")
         if resource_nodes is not None:
@@ -145,7 +145,8 @@ def make_env_mach_specific_file(self, shell, case):
         sh_init_cmd = self.get_module_system_init_path(shell)
         sh_mod_cmd = self.get_module_system_cmd_path(shell)
 
-        lines = ["source {}".format(sh_init_cmd)]
+        if sh_init_cmd:
+            lines = ["source {}".format(sh_init_cmd)]
 
         if "SOFTENV_ALIASES" in os.environ:
             lines.append("source $SOFTENV_ALIASES")


Test patch:
Repository: ESMCI/cime

Issue:
Test NODEFAIL_P1.f09_g16 Fails on cheyenne
NODEFAIL_P1.f09_g16 is set to use 1 task but 4 nodes.   It gets submitted to the share queue on cheyenne which is correct for 1 task but wrong for 4 nodes.    


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -329,8 +329,8 @@ def get_batch_directives(self, case, job, overrides=None):
                 dnodes = self.get_children("directives", root=root)
                 for dnode in dnodes:
                     nodes = self.get_children("directive", root=dnode)
-                    for node in nodes:
-                        if self._match_attribs(self.attrib(node), case, queue):
+                    if self._match_attribs(self.attrib(dnode), case, queue):
+                        for node in nodes:
                             directive = self.get_resolved_value("" if self.text(node) is None else self.text(node))
                             default = self.get(node, "default")
                             if default is None:


Test patch:
diff --git a/scripts/lib/CIME/test_scheduler.py b/scripts/lib/CIME/test_scheduler.py
--- a/scripts/lib/CIME/test_scheduler.py
+++ b/scripts/lib/CIME/test_scheduler.py
@@ -474,6 +474,14 @@ def _create_newcase_phase(self, test):
 
         if self._queue is not None:
             create_newcase_cmd += " --queue={}".format(self._queue)
+        else:
+            # We need to hard code the queue for this test on cheyenne
+            # otherwise it runs in share and fails intermittently
+            test_case = CIME.utils.parse_test_name(test)[0]
+            if test_case == "NODEFAIL":
+                machine = machine if machine is not None else self._machobj.get_machine_name()
+                if machine == "cheyenne":
+                    create_newcase_cmd += " --queue=regular"
 
         if self._walltime is not None:
             create_newcase_cmd += " --walltime {}".format(self._walltime)
Repository: ESMCI/cime

Issue:
The string in `--compiler=` is being matched unexpectedly
We have compilers such as: "intel,intel18,gnu,gnu7".
And currently, using `--compiler=gnu` does what is expected, but `--compiler=gnu7` will match BOTH "gnu" and "gnu7", which means the module command for both of these will be executed.  Is this a bug or feature?

For example, when using `--compiler=intel8` on edison, I see the following module commands (as shown in the csh file).  Note that the intel17 module commands are executed as well as the intel18 commands.  This still works (intel18 will be used), but only because of the order in config_machines.xml.

```
source /opt/modules/default/init/csh
module rm PrgEnv-intel PrgEnv-cray PrgEnv-gnu intel cce gcc cray-parallel-netcdf cray-parallel-hdf5 pmi cray-libsci cray-mpich2 cray-mpich cray-netcdf cray-hdf5 cray-netcdf-hdf5parallel craype-san
dybridge craype-ivybridge craype papi cray-petsc esmf craype
module load craype/2.5.12.3 craype-ivybridge
module rm pmi
module load pmi/5.0.12
module rm cray-mpich
module load cray-mpich/7.6.0 PrgEnv-intel/6.0.4
module rm intel
module load intel/17.0.2.174
module rm cray-libsci
module load PrgEnv-intel/6.0.4
module rm intel
module load intel/18.0.2.199
```



Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_mach_specific.py b/scripts/lib/CIME/XML/env_mach_specific.py
--- a/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/scripts/lib/CIME/XML/env_mach_specific.py
@@ -245,12 +245,12 @@ def _match_attribs(self, attribs, case, job=None):
 
     def _match(self, my_value, xml_value):
         if xml_value.startswith("!"):
-            result = re.match(xml_value[1:],str(my_value)) is None
+            result = re.match(xml_value[1:] + "$",str(my_value)) is None
         elif isinstance(my_value, bool):
             if my_value: result = xml_value == "TRUE"
             else: result = xml_value == "FALSE"
         else:
-            result = re.match(xml_value,str(my_value)) is not None
+            result = re.match(xml_value + "$",str(my_value)) is not None
 
         logger.debug("(env_mach_specific) _match {} {} {}".format(my_value, xml_value, result))
         return result


Test patch:
Repository: ESMCI/cime

Issue:
settings from env_test are not being reapplied to tests
It seems that some recent change causes the settings from env_test.xml not to be applied when a test is resubmitted.   This is supposed to be tested in scripts_regression_tests so a second question is - why is that test not failing?


Discussion:
@jgfouca  We need to call _resetup_case (called from build and run functions in system_tests_common.py) before we check the value of continue_run in _submit


Resolving patch:
diff --git a/scripts/lib/CIME/case/case_submit.py b/scripts/lib/CIME/case/case_submit.py
--- a/scripts/lib/CIME/case/case_submit.py
+++ b/scripts/lib/CIME/case/case_submit.py
@@ -20,9 +20,10 @@ def _submit(case, job=None, no_batch=False, prereq=None, resubmit=False,
         job = case.get_primary_job()
 
     rundir = case.get_value("RUNDIR")
-    continue_run = case.get_value("CONTINUE_RUN")
-    expect(os.path.isdir(rundir) or not continue_run,
-           " CONTINUE_RUN is true but RUNDIR {} does not exist".format(rundir))
+    if job != "case.test":
+        continue_run = case.get_value("CONTINUE_RUN")
+        expect(os.path.isdir(rundir) or not continue_run,
+               " CONTINUE_RUN is true but RUNDIR {} does not exist".format(rundir))
 
     # if case.submit is called with the no_batch flag then we assume that this
     # flag will stay in effect for the duration of the RESUBMITs


Test patch:
Repository: ESMCI/cime

Issue:
notification options not preserved on resubmit
... got emails regarding a job I submitted via case.submit, had RESUBMIT=1, and I didn't get emails from the resubmission. 


Discussion:
Note that you can set these options permanently for your account in the file
$HOME/.cime/config using
```
[main]
MAIL_TYPE=
MAIL_USER=
```

Resolving patch:
diff --git a/scripts/lib/CIME/case/case_submit.py b/scripts/lib/CIME/case/case_submit.py
--- a/scripts/lib/CIME/case/case_submit.py
+++ b/scripts/lib/CIME/case/case_submit.py
@@ -7,6 +7,7 @@
 submit, check_case and check_da_settings are members of class Case in file case.py
 """
 import socket
+from six.moves                   import configparser
 from CIME.XML.standard_module_setup import *
 from CIME.utils                     import expect, run_and_log_case_status, verbatim_success_msg
 from CIME.locked_files              import unlock_file, lock_file
@@ -115,12 +116,28 @@ def submit(self, job=None, no_batch=False, prereq=None, resubmit=False,
             if phase_status != TEST_PASS_STATUS:
                 ts.set_status(SUBMIT_PHASE, TEST_PASS_STATUS)
 
+    # If this is a resubmit check the hidden file .submit_options for
+    # any submit options used on the original submit and use them again
+    caseroot = self.get_value("CASEROOT")
+    submit_options = os.path.join(caseroot, ".submit_options")
+    if resubmit and os.path.exists(submit_options):
+        config = configparser.SafeConfigParser()
+        config.read(submit_options)
+        if not skip_pnl and config.has_option('SubmitOptions','skip_pnl'):
+            skip_pnl = config.getboolean('SubmitOptions', 'skip_pnl')
+        if mail_user is None and config.has_option('SubmitOptions', 'mail_user'):
+            mail_user = config.get('SubmitOptions', 'mail_user')
+        if mail_type is None and config.has_option('SubmitOptions', 'mail_type'):
+            mail_type = config.get('SubmitOptions', 'mail_type').split(',')
+        if batch_args is None and config.has_option('SubmitOptions', 'batch_args'):
+            batch_args = config.get('SubmitOptions', 'batch_args')
+
     try:
         functor = lambda: _submit(self, job=job, no_batch=no_batch, prereq=prereq,
                                   resubmit=resubmit, skip_pnl=skip_pnl,
                                   mail_user=mail_user, mail_type=mail_type,
                                   batch_args=batch_args)
-        run_and_log_case_status(functor, "case.submit", caseroot=self.get_value("CASEROOT"),
+        run_and_log_case_status(functor, "case.submit", caseroot=caseroot,
                                 custom_success_msg_functor=verbatim_success_msg)
     except:
         # If something failed in the batch system, make sure to mark


Test patch:
Repository: ESMCI/cime

Issue:
check_input_data too enthusiastic 
It seems that check_input_data is currently loading the entire directory of ozone files for cam when just one file will do. 


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case/check_input_data.py b/scripts/lib/CIME/case/check_input_data.py
--- a/scripts/lib/CIME/case/check_input_data.py
+++ b/scripts/lib/CIME/case/check_input_data.py
@@ -167,6 +167,8 @@ def check_input_data(case, protocal="svn", address=None, input_data_root=None, d
             if (line and not line.startswith("#")):
                 tokens = line.split('=')
                 description, full_path = tokens[0].strip(), tokens[1].strip()
+                if description.endswith('datapath'):
+                    continue
                 if(full_path):
                     # expand xml variables
                     full_path = case.get_resolved_value(full_path)


Test patch:
Repository: ESMCI/cime

Issue:
With change in #2588, xmlquery sometimes doesn't give you an error when it should
This is a minor issue, so I don't feel like it needs to be fixed for the CESM2 release, but I feel like it would eventually be good to fix this to work more robustly:

With the fix that @jedwards4b put in in #2588 , a new issue appears: If you xmlquery a non-existent variable with flag `--file`, you no longer get an error: e.g.:

```
$ ./xmlquery STOP_FOO --file env_run.xml
setting case file to env_run.xml
$ echo $?
0
```

(This still works fine - giving an error as expected - if you don't include the --file option.)

Since specifying --file in this single-variable usage of xmlquery seems uncommon, this doesn't feel critical to fix immediately.

From a quick look through the logic, I'm wondering if a better solution to #2580 would be to revert the logic on these lines to the pre-PR #2588 logic:

https://github.com/ESMCI/cime/blob/6c4b1524cd0cdab8257dc98ea6c7e6024a1ac9d9/scripts/Tools/xmlquery#L255-L258

and instead adding an option to `case.get_record_fields`, as called on this line:

https://github.com/ESMCI/cime/blob/6c4b1524cd0cdab8257dc98ea6c7e6024a1ac9d9/scripts/Tools/xmlquery#L286

to pass in an optional file; if specified, it would only query that file, rather than querying all files.

My understanding of the current --listall logic when --file is given is that it gets variables from all files, then builds up a result list containing just those variables in the given file; with the above suggestion, the subsetting on file would happen up-front. I'm not sure if this would work right, though.


Discussion:
@billsacks thanks - this makes sense and I think I can still do it before the freeze tomorrow.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_base.py b/scripts/lib/CIME/XML/env_base.py
--- a/scripts/lib/CIME/XML/env_base.py
+++ b/scripts/lib/CIME/XML/env_base.py
@@ -106,7 +106,7 @@ def check_if_comp_var(self, vid, attribute=None, node=None):
                     comp = attribute["compclass"]
                 return vid, comp, True
         else:
-            if hasattr(self, "_components"):
+            if hasattr(self, "_components") and self._components:
                 new_vid = None
                 for comp in self._components:
                     if vid.endswith('_'+comp):


Test patch:
Repository: ESMCI/cime

Issue:
GET_REFCASE wont change from TRUE to FALSE
If the user has GET_REFCASE=TRUE a file Buildconf/refcase.input_data_list is created, if in a subsequent run the user sets GET_REFCASE=FALSE this file must be deleted or the model will still attempt to find a refcase.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case/check_input_data.py b/scripts/lib/CIME/case/check_input_data.py
--- a/scripts/lib/CIME/case/check_input_data.py
+++ b/scripts/lib/CIME/case/check_input_data.py
@@ -48,7 +48,7 @@ def check_all_input_data(self, protocol=None, address=None, input_data_root=None
                                    input_data_root=input_data_root, data_list_dir=data_list_dir)
         if download and not success:
             success = _downloadfromserver(self, input_data_root, data_list_dir)
-    
+
     expect(not download or (download and success), "Could not find all inputdata on any server")
     self.stage_refcase(input_data_root=input_data_root, data_list_dir=data_list_dir)
     return success
@@ -123,7 +123,10 @@ def stage_refcase(self, input_data_root=None, data_list_dir=None):
         for cam2file in  glob.iglob(os.path.join("{}","*.cam2.*").format(rundir)):
             camfile = cam2file.replace("cam2", "cam")
             os.symlink(cam2file, camfile)
-
+    elif not get_refcase and run_type != "startup":
+        logger.info("GET_REFCASE is false, the user is expected to stage the refcase to the run directory.")
+        if os.path.exists(os.path.join("Buildconf","refcase.input_data_list")):
+            os.remove(os.path.join("Buildconf","refcase.input_data_list"))
     return True
 
 def check_input_data(case, protocol="svn", address=None, input_data_root=None, data_list_dir="Buildconf", download=False):


Test patch:
Repository: ESMCI/cime

Issue:
component ww3 buildnml has inputdata dependancy
Component ww3 issues error 
           WAVEWATCH III ERROR IN W3IOGR : ERROR IN OPENING FILE
  Workaround:  The first time you run the ww3 buildnml script on a new system the data must be staged, but the staging step happens after the buildnml step.   Simply submitting the case again will
resolve the problem



Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case/case_submit.py b/scripts/lib/CIME/case/case_submit.py
--- a/scripts/lib/CIME/case/case_submit.py
+++ b/scripts/lib/CIME/case/case_submit.py
@@ -169,6 +169,11 @@ def check_case(self):
     logger.info("Checking that inputdata is available as part of case submission")
     self.check_all_input_data()
 
+    if self.get_value('COMP_WAV') == 'ww':
+        # the ww3 buildnml has dependancies on inputdata so we must run it again
+        self.create_namelists(component='WAV')
+
+
     expect(self.get_value("BUILD_COMPLETE"), "Build complete is "
            "not True please rebuild the model by calling case.build")
     logger.info("Check case OK")


Test patch: