internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ESMCI/cime

Issue:
Output build time per component
In CIME2, there was a timestamp output to the screen when each component build started.  Example from an F-case:
```
      Wed Dec  7 14:06:40 2016 /lcrc/project/ACME/jacob/acme_scratch/FC5ne30c2gnuthrdblues/bld/lnd.bldlog.161207-140501
      Wed Dec  7 14:06:58 2016 /lcrc/project/ACME/jacob/acme_scratch/FC5ne30c2gnuthrdblues/bld/ice.bldlog.161207-140501
      Wed Dec  7 14:07:18 2016 /lcrc/project/ACME/jacob/acme_scratch/FC5ne30c2gnuthrdblues/bld/ocn.bldlog.161207-140501
      Wed Dec  7 14:07:20 2016 /lcrc/project/ACME/jacob/acme_scratch/FC5ne30c2gnuthrdblues/bld/glc.bldlog.161207-140501
      Wed Dec  7 14:07:21 2016 /lcrc/project/ACME/jacob/acme_scratch/FC5ne30c2gnuthrdblues/bld/wav.bldlog.161207-140501
      Wed Dec  7 14:07:22 2016 /lcrc/project/ACME/jacob/acme_scratch/FC5ne30c2gnuthrdblues/bld/rof.bldlog.161207-140501
      Wed Dec  7 14:07:28 2016 /lcrc/project/ACME/jacob/acme_scratch/FC5ne30c2gnuthrdblues/bld/cesm.bldlog.161207-140501

```

Those time stamps don't appear when building with cime5.  They are useful for finding which component is responsible for an increase in total build time.

Would like to at least get the build timestamps back in output.  Ideally, the build start and end times would also be logged in CaseStatus for each component.


Discussion:
Build start and end times are written to the CaseStatus file.  Since component builds now occur in parallel it may be a little trickier to get individual timing output.  
Isn't that the full build start and end time written to CaseStatus (just one each)?
correct
I agree the parallelism means the starts and end timestamps could be out-of-order.  In that case, we would need to internally calculate the total build time and output that.

Resolving patch:
diff --git a/scripts/lib/CIME/build.py b/scripts/lib/CIME/build.py
--- a/scripts/lib/CIME/build.py
+++ b/scripts/lib/CIME/build.py
@@ -301,6 +301,7 @@ def _build_model_thread(config_dir, compclass, caseroot, libroot, bldroot, incro
                         thread_bad_results, smp, compiler):
 ###############################################################################
     logger.info("Building %s with output to %s"%(compclass, file_build))
+    t1 = time.time()
     with open(file_build, "w") as fd:
         stat = run_cmd("MODEL=%s SMP=%s %s/buildlib %s %s %s " %
                        (compclass, stringify_bool(smp), config_dir, caseroot, libroot, bldroot),
@@ -313,6 +314,8 @@ def _build_model_thread(config_dir, compclass, caseroot, libroot, bldroot, incro
     for mod_file in glob.glob(os.path.join(bldroot, "*_[Cc][Oo][Mm][Pp]_*.mod")):
         shutil.copy(mod_file, incroot)
 
+    t2 = time.time()
+    logger.info("%s built in %f seconds" % (compclass, (t2 - t1)))
 
 ###############################################################################
 def _clean_impl(case, cleanlist, clean_all):


Test patch:
Repository: ESMCI/cime

Issue:
case directory xml files do not validate
It would be nice if the case xml files were validated with xmllint using the same schema used for
config xml files.  They are close but not quite there.


Discussion:
This is partially fixed by #1184, files env_mach_specific, env_mach_pes, env_batch still need validation schemas.   

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -24,7 +24,8 @@ def __init__(self, case_root=None, infile="env_batch.xml"):
         self.batchtype = None
         # This arbitrary setting should always be overwritten
         self._default_walltime = "00:20:00"
-        EnvBase.__init__(self, case_root, infile)
+        schema = os.path.join(get_cime_root(), "config", "xml_schemas", "env_batch.xsd")
+        EnvBase.__init__(self, case_root, infile, schema=schema)
 
     def set_value(self, item, value, subgroup=None, ignore_type=False):
         """
diff --git a/scripts/lib/CIME/XML/env_mach_pes.py b/scripts/lib/CIME/XML/env_mach_pes.py
--- a/scripts/lib/CIME/XML/env_mach_pes.py
+++ b/scripts/lib/CIME/XML/env_mach_pes.py
@@ -16,7 +16,8 @@ def __init__(self, case_root=None, infile="env_mach_pes.xml", components=None):
         self._components = components
         self._component_value_list = ["NTASKS", "NTHRDS", "NINST",
                                       "ROOTPE", "PSTRID", "NINST_LAYOUT"]
-        EnvBase.__init__(self, case_root, infile)
+        schema = os.path.join(get_cime_root(), "config", "xml_schemas", "env_mach_pes.xsd")
+        EnvBase.__init__(self, case_root, infile, schema=schema)
 
     def get_value(self, vid, attribute=None, resolved=True, subgroup=None, pes_per_node=None): # pylint: disable=arguments-differ
         value = EnvBase.get_value(self, vid, attribute, resolved, subgroup)
diff --git a/scripts/lib/CIME/XML/env_mach_specific.py b/scripts/lib/CIME/XML/env_mach_specific.py
--- a/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/scripts/lib/CIME/XML/env_mach_specific.py
@@ -4,7 +4,7 @@
 from CIME.XML.standard_module_setup import *
 
 from CIME.XML.env_base import EnvBase
-from CIME.utils import transform_vars
+from CIME.utils import transform_vars, get_cime_root
 import string
 
 logger = logging.getLogger(__name__)
@@ -19,7 +19,9 @@ def __init__(self, caseroot=None, infile="env_mach_specific.xml",
         initialize an object interface to file env_mach_specific.xml in the case directory
         """
         fullpath = infile if os.path.isabs(infile) else os.path.join(caseroot, infile)
-        EnvBase.__init__(self, caseroot, fullpath)
+        schema = os.path.join(get_cime_root(), "config", "xml_schemas", "env_mach_specific.xsd")
+        EnvBase.__init__(self, caseroot, fullpath,schema=schema)
+
         self._unit_testing = unit_testing
 
     def populate(self, machobj):


Test patch:
Repository: ESMCI/cime

Issue:
test for -- options does not handle -- in quotes 
example CLM_OPTIONS="-dothis -dothat"  causes problems (replace this with a better example)


Discussion:
./xmlchange --id CLM_BLDNML_OPTS      --val "-bgc bgc -crop -irrig=.true."

Resolving patch:
diff --git a/scripts/lib/CIME/utils.py b/scripts/lib/CIME/utils.py
--- a/scripts/lib/CIME/utils.py
+++ b/scripts/lib/CIME/utils.py
@@ -1238,7 +1238,8 @@ def run_and_log_case_status(func, phase, caseroot='.'):
 
 def _check_for_invalid_args(args):
     for arg in args:
-        if arg.startswith("--"):
+        # if arg contains a space then it was originally quoted and we can ignore it here.
+        if " " in arg or arg.startswith("--"):
             continue
         if arg.startswith("-") and len(arg) > 2:
             if arg == "-value" or arg == "-noecho":


Test patch:
Repository: ESMCI/cime

Issue:
python 3 format causing pylint fails 
In PR #1471 I objected to the usage of python3 syntax format statements instead of %s as used throughout the code.   Now I've noticed that this usage was introduced to the trunk in PR #1388

Master is failing pylint tests due to #1388 - while fixing the pylint test can you please revert to %s usage throughout?


Discussion:
I can do that.
So the question is why wasn't pylint failing with the other python3 string formats that already existed in the code?
I don't know, it's also not failing on cheyenne - different pylint version maybe?

Resolving patch:
diff --git a/scripts/lib/CIME/XML/compsets.py b/scripts/lib/CIME/XML/compsets.py
--- a/scripts/lib/CIME/XML/compsets.py
+++ b/scripts/lib/CIME/XML/compsets.py
@@ -80,10 +80,10 @@ def get_value(self, name, attribute=None, resolved=False, subgroup=None):
                 compsets[alias] = lname
             return compsets
 
-    def print_values(self, help=True):
+    def print_values(self, arg_help=True):
         help_text = self.get_value(name="help")
         compsets_text = self.get_value("names")
-        if help:
+        if arg_help:
             logger.info(" {} ".format(help_text))
 
         logger.info("       --------------------------------------")
diff --git a/scripts/lib/CIME/code_checker.py b/scripts/lib/CIME/code_checker.py
--- a/scripts/lib/CIME/code_checker.py
+++ b/scripts/lib/CIME/code_checker.py
@@ -16,7 +16,7 @@ def _run_pylint(on_file, interactive):
 ###############################################################################
     pylint = find_executable("pylint")
 
-    cmd_options = " --disable=I,C,R,logging-not-lazy,wildcard-import,unused-wildcard-import,fixme,broad-except,bare-except,eval-used,exec-used,global-statement"
+    cmd_options = " --disable=I,C,R,logging-not-lazy,wildcard-import,unused-wildcard-import,fixme,broad-except,bare-except,eval-used,exec-used,global-statement,logging-format-interpolation"
     cimeroot = get_cime_root()
 
     if "scripts/Tools" in on_file:


Test patch:
Repository: ESMCI/cime

Issue:
short-term archive making empty rest sub-directories
When using short-term archiving  a subdirectory is created for each set of interim restart files in archive/$CASE/rest but they stay empty since interim restarts aren't saved by default.   Only create a sub-dir in archive/$CASE/rest if files are going to be copied there.


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case_st_archive.py b/scripts/lib/CIME/case_st_archive.py
--- a/scripts/lib/CIME/case_st_archive.py
+++ b/scripts/lib/CIME/case_st_archive.py
@@ -212,8 +212,9 @@ def _archive_restarts(case, archive, archive_entry,
     rundir = case.get_value("RUNDIR")
     casename = case.get_value("CASE")
     archive_restdir = join(dout_s_root, 'rest', datename)
-    if not os.path.exists(archive_restdir):
-        os.makedirs(archive_restdir)
+    if not datename_is_last or case.get_value('DOUT_S_SAVE_INTERIM_RESTART_FILES'):
+        if not os.path.exists(archive_restdir):
+            os.makedirs(archive_restdir)
 
     # archive the rpointer file(s) for this datename and all possible ninst_strings
     _archive_rpointer_files(case, archive, archive_entry, archive_restdir,


Test patch:
Repository: ESMCI/cime

Issue:
Need to change modify_via_xml for more meaningful error messages
If you try to modify a namelist variable that has a modify_via_xml attribute, the namelist generation prints out the value of the modify_via_xml attribute. But in many cases, this doesn't give an understandable error message. e.g., when trying to modify atm_cpl_dt, I get:

```
ERROR: Cannot change atm_cpl_dt in user_nl_xxx file, ATM_NCPL
```

(1) user_nl_xxx should be fixed or removed

(2) the modify_via_xml should be changed to the full message. e.g., rather than this (in namelist_definition_drv.xml)

```xml
  <entry id="atm_cpl_dt" modify_via_xml="ATM_NCPL" skip_default_entry="true">
```

it should say

```xml
  <entry id="atm_cpl_dt" modify_via_xml="set via ATM_NCPL" skip_default_entry="true">
```


Discussion:
It's difficult to know at the point of the error which user_nl_ file is the problem.  What is wrong with user_nl_xxx ?   It's really not that cryptic.    I can change the message to :
ERROR: Cannot change atm_cpl_dt in user_nl file use xml variable ATM_NCPL.
If it's hard to determine which user_nl file is the problem, I'm fine with (1) not being addressed - though @jedwards4b , I do like your suggestion better - `in user_nl file`.

As for (2), I looked briefly into this: What I had forgotten in the telecon discussion a few minutes ago was that @mvertens had addressed this by adding the capability for a separate attribute named `cannot_modify_by_user_nl`. So, @jedwards4b I actually think your suggestion of changing the python is right after all.

Given this, I'm happy to take this and make the changes you suggest.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/namelist_definition.py b/scripts/lib/CIME/XML/namelist_definition.py
--- a/scripts/lib/CIME/XML/namelist_definition.py
+++ b/scripts/lib/CIME/XML/namelist_definition.py
@@ -321,11 +321,11 @@ def _user_modifiable_in_variable_definition(self, name):
         user_modifiable_only_by_xml = node.get('modify_via_xml')
         if user_modifiable_only_by_xml is not None:
             expect(False,
-                   "Cannot change %s in user_nl_xxx file, %s" %(name, user_modifiable_only_by_xml))
+                   "Cannot change %s in user_nl file: set via xml variable %s" %(name, user_modifiable_only_by_xml))
         user_cannot_modify = node.get('cannot_modify_by_user_nl')
         if user_cannot_modify is not None:
             expect(False,
-                   "Cannot change %s in user_nl_xxx file, %s" %(name, user_cannot_modify))
+                   "Cannot change %s in user_nl file: %s" %(name, user_cannot_modify))
 
     def validate(self, namelist,filename=None):
         """Validate a namelist object against this definition.


Test patch:
Repository: ESMCI/cime

Issue:
File inquiry operators: csh VS tcsh
In cime_config/buildlib.mct, last file modification time is obtained with file inquiry operator M (supported by tcsh)
`if ( (-M $MCT_DIR/Makefile ) >= (-M Makefile ) ) then`

On the header of this file, it is indicated to use csh as the shell
`#! /bin/csh -fx`

On most machines, csh is actually tcsh so the command above works.
/usr/bin/csh -> /bin/tcsh
/bin/csh -> /etc/alternatives/csh, /etc/alternatives/csh -> /bin/tcsh

On some machines where csh is not tcsh (e.g. bsd-csh on Ubuntu) we will get an error:
if: Badly formed number.

If CIME wants to support both csh and tcsh, would it be better to test the csh version and suggest users to install tcsh (if csh is not tcsh)?




Discussion:
My suggestion is just to change the header to be
#! /bin/tcsh -fx 
to correctly reflect the language requirement of this script.  

Resolving patch:
diff --git a/scripts/lib/CIME/XML/grids.py b/scripts/lib/CIME/XML/grids.py
--- a/scripts/lib/CIME/XML/grids.py
+++ b/scripts/lib/CIME/XML/grids.py
@@ -254,9 +254,10 @@ def _get_domains_v2(self, component_grids, atmlevregex, lndlevregex):
                 if not comp_name == "MASK":
                     domains[comp_name + "_NX"] = int(self.get_element_text("nx", root=domain_node))
                     domains[comp_name + "_NY"] = int(self.get_element_text("ny", root=domain_node))
+
+                    file_name = comp_name + "_DOMAIN_FILE"
+                    path_name = comp_name + "_DOMAIN_PATH"
                 domains[comp_name + "_GRID"] = grid_name
-                file_name = comp_name + "_DOMAIN_FILE"
-                path_name = comp_name + "_DOMAIN_PATH"
                 file_nodes = self.get_nodes(nodename="file", root=domain_node)
                 for file_node in file_nodes:
                     grid_attrib = file_node.get("grid")
@@ -341,7 +342,7 @@ def _get_domains_v1(self, component_grids):
                 if mask_name is not None:
                     file_ = self.get_element_text("file", attributes={mask_name:mask}, root=root)
                     path  = self.get_element_text("path", attributes={mask_name:mask}, root=root)
-                    if file_ is not None:
+                    if file_ is not None and grid[0] != "MASK":
                         domains[file_name] = file_
                     if path is not None:
                         domains[path_name] = path


Test patch:
Repository: ESMCI/cime

Issue:
logic of mpirun attributes
In env_mach_specific.py there is code that requires you to specify all of the possible attributes when calling get_mpirun.   This should not be a requirement.  The code should read the list of possible attributes from the config_machines.xsd file and make sure that the attribs argument is a subset of that list.   From a discussion in PR #1343 


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_mach_specific.py b/scripts/lib/CIME/XML/env_mach_specific.py
--- a/scripts/lib/CIME/XML/env_mach_specific.py
+++ b/scripts/lib/CIME/XML/env_mach_specific.py
@@ -21,7 +21,7 @@ def __init__(self, caseroot=None, infile="env_mach_specific.xml",
         fullpath = infile if os.path.isabs(infile) else os.path.join(caseroot, infile)
         schema = os.path.join(get_cime_root(), "config", "xml_schemas", "env_mach_specific.xsd")
         EnvBase.__init__(self, caseroot, fullpath,schema=schema)
-
+        self._allowed_mpi_attributes = ("compiler", "mpilib", "threaded", "unit_testing")
         self._unit_testing = unit_testing
 
     def populate(self, machobj):
@@ -344,7 +344,9 @@ def get_mpirun(self, case, attribs, check_members=None, job="case.run", exe_only
             all_match = True
             matches = 0
             is_default = False
+
             for key, value in attribs.iteritems():
+                expect(key in self._allowed_mpi_attributes, "Unexpected key %s in mpirun attributes"%key)
                 if key in xml_attribs:
                     if xml_attribs[key].lower() == "false":
                         xml_attrib = False
@@ -361,8 +363,7 @@ def get_mpirun(self, case, attribs, check_members=None, job="case.run", exe_only
                         all_match = False
                         break
 
-            for key in xml_attribs:
-                expect(key in attribs, "Unhandled MPI property '%s'" % key)
+
 
             if all_match:
                 if is_default:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -146,7 +146,6 @@ def initialize_derived_attributes(self):
             "compiler" : self.get_value("COMPILER"),
             "mpilib"   : self.get_value("MPILIB"),
             "threaded" : self.get_build_threaded(),
-            "unit_testing" : False
             }
 
         executable = env_mach_spec.get_mpirun(self, mpi_attribs, job="case.run", exe_only=True)[0]


Test patch:
Repository: ESMCI/cime

Issue:
Do we really want this warning for --user-compset?
If you create a user-defined compset with the --user-compset option, the first thing you see is the following warning:

```
Could not find a compset match for either alias or longname in 2000_DATM%QIA_CLM50%SP_SICE_SOCN_RTM_SGLC_SWAV
```

But typically you are using --user-compset exactly because there is no alias for your compset, so this message is confusing, making it seem like there are problems when really everything is okay.

So I'd vote to remove this warning. What do others think?


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -450,11 +450,12 @@ def _set_compset_and_pesfile(self, compset_name, files, user_compset=False, pesf
                     logger.info("Compset longname is %s"%(match))
                     logger.info("Compset specification file is %s" %(compsets_filename))
                     logger.info("Pes     specification file is %s" %(self._pesfile))
+                    if user_compset is True:
+                        logger.info("Found a compset match for longname %s in alias %s" %(compset_name, compset_alias))
+
                     return compset_alias, science_support
 
         if user_compset is True:
-            #Do not error out for user_compset
-            logger.warn("Could not find a compset match for either alias or longname in %s" %(compset_name))
             self._compsetname = compset_name
         else:
             expect(False,


Test patch:
Repository: ESMCI/cime

Issue:
gx1v7 domain file
The scripts are pointing to domain.ocn.gx1v7.151020.nc and should be pointing to domain.ocn.gx1v7.151008.nc. This causes errors in cases that use the DOCN.

Dave 


Discussion:
This also affects cases that use the DICE component (C compsets fail because DICE needs the domain file; G compsets run fine because neither POP nor CICE need it).
Could you please assign this to Jim or me with a critical issue and we'll
try to get it through for the next alpha tag.

On Mon, May 15, 2017 at 2:31 PM, dabail10 <notifications@github.com> wrote:

> The scripts are pointing to domain.ocn.gx1v7.151020.nc and should be
> pointing to domain.ocn.gx1v7.151008.nc. This causes errors in cases that
> use the DOCN.
>
> Dave
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/ESMCI/cime/issues/1538>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AHlxEwekXHiy43ATERzxPQux9QVHN4h7ks5r6LYOgaJpZM4Nbp0e>
> .
>

How does one make it critical?

Resolving patch:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -910,7 +910,7 @@ def get_compset_var_settings(self):
         for name, value in matches:
             if len(value) > 0:
                 logger.debug("Compset specific settings: name is %s and value is %s"%(name,value))
-                self.set_lookup_value(name, value)
+                self.set_value(name, value)
 
 
     def set_initial_test_values(self):


Test patch:
Repository: ESMCI/cime

Issue:
Short-term archiving not working if DOUT_S_SAVE_INTERIM_RESTART_FILES is false
I just tried creating an A compset case from cime master, using create_newcase (not create_test). The short-term archiver failed with this error:

```
[Errno 2] No such file or directory: '/glade/scratch/sacks/archive/check_int_rest_0519a/rest/0001-01-06-00000/rpointer.ice'
```

I think the problem is this code in case_st_archive.py:

```python
    if not datename_is_last or case.get_value('DOUT_S_SAVE_INTERIM_RESTART_FILES') \
       or case.get_value("TEST"):
        if not os.path.exists(archive_restdir):
            os.makedirs(archive_restdir)
```

It seems to not create the necessary directory tree in a regular case with DOUT_S_SAVE_INTERIM_RESTART_FILES=FALSE.

Why does this have `or case.get_value("TEST")`? That is presumably why we haven't picked up any problems in our testing.


Discussion:
CC @mfdeakin-sandia 

Resolving patch:
diff --git a/scripts/lib/CIME/case_st_archive.py b/scripts/lib/CIME/case_st_archive.py
--- a/scripts/lib/CIME/case_st_archive.py
+++ b/scripts/lib/CIME/case_st_archive.py
@@ -212,8 +212,7 @@ def _archive_restarts(case, archive, archive_entry,
     rundir = case.get_value("RUNDIR")
     casename = case.get_value("CASE")
     archive_restdir = join(dout_s_root, 'rest', datename)
-    if not datename_is_last or case.get_value('DOUT_S_SAVE_INTERIM_RESTART_FILES') \
-       or case.get_value("TEST"):
+    if datename_is_last or case.get_value('DOUT_S_SAVE_INTERIM_RESTART_FILES'):
         if not os.path.exists(archive_restdir):
             os.makedirs(archive_restdir)
 


Test patch:
Repository: ESMCI/cime

Issue:
compilers.py get_value doesn't use attributes argument
I noticed that class Compilers in compilers.py has a get_value method that doesn't use the attributes argument. Presumably it should?


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/compilers.py b/scripts/lib/CIME/XML/compilers.py
--- a/scripts/lib/CIME/XML/compilers.py
+++ b/scripts/lib/CIME/XML/compilers.py
@@ -122,7 +122,7 @@ def get_value(self, name, attribute=None, resolved=True, subgroup=None):
         expect(subgroup is None, "This class does not support subgroups")
         value = None
 
-        node = self.get_optional_compiler_node(name)
+        node = self.get_optional_compiler_node(name, attributes=attribute)
         if node is not None:
             value = node.text
 


Test patch: