internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ESMCI/cime

Issue:
scripts_regression_tests needs --machine option
There are two machines possible from a cori (nersc) login node: cori-haswell and cori-knl
since the regex in config_machines.xml will match both, one must be selected as the default system.
This is fine except that scripts_regression_tests doesn't have any way to specify that the non-default system should be used for testing.   


Discussion:

Resolving patch:
diff --git a/utils/python/CIME/XML/machines.py b/utils/python/CIME/XML/machines.py
--- a/utils/python/CIME/XML/machines.py
+++ b/utils/python/CIME/XML/machines.py
@@ -41,7 +41,10 @@ def __init__(self, infile=None, files=None, machine=None):
             GenericXML.read(self, local_infile)
 
         if machine is None:
-            machine = self.probe_machine_name()
+            if "CIME_MACHINE" in os.environ:
+                machine = os.environ["CIME_MACHINE"]
+            else:
+                machine = self.probe_machine_name()
 
         expect(machine is not None, "Could not initialize machine object from %s or %s" % (infile, local_infile))
         self.set_machine(machine)


Test patch:
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -28,7 +28,7 @@
 FAST_ONLY   = False
 NO_BATCH    = False
 NO_CMAKE    = False
-TEST_ROOT = None
+TEST_ROOT   = None
 
 os.environ["CIME_GLOBAL_WALLTIME"] = "0:05:00"
 
@@ -1887,9 +1887,12 @@ def write_provenance_info():
     curr_commit = get_current_commit(repo=LIB_DIR)
     logging.info("\nTesting commit %s" % curr_commit)
     cime_model = CIME.utils.get_model()
-    logging.info("Using cime_model = %s\n" % cime_model)
+    logging.info("Using cime_model = %s" % cime_model)
+    logging.info("Testing machine = %s" % MACHINE.get_machine_name())
+    logging.info("Test root: %s\n" % TEST_ROOT)
 
 def _main_func():
+    global MACHINE
 
     if "--fast" in sys.argv:
         sys.argv.remove("--fast")
@@ -1906,6 +1909,14 @@ def _main_func():
         global NO_CMAKE
         NO_CMAKE = True
 
+    if "--machine" in sys.argv:
+        midx = sys.argv.index("--machine")
+        mach_name = sys.argv[midx + 1]
+        MACHINE = Machines(machine=mach_name)
+        os.environ["CIME_MACHINE"] = mach_name
+        del sys.argv[midx + 1]
+        del sys.argv[midx]
+
     if "--test-root" in sys.argv:
         global TEST_ROOT
         trindex = sys.argv.index("--test-root")
@@ -1916,6 +1927,7 @@ def _main_func():
         TEST_ROOT = os.path.join(MACHINE.get_value("CIME_OUTPUT_ROOT"),
                                  "scripts_regression_test.%s"% CIME.utils.get_timestamp())
 
+
     args = lambda: None # just something to set attrs on
     for log_param in ["debug", "silent", "verbose"]:
         flag = "--%s" % log_param
Repository: ESMCI/cime

Issue:
Hard to find reason for NLCOMP failure in TestStatus.log
Apologies for not giving concrete examples, I cleaned up a lot of old testing directories before realizing I should keep the results to include in this ticket. But basically, I had two different causes for `NLCOMP` failures in various tags -- one was a change in `user_nl_pop` and the other a change in `pop_in`. The `TestStatus` file just reads `FAIL ... NLCOMP`, and then the `TestStatus.log` file is several hundred lines long with no clear way to search for these namelist-generated fails. Could we include the `NLCOMP` keyword along with a message about which file(s) is/are to blame for the fail somewhere in the log?


Discussion:
@mnlevy1981 sorry for taking so long to get around to this. The modifications were simple.

Resolving patch:
diff --git a/utils/python/CIME/case_cmpgen_namelists.py b/utils/python/CIME/case_cmpgen_namelists.py
--- a/utils/python/CIME/case_cmpgen_namelists.py
+++ b/utils/python/CIME/case_cmpgen_namelists.py
@@ -32,7 +32,7 @@ def _do_full_nl_comp(case, test, compare_name, baseline_root=None):
                             and not os.path.basename(item).startswith(".")] + \
                             glob.glob("%s/*user_nl*" % test_dir)
 
-    comments = ""
+    comments = "NLCOMP\n"
     for item in all_items_to_compare:
         baseline_counterpart = os.path.join(baseline_casedocs \
                                             if os.path.dirname(item).endswith("CaseDocs") \
@@ -47,6 +47,9 @@ def _do_full_nl_comp(case, test, compare_name, baseline_root=None):
                 success, current_comments = compare_files(baseline_counterpart, item, test)
 
             all_match &= success
+            if not success:
+                comments += "Comparison failed between '%s' with '%s'\n" % (item, baseline_counterpart)
+
             comments += current_comments
 
     logging.info(comments)


Test patch:
Repository: ESMCI/cime

Issue:
scripts_regression_tests.py O_TestTestScheduler
This test fails with error SystemExit: ERROR: Leftover threads?
when run as part of the full scripts_regression_tests.py 
but passes when run using ctest or when run as an individual test.




Discussion:
Looks like my pylint changes are causing threads to hang around. Fix incoming.

Resolving patch:
diff --git a/utils/python/CIME/code_checker.py b/utils/python/CIME/code_checker.py
--- a/utils/python/CIME/code_checker.py
+++ b/utils/python/CIME/code_checker.py
@@ -106,4 +106,6 @@ def check_code(files, num_procs=10, interactive=False):
 
     pool = ThreadPool(num_procs)
     results = pool.map(lambda x : _run_pylint(x, interactive), files_to_check)
+    pool.close()
+    pool.join()
     return dict(results)


Test patch:
Repository: ESMCI/cime

Issue:
Missing f19_g16_r01 high resolution river flow grid
Between cime5.2.0-alpha.9 and cime5.2.0-alpha.20 the config_grids file format was changed, and one grid needed for CLM testing was removed. The change to add it back again is as follows...

```
[erik@yslogin4 scripts]$ svn diff ../cime_config/cesm/config_grids.xml 
Index: ../cime_config/cesm/config_grids.xml
===================================================================
--- ../cime_config/cesm/config_grids.xml	(revision 7095)
+++ ../cime_config/cesm/config_grids.xml	(working copy)
@@ -294,6 +294,15 @@
       <grid name="ocnice">gx1v6</grid>
     </model_grid>
 
+    <model_grid alias="f19_g16_r01">
+      <grid name="atm">1.9x2.5</grid>
+      <grid name="lnd">1.9x2.5</grid>
+      <grid name="ocnice">gx1v6</grid>
+      <grid name="rof">r01</grid>
+      <mask>gx1v6</mask>
+    </model_grid>
+
+
     <model_grid alias="f19_g16_gl4" compset="_CISM">
       <grid name="atm">1.9x2.5</grid>
       <grid name="lnd">1.9x2.5</grid>

```
@mvertens @jedwards4b @billsacks


Discussion:

Resolving patch:
diff --git a/utils/python/CIME/BuildTools/configure.py b/utils/python/CIME/BuildTools/configure.py
--- a/utils/python/CIME/BuildTools/configure.py
+++ b/utils/python/CIME/BuildTools/configure.py
@@ -49,16 +49,20 @@ def configure(machobj, output_dir, macros_format, compiler, mpilib, debug, sysos
 def _copy_depends_files(machine_name, machines_dir, output_dir, compiler):
     """
     Copy any system or compiler Depends files if they do not exist in the output directory
+    If there is a match for Depends.machine_name.compiler copy that and ignore the others
     """
-    for dep in (machine_name, compiler):
-        dfile = os.path.join(machines_dir, "Depends.%s"%dep)
-        outputdfile = os.path.join(output_dir, "Depends.%s"%dep)
-        if os.path.isfile(dfile) and not os.path.isfile(outputdfile):
-            shutil.copyfile(dfile, outputdfile)
     dfile = os.path.join(machines_dir, "Depends.%s.%s"%(machine_name,compiler))
     outputdfile = os.path.join(output_dir, "Depends.%s.%s"%(machine_name,compiler))
-    if os.path.isfile(dfile) and not os.path.isfile(outputdfile):
-        shutil.copyfile(dfile, outputdfile)
+    if os.path.isfile(dfile):
+        if not os.path.isfile(outputdfile):
+            shutil.copyfile(dfile, outputdfile)
+    else:
+        for dep in (machine_name, compiler):
+            dfile = os.path.join(machines_dir, "Depends.%s"%dep)
+            outputdfile = os.path.join(output_dir, "Depends.%s"%dep)
+            if os.path.isfile(dfile) and not os.path.isfile(outputdfile):
+                shutil.copyfile(dfile, outputdfile)
+
 
 def _generate_env_mach_specific(output_dir, machobj, compiler, mpilib, debug, sysos):
     """


Test patch:
Repository: ESMCI/cime

Issue:
Support some interface to the shell in our XML
Users like @amametjanov are asking for access to the shell within our XML files. Something like:

```
<env name="NETCDF_PATH">$SHELL{which nc-config | xargs dirname | xargs dirname}</env>
```


Discussion:
I would also appreciate this power
A new package manager called "Spack" uses hashes in the path names leading to things like this:  <env name="NETCDFROOT">/soft/spack/opt/spack/linux-centos6-x86_64/intel-16.0.3/netcdf-4.4.0-ds5gqquzjytf4efz3d5njsvpkfmmrwq7</env>

Spack is used by Blues and probably coming soon to a cluster near you.  Easier to just use nc-config.
https://spack.io/

Resolving patch:
diff --git a/utils/python/CIME/XML/generic_xml.py b/utils/python/CIME/XML/generic_xml.py
--- a/utils/python/CIME/XML/generic_xml.py
+++ b/utils/python/CIME/XML/generic_xml.py
@@ -7,6 +7,8 @@
 from xml.dom import minidom
 from CIME.utils import expect, get_cime_root
 
+import getpass
+
 logger = logging.getLogger(__name__)
 
 class GenericXML(object):
@@ -198,10 +200,13 @@ def get_resolved_value(self, raw_value):
         '4'
         >>> obj.get_resolved_value("0001-01-01")
         '0001-01-01'
+        >>> obj.get_resolved_value("$SHELL{echo hi}")
+        'hi'
         """
         logger.debug("raw_value %s" % raw_value)
         reference_re = re.compile(r'\${?(\w+)}?')
         env_ref_re   = re.compile(r'\$ENV\{(\w+)\}')
+        shell_ref_re = re.compile(r'\$SHELL\{([^}]+)\}')
         math_re = re.compile(r'\s[+-/*]\s')
         item_data = raw_value
 
@@ -217,6 +222,11 @@ def get_resolved_value(self, raw_value):
             expect(env_var in os.environ, "Undefined env var '%s'" % env_var)
             item_data = item_data.replace(m.group(), os.environ[env_var])
 
+        for s in shell_ref_re.finditer(item_data):
+            logger.debug("execute %s in shell" % item_data)
+            shell_cmd = s.groups()[0]
+            item_data = item_data.replace(s.group(), run_cmd_no_fail(shell_cmd))
+
         for m in reference_re.finditer(item_data):
             var = m.groups()[0]
             logger.debug("find: %s" % var)
@@ -230,11 +240,9 @@ def get_resolved_value(self, raw_value):
             elif var == "SRCROOT":
                 srcroot = os.path.join(get_cime_root(),"..")
                 item_data = item_data.replace(m.group(), srcroot)
-            elif var in os.environ:
-                # this is a list of suppressed warnings (things normally expected to be resolved in env)
-                if var not in ("USER",):
-                    logging.debug("Resolved from env: " + var)
-                item_data = item_data.replace(m.group(), os.environ[var])
+            elif var == "USER":
+                item_data = item_data.replace(m.group(), getpass.getuser())
+
         if math_re.search(item_data):
             try:
                 tmp = eval(item_data)


Test patch:
Repository: ESMCI/cime

Issue:
grid mask in grid longname
The grid mask got dropped from the grid long name - put it back.


Discussion:

Resolving patch:
diff --git a/utils/python/CIME/XML/grids.py b/utils/python/CIME/XML/grids.py
--- a/utils/python/CIME/XML/grids.py
+++ b/utils/python/CIME/XML/grids.py
@@ -35,6 +35,8 @@ def _get_grid_names(self):
                 gn = node.get("name")
                 if gn not in gridnames:
                     gridnames.append(gn)
+            if "mask" not in gridnames:
+                gridnames.append("mask")
         else:
             expect(False,"Did not recognize config_grids.xml file version")
 
@@ -215,7 +217,7 @@ def _get_domains_v2(self, component_grids):
         """ determine domains dictionary for config_grids.xml v2 schema"""
         # use component_grids to create grids dictionary
         # TODO: this should be in XML, not here
-        grids = [("atm", "a%"), ("lnd", "l%"), ("ocn", "o%"), \
+        grids = [("atm", "a%"), ("lnd", "l%"), ("ocn", "o%"), ("mask", "m%"),\
                  ("ice", "i%"), ("rof", "r%"), ("glc", "g%"), ("wav", "w%")]
         domains = {}
         mask_name = None


Test patch:
Repository: ESMCI/cime

Issue:
case.lt_archive
lt_archive script has several problems preventing functionality.   


Discussion:

Resolving patch:
diff --git a/utils/python/CIME/case_lt_archive.py b/utils/python/CIME/case_lt_archive.py
--- a/utils/python/CIME/case_lt_archive.py
+++ b/utils/python/CIME/case_lt_archive.py
@@ -12,17 +12,16 @@ def case_lt_archive(case):
     caseroot = case.get_value("CASEROOT")
 
     # max number of threads needed by scripts
-    os.environ["maxthrds"] = 1
+    os.environ["maxthrds"] = "1"
 
     # document start
     append_status("lt_archive starting",caseroot=caseroot,sfile="CaseStatus")
 
     # determine status of run and short term archiving
     runComplete = does_file_have_string(os.path.join(caseroot, "CaseStatus"),
-                                        "run SUCCESSFUL")
-    staComplete = does_file_have_string(os.path.join(caseroot, "stArchiveStatus"),
-                                        "st_archive_complete")
-
+                                        "Run SUCCESSFUL")
+    staComplete = does_file_have_string(os.path.join(caseroot, "CaseStatus"),
+                                        "st_archiving completed")
     # set up envrionment vars and call the lt_archive.sh script
     if runComplete and staComplete:
         os.environ["DOUT_S_ROOT"] = case.get_value("DOUT_S_ROOT")
@@ -32,10 +31,13 @@ def case_lt_archive(case):
         lid = time.strftime("%y%m%d-%H%M%S")
         lt_archive = LTArchive(case.get_value("MACH"))
         lt_archive_args = lt_archive.get_lt_archive_args()
-        cmd = os.path.join(caseroot, "Tools/lt_archive.sh") \
+        if lt_archive_args is None:
+            lt_archive_args = " "
+        cmd = os.path.join(caseroot, "Tools", "lt_archive.sh") \
             + lt_archive_args + "ltArchiveStatus." + lid + " 2>&1"
         run_cmd_no_fail(cmd, from_dir=caseroot)
     else:
+        logger.warn("runComplete %s staComplete %s"%(runComplete, staComplete))
         expect(False,
                "lt_archive: run or st_archive is not yet complete or was not successful."
                "Unable to perform long term archive...")


Test patch:
Repository: ESMCI/cime

Issue:
new cdash failure on master
ERRPUT: In TestStatus file for test 'DAE.f19_f19.A.yellowstone_intel', line 'ak test' not in expected format
In TestStatus file for test 'ERIO.f45_g37.X.yellowstone_intel', line 'eak test' not in expected format




Discussion:
When I create_test DAE.f19_f19.A.yellowstone_intel on yellowstone it appears that the first run of the test passes, but a subsequent run from the same directory fails with 
FAIL DAE.f19_f19.A.yellowstone_intel COMPARE_base_da
The two runs of the base case are identical, but it looks like the first da run begins at date 00010101
while the second begins at date 00010105
I assume the first run was done as part of the `./create_test` call. How did you do the second run?
After the first run completes I just go to the original case directory and issue case.submit
This will read the env_test.xml file and reset the case to those values before running.

Resolving patch:
diff --git a/utils/python/CIME/SystemTests/dae.py b/utils/python/CIME/SystemTests/dae.py
--- a/utils/python/CIME/SystemTests/dae.py
+++ b/utils/python/CIME/SystemTests/dae.py
@@ -69,6 +69,9 @@ def run_phase(self):
         for file_ in da_files:
             os.remove(file_)
         # End for
+        # CONTINUE_RUN ends up TRUE, set it back in case this is a re-run.
+        self._case.set_value("CONTINUE_RUN", False)
+        # Start normal run here
         self._activate_case1()
         SystemTestsCompareTwo.run_phase(self)
         # Do some checks on the data assimilation 'output' from case2


Test patch:
Repository: ESMCI/cime

Issue:
long compset name needs to be included in cesm_timing files
The compset longname is not currently listed in the cesm_timing output files and probably should be. The grid long name is listed. Please close this issue if this will be resolved once all the
final compsets have been defined. 

Here's an example with the missing compset longname:  

  Curr Date   : Wed Mar  1 11:04:13 2017
  grid        : a%0.9x1.25_l%0.9x1.25_oi%0.9x1.25_r%r05_g%gland4_w%null_m%gx1v6
  compset     : 
  run_type    : startup, continue_run = FALSE (inittype = TRUE)
  stop_option : ndays, stop_n = 20
  run_length  : 20 days (19.9791666667 for ocean)



Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/get_timing.py b/scripts/lib/CIME/get_timing.py
--- a/scripts/lib/CIME/get_timing.py
+++ b/scripts/lib/CIME/get_timing.py
@@ -122,9 +122,7 @@ def getTiming(self):
         ncpl_base_period = self.case.get_value("NCPL_BASE_PERIOD")
         atm_ncpl = self.case.get_value("ATM_NCPL")
         ocn_ncpl = self.case.get_value("OCN_NCPL")
-        ccsm_compset = self.case.get_value("CCSM_COMPSET")
-        if ccsm_compset is None:
-            ccsm_compset = ""
+        compset = self.case.get_value("COMPSET")
         grid = self.case.get_value("GRID")
         run_type = self.case.get_value("RUN_TYPE")
         stop_option = self.case.get_value("STOP_OPTION")
@@ -232,7 +230,7 @@ def getTiming(self):
         self.write("  Curr Date   : %s\n" % now)
 
         self.write("  grid        : %s\n" % grid)
-        self.write("  compset     : %s\n" % ccsm_compset)
+        self.write("  compset     : %s\n" % compset)
         self.write("  run_type    : %s, continue_run = %s "
                    "(inittype = %s)\n" %
                    (run_type, str(continue_run).upper(), inittype))


Test patch:
Repository: ESMCI/cime

Issue:
Add check on symbolic links in sourcemods directories when cloning
When working with SourceMods I tend to create symbolic links to various places, notably the sandbox from where the source originates. This is helpful for diffing code and for archiving purposes. When cloning a case these symbolic links are not re-created, instead, the folder to which it points is deepcopied. This could potentially lead to problems (disc space usage) when symlinking to very large directories. 

In my head, the solution would be to check for symbolic links in Sourcemods directories and threat these differently:
(1) no copy at all
(2) re-create symlink


Discussion:
The intention of the SourceMods directory is to isolate changes so that they only affect a single case.  If you are making modifications in the source sandbox then you don't need to put files into the SourceMods directory at all.   If you copy links from one SourceMods to another and then edit the link you may unintentionally edit the initial experiment an unintended and perhaps dangerous result.   
Let me try to explain @lvankampenhout 's use case. Unlike what @jedwards4b is suggesting, he doesn't link directly to the source files he's changing. Rather, his SourceMods/src.clm directory looks like this:

```
-rw-r--r-- 1 lvank ncar 31303 Feb  3 08:56 atm2lndMod.F90
-rw-r--r-- 1 lvank ncar 45311 Feb  3 08:26 CanopyHydrologyMod.F90
-rw-r--r-- 1 lvank ncar 22918 Feb  3 07:33 clm_instMod.F90
-rw-r--r-- 1 lvank ncar 18362 Feb  3 08:57 clm_varcon.F90
-rw-r--r-- 1 lvank ncar 54415 Feb  3 07:47 EnergyFluxType.F90
-rw-r--r-- 1 lvank ncar 32382 Feb  3 08:44 initVerticalMod.F90
-rw-r--r-- 1 lvank ncar 38167 Feb  3 08:33 LakeHydrologyMod.F90
lrwxrwxrwx 1 lvank ncar    69 Feb  3 07:40 src -> /glade/u/home/lvank/cesm/cesm_tags2/clm4_5_14_r222/components/clm/src
-rw-r--r-- 1 lvank ncar 54799 Feb  3 07:20 WaterStateType.F90
```

Having worked with one of his cases, I agree that it's very useful to have that src symlink in there.
Thanks @billsacks for your explainer. I took a quick look at the commit that @bertinia did 2 days ago that referenced this issue, but it appears to be unrelated? 

Resolving patch:
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -1016,8 +1016,10 @@ def create_clone(self, newcase, keepexe=False, mach_dir=None, project=None, cime
             shutil.copy(item, newcaseroot)
 
         # copy SourceMod and Buildconf files
+        # if symlinks exist, copy rather than follow links
         for casesub in ("SourceMods", "Buildconf"):
-            shutil.copytree(os.path.join(cloneroot, casesub), os.path.join(newcaseroot, casesub))
+            shutil.copytree(os.path.join(cloneroot, casesub), os.path.join(newcaseroot, casesub)
+                            , symlinks=True)
 
         # lock env_case.xml in new case
         lock_file("env_case.xml", newcaseroot)


Test patch:
Repository: ESMCI/cime

Issue:
case.submit --job case.st_archive errors if you don't have a built case
Testing the case.st_archive job submission with case.submit and working with pre-existing run directory files generated from a  DART test run returns the following error:

./case.submit --job case.st_archive
Submitting job 'case.st_archive', resubmit=False
Checking file env_case
Checking file env_mach_pes
Running ... [component buildnml] 
ERROR: Build complete is not True please rebuild the model by calling case.build



Discussion:
Case.submit should be able to ignore build status for certain cases.   


Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -55,9 +55,9 @@ def set_value(self, item, value, subgroup=None, ignore_type=False):
                 val = value
         else:
             group = self.get_optional_node("group", {"id":subgroup})
-            if group:
+            if group is not None:
                 node = self.get_optional_node("entry", {"id":item}, root=group)
-                if node:
+                if node is not None:
                     val = self._set_value(node, value, vid=item, ignore_type=ignore_type)
 
         return val
@@ -282,6 +282,7 @@ def submit_jobs(self, case, no_batch=False, job=None, batch_args=None):
         alljobs = self.get_jobs()
         startindex = 0
         jobs = []
+        firstjob = job
         if job is not None:
             expect(job in alljobs, "Do not know about batch job %s"%job)
             startindex = alljobs.index(job)
@@ -292,7 +293,7 @@ def submit_jobs(self, case, no_batch=False, job=None, batch_args=None):
                 continue
             try:
                 prereq = self.get_value('prereq', subgroup=job, resolved=False)
-                if prereq is None:
+                if prereq is None or job == firstjob:
                     prereq = True
                 else:
                     prereq = case.get_resolved_value(prereq)


Test patch: