internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ESMCI/cime

Issue:
-mach no longer a required argument to create_newcase
-mach is still labeled as a required argument to create_newcase, but it no longer is. Here's a sample change that clarifies it...
# Index: create_newcase

--- create_newcase  (revision 10787)
+++ create_newcase  (working copy)
@@ -33,7 +33,7 @@
                         "To see list of current compsets, use the utility manage_case in this directory")

```
 parser.add_argument("--mach", "-mach",
```
-                        help="(required) Specify a machine. "
- ```
                     help="Specify a machine (if not set, base it off of the hostname). "
                   "To see list of current  machines, use the utility manage_case in this directory")
  ```
  
   parser.add_argument("--compiler", "-compiler",



Discussion:
Erik - can you please submit a PR with this change.  


Resolving patch:
diff --git a/utils/python/CIME/XML/machines.py b/utils/python/CIME/XML/machines.py
--- a/utils/python/CIME/XML/machines.py
+++ b/utils/python/CIME/XML/machines.py
@@ -128,7 +128,9 @@ def set_machine(self, machine):
         ...
         SystemExit: ERROR: No machine trump found
         """
-        if self.machine != machine or self.machine_node is None:
+        if machine == "Query":
+            self.machine = machine
+        elif self.machine != machine or self.machine_node is None:
             self.machine_node = self.get_optional_node("machine", {"MACH" : machine})
             expect(self.machine_node is not None, "No machine %s found" % machine)
             self.machine = machine


Test patch:
Repository: ESMCI/cime

Issue:
baseline compare error can be recorded as RUN FAIL
The expect statement at line 203 of hist_utils.py: 

```
    expect(num_compared > 0, "Did not compare any hist files for suffix1='%s' suffix2='%s', dir1='%s', dir2='%s'\nComments=%s" %
           (suffix1, suffix2, from_dir1, from_dir2, comments))
```

Is causing an abort with the TESTSTATUS file recording a FAIL in the RUN phase,
this fail should be in baseline compare and not cause a complete abort. 



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/hist_utils.py b/utils/python/CIME/hist_utils.py
--- a/utils/python/CIME/hist_utils.py
+++ b/utils/python/CIME/hist_utils.py
@@ -200,8 +200,10 @@ def _compare_hists(case, from_dir1, from_dir2, suffix1="", suffix2=""):
                 comments += cprnc_comments + "\n"
                 all_success = False
 
-    expect(num_compared > 0, "Did not compare any hist files for suffix1='%s' suffix2='%s', dir1='%s', dir2='%s'\nComments=%s" %
-           (suffix1, suffix2, from_dir1, from_dir2, comments))
+    if num_compared == 0:
+        all_success = False
+        comments += "Did not compare any hist files! Missing baselines?\n"
+
     return all_success, comments
 
 def compare_test(case, suffix1, suffix2):


Test patch:
Repository: ESMCI/cime

Issue:
Latest CIME not working on blues/Cetus
 The latest CIME master (cime5.0.10-10-gf21e864) does not work on blues.
The case.setup (-res f45_g37_rx1 -compset A) fails with the following error message,

```
ERROR: Expected full directory path, got 'UNSET'
```



Discussion:
I suspect a bash/tcsh thing.

This also happens on Cetus.
Seems to be a bug in master.
This happens when case_setup.py tries to read value of USER_MODS_FULLPATH (the value of this var in env_case.xml is "UNSET")

@jedwards4b already has a fix for this bug.


Resolving patch:
diff --git a/utils/python/CIME/user_mod_support.py b/utils/python/CIME/user_mod_support.py
--- a/utils/python/CIME/user_mod_support.py
+++ b/utils/python/CIME/user_mod_support.py
@@ -92,12 +92,14 @@ def build_include_dirs_list(user_mods_path, include_dirs=None):
     each of those directories for further directories.
     The file may also include comments deleneated with # in the first column
     '''
+    include_dirs = [] if include_dirs is None else include_dirs
+    if user_mods_path is None or user_mods_path == 'UNSET':
+        return include_dirs
     expect(os.path.isabs(user_mods_path),
            "Expected full directory path, got '%s'"%user_mods_path)
     expect(os.path.isdir(user_mods_path),
            "Directory not found %s"%user_mods_path)
     logger.info("Adding user mods directory %s"%user_mods_path)
-    include_dirs = [] if include_dirs is None else include_dirs
     include_dirs.append(os.path.normpath(user_mods_path))
     include_file = os.path.join(include_dirs[-1],"include_user_mods")
     if os.path.isfile(include_file):


Test patch:
Repository: ESMCI/cime

Issue:
.cesm_proj causes screwy things to happen when it has multiple lines
Danica noticed this. She had a list of project numbers in her ~/.cesm_proj and in cime4 -- it just took the first and runs happily. In cime5.0.9 -- however, it reads in the whole list, and then hangs when it tries to submit. The utility in having a list is to list all your project numbers so you can flip between the one you want to be active.

So "get_project" should do more checking to only read a single line and maybe make sure the input isn't garbage.



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/utils.py b/utils/python/CIME/utils.py
--- a/utils/python/CIME/utils.py
+++ b/utils/python/CIME/utils.py
@@ -530,7 +530,10 @@ def get_project(machobj=None):
     projectfile = os.path.abspath(os.path.join(os.path.expanduser("~"), ".cesm_proj"))
     if (os.path.isfile(projectfile)):
         with open(projectfile,'r') as myfile:
-            project = myfile.read().rstrip()
+            for line in myfile:
+                project = line.rstrip()
+                if not project.startswith("#"):
+                    break
             logger.info("Using project from .cesm_proj: " + project)
             cime_config.set('main','PROJECT',project)
             return project


Test patch:
diff --git a/utils/python/tests/scripts_regression_tests.py b/utils/python/tests/scripts_regression_tests.py
--- a/utils/python/tests/scripts_regression_tests.py
+++ b/utils/python/tests/scripts_regression_tests.py
@@ -1010,7 +1010,7 @@ def simple_test(self, manual_timing=False):
         if manual_timing:
             run_cmd_assert_result(self, "cd %s && %s/save_provenance postrun" % (casedir, TOOLS_DIR))
 
-        if CIME.utils.get_model() != "acme":
+        if CIME.utils.get_model() == "acme":
             provenance_dir = os.path.join(timing_dir, "performance_archive", getpass.getuser(), casename, lids[0])
             self.assertTrue(os.path.isdir(provenance_dir), msg="'%s' was missing" % provenance_dir)
Repository: ESMCI/cime

Issue:
Short term and long-term archiving issues in cim5.0.10
case.st_archive and case.lt_archive setup for the number of tasks as in case.run rather than 1 as they should. This is on yellowstone.

Also DOUT_S is default FALSE, and we probably want it default TRUE? 

For example, the testcase SMS_D.f10_f10.A.yellowstone_intel. But, you can probably see it with other cases as well.



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/case.py b/utils/python/CIME/case.py
--- a/utils/python/CIME/case.py
+++ b/utils/python/CIME/case.py
@@ -580,11 +580,12 @@ def configure(self, compset_name, grid_name, machine_name=None,
             mach_pes_obj.set_value(key,int(value), pes_per_node=pes_per_node)
 
         maxval = 1
-        for key, val in totaltasks.items():
-            if val < 0:
-                val = -1*val*pes_per_node
-            if val > maxval:
-                maxval = val
+        if mpilib != "mpi-serial":
+            for key, val in totaltasks.items():
+                if val < 0:
+                    val = -1*val*pes_per_node
+                if val > maxval:
+                    maxval = val
 
         # Make sure that every component has been accounted for
         # set, nthrds and ntasks to 1 otherwise. Also set the ninst values here.
@@ -650,6 +651,13 @@ def configure(self, compset_name, grid_name, machine_name=None,
         if self.get_value("RUN_TYPE") == 'hybrid':
             self.set_value("GET_REFCASE", True)
 
+        # Turn on short term archiving as cesm default setting
+        model = get_model()
+        if model == "cesm" and not test:
+            self.set_value("DOUT_S",True)
+
+
+
     def get_compset_var_settings(self):
         compset_obj = Compsets(infile=self.get_value("COMPSETS_SPEC_FILE"))
         matches = compset_obj.get_compset_var_settings(self._compsetname, self._gridname)
@@ -765,7 +773,7 @@ def _create_caseroot_sourcemods(self):
         if not os.path.exists(directory):
             os.makedirs(directory)
 
-        if get_model() is "cesm":
+        if get_model() == "cesm":
         # Note: this is CESM specific, given that we are referencing cism explitly
             if "cism" in components:
                 directory = os.path.join(self._caseroot, "SourceMods", "src.cism", "glimmer-cism")


Test patch:
Repository: ESMCI/cime

Issue:
PET tests do not work on skybridge
Skybridge insta-fails the single-threaded case because it tries to use 16 procs-per-node and the sbatch only requested 8 ppn.



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/SystemTests/pet.py b/utils/python/CIME/SystemTests/pet.py
--- a/utils/python/CIME/SystemTests/pet.py
+++ b/utils/python/CIME/SystemTests/pet.py
@@ -40,5 +40,14 @@ def _case_two_setup(self):
         for comp in self._COMPONENT_LIST:
             self._case.set_value("NTHRDS_%s"%comp, 1)
 
+        # The need for this is subtle. On batch systems, the entire PET test runs
+        # under a single submission and that submission is configured based on
+        # the case settings for case 1, IE 2 threads for all components. This causes
+        # the procs-per-node to be half of what it would be for single thread. On some
+        # machines, if the mpiexec tries to exceed the procs-per-node that were given
+        # to the batch submission, things break. Setting MAX_TASKS_PER_NODE to half of
+        # it original value prevents this.
+        self._case.set_value("MAX_TASKS_PER_NODE", self._case.get_value("MAX_TASKS_PER_NODE") / 2)
+
         # Need to redo case_setup because we may have changed the number of threads
         case_setup(self._case, reset=True)


Test patch:
Repository: ESMCI/cime

Issue:
Get PFS test working.
PFS does not produce history files because it does not set HIST_N or HIST_OPTION. The test fails because it tries to move history files that aren't there.



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/SystemTests/pfs.py b/utils/python/CIME/SystemTests/pfs.py
--- a/utils/python/CIME/SystemTests/pfs.py
+++ b/utils/python/CIME/SystemTests/pfs.py
@@ -18,11 +18,5 @@ def __init__(self, case):
         SystemTestsCommon.__init__(self, case)
 
     def run_phase(self):
-        self._case.set_value("STOP_OPTION", "ndays")
-        self._case.set_value("STOP_N", 20)
-        self._case.set_value("REST_OPTION","none")
-        self._case.set_value("CONTINUE_RUN", False)
-        self._case.flush()
-
         logger.info("doing an 20 day initial test, no restarts written")
-        self.run_indv()
+        self.run_indv(suffix=None)


Test patch:
diff --git a/cime_config/config_tests.xml b/cime_config/config_tests.xml
--- a/cime_config/config_tests.xml
+++ b/cime_config/config_tests.xml
@@ -388,6 +388,7 @@ LII    CLM initial condition interpolation test
     <REST_OPTION>none</REST_OPTION>
     <CCSM_TCOST>0</CCSM_TCOST>
     <DOUT_S>FALSE</DOUT_S>
+    <CONTINUE_RUN>FALSE</CONTINUE_RUN>
   </test>
 
   <test NAME="NCK">
Repository: ESMCI/cime

Issue:
xmlquery resolve option not functioning 
The xmlquery routine has an option --resolve.   But xmlquery should resolve variables by default and allow the user to override this and get the unresolved value.   As far as I can tell the --resolve option does nothing at all.   This option should be changed to --no-resolve and when set should change the resolved option to get_value to False.   



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/XML/generic_xml.py b/utils/python/CIME/XML/generic_xml.py
--- a/utils/python/CIME/XML/generic_xml.py
+++ b/utils/python/CIME/XML/generic_xml.py
@@ -191,6 +191,7 @@ def get_resolved_value(self, raw_value):
         logger.debug("raw_value %s" % raw_value)
         reference_re = re.compile(r'\${?(\w+)}?')
         env_ref_re   = re.compile(r'\$ENV\{(\w+)\}')
+        math_re = re.compile(r'[+-/*]')
         item_data = raw_value
 
         if item_data is None:
@@ -218,5 +219,13 @@ def get_resolved_value(self, raw_value):
             elif var == "SRCROOT":
                 srcroot = os.path.join(get_cime_root(),"..")
                 item_data = item_data.replace(m.group(), srcroot)
-
+            elif var in os.environ:
+                logging.debug("resolve from env: " + var)
+                item_data = item_data.replace(m.group(), os.environ[var])
+        if math_re.search(item_data):
+            try:
+                tmp = eval(item_data)
+            except:
+                tmp = item_data
+            item_data = str(tmp)
         return item_data


Test patch:
diff --git a/utils/python/update_acme_tests.py b/utils/python/update_acme_tests.py
--- a/utils/python/update_acme_tests.py
+++ b/utils/python/update_acme_tests.py
@@ -44,7 +44,7 @@
                              "ERS_Ld3.ne30_g16_rx1.A",
                              "ERS_N2_Ld3.f19_g16_rx1.A",
                              "ERR_Ld3.f45_g37_rx1.A",
-                             "SMS_D_Ln9_Mmpi-serial.f19_g16_rx1.A")
+                             "SMS_D_Ln9.f19_g16_rx1.A")
                             ),
 
     #
Repository: ESMCI/cime

Issue:
COST_PES value not always updated by case.setup
If the value of COST_PES is non-zero and changes to env_mach_pes.xml should cause it to 
be reset to 0 by case.setup that is not happening and so timing results are incorrect. 



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/case_setup.py b/utils/python/CIME/case_setup.py
--- a/utils/python/CIME/case_setup.py
+++ b/utils/python/CIME/case_setup.py
@@ -200,6 +200,10 @@ def _case_setup_impl(case, caseroot, casebaseid, clean=False, test_mode=False, r
             if mtpn > pespn:
                 pestot = pestot * (mtpn // pespn)
                 case.set_value("COST_PES", tm.num_nodes*pespn)
+            else:
+                # reset cost_pes to totalpes
+                case.set_value("COST_PES", 0)
+
             case.set_value("TOTALPES", pestot)
 
             # Compute cost based on PE count
diff --git a/utils/python/CIME/task_maker.py b/utils/python/CIME/task_maker.py
--- a/utils/python/CIME/task_maker.py
+++ b/utils/python/CIME/task_maker.py
@@ -177,7 +177,10 @@ def _compute_values(self):
         self.pbsrs = pbsrs + "%d:ncpus=%d:mpiprocs=%d:ompthreads=%d:model=$MODEL" % (max_total_node_count, self.MAX_TASKS_PER_NODE, tasks_per_node, thread_count)
 
         # calculate ptile..
-        ptile = self.MAX_TASKS_PER_NODE / 2
+        if self.MAX_TASKS_PER_NODE > self.PES_PER_NODE:
+            ptile = self.MAX_TASKS_PER_NODE / 2
+        else:
+            ptile = self.PES_PER_NODE
         if self.maxthreads > 1:
             ptile = int(math.floor(float(self.MAX_TASKS_PER_NODE) / self.maxthreads))
 


Test patch:
Repository: ESMCI/cime

Issue:
inputdata download error 
When inputdata is missing on a local machine it appears that cime will download the
data correctly but exit with an error anyway:

```
ERROR: Failed to download input data
Traceback (most recent call last):
  File "/picnic/u/home/jedwards/cesm2_0_alpha/cime/scripts/Tools/../../utils/python/CIME/SystemTests/system_tests_common.py", line 90, in build
    model_only=(phase_name==MODEL_BUILD_PHASE))
  File "/picnic/u/home/jedwards/cesm2_0_alpha/cime/scripts/Tools/../../utils/python/CIME/SystemTests/system_tests_common.py", line 114, in build_phase
    self.build_indv(sharedlib_only=sharedlib_only, model_only=model_only)
  File "/picnic/u/home/jedwards/cesm2_0_alpha/cime/scripts/Tools/../../utils/python/CIME/SystemTests/system_tests_common.py", line 121, in build_indv
    sharedlib_only=sharedlib_only, model_only=model_only)
  File "/picnic/u/home/jedwards/cesm2_0_alpha/cime/scripts/Tools/../../utils/python/CIME/build.py", line 169, in case_build
    check_all_input_data(case)
  File "/picnic/u/home/jedwards/cesm2_0_alpha/cime/scripts/Tools/../../utils/python/CIME/build.py", line 322, in check_all_input_data
    expect(success, "Failed to download input data")
  File "/picnic/u/home/jedwards/cesm2_0_alpha/cime/scripts/Tools/../../utils/python/CIME/utils.py", line 26, in expect
    raise exc_type("ERROR: %s" % error_msg)
SystemExit: ERROR: Failed to download input data
```

This is an exit in the build phase - going to the case directory and reissuing the case.build sometimes works correctly but I've also noticed that file permissions in downloaded files may look pretty screwy - for example in a recent case the directory inputdata/atm/cam/ozone did not have the executable bit on and so access below that directory was failing.   



Discussion:
When making a branch to fix this, please start from the cime5.1.1 tag so the fix can be merge to master and cime5.1.


Resolving patch:
diff --git a/utils/python/CIME/check_input_data.py b/utils/python/CIME/check_input_data.py
--- a/utils/python/CIME/check_input_data.py
+++ b/utils/python/CIME/check_input_data.py
@@ -69,7 +69,6 @@ def check_input_data(case, svn_loc=None, input_data_root=None, data_list_dir="Bu
     expect(data_list_files, "No .input_data_list files found in dir '%s'" % data_list_dir)
 
     no_files_missing = True
-
     for data_list_file in data_list_files:
         logging.info("Loading input file: '%s'" % data_list_file)
         with open(data_list_file, "r") as fd:
@@ -105,9 +104,9 @@ def check_input_data(case, svn_loc=None, input_data_root=None, data_list_dir="Bu
                                         no_files_missing = False
                                 else:
                                     no_files_missing = False
-                            else:
-                                no_files_missing = False
-
+                        # if not download
+                        else:
+                            no_files_missing = False
                     else:
                         logging.debug("Already had input file: '%s'" % full_path)
 


Test patch:
Repository: ESMCI/cime

Issue:
Bug in hist_utils.py causes tests to fail when just the baselines are missing
TestStatus is reporting that the RUN phase is failing when it's just the baselines missing.
Looks like I made a typo in hist_utils.py when I added BFAIL to the comment when baselines are missing. @jedwards4b 



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/hist_utils.py b/utils/python/CIME/hist_utils.py
--- a/utils/python/CIME/hist_utils.py
+++ b/utils/python/CIME/hist_utils.py
@@ -291,7 +291,7 @@ def compare_baseline(case, baseline_dir=None):
 
     for bdir in dirs_to_check:
         if not os.path.isdir(bdir):
-            return False, "ERROR %s baseline directory '%s' does not exist" % TEST_NO_BASELINES_COMMENT,bdir
+            return False, "ERROR %s baseline directory '%s' does not exist" % (TEST_NO_BASELINES_COMMENT,bdir)
 
     return _compare_hists(case, rundir, basecmp_dir)
 


Test patch:
Repository: ESMCI/cime

Issue:
Files copied to baseline directories should be made world readable
On hobart, the *.nc files output from CESM have permission bits `-rw-------`. When these files are copied to the baseline directory, they should be made world readable (or at the very least group readable).



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/hist_utils.py b/utils/python/CIME/hist_utils.py
--- a/utils/python/CIME/hist_utils.py
+++ b/utils/python/CIME/hist_utils.py
@@ -5,7 +5,7 @@
 from CIME.XML.standard_module_setup import *
 from CIME.test_status import TEST_NO_BASELINES_COMMENT
 
-import logging, glob, os, shutil, re
+import logging, glob, os, shutil, re, stat
 logger = logging.getLogger(__name__)
 
 def _iter_model_file_substrs(case):
@@ -382,5 +382,9 @@ def generate_baseline(case, baseline_dir=None, allow_baseline_overwrite=False):
             comments += "    generating baseline '%s' from file %s\n" % (baseline, hist)
 
     expect(num_gen > 0, "Could not generate any hist files for case '%s', something is seriously wrong" % testcase)
+    #make sure permissions are open in baseline directory
+    for root,dirs,files in os.walk(basegen_dir):
+        for name in files:
+            os.chmod(os.path.join(root,name), stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IWGRP | stat.S_IROTH)        
 
     return True, comments


Test patch:
diff --git a/scripts/create_test b/scripts/create_test
--- a/scripts/create_test
+++ b/scripts/create_test
@@ -257,7 +257,7 @@ OR
         args.no_batch = True
 
     if args.test_id is None:
-        args.test_id = "%s.%s"%(CIME.utils.get_timestamp(), CIME.utils.id_generator())
+        args.test_id = "%s_%s"%(CIME.utils.get_timestamp(), CIME.utils.id_generator())
 
     if args.testfile is not None:
         with open(args.testfile, "r") as fd:
Repository: ESMCI/cime

Issue:
code_checker should not be so verbose in debug mode



Discussion:

Resolving patch:
diff --git a/utils/python/CIME/utils.py b/utils/python/CIME/utils.py
--- a/utils/python/CIME/utils.py
+++ b/utils/python/CIME/utils.py
@@ -156,7 +156,7 @@ def run_cmd(cmd, input_str=None, from_dir=None, verbose=None,
     if (arg_stderr is _hack):
         arg_stderr = subprocess.PIPE
 
-    if (verbose or logger.isEnabledFor(logging.DEBUG)):
+    if (verbose != False and (verbose or logger.isEnabledFor(logging.DEBUG))):
         logger.info("RUN: %s" % cmd)
 
     if (input_str is not None):
@@ -177,7 +177,7 @@ def run_cmd(cmd, input_str=None, from_dir=None, verbose=None,
     errput = errput.strip() if errput is not None else errput
     stat = proc.wait()
 
-    if (verbose or logger.isEnabledFor(logging.DEBUG)):
+    if (verbose != False and (verbose or logger.isEnabledFor(logging.DEBUG))):
         if stat != 0:
             logger.info("  stat: %d\n" % stat)
         if output:


Test patch: