Repository: ESMCI/cime

Issue:
Add pstrid as value to modify in config_pes.xml file
Here is an example of the file:

```
        <ntasks>
          <ntasks_atm>43200</ntasks_atm>
          <ntasks_lnd>43200</ntasks_lnd>
          <ntasks_rof>43200</ntasks_rof>
          <ntasks_ice>43200</ntasks_ice>
          <ntasks_ocn>43200</ntasks_ocn>
          <ntasks_glc>64</ntasks_glc>
          <ntasks_wav>64</ntasks_wav>
          <ntasks_cpl>43200</ntasks_cpl>
        </ntasks>
        <nthrds>
          <nthrds_atm>2</nthrds_atm>
          <nthrds_lnd>2</nthrds_lnd>
          <nthrds_rof>1</nthrds_rof>
          <nthrds_ice>1</nthrds_ice>
          <nthrds_ocn>1</nthrds_ocn>
          <nthrds_glc>1</nthrds_glc>
          <nthrds_wav>1</nthrds_wav>
          <nthrds_cpl>2</nthrds_cpl>
        </nthrds>
        <rootpe>
          <rootpe_atm>0</rootpe_atm>
          <rootpe_lnd>0</rootpe_lnd>
          <rootpe_rof>0</rootpe_rof>
          <rootpe_ice>0</rootpe_ice>
          <rootpe_ocn>0</rootpe_ocn>
          <rootpe_glc>0</rootpe_glc>
          <rootpe_wav>0</rootpe_wav>
          <rootpe_cpl>0</rootpe_cpl>
        </rootpe>
```

We've found cases that benefit from changing the PSTRID value (default is 1).  Can we add pstrid as another set of values to change?

And, can we have defaults for all 4 of these:
For any component where ntasks is not listed, use MPITASKS_PER_NODE
For any component where nthrds is not listed, use 1
For any component where rootpe is not listed, use 0
For any component where pstrid is not listed, use 1



Discussion:
Add a block like this:        
```
<pstrid>
          <pstrid_atm>0</pstrid_atm>
          <pstrid_lnd>0</pstrid_lnd>
          <pstrid_rof>0</pstrid_rof>
          <pstrid_ice>0</pstrid_ice>
          <pstrid_ocn>0</pstrid_ocn>
          <pstrid_glc>0</pstrid_glc>
          <pstrid_wav>0</pstrid_wav>
          <pstrid_cpl>0</pstrid_cpl>
        </pstrid>
```
@jedwards4b  does that look right?   Can this be added to just one entry in config_pes.xml?
Well that's the first thing I tried. I added
```
        <pstrid>
          <pstrid_atm>1</pstrid_atm>
          <pstrid_lnd>1</pstrid_lnd>
          <pstrid_rof>1</pstrid_rof>
          <pstrid_ice>1</pstrid_ice>
          <pstrid_ocn>1</pstrid_ocn>
          <pstrid_glc>1</pstrid_glc>
          <pstrid_wav>1</pstrid_wav>
          <pstrid_cpl>2</pstrid_cpl>
        </pstrid>
```

```
cori11% create_test SMS.ne30_oECv3_ICG.A_WCYCL2000S
Using project from .cesm_proj: acme
Creating test directory /global/cscratch1/sd/ndk/acme_scratch/cori-knl/mfpeat03/SMS.ne30_oECv3_ICG.A_WCYCL2000S.cori-knl_intel.20180123_172353_akk2gf
RUNNING TESTS:
  SMS.ne30_oECv3_ICG.A_WCYCL2000S.cori-knl_intel
Starting CREATE_NEWCASE for test SMS.ne30_oECv3_ICG.A_WCYCL2000S.cori-knl_intel with 1 procs
Finished CREATE_NEWCASE for test SMS.ne30_oECv3_ICG.A_WCYCL2000S.cori-knl_intel in 1.868962 seconds (FAIL). [COMPLETED 1 of 1]
    Case dir: /global/cscratch1/sd/ndk/acme_scratch/cori-knl/mfpeat03/SMS.ne30_oECv3_ICG.A_WCYCL2000S.cori-knl_intel.20180123_172353_akk2gf
    Errors were:
        ERROR: No variable pstrid found in case

```

So I assumed this was not something we could adjust in that file.  Also, that implies that we can leave sections out?  That is, if I remove the rootpe section from just one of them, is that OK?
To test pstrid, would I need to have that section in all of them?

(note: you would not want to use a pstrid of 0 as in your example)
Any thoughts/progress on this?
This is something E3SM wants so maybe @jgfouca should do it?
@jedwards4b could probably do it faster, but I could take it on.
I should be able to work on this this afternoon.

Resolving patch:
diff --git a/scripts/lib/CIME/XML/machines.py b/scripts/lib/CIME/XML/machines.py
--- a/scripts/lib/CIME/XML/machines.py
+++ b/scripts/lib/CIME/XML/machines.py
@@ -293,7 +293,6 @@ def get_suffix(self, suffix_type):
         return None
 
     def set_value(self, vid, value, subgroup=None, ignore_type=True):
-        expect(False, "This object is not mutable")
         tmproot = self.root
         self.root = self.machine_node
         result = super(Machines, self).set_value(vid, value, subgroup=subgroup,
diff --git a/scripts/lib/CIME/XML/pes.py b/scripts/lib/CIME/XML/pes.py
--- a/scripts/lib/CIME/XML/pes.py
+++ b/scripts/lib/CIME/XML/pes.py
@@ -24,6 +24,7 @@ def find_pes_layout(self, grid, compset, machine, pesize_opts='M', mpilib=None):
         opes_ntasks = {}
         opes_nthrds = {}
         opes_rootpe = {}
+        opes_pstrid = {}
         oother_settings = {}
         other_settings = {}
         o_grid_nodes = []
@@ -31,7 +32,7 @@ def find_pes_layout(self, grid, compset, machine, pesize_opts='M', mpilib=None):
         overrides = self.get_optional_child("overrides")
         if overrides is not None:
             o_grid_nodes = self.get_children("grid", root = overrides)
-            opes_ntasks, opes_nthrds, opes_rootpe, oother_settings = self._find_matches(o_grid_nodes, grid, compset, machine, pesize_opts, True)
+            opes_ntasks, opes_nthrds, opes_rootpe, opes_pstrid, oother_settings = self._find_matches(o_grid_nodes, grid, compset, machine, pesize_opts, True)
         # Get all the nodes
         grid_nodes = self.get_children("grid")
         if o_grid_nodes:
@@ -41,10 +42,11 @@ def find_pes_layout(self, grid, compset, machine, pesize_opts='M', mpilib=None):
             grid_nodes = list(gn_set)
 
 
-        pes_ntasks, pes_nthrds, pes_rootpe, other_settings = self._find_matches(grid_nodes, grid, compset, machine, pesize_opts, False)
+        pes_ntasks, pes_nthrds, pes_rootpe, pes_pstrid, other_settings = self._find_matches(grid_nodes, grid, compset, machine, pesize_opts, False)
         pes_ntasks.update(opes_ntasks)
         pes_nthrds.update(opes_nthrds)
         pes_rootpe.update(opes_rootpe)
+        pes_pstrid.update(opes_pstrid)
         other_settings.update(oother_settings)
 
         if mpilib == "mpi-serial":
@@ -52,15 +54,17 @@ def find_pes_layout(self, grid, compset, machine, pesize_opts='M', mpilib=None):
                 pes_ntasks[i] = 1
             for i in iter(pes_rootpe):
                 pes_rootpe[i] = 0
+            for i in iter(pes_pstrid):
+                pes_pstrid[i] = 0
 
         logger.info("Pes setting: grid          is {} ".format(grid))
         logger.info("Pes setting: compset       is {} ".format(compset))
         logger.info("Pes setting: tasks       is {} ".format(pes_ntasks))
         logger.info("Pes setting: threads     is {} ".format(pes_nthrds))
         logger.info("Pes setting: rootpe      is {} ".format(pes_rootpe))
-
+        logger.info("Pes setting: pstrid      is {} ".format(pes_pstrid))
         logger.info("Pes other settings: {}".format(other_settings))
-        return pes_ntasks, pes_nthrds, pes_rootpe, other_settings
+        return pes_ntasks, pes_nthrds, pes_rootpe, pes_pstrid, other_settings
 
     def _find_matches(self, grid_nodes, grid, compset, machine, pesize_opts, override=False):
         grid_choice = None
@@ -68,7 +72,7 @@ def _find_matches(self, grid_nodes, grid, compset, machine, pesize_opts, overrid
         compset_choice = None
         pesize_choice = None
         max_points = -1
-        pes_ntasks, pes_nthrds, pes_rootpe, other_settings = {}, {}, {}, {}
+        pes_ntasks, pes_nthrds, pes_rootpe, pes_pstrid, other_settings = {}, {}, {}, {}, {}
         pe_select = None
         for grid_node in grid_nodes:
             grid_match = self.get(grid_node, "name")
@@ -101,6 +105,9 @@ def _find_matches(self, grid_nodes, grid, compset, machine, pesize_opts, overrid
                                         elif "rootpe" in vid:
                                             for child in self.get_children(root=node):
                                                 pes_rootpe[self.name(child).upper()] = int(self.text(child))
+                                        elif "pstrid" in vid:
+                                            for child in self.get_children(root=node):
+                                                pes_pstrid[self.name(child).upper()] = int(self.text(child))
                                     # if the value is already upper case its something else we are trying to set
                                         elif vid == self.name(node):
                                             other_settings[vid] = self.text(node)
@@ -133,6 +140,9 @@ def _find_matches(self, grid_nodes, grid, compset, machine, pesize_opts, overrid
                 elif "rootpe" in vid:
                     for child in self.get_children(root=node):
                         pes_rootpe[self.name(child).upper()] = int(self.text(child))
+                elif "pstrid" in vid:
+                    for child in self.get_children(root=node):
+                        pes_pstrid[self.name(child).upper()] = int(self.text(child))
             # if the value is already upper case its something else we are trying to set
                 elif vid == self.name(node) and vid != "comment":
                     other_settings[vid] = self.text(node)
@@ -145,4 +155,4 @@ def _find_matches(self, grid_nodes, grid, compset, machine, pesize_opts, overrid
             if pesize_choice != 'any' or logger.isEnabledFor(logging.DEBUG):
                 logger.info("Pes setting: pesize match  is {} ".format(pesize_choice))
 
-        return pes_ntasks, pes_nthrds, pes_rootpe, other_settings
+        return pes_ntasks, pes_nthrds, pes_rootpe, pes_pstrid, other_settings
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -669,6 +669,7 @@ def _setup_mach_pes(self, pecount, multi_driver, ninst, machine_name, mpilib):
         pes_ntasks = {}
         pes_nthrds = {}
         pes_rootpe = {}
+        pes_pstrid = {}
         other      = {}
 
         force_tasks = None
@@ -686,7 +687,7 @@ def _setup_mach_pes(self, pecount, multi_driver, ninst, machine_name, mpilib):
             force_tasks = int(match2.group(1))
             pes_nthrds = pesobj.find_pes_layout(self._gridname, self._compsetname, machine_name, mpilib=mpilib)[1]
         else:
-            pes_ntasks, pes_nthrds, pes_rootpe, other = pesobj.find_pes_layout(self._gridname, self._compsetname,
+            pes_ntasks, pes_nthrds, pes_rootpe, pes_pstrid, other = pesobj.find_pes_layout(self._gridname, self._compsetname,
                                                                                machine_name, pesize_opts=pecount, mpilib=mpilib)
 
         if match1 or match2:
@@ -714,16 +715,19 @@ def _setup_mach_pes(self, pecount, multi_driver, ninst, machine_name, mpilib):
             ntasks_str = "NTASKS_{}".format(comp_class)
             nthrds_str = "NTHRDS_{}".format(comp_class)
             rootpe_str = "ROOTPE_{}".format(comp_class)
+            pstrid_str = "PSTRID_{}".format(comp_class)
 
             ntasks = pes_ntasks[ntasks_str] if ntasks_str in pes_ntasks else 1
             nthrds = pes_nthrds[nthrds_str] if nthrds_str in pes_nthrds else 1
             rootpe = pes_rootpe[rootpe_str] if rootpe_str in pes_rootpe else 0
+            pstrid = pes_pstrid[pstrid_str] if pstrid_str in pes_pstrid else 1
 
             totaltasks.append( (ntasks + rootpe) * nthrds )
 
             mach_pes_obj.set_value(ntasks_str, ntasks)
             mach_pes_obj.set_value(nthrds_str, nthrds)
             mach_pes_obj.set_value(rootpe_str, rootpe)
+            mach_pes_obj.set_value(pstrid_str, pstrid)
 
         pesize = 1
         max_mpitasks_per_node = self.get_value("MAX_MPITASKS_PER_NODE")


Test patch: