Repository: ESMCI/cime

Issue:
pio strides reset by default in CIME
On master (cime5.3.0-alpha.07-144-g8924bb3) case.setup always resets
pio strides/numtasks to defaults (PES_PER_NODE) independent of the current
values in env_run.xml, if I don't specify "--no-adjust-pio". 
The default value of PIO_STRIDE in env_run.xml (after running ./create_newcase,
and before ./case.setup is run) comes from config_pio.xml .
 There are two cases that fail due to ./case.setup adjusting pio strides/numiotasks by
default,
* There is a machine specific setting for stride in config_pio.xml 
* The user manually modifies env_run.xml, to set pio strides to something different
  from PES_PER_NODE, after creating a new case (before ./case.setup is run)


Discussion:
For example, in ACME, we have specific pio_stride for Cetus/Mira that don't work unless the user specifies "--no-adjust-pio" when running ./case.setup.
I think the option for ./case.setup should be "--adjust-pio" (By default the pio strides etc is not adjusted, it is whatever is there in env_run.xml. However user can request it to be adjusted like for some tests where the user knows that the values needs to be readjusted. The pio strides could be reset by default only if thay are invalid.)
The relevant part of the code in discussion is the following function in scripts/lib/CIME/case_setup.py that is always called by default: 
```
def adjust_pio_layout(case, new_pio_stride):

    models = case.get_values("COMP_CLASSES")
    for comp in models:
        pio_stride = case.get_value("PIO_STRIDE_%s"%comp)
        pio_numtasks = case.get_value("PIO_NUMTASKS_%s"%comp)
        ntasks = case.get_value("NTASKS_%s"%comp)
        new_stride = min(ntasks, new_pio_stride)
        new_numtasks = max(1, ntasks//new_stride)
        if pio_stride != new_stride:
            logger.info("Resetting  PIO_STRIDE_%s to %s"%(comp, new_stride))
            case.set_value("PIO_STRIDE_%s"%comp, new_stride)
        if pio_numtasks != new_numtasks:
            logger.info("Resetting  PIO_NUMTASKS_%s to %s"%(comp, new_numtasks))
            case.set_value("PIO_NUMTASKS_%s"%comp, new_numtasks)
```
Calls to this function provide "tasks_per_node" as the "new_pio_stride" arg
I agree that this is a problem - but changing to --adjust-pio  will have issues to.  I know that some of the tests will break unless we change them to add the adjust-pio flag.  But if we add the flag then presumably those tests will have problems on the machines cetus/mira.    How about if we add an optional  DEFAULT_PIO_STRIDE field to config_machines.xml  - if this field is present for a given machine it will be used and the default (if it isn't present) will be TASKS_PER_NODE.   
I will prototype something for your review...

It may be confusing to add another field to config_machines.xml (there are already similar settings in config_pio.xml and that setting could be used instead)
There is still the issue of the user modifying env_run.xml before case.setup is run.
What I think we want to find out is whether the user explicitly set the stride (config_pio.xml or modifying env_run.xml) or not.
Please let me know your thoughts
I am not sure if adding another machine-specific field is the right approach. We should be able to distinguish between cases where the user explicitly sets the stride (config_pio.xml or modifying env_run.xml) vs cases where CIME is free to choose the value.
DId you look at or try the PR?   It doesn't add a new machine specific field, only a mechanism to remember and reuse the default value for a given machine.
I haven't tried it yet, just glanced through the changes. I am looking at it now.
So what should I do to get custom pio_strides on Mira (custom pio strides are set in config_pio.xml for Mira)?
You get them when you create a case.
I tried an X case on Mira and I am still getting "Resetting PIO_STRIDE_* to 4" messages. Where did you test the code (and are there any special args that I need to pass to case_setup)?
what is happening in an X case is that the default you have in config_pio.xml  (128) is more tasks than you are using for the case.   Resetting to 4 is a last ditch effort to find something that will work at run time.   This  should be okay for an X case.   Now try a case that uses more tasks.  
IMO, CIME should not reset strides explicitly set by the user (only reset numiotasks if it is invalid).
That reset is in the fortran, not in the scripting.
It won't work at all if you are setting PIO_STRIDE to 128 and TOTALTASKS to 64.  Would you prefer it just fail outright?
Hmm, a stride of 128, totaltasks = 64 and a pio_root=0 should result in just 1 io task, right (the stride is not invalid)?
Can you confirm that the code works as expected when you have enough tasks to start with - I think that this argument about what happens if you request too few tasks is another issue.
And the reset should be happening in the scripts, I just ran create_newcase and case_setup.
Let me try a larger case.
> I tried an X case on Mira and I am still getting "Resetting PIO_STRIDE_* to 4" messages. Where did you test the code (and are there any special args that I need to pass to case_setup)?

This was a runtime error wasn't it?   I don't think that this was reset in the scripts.   Can you look in the drv_in namelist and confirm the pio_stride value.
"Resetting PIO_STRIDE_* to 4" messages (for X case on Mira) are coming from lib/CIME/case_setup.py
What is the pelayout for this job?   xmlquery NTASKS NTHRDS ROOTPE
       new_stride = min(ntasks, new_pio_stride)  so resetting to 4 should only happen if ntasks is 4 and should not be a problem.
```
[jayesh@cetuslac1 tmp_X_f19_g16 (jedwards4b-pio_readjust)]$ ./xmlquery NTASKS NTHRDS ROOTPE

Results in group mach_pes
	NTASKS: ['CPL:4', 'ATM:4', 'LND:4', 'ICE:4', 'OCN:4', 'ROF:4', 'GLC:4', 'WAV:4', 'ESP:4']
	NTHRDS: ['CPL:1', 'ATM:1', 'LND:1', 'ICE:1', 'OCN:1', 'ROF:1', 'GLC:1', 'WAV:1', 'ESP:1']
	ROOTPE: ['CPL:0', 'ATM:0', 'LND:0', 'ICE:0', 'OCN:0', 'ROF:0', 'GLC:0', 'WAV:0', 'ESP:0']

```
So there is nothing wrong with that reset.

Now in that same case try
xmlchange NTASKS=512
case.setup --reset
./xmlquery PIO_STRIDE

A case with 512/1024 tasks reflects the stride (128) from config_pio.xml .
* create_newcase
* change NTASKS=512
* case_setup (case_setup no longer resets the stride to 4, the stride in env_run.xml is 128)

How about the case where user explicitly sets the stride for each component (different stride for each component) in env_run.xml?
It looks like the code resets all strides to default (On Mira+X+512tasks+128 stride in config_pio.xml & different strides for each component in env_run.xml)
That is the case that would use the --no-adjust-pio option
I would have preferred an "--adjust-pio" option instead, where the user explicitly tells CIME to adjust the pio layouts. I think the default behaviour should not be resetting the values already in env_run.xml .
So we need to explain to 90% of our users (who don't customize pio settings) to use this additional option so that the 10% who do make pio adjustments don't need to?   
The example way at the top:  an OOB setting from config_pio.xml was overwritten by case.setup.  That should never happen.  If OOB settings for PIO are bad, why can't that be fixed in the config files?  Why does case.setup have to correct it?
@rljacob that example is now fixed
The issue is that changing the pe-layout may cause the pio settings to become bad and so they need to be readjusted

The 90% of users who don't customize pio settings and adjust the pe layout also probably don't have customized settings for pio (in config_pio.xml, manual modification of env_run.xml) etc. So the code should be modified so that "--adjust-pio" option is not needed (CIME is choosing the stride etc in this case because the user chose not to set it explicitly) if the user leaves the settings as default (no custom settings in config_pio.xml etc)
I think what we need is more communication with the user.  If case.setup is going to change the pio settings, it should prompt the user:  "Changing pio settings to match pe layout. OK?" y/n.

Then we can have both --adjust-pio (assume "y") and --no-adjust-pio (assume "n") command line options.
I sort of agree with @jayeshkrishna. With no user action, the scripts should set up a default layout that is functional (if not optimal). To this end, I think the default values for `PIO_STRIDE` and `PIO_NUMTASKS` should be `-99`, not `$PES_PER_NODE` and `1` respectively.
If this suggestion is followed, then no switch (`--[no-]adjust-pio`) is needed.
If a user decides to customize their layout, they should be able to do so by setting variables such as `PIO_STRIDE` and `PIO_NUMTASKS`.
If a user decides to revert to letting the scripts decide the layout, they can follow the existing documentation and set those values to `-99`.
@gold2718 we removed that code from the fortran quite some time ago and determined that it was better handled in the scriipts.
@jedwards4b, I didn't say anything about Fotrran, what am I missing? The XML files still say use `-99` to have the value automatically computed. Is this not correct?
Alright I'll go back to the drawing board on this one...

Resolving patch:
diff --git a/scripts/lib/CIME/SystemTests/seq.py b/scripts/lib/CIME/SystemTests/seq.py
--- a/scripts/lib/CIME/SystemTests/seq.py
+++ b/scripts/lib/CIME/SystemTests/seq.py
@@ -3,7 +3,7 @@
 """
 from CIME.XML.standard_module_setup import *
 from CIME.SystemTests.system_tests_common import SystemTestsCommon
-from CIME.case_setup import case_setup, adjust_pio_layout
+from CIME.case_setup import case_setup
 from CIME.check_lockedfiles import *
 import shutil
 
@@ -90,7 +90,6 @@ def run_phase(self):
 
         # update the pelayout settings for this run
         self._case.read_xml()
-        adjust_pio_layout(self._case, self._case.get_value("PES_PER_NODE"))
 
         self.run_indv()
 
@@ -103,7 +102,6 @@ def run_phase(self):
         logger.info("doing a second %d %s test with rootpes set to zero" % (stop_n, stop_option))
         # update the pelayout settings for this run
         self._case.read_xml()
-        adjust_pio_layout(self._case, self._case.get_value("PES_PER_NODE"))
 
         self.run_indv(suffix="seq")
         self._component_compare_test("base", "seq")
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -296,12 +296,9 @@ def submit_jobs(self, case, no_batch=False, job=None, batch_args=None, dry_run=F
                 continue
             try:
                 prereq = self.get_value('prereq', subgroup=job, resolved=False)
-                if prereq is None or job == firstjob:
+                if prereq is None or job == firstjob or (dry_run and prereq == "$BUILD_COMPLETE"):
                     prereq = True
                 else:
-                    if dry_run:
-                        # Assume build is complete
-                        prereq = prereq.replace("$BUILD_COMPLETE", "True")
                     prereq = case.get_resolved_value(prereq)
                     prereq = eval(prereq)
             except:
diff --git a/scripts/lib/CIME/case_setup.py b/scripts/lib/CIME/case_setup.py
--- a/scripts/lib/CIME/case_setup.py
+++ b/scripts/lib/CIME/case_setup.py
@@ -59,7 +59,7 @@ def _build_usernl_files(case, model, comp):
                     shutil.copy(model_nl, nlfile)
 
 ###############################################################################
-def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False, adjust_pio=True):
+def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False):
 ###############################################################################
     os.chdir(caseroot)
 
@@ -163,8 +163,6 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False,
 
             # Make sure pio settings are consistent
             tasks_per_node = env_mach_pes.get_tasks_per_node(pestot, thread_count)
-            if adjust_pio:
-                adjust_pio_layout(case, tasks_per_node)
 
             case.initialize_derived_attributes()
 
@@ -221,29 +219,12 @@ def _case_setup_impl(case, caseroot, clean=False, test_mode=False, reset=False,
         env_module.make_env_mach_specific_file(compiler, debug, mpilib, "csh")
         env_module.save_all_env_info("software_environment.txt")
 
-def adjust_pio_layout(case, new_pio_stride):
-
-    models = case.get_values("COMP_CLASSES")
-    for comp in models:
-        pio_stride = case.get_value("PIO_STRIDE_%s"%comp)
-        pio_numtasks = case.get_value("PIO_NUMTASKS_%s"%comp)
-        ntasks = case.get_value("NTASKS_%s"%comp)
-        new_stride = min(ntasks, new_pio_stride)
-        new_numtasks = max(1, ntasks//new_stride)
-        if pio_stride != new_stride:
-            logger.info("Resetting  PIO_STRIDE_%s to %s"%(comp, new_stride))
-            case.set_value("PIO_STRIDE_%s"%comp, new_stride)
-        if pio_numtasks != new_numtasks:
-            logger.info("Resetting  PIO_NUMTASKS_%s to %s"%(comp, new_numtasks))
-            case.set_value("PIO_NUMTASKS_%s"%comp, new_numtasks)
-
-
 ###############################################################################
-def case_setup(case, clean=False, test_mode=False, reset=False, adjust_pio=True):
+def case_setup(case, clean=False, test_mode=False, reset=False):
 ###############################################################################
     caseroot, casebaseid = case.get_value("CASEROOT"), case.get_value("CASEBASEID")
     phase = "setup.clean" if clean else "case.setup"
-    functor = lambda: _case_setup_impl(case, caseroot, clean, test_mode, reset, adjust_pio)
+    functor = lambda: _case_setup_impl(case, caseroot, clean, test_mode, reset)
 
     if case.get_value("TEST") and not test_mode:
         test_name = casebaseid if casebaseid is not None else case.get_value("CASE")


Test patch: