Repository: ESMCI/cime

Issue:
Comment at the top of env_batch.xml is wrong
The comment at the top of env_batch.xml (imported from config/config_headers.xml) is:

>       These variables may be changed anytime during a run, they
>       control arguments to the batch submit command.
> 
However, if I submit a job and then try:
```
./xmlchange JOB_WALLCLOCK_TIME=40:00:00
./xmlchange JOB_QUEUE=long
```
My job is killed (if it is in the queue when I make the change) or resubmit fails and I see the following message in my output file:
```
goldy@hobart^GFile /scratch/cluster/goldy/FQ3D_ne5pg3_ne5pg3_mg37/LockedFiles/env_batch.xml has been modified
  found difference in JOB_WALLCLOCK_TIME : case '40:00:00' locked '80:00:00'
  found difference in USER_REQUESTED_WALLTIME : case '40:00:00' locked ''
ERROR: Batch configuration has changed, please run case.setup --reset
```
The CIME documentation does not seem to mention this ability to change (cf http://esmci.github.io/cime/users_guide/running-a-case.html?highlight=job_wallclock_time).

Please fix the header with correct information.

Question: Is there a way to see the `<header>` information via `xmlquery`?

Reported by a CESM 2.0 user.


Discussion:
I ran a new case (in /scratch/cluster/goldy/FKESSLER_f19_f19_mg17) and while the first run was running tried:
```
./xmlchange JOB_QUEUE=medium
```
However, when the next run started up, it crashed with the following output:
> goldy@hobart File /scratch/cluster/goldy/FKESSLER_f19_f19_mg17/LockedFiles/env_batch.xml has been modified
>   found difference in JOB_QUEUE : case 'medium' locked 'long'
>   found difference in USER_REQUESTED_QUEUE : case 'medium' locked 'long'
> ERROR: Batch configuration has changed, please run case.setup --reset
> -------------------- Post Job Clean Up --------------------
> 
You must run case.setup --reset when changing the JOB_QUEUE.  The issue here is that the number of mpi tasks must be specified in the PBS (or other scheduler) code at the top of the run script and changing the queue may change the allowed number of tasks.
This does not work during a run. The resubmit notices the reset and reports:
```
goldy@hobart ERROR: BUILD_COMPLETE is not true
Please rebuild the model interactively
```
However, note that if instead of `./case.setup --reset`, I change the queue and then hit `./case.submit`, I get the following:
```
File /scratch/cluster/goldy/FKESSLER_f19_f19_mg17/LockedFiles/env_batch.xml has been modified
  found difference in JOB_QUEUE : case 'medium' locked 'long'
  found difference in USER_REQUESTED_QUEUE : case 'medium' locked 'long'

env_batch.xml appears to have changed, regenerating batch scripts
manual edits to these file will be lost!
```
Then the job is changed and resubmitted to the requested queue. Can something like that be done on resubmit?

Resolving patch:
diff --git a/scripts/lib/CIME/case/case_submit.py b/scripts/lib/CIME/case/case_submit.py
--- a/scripts/lib/CIME/case/case_submit.py
+++ b/scripts/lib/CIME/case/case_submit.py
@@ -37,10 +37,35 @@ def _submit(case, job=None, no_batch=False, prereq=None, allow_fail=False, resub
     # if case.submit is called with the no_batch flag then we assume that this
     # flag will stay in effect for the duration of the RESUBMITs
     env_batch = case.get_env("batch")
-    if resubmit:
-        if env_batch.get_batch_system_type() == "none":
-            no_batch = True
 
+    if resubmit and env_batch.get_batch_system_type() == "none":
+        no_batch = True
+    if no_batch:
+        batch_system = "none"
+    else:
+        batch_system = env_batch.get_batch_system_type()
+
+    case.set_value("BATCH_SYSTEM", batch_system)
+
+    env_batch_has_changed = False
+    try:
+        case.check_lockedfile(os.path.basename(env_batch.filename))
+    except:
+        env_batch_has_changed = True
+
+    if batch_system != "none" and env_batch_has_changed:
+        # May need to regen batch files if user made batch setting changes (e.g. walltime, queue, etc)
+        logger.warning(\
+"""
+env_batch.xml appears to have changed, regenerating batch scripts
+manual edits to these file will be lost!
+""")
+        env_batch.make_all_batch_files(case)
+
+    unlock_file(os.path.basename(env_batch.filename))
+    lock_file(os.path.basename(env_batch.filename))
+
+    if resubmit:
         # This is a resubmission, do not reinitialize test values
         if job == "case.test":
             case.set_value("IS_FIRST_RUN", False)
@@ -55,31 +80,6 @@ def _submit(case, job=None, no_batch=False, prereq=None, allow_fail=False, resub
         if job == "case.test":
             case.set_value("IS_FIRST_RUN", True)
 
-        if no_batch:
-            batch_system = "none"
-        else:
-            batch_system = env_batch.get_batch_system_type()
-
-        case.set_value("BATCH_SYSTEM", batch_system)
-
-        env_batch_has_changed = False
-        try:
-            case.check_lockedfile(os.path.basename(env_batch.filename))
-        except SystemExit:
-            env_batch_has_changed = True
-
-        if env_batch.get_batch_system_type() != "none" and env_batch_has_changed:
-            # May need to regen batch files if user made batch setting changes (e.g. walltime, queue, etc)
-            logger.warning(\
-"""
-env_batch.xml appears to have changed, regenerating batch scripts
-manual edits to these file will be lost!
-""")
-            env_batch.make_all_batch_files(case)
-
-        unlock_file(os.path.basename(env_batch.filename))
-        lock_file(os.path.basename(env_batch.filename))
-
         if job == case.get_primary_job():
             case.check_case()
             case.check_DA_settings()


Test patch: