Repository: ESMCI/cime

Issue:
e3sm batch commands not working after CIME update
The reason seems to be this PR:
https://github.com/ESMCI/cime/pull/4440

When I revert the above, everything works.

The error I got in all my cases was this:
```
Finished RUN for test ERS.f19_g16_rx1.A.mappy_gnu in 1.033817 seconds (FAIL). [COMPLETED 1 of 2]
    Case dir: /ascldap/users/jgfouca/acme/scratch/ERS.f19_g16_rx1.A.mappy_gnu.20230724_125551_eiymy6
    Errors were:
        env_batch.xml appears to have changed, regenerating batch scripts
        manual edits to these file will be lost!
        
        submit_jobs case.test
        Submit job case.test
        ERROR: Command: 'sbatch --time 00:31:40 -p batch --account  .case.test --skip-preview-namelist' failed with error 'sbatch: unrecognized option '--skip-preview-namelist'
        Try "sbatch --help" for more information' from dir '/home/jgfouca/acme/scratch/ERS.f19_g16_rx1.A.mappy_gnu.20230724_125551_eiymy6'
```


Discussion:
What is the unresolved value of BATCH_COMMAND_FLAGS?  
And what does the sbatch command look like when you revert to a working version?
@jedwards4b 

```
% ./xmlquery BATCH_COMMAND_FLAGS

Results in group case.post_run_io
	BATCH_COMMAND_FLAGS:   --time 00:31:40 -p batch --account 

Results in group case.st_archive
	BATCH_COMMAND_FLAGS:   --time 00:31:40 -p batch --account 

Results in group case.test
	BATCH_COMMAND_FLAGS:   --time 00:31:40 -p batch --account 

% ./preview_run 
CASE INFO:
  nodes: 1
  total tasks: 64
  tasks per node: 64
  thread count: 1
  ngpus per node: 0

BATCH INFO:
Deprecated "arg" node detected in /home/jgfouca/acme/scratch/ERS.MOS_USRDAT.RMOSNLDAS.mappy_gnu.mosart-sediment.C.20230724_124931_pk9ocx/env_batch.xml, check files /home/jgfouca/E3SM-clean/cime_config/machines/config_batch.xml
  FOR JOB: case.test
    ENV:
      Setting Environment NETCDFROOT=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/8.1.0/openmpi/4.1.4/acme
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_PLACES=threads
      Setting Environment OMP_PROC_BIND=spread
      Setting Environment OMP_STACKSIZE=64M

    SUBMIT CMD:
      sbatch --time 00:31:40 -p batch --account .case.test 

    MPIRUN (job=case.test):
      srun  --cpu_bind=threads /ascldap/users/jgfouca/acme/scratch/ERS.MOS_USRDAT.RMOSNLDAS.mappy_gnu.mosart-sediment.C.20230724_124931_pk9ocx/bld/e3sm.exe   >> e3sm.log.$LID 2>&1

% tail TestStatus.log

Submitting job script sbatch --time 00:31:40 -p batch --account  .case.test --skip-preview-namelist
env_batch.xml appears to have changed, regenerating batch scripts
manual edits to these file will be lost!
```

From a case where the PR has been reverted (as so it works):
```
./xmlquery BATCH_COMMAND_FLAGS

Results in group case.post_run_io
	BATCH_COMMAND_FLAGS: 

Results in group case.st_archive
	BATCH_COMMAND_FLAGS: 

Results in group case.test
	BATCH_COMMAND_FLAGS: 

% ./preview_run 
CASE INFO:
  nodes: 1
  total tasks: 64
  tasks per node: 64
  thread count: 1
  ngpus per node: 0

BATCH INFO:
Deprecated "arg" node detected in /home/jgfouca/acme/scratch/ERS.f19_g16_rx1.A.mappy_gnu.20230724_130613_jbwhnw/env_batch.xml, check files /home/jgfouca/E3SM-clean/cime_config/machines/config_batch.xml
  FOR JOB: case.test
    ENV:
      Setting Environment NETCDFROOT=/projects/sems/install/rhel7-x86_64/acme/tpl/netcdf/4.7.4/gcc/8.1.0/openmpi/4.1.4/acme
      Setting Environment OMP_NUM_THREADS=1
      Setting Environment OMP_PLACES=threads
      Setting Environment OMP_PROC_BIND=spread
      Setting Environment OMP_STACKSIZE=64M

    SUBMIT CMD:
      sbatch --time 00:31:40 -p batch --account PROJECT .case.test 

    MPIRUN (job=case.test):
      srun  --cpu_bind=threads /ascldap/users/jgfouca/acme/scratch/ERS.f19_g16_rx1.A.mappy_gnu.20230724_130613_jbwhnw/bld/e3sm.exe   >> e3sm.log.$LID 2>&1 

% tail TestStatus.log

Submitting job script sbatch --time 00:31:40 -p batch --account PROJECT .case.test --skip-preview-namelist
Submitted job id is 659406
Submitted job case.test with id 659406
env_batch.xml appears to have changed, regenerating batch scripts
manual edits to these file will be lost!
```

Weird, these both appear to be doing the same batch command, but it in first case, things crash.
Oh, it's the project causing the issue!
I think that I fixed this issue with https://github.com/ESMCI/cime/pull/4461. 
Are you using the latest cime?
@jedwards4b , this particular machine (mappy) did not fit the assumption being described in the comment here:

```
        if not project:
            # If there is no project then we need to remove the project flag                                                                                                                                                  
            # slurm defines --account only on machines that require it, so this strip isn't required                                                                                                                          
            if batch_system == "pbs" or batch_system == "cobalt":
                submitargs = submitargs.replace("-A", "")
            elif batch_system == "lsf":
                submitargs = submitargs.replace("-P", "")
```

I (mistakenly) had `--account` in the batch args, even though there's no PROJECT on this machine. It may be nice to add an `expect(--account not in submitargs)` here to validate this assumption. In the meantime, I can easily fix the batch configuration for this machine.

Resolving patch:
diff --git a/CIME/XML/env_batch.py b/CIME/XML/env_batch.py
--- a/CIME/XML/env_batch.py
+++ b/CIME/XML/env_batch.py
@@ -969,11 +969,14 @@ def _submit_single_job(
 
         if not project:
             # If there is no project then we need to remove the project flag
-            # slurm defines --account only on machines that require it, so this strip isn't required
-            if batch_system == "pbs" or batch_system == "cobalt":
+            if (
+                batch_system == "pbs" or batch_system == "cobalt"
+            ) and " -A " in submitargs:
                 submitargs = submitargs.replace("-A", "")
-            elif batch_system == "lsf":
+            elif batch_system == "lsf" and " -P " in submitargs:
                 submitargs = submitargs.replace("-P", "")
+            elif batch_system == "slurm" and " --account " in submitargs:
+                submitargs = submitargs.replace("--account", "")
 
         if dep_jobs is not None and len(dep_jobs) > 0:
             logger.debug("dependencies: {}".format(dep_jobs))


Test patch: