Repository: ESMCI/cime

Issue:
cheyenne batch directives incorrect when using an unknown queue
This came up in the CTSM tutorial this week, in which participants were directed to use a special queue on cheyenne. I imagine it could impact other, future tutorials, as well as other use cases where users use some special queue for their work.

When you set a cheyenne queue to something unknown to cime, using something like:

```
./xmlchange JOB_QUEUE=R4231261 --subgroup case.run --force
```

you get batch directives that are appropriate for a single-node run rather than a multi-node run.

Here is an example: For `./create_newcase --case $scratch/test_queue_0208a --compset A --res f45_g37 --run-unsupported` followed by `./xmlchange NTASKS=144`: with the default queue, I have the following in `.case.run`:

```
 #PBS  -l select=4:ncpus=36:mpiprocs=36:ompthreads=1
```

But if I use the above `./xmlchange JOB_QUEUE` command, I instead get:

```
 #PBS  -l select=1:mpiprocs=144:ompthreads=1
```

which seems to put the whole job on a single node, leading the job to hang.

A couple of possible solutions come to mind, but I'm not sure which is the right one to do here:

(1) In CESM's config_batch.xml, for cheyenne, have a default directives block that applies to any queue not explicitly specified:

```xml
  <batch_system MACH="cheyenne" type="pbs">
    <directives>
      <directive default="/bin/bash" > -S {{ shell }}  </directive>
      <directive> -l select={{ num_nodes }}:ncpus={{ max_tasks_per_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}</directive>
    </directives>
```

(Would this let us remove the duplicate directives blocks that are currently specified for regular, premium and economy, or would those still be needed?)

(2) It seems like there is some logic that picks up the last queue specified in the directives blocks for an unknown queue; if that were either changed to pick up the first, or if we reorder those blocks for cheyenne, then maybe this would work correctly. (Not yet tested.)

(This seems somewhat related to issue #2551 , which @jedwards4b resolved last May.)


Discussion:

Resolving patch:
diff --git a/scripts/lib/CIME/XML/env_batch.py b/scripts/lib/CIME/XML/env_batch.py
--- a/scripts/lib/CIME/XML/env_batch.py
+++ b/scripts/lib/CIME/XML/env_batch.py
@@ -326,14 +326,25 @@ def get_batch_directives(self, case, job, overrides=None):
         roots = self.get_children("batch_system")
         queue = self.get_value("JOB_QUEUE", subgroup=job)
         if self._batchtype != "none" and not queue in self._get_all_queue_names():
+            unknown_queue = True
             qnode = self.get_default_queue()
-            queue = self.text(qnode)
+            default_queue = self.text(qnode)
+        else:
+            unknown_queue = False
 
         for root in roots:
             if root is not None:
                 if directive_prefix is None:
                     directive_prefix = self.get_element_text("batch_directive", root=root)
 
+                if unknown_queue:
+                    unknown_queue_directives = self.get_element_text("unknown_queue_directives",
+                                                                     root=root)
+                    if unknown_queue_directives is None:
+                        queue = default_queue
+                    else:
+                        queue = unknown_queue_directives
+
                 dnodes = self.get_children("directives", root=root)
                 for dnode in dnodes:
                     nodes = self.get_children("directive", root=dnode)


Test patch:
diff --git a/scripts/lib/CIME/test_scheduler.py b/scripts/lib/CIME/test_scheduler.py
--- a/scripts/lib/CIME/test_scheduler.py
+++ b/scripts/lib/CIME/test_scheduler.py
@@ -760,7 +760,7 @@ def _consumer(self, test, test_phase, phase_method):
 
         if not success:
             status_str += "\n    Case dir: {}\n".format(self._get_test_dir(test))
-            status_str += "    Errors were:\n        {}\n".format("\n        ".join(errors.encode('utf-8').splitlines()))
+            status_str += "    Errors were:\n        {}\n".format("\n        ".join(str(errors.encode('utf-8')).splitlines()))
 
         logger.info(status_str)