Repository: Parsl/parsl

Issue:
PBSPro scheduler options ignored
**Describe the bug**
The PBSPro provider template adds `${scheduler_options}` to the bottom of the list of #PBS commands. However, PBSPro seems to only consider the first #PBS option and ignore any later competing ones. This means specifying a new select option with gpus is superseded by the default one and ignored. We can resolve this by moving the user defined scheduler options to the top of the #PBS list in the template.

**To Reproduce**
Use the PBSPro provider (e.g., at ALCF's edge testbed for Polaris) and try to specify a new select option:`-l select=1:ncpus=32:ngpus=1`. These options are ignored in favor of the default specified above in the template.

**Expected behavior**
I would like scheduler_options to be put first so they can be used to enter specific options.

**Environment**
 - Parsl 1.2.0

**Distributed Environment**
- Where are you running the Parsl script from ? ALCF
- Where do you need the workers to run ? Compute nodes



Discussion:

Resolving patch:
diff --git a/parsl/providers/pbspro/pbspro.py b/parsl/providers/pbspro/pbspro.py
--- a/parsl/providers/pbspro/pbspro.py
+++ b/parsl/providers/pbspro/pbspro.py
@@ -43,6 +43,9 @@ class PBSProProvider(TorqueProvider):
         Walltime requested per block in HH:MM:SS.
     scheduler_options : str
         String to prepend to the #PBS blocks in the submit script to the scheduler.
+    select_options : str
+        String to append to the #PBS -l select block in the submit script to the scheduler. This can be used to
+        specify ngpus.
     worker_init : str
         Command to be run before starting a worker, such as 'module load Anaconda; source activate env'.
     launcher : Launcher
@@ -54,6 +57,7 @@ def __init__(self,
                  account=None,
                  queue=None,
                  scheduler_options='',
+                 select_options='',
                  worker_init='',
                  nodes_per_block=1,
                  cpus_per_node=1,
@@ -81,6 +85,7 @@ def __init__(self,
         self.template_string = template_string
         self._label = 'pbspro'
         self.cpus_per_node = cpus_per_node
+        self.select_options = select_options
 
     def submit(self, command, tasks_per_node, job_name="parsl"):
         """Submits the command job.
@@ -119,6 +124,12 @@ def submit(self, command, tasks_per_node, job_name="parsl"):
         job_config["worker_init"] = self.worker_init
         job_config["user_script"] = command
 
+        # Add a colon to select_options if one isn't included
+        if self.select_options and not self.select_options.startswith(":"):
+            self.select_options = ":" + self.select_options
+
+        job_config["select_options"] = self.select_options
+
         # Wrap the command
         job_config["user_script"] = self.launcher(command,
                                                   tasks_per_node,
diff --git a/parsl/providers/pbspro/template.py b/parsl/providers/pbspro/template.py
--- a/parsl/providers/pbspro/template.py
+++ b/parsl/providers/pbspro/template.py
@@ -4,7 +4,7 @@
 #PBS -N ${jobname}
 #PBS -m n
 #PBS -l walltime=$walltime
-#PBS -l select=${nodes_per_block}:ncpus=${ncpus}
+#PBS -l select=${nodes_per_block}:ncpus=${ncpus}${select_options}
 #PBS -o ${submit_script_dir}/${jobname}.submit.stdout
 #PBS -e ${submit_script_dir}/${jobname}.submit.stderr
 ${scheduler_options}


Test patch: