Repository: ESMCI/cime

Issue:
Unable to use DDT with aprun on Titan
We determined the cause of the problem of trying to use DDT on Titan. For DDT to work, we need to call "ddt --connect aprun" instead of "aprun". I tried to do this through env_mach_specific.xml, changing the following:

122     <executable args="default">aprun</executable>
122     <executable args="default">ddt --connect aprun</executable>

But when I do this, the flags (e.g., -S8 -N16 -n16 -d1) all get dropped. The output from the standard "aprun" is:

run command is aprun -S 8 -n 16 -N 16 -d 1 /lustre/atlas1/stf006/scratch/imn/enable_debugger17.3/bld/acme.exe    >> acme.log.$LID 2>&1  

But the output from "ddt --connect aprun" is:

run command is ddt --connect aprun   /lustre/atlas1/stf006/scratch/imn/enable_debugger17.3/bld/acme.exe  >> acme.log.$LID 2>&1  

The DDT job successfully connects to my debugger but, of course, fails because it's only running one MPI task. I need some way of specifying "ddt --connect aprun" instead of "aprun" that keeps those flags in play.

I should also mention that this is pretty critical because PGI said they'd make much better progress if they could use a debugger on Titan to diagnose what's going on with their compiler bugs.


Discussion:
@rljacob @jgfouca
This is related to issue #643 and we would prefer to have a generic debug option to one specific to a single system.  Currently the best way to launch a debugger is outside of the cime scripts.
How does one launch a debugger outside the cime scripts but with a model built by create_newcase?
1. build the model executable.
2. set the environment by sourcing the .env_mach_specific.[sh,csh]
3. launch ddt from the command line, the rest is done in the ddt gui
4. set the run directory to the model run dir
5. set the executable 
6. set the proper number of tasks and threads
7. start the job.
It would be significantly easier for everyone if we could just alter the aprun command without it breaking CIME. We need some way of supporting the reverse connect feature of DDT, especially for large debug jobs.
@jedwards4b the reverse connect method is what the Allinea people recommend. They may start deprecating the above method that you give. In practice I've found doing the above for CESM to be problematic. It makes the debugging very tied to the specific case and getting all the options set exactly right. And it seems to be especially so, if you aren't just using the intel compiler.

Reverse connect can set the right number of tasks and threads, as shown in #643 and can make it much easier to setup DDT.
I'd like to try the method @ekluzek recommended in the other Issue, but my parameters for aprun on Titan will be different than that example. The aprun parameters for Titan are commented out in cime_config/acme/machines/config_machines.xml. Therefore, it's not clear to me where these flags are actually coming from, and I don't know where to copy and paste from.
Okay, I'm up to speed now.   @mrnorman the problem is on line 152 and 1122 of case.py 
```
       if executable == "aprun":
```
should be
```
       if "aprun" in executable:
```
Thanks Jim! I have an older version of CIME, but I found the correct line and changed it. This does restore the aprun flags. However, I still have the problem that it starts the aprun command with "aprun" rather than:

\<executable args="default"\>ddt --connect aprun\</executable\>

Is there an environment variable in the python scripts that can reference the above XML executable name instead of just "aprun"?
It looks like aprun.py is hardcoding aprun in its argument string.   Instead of returning aprun plus the arguments it should just return aprun_arguments and use executable for the command.   

Resolving patch:
diff --git a/scripts/lib/CIME/aprun.py b/scripts/lib/CIME/aprun.py
--- a/scripts/lib/CIME/aprun.py
+++ b/scripts/lib/CIME/aprun.py
@@ -30,10 +30,10 @@ def _get_aprun_cmd_for_case_impl(ntasks, nthreads, rootpes, pstrids,
     >>> machine = "titan"
     >>> run_exe = "acme.exe"
     >>> _get_aprun_cmd_for_case_impl(ntasks, nthreads, rootpes, pstrids, max_tasks_per_node, pes_per_node, pio_numtasks, pio_async_interface, compiler, machine, run_exe)
-    ('aprun -S 4 -n 680 -N 8 -d 2 acme.exe : -S 2 -n 128 -N 4 -d 4 acme.exe ', 117)
+    (' -S 4 -n 680 -N 8 -d 2 acme.exe : -S 2 -n 128 -N 4 -d 4 acme.exe ', 117)
     >>> compiler = "intel"
     >>> _get_aprun_cmd_for_case_impl(ntasks, nthreads, rootpes, pstrids, max_tasks_per_node, pes_per_node, pio_numtasks, pio_async_interface, compiler, machine, run_exe)
-    ('aprun -S 4 -cc numa_node -n 680 -N 8 -d 2 acme.exe : -S 2 -cc numa_node -n 128 -N 4 -d 4 acme.exe ', 117)
+    (' -S 4 -cc numa_node -n 680 -N 8 -d 2 acme.exe : -S 2 -cc numa_node -n 128 -N 4 -d 4 acme.exe ', 117)
 
     """
     max_tasks_per_node = 1 if max_tasks_per_node < 1 else max_tasks_per_node
@@ -64,8 +64,8 @@ def _get_aprun_cmd_for_case_impl(ntasks, nthreads, rootpes, pstrids,
             maxt[c1] = 1
 
     # Compute task and thread settings for batch commands
-    tasks_per_node, task_count, thread_count, max_thread_count, total_node_count, aprun = \
-        0, 1, maxt[0], maxt[0], 0, "aprun"
+    tasks_per_node, task_count, thread_count, max_thread_count, total_node_count, aprun_args = \
+        0, 1, maxt[0], maxt[0], 0, ""
     for c1 in xrange(1, total_tasks):
         if maxt[c1] != thread_count:
             tasks_per_node = min(pes_per_node, max_tasks_per_node / thread_count)
@@ -76,11 +76,11 @@ def _get_aprun_cmd_for_case_impl(ntasks, nthreads, rootpes, pstrids,
             task_per_numa = int(math.ceil(tasks_per_node / 2.0))
             # Option for Titan
             if machine == "titan" and tasks_per_node > 1:
-                aprun += " -S %d" % task_per_numa
+                aprun_args += " -S %d" % task_per_numa
                 if compiler == "intel":
-                    aprun += " -cc numa_node"
+                    aprun_args += " -cc numa_node"
 
-            aprun += " -n %d -N %d -d %d %s :" % (task_count, tasks_per_node, thread_count, run_exe)
+            aprun_args += " -n %d -N %d -d %d %s :" % (task_count, tasks_per_node, thread_count, run_exe)
 
             node_count = int(math.ceil(float(task_count) / tasks_per_node))
             total_node_count += node_count
@@ -105,13 +105,13 @@ def _get_aprun_cmd_for_case_impl(ntasks, nthreads, rootpes, pstrids,
 
     # Special option for Titan with intel compiler
     if machine == "titan" and tasks_per_node > 1:
-        aprun += " -S %d" % task_per_numa
+        aprun_args += " -S %d" % task_per_numa
         if compiler == "intel":
-            aprun += " -cc numa_node"
+            aprun_args += " -cc numa_node"
 
-    aprun += " -n %d -N %d -d %d %s " % (task_count, tasks_per_node, thread_count, run_exe)
+    aprun_args += " -n %d -N %d -d %d %s " % (task_count, tasks_per_node, thread_count, run_exe)
 
-    return aprun, total_node_count
+    return aprun_args, total_node_count
 
 ###############################################################################
 def get_aprun_cmd_for_case(case, run_exe):
diff --git a/scripts/lib/CIME/case.py b/scripts/lib/CIME/case.py
--- a/scripts/lib/CIME/case.py
+++ b/scripts/lib/CIME/case.py
@@ -149,7 +149,7 @@ def _initialize_derived_attributes(self):
             }
 
         executable = env_mach_spec.get_mpirun(self, mpi_attribs, job="case.run", exe_only=True)[0]
-        if executable == "aprun":
+        if "aprun" in executable:
             self.num_nodes = get_aprun_cmd_for_case(self, "acme.exe")[1]
             self.spare_nodes = env_mach_pes.get_spare_nodes(self.num_nodes)
             self.num_nodes += self.spare_nodes
@@ -1119,10 +1119,10 @@ def get_mpirun_cmd(self, job="case.run"):
         executable, args = env_mach_specific.get_mpirun(self, mpi_attribs, job=job)
 
         # special case for aprun
-        if executable == "aprun":
-            aprun_cmd, num_nodes = get_aprun_cmd_for_case(self, run_exe)
+        if  "aprun" in executable:
+            aprun_args, num_nodes = get_aprun_cmd_for_case(self, run_exe)
             expect(num_nodes == self.num_nodes, "Not using optimized num nodes")
-            return aprun_cmd + " " + run_misc_suffix
+            return executable + aprun_args + " " + run_misc_suffix
         else:
             mpi_arg_string = " ".join(args.values())
 


Test patch: