Repository: Parsl/parsl

Issue:
Running on a local cluster
I'd like to have parsl push jobs out to a small cluster we've got in the HEP group. About 120 cores, split over about 10 machines. There is no batch manager or similar.

The pilot model seemed appropriate. I was thinking to use the IPyParallelExecutor, and then have each on a seperate note.

However, I don't know what to use for a provider. LocalProvider seems like what I want - tho it needs to be on another machine. I tried to hack it by giving it an SSHChannel, but that didn't work.

Here is the configuration I've been playing with:

        remote_config = Config(
            executors=[
                IPyParallelExecutor(
                    label='remote_ipp',
                    provider=LocalProvider(
                        min_blocks=1,
                        init_blocks=1,
                        max_blocks=4,
                        tasks_per_node=2,
                        nodes_per_block=1,
                        parallelism=0.5,
                        channel=SSHChannel(hostname=m)
                    )
                ) for m in args.remote
            ]
        )
        parsl.load(remote_config)

If I remove the "channel=" line, btw, then my code runs just fine. All the jobs seem to execute without difficulty. Though the IPy executor generates a huge number of log messages!

I'm a totally newbee here - This is the first time I've used parsl. Very nice, btw, in about 2 hours I had something up and running on a single machine. Took me another 2 days to make it "robust". :-)


Discussion:
Welcome @gordonwatts!!!

Your approach sounds reasonable to me. What error are you getting when you run with `channel=...`? Note also that you should use a unique label for each executor.
Ops - thanks for the note about the label. I would have run into that as soon as I tried more than one machine.

So, I did see some crashes. But what made me think I was really taking the wrong approach was that I ran this on a machine called tev01. And had it connect to tev02 for running. But the processes and jobs were actually executing on tev01.

I will capture a log file for you. I am sure I'm doing something wrong here - I didn't find a working example, so I was looking through the source code trying to figure out how to make this work.
I think your idea of a "local cluster" is probably different than ours, so this will probably take some exchange to understand fully
Awesome job, @gordonwatts, sounds like you've discovered a mode of running we haven't tried before. You're a trailblazer! @yadudoc is going to test this on our side. It sounds like there may be some bugs we need to fix when using LocalProvider+SSHChannel.
The docs for LocalProvider say for "running on localhost" - so I was thinking LocalProvider was the wrong thing for me to use. Log file in a second (edit: shoot, meeting hell starting, will get it up in a short while). I can also point you to the source code if you think it will help. But it is a mess (first time I've used this, and I always get stuck in monad hell when working with futures). And thanks for your help!

Resolving patch:
diff --git a/parsl/providers/local/local.py b/parsl/providers/local/local.py
--- a/parsl/providers/local/local.py
+++ b/parsl/providers/local/local.py
@@ -52,18 +52,22 @@ def __init__(self,
                  min_blocks=0,
                  max_blocks=10,
                  walltime="00:15:00",
+                 worker_init='',
+                 cmd_timeout=30,
                  parallelism=1):
         self.channel = channel
         self.label = 'local'
         self.provisioned_blocks = 0
         self.nodes_per_block = nodes_per_block
         self.launcher = launcher
+        self.worker_init = worker_init
         self.init_blocks = init_blocks
         self.min_blocks = min_blocks
         self.max_blocks = max_blocks
         self.parallelism = parallelism
         self.walltime = walltime
         self.script_dir = None
+        self.cmd_timeout = cmd_timeout
 
         # Dictionary that keeps track of jobs, keyed on job_id
         self.resources = {}
@@ -81,16 +85,31 @@ def status(self, job_ids):
 
         logging.debug("Checking status of : {0}".format(job_ids))
         for job_id in self.resources:
-            poll_code = self.resources[job_id]['proc'].poll()
-            if self.resources[job_id]['status'] in ['COMPLETED', 'FAILED']:
-                continue
 
-            if poll_code is None:
-                self.resources[job_id]['status'] = 'RUNNING'
-            elif poll_code == 0 and self.resources[job_id]['status'] != 'RUNNING':
-                self.resources[job_id]['status'] = 'COMPLETED'
-            elif poll_code < 0 and self.resources[job_id]['status'] != 'RUNNING':
-                self.resources[job_id]['status'] = 'FAILED'
+            if self.resources[job_id]['proc']:
+
+                poll_code = self.resources[job_id]['proc'].poll()
+                if self.resources[job_id]['status'] in ['COMPLETED', 'FAILED']:
+                    continue
+
+                if poll_code is None:
+                    self.resources[job_id]['status'] = 'RUNNING'
+                elif poll_code == 0 and self.resources[job_id]['status'] != 'RUNNING':
+                    self.resources[job_id]['status'] = 'COMPLETED'
+                elif poll_code < 0 and self.resources[job_id]['status'] != 'RUNNING':
+                    self.resources[job_id]['status'] = 'FAILED'
+
+            elif self.resources[job_id]['remote_pid']:
+
+                retcode, stdout, stderr = self.channel.execute_wait('ps -p {} &> /dev/null; echo "STATUS:$?" ',
+                                                                    self.cmd_timeout)
+                for line in stdout.split('\n'):
+                    if line.startswith("STATUS:"):
+                        status = line.split("STATUS:")[1].strip()
+                        if status == "0":
+                            self.resources[job_id]['status'] = 'RUNNING'
+                        else:
+                            self.resources[job_id]['status'] = 'FAILED'
 
         return [self.resources[jid]['status'] for jid in job_ids]
 
@@ -158,12 +177,38 @@ def submit(self, command, blocksize, tasks_per_node, job_name="parsl.auto"):
         script_path = "{0}/{1}.sh".format(self.script_dir, job_name)
         script_path = os.path.abspath(script_path)
 
-        wrap_command = self.launcher(command, tasks_per_node, self.nodes_per_block)
+        wrap_command = self.worker_init + '\n' + self.launcher(command, tasks_per_node, self.nodes_per_block)
 
         self._write_submit_script(wrap_command, script_path)
 
-        job_id, proc = self.channel.execute_no_wait('bash {0}'.format(script_path), 3)
-        self.resources[job_id] = {'job_id': job_id, 'status': 'RUNNING', 'blocksize': blocksize, 'proc': proc}
+        job_id = None
+        proc = None
+        remote_pid = None
+        if not isinstance(self.channel, LocalChannel):
+            logger.debug("Not a localChannel, files need to be moved")
+            script_path = self.channel.push_file(script_path, self.channel.script_dir)
+
+            # Bash would return until the streams are closed. So we redirect to a outs file
+            cmd = 'bash {0} &> {0}.out & \n echo "PID:$!" '.format(script_path)
+            retcode, stdout, stderr = self.channel.execute_wait(cmd, self.cmd_timeout)
+            for line in stdout.split('\n'):
+                if line.startswith("PID:"):
+                    remote_pid = line.split("PID:")[1].strip()
+                    job_id = remote_pid
+            if job_id is None:
+                logger.warning("Channel failed to start remote command/retrieve PID")
+        else:
+
+            try:
+                job_id, proc = self.channel.execute_no_wait('bash {0}'.format(script_path), self.cmd_timeout)
+            except Exception as e:
+                logger.debug("Channel execute failed for:{}, {}".format(self.channel, e))
+                raise
+
+        self.resources[job_id] = {'job_id': job_id, 'status': 'RUNNING',
+                                  'blocksize': blocksize,
+                                  'remote_pid': remote_pid,
+                                  'proc': proc}
 
         return job_id
 
@@ -179,11 +224,19 @@ def cancel(self, job_ids):
         for job in job_ids:
             logger.debug("Terminating job/proc_id : {0}".format(job))
             # Here we are assuming that for local, the job_ids are the process id's
-            proc = self.resources[job]['proc']
-            os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
-            self.resources[job]['status'] = 'CANCELLED'
-        rets = [True for i in job_ids]
+            if self.resources[job]['proc']:
+                proc = self.resources[job]['proc']
+                os.killpg(os.getpgid(proc.pid), signal.SIGTERM)
+                self.resources[job]['status'] = 'CANCELLED'
+
+            elif self.resources[job]['remote_pid']:
+                cmd = "kill -- -$(ps -o pgid={} | grep -o '[0-9]*')".format(self.resources[job]['remote_pid'])
+                retcode, stdout, stderr = self.channel.execute_wait(cmd, self.cmd_timeout)
+                if retcode != 0:
+                    logger.warning("Failed to kill PID:{} and child processes on {}".format(self.resources[job]['remote_pid'],
+                                                                                            self.label))
 
+        rets = [True for i in job_ids]
         return rets
 
     @property


Test patch:
diff --git a/parsl/tests/configs/htex_ad_hoc_cluster.py b/parsl/tests/configs/htex_ad_hoc_cluster.py
new file mode 100644
--- /dev/null
+++ b/parsl/tests/configs/htex_ad_hoc_cluster.py
@@ -0,0 +1,29 @@
+from parsl.providers import LocalProvider
+from parsl.channels import SSHChannel
+from parsl.executors import HighThroughputExecutor
+
+from parsl.config import Config
+
+username = "yadunand"
+remotes = ['midway2-login1.rcc.uchicago.edu', 'midway2-login2.rcc.uchicago.edu']
+
+config = Config(
+    executors=[
+        HighThroughputExecutor(
+            label='remote_htex_{}'.format(m),
+            cores_per_worker=4,
+            worker_debug=False,
+            address="128.135.112.73",
+            provider=LocalProvider(
+                init_blocks=1,
+                nodes_per_block=1,
+                parallelism=0.5,
+                worker_init="source /scratch/midway2/yadunand/parsl_env_setup.sh",
+                channel=SSHChannel(hostname=m,
+                                   username=username,
+                                   script_dir="/scratch/midway2/{}/parsl_tests/".format(username)
+                )
+            )
+        ) for m in remotes
+    ],
+)
diff --git a/parsl/tests/configs/ipp_ad_hoc_cluster.py b/parsl/tests/configs/ipp_ad_hoc_cluster.py
new file mode 100644
--- /dev/null
+++ b/parsl/tests/configs/ipp_ad_hoc_cluster.py
@@ -0,0 +1,30 @@
+from parsl.providers import LocalProvider
+from parsl.channels import SSHChannel
+from parsl.executors.ipp_controller import Controller
+from parsl.executors.ipp import IPyParallelExecutor
+
+from parsl.config import Config
+
+username = "yadunand"
+remotes = ['midway2-login1.rcc.uchicago.edu', 'midway2-login2.rcc.uchicago.edu']
+
+config = Config(
+    executors=[
+        IPyParallelExecutor(
+            label='remote_ipp_{}'.format(m),
+            workers_per_node=2,  # Replaces provider.tasks_per_node
+            engine_debug_level="DEBUG",
+            controller=Controller(public_ip="128.135.112.73"),
+            provider=LocalProvider(
+                init_blocks=1,
+                nodes_per_block=1,
+                parallelism=0.5,
+                worker_init="source /scratch/midway2/yadunand/parsl_env_setup.sh",
+                channel=SSHChannel(hostname=m,
+                                   username=username,
+                                   script_dir="/scratch/midway2/{}/parsl_tests/".format(username)
+                )
+            ),
+        ) for m in remotes
+    ],
+)