cooperative-computing-lab/cctools — pull request #179 by leeping (2013-08-29 20:37:28+00:00)
Title: Add multiple-node batch jobs and SSH port forwarding to job submission scripts.

Hi there,

This is related to an email discussion that we had on the CCTools discussion list. Please let me know what you think.

From Dinesh:

```
Lee-Ping,

Thank you for volunteering your contributions!

> I have a couple of customized Work Queue scripts on various clusters which
> implement various simple "features" that make it easier to run on those
> clusters. The features include:
>
> 1) SSH port forwarding capability, which helps if the compute node
> doesn't have Internet connectivity, or if the master is behind a firewall.
> The procedure goes like this:
>
> a. First, manually establish a connection from the master node to an
> intermediate node (e.g. the cluster head node) using SSH remote or local
> port forwarding.
>
> b. The submitted job establishes a connection to the master node using
> SSH local port forwarding.
>
> c. The worker connects to a port on localhost, which actually connects
> to the master.

This will be a good addition to the How-To link here:
http://www.nd.edu/~ccl/software/howto/wq-portforward.shtml

If you can send us the script, we can link it there.

> 2) Multiple node submission for trivially parallel tasks; you can
> submit a single multi-node job and it launches workers on each of the nodes
> using "pbsdsh".

This is useful. We can integrate this into our pbs_submit_workers
script (this feature sounds similar to the job array feature, the -j
option, in sge_submit_workers).

Will it be possible for you to send us a pull request including these
changes in pbs_submit_workers in GitHub
(https://github.com/cooperative-computing-lab/cctools)?
```