CGATOxford/cgat — pull request #27 by IanSudbery (2014-09-16 13:54:10+00:00)
Title: Cluster_runnable

Introduction
-----------------

Many of our pipelines run short pieces of possibly computationally intensive code. At the moment there are three options for this code:

1. Package it up as a seperate script and submit this script to the cluster via P.run()

2. Write it in a module file associated with the pipeline, and then submit it to the cluster using the P.submit framework.

3. Write it as above, but call directly from the pipeline, executing the code on the head node.

cluster_submit gives you the best of each of these, without the downsides:

* You get the advantages of remote execution: you do not hog CPU or memory on CGAT150, and yet can execute an unlimited number of jobs in parallel.

* You don't have to know or remember what the absolute path to the module containing function you wish to call is.

* You don't have to encode your function arguements as a list of infiles, outfiles plus an arbitrary list of params, that are unlabels, can only be strings, and cannot contain ',' or shell interpretable charactors, instead you can use as rich a call signature as you like, with both positional and keyword arguements.

* You get a nice clean function call in your pipeline, that looks just like a local function call

* You do not have a seperate script for each task, that will only ever be called by this pipeline. You do not have to write all the boilerplate code that goes along creating a fully documented and readable script.

How to use it
------------------

To use this first import it into your script:

```python

from CGAT.Pipeline import cluster_runnable

```

Now when we wish to write a function that can be submitted to the cluster, we decorate the function with cluster_runnable

```python

@cluster_runnable
def my_function(some_list, some_dict):
for item in some_list:
print some_dict[item]

```

The decorated function now takes an additional arguement - submit, which tells it to submit the function as a job to run on the cluster:

```python

my_function(my_list, my_dict, submit=True)

```

Note that the function doesn't have to be submitted. Omitting submit, or passing submit=False will lead to the function being executed in the local process exactly as if it hadn't been decorated.

The decorator will take care of encoding the arguements of your function in a way that can be passed to the cluster, and decoding them once running on the cluster.

Behind the scenes, the P.submit infrastructure is still used and you can also pass the P.submit toCluster, jobOptions, and logfile arguements to a decorated function.

There are two limitations on the arguments that a decorated function can use:
1. The must be picklable.

2. You can't use submit, toCluster, jobOptions or logfile as arguements to the function as they are reserved for passing to P.submit.

Behind the scenes
------------------------

cluster_runnable replaces your function with one that tests for submit=True, if it doesn't find it, it just returns the original function. If it does find it, it removes the P.submit arguments from kwargs and the pickles any remaining args and kwargs in a temporary file stored in a cluster readable location. Uses P.submit to call its own run_picked function, passing as params, the module file (automatically detected), function name and the name of the file containing the pickled arguements.

run_picked is then executed by run_function.py on the cluster, it reads in the args and kwargs from the provided file, imports the specified module and runs the function with submit=False. The args file is deleted upon sucessful execution.