Package: cluster-func
# Cluster-funcCluster-func is a command line tool that lets you scale embarassingly parallelsolution written in Python with zero additional code.## InstallWith pip```bashpip install cluster-func```From source```bashgit clone git://github.com/enewel101/cluster-funccd cluster-funcpython setup.py develop```## Why?Too often I find myself writing multiprocessing and job-division / schedulingboilerplate code, which, despite being conceptually straightforward, is tediousand error-prone.Sure, `xargs` is nice, assuming that you can conveniently get the shell tospit out your argument values.  But that's often not the case, andyou may want your arguments to be arbitrary python types instead of strings.And, sure, Hadoop is nice too, assuming you've got a lot of time to burnconfiguring it, and that your mappers and reducers don't use too muchmemory, and that you've loaded all your data in the hdfs,and that the results from your maps don'tyield too much data and overwhelm the network.  Oh, and assumingyou enjoy writing boilerplate mapper and reducer code... wait, maybe hadoopisn't so nice... (OK, OK, it does have its place!)## Basic usageCluster-func is designed for situations where you need to run a single functionon many different arguments.  This kind embarassingly parallelizable problemcomes up pretty often.  At it's minimum, a solution involves defining**A)** the function to be repeatedly called, and **B)** all of the differentarguments on which it should be called.Cluster-func assumes that you have definedthese in the form of a *callable* and an *iterable* within a python script, andthen it handles the business of spreading the work across cores and machines.The nice thing about this approach is that you unavoidably define these twothings when you write your code for a single process anyway.  So you'll getmultiprocessing and cluster processing basically for free!The tool has two modes.  In **direct mode**, it runs your function on the cpus ofa single machine.  In **dispatch mode**, it breaks the work into subjobs that canbe run on separate machines, and optionally submits them to a job schedulerusing `qsub`.### Direct modeThis command:```bash$ cluf my_script.py```will look in `my_script.py` for a function called `target` and an iterable called`args`, and it will run `target` on every single value yielded from `args`,spawning as many workers as there are cpu cores available to your user on themachine.To use other names for you target function or arguments iterable,or to use a different number of worker processes, do this:```bash$ cluf my_script.py --target=my_func --args=my_iterable --processes=12	# short options -t, -a, -p```If `args` yields a tuple, its contents will be unpacked and interpreted as thepositional arguments for one invocation of the target function.  If you needgreater control, for example, to provide keyword arguments, then see**<a href="#arguments-iterable">Arguments iterable</a>**`args` can also be a callable that *returns* an iterable (including a generator),which is often more convenient.So, using `cluf` in direct mode lets you multiprocess any script without poolsor queues or even importing multiprocessing.  But, if you really need to scaleup, then you'll want to use the dispatch mode.### Dispatch modeThe main use of dispatch mode is to spread the work in a cluster thatuses the `qsub` command for job submission.  But you can still use `cluf` tospread work between machines don't use `qsub`.Dispatch mode is implicitly invoked when you specify a number of compute nodesto use (you can force the running mode using `--mode`, see **<ahref="#reference">Reference</a>**).For example, this command:```bash$ cluf my_script.py --nodes=10 --queue	# short options -n, -q```will break the work into 10 subjobs, and submit them using qsub.  It doesthis by writing small shell scripts, each of which is responsible for callingthe target function on a subset of the arguments yielded by the iterable.(To learn about setting PBS directives for your subjob scripts, see **<ahref="#pbs-options">PBSoptions</a>** below.)Because each subjob script is a valid shell script, you can manually run themon separate machines in case they aren't part of a cluster that uses `qsub`.Just leave off the `--queue` option, and the scripts will be created but notenqueued.  This is also a good way to test run one of the subjobsbefore submitting it.To divide the work properly, it's important that your argument iterable yieldsthe same arguments in the same order on each machine.  If you can't or don'twant write your iterable that way, see **<a href="#how-work-is-divided">How work is divided</a>** for other options.And that's the end of the basic usage guide.  This will cover you for the mostbasic usecases.  To learn about more cool features, read on!## <a name="arguments-iterable">Arguments iterable</a>The main usecase imagined is one where the arguments iterable yields eithersingle, bare arguments, or tuples of positional arguments.  Of course, thePython language provides a veryflexible way of calling functions, allowing you to mix positional argumentsand keyword arguments.  If you need that flexibility, then set up your iteratorto yield `cluster_func.Arguments` objects.  This class acts as a proxy, andwill be used to call your target function in exactly the way you called the`Arguments` constructor.Here is the `Arguments` class in action:```python>>> from cluster_func import Arguments>>> my_args = Arguments(0, *[1,2,3], four=4, **{'five':5, 'six':6})>>> my_argsArguments(0, 1, 2, 3, four=4, five=5, six=6)>>>>>> # Your target function would be called the same way you called Arguments, e.g.:>>> my_target(0, *[1,2,3], four=4, **{'five':5, 'six':6})```## <a name="how-work-is-divided">How work is divided</a>By default, work is divided by assuming that the arguments iterator will yieldthe same arguments in the same order during each subjob.  Each subjob can thenexecute the target function only on those arguments assigned to it.For example, if there were 10 subjobs, subjob 7 would run arguments 7, 17, 27,... etc.  For ease of explanation, we'll call that subset "bin 7".If you open the subjob scripts in an editor, you'll find that they actuallycall `cluf` itself in *direct mode*.  In other words, when you run`cluf` in dispatch mode, it creates scripts that call `cluf` in direct mode.These direct-mode invocations use the `--bin` option, whichis what instructs `cluf` to only run arguments that fall into that subjob'sbin.  For example, this command:```bash$ cluf my_script --bin=0/3		# short option: -b```would run `cluf` in direct mode, but only execute iterations falling into bin 0out of 3, i.e., iterations 0, 3, 6, 9, etc.  (Bins are zero-indexed.)You can use this to start subjobs manually if you like.You can assign multiple bins to one subjob, For example, the option`--bins=0-2,5/10` will assign bins 0, 1, 2, and 5 (out of a total of 10 bins).### If your iterable is not stableThe default approach to binning assumes that the arguments iterable willyield the same arguments in the same order during execution of each subjob.If you can't ensure that, then binning can be based on the arguments themselves,instead of their order.There are two alernative ways to handle binning: using *argument hashing* and*direct assignment*.### Argument hashingBy specifying the `--hash` option, you can instruct `cluf` to hash one or moreof the arguments to determine its bin.For example, doing:```bash$ cluf example --nodes=12 --hash=0		# short options: -n and -x```will instruct `cluf` to hash the first argument of each iteration to decidewhich bin the iteration belongs to.Before hashing, `cluf` calls `str` on the argument.It's important that the argument selected for hashing has astable string representation that reflects its value. Using objects that don'timplement `__str__` won't work, both because their string representationdoesn't reflect their value, and because their memory address appears withinit, which will be different in each subjob.  However, for this purpose, a listwould be considered "hashable" (provided it's individual elements are).  On theother hand, dict's and sets are not suitable, because they are unordered, sotheir string representation is not stable.  One safe approach is to simplyprovide one argument that is a unique ID, and select it for hashing.Ideally the argument selected for hashing should be unique throughoutiteration, since repeated values would be assigned to the same subjob.  Butoccaisional repetitions won't imbalance load much.  To help achieve uniquenessyou can provide combinations of arguments to be hashed.  If you providedarguments as keyword arguments (using an Arguments object) you can select themtoo.For example, this:```bash$ cluf example --nodes=12 --hash=0-2,5,my_kwarg	# short options: -n and -x```will hash arguents in positions 0,1,2, and 5, along with the keyword argument`my_kwarg`.  If any hashed arguments are missing in aniteration (becase, recall, invocations may use different numbers of arguments),they are simply ommitted when calculating the hash.### Direct assignmentThe final method for dividing work is to include an argument that explicitlyspecifies thebin for each iteration.  To activate direct assignment, and to specify whichargument should be interpreted as the bin, use `--key` option:```bash$ cluf example --nodes=12 --key=2 		# short options: -n and -k```In the command above, the argument in position 2 (the third argument)will be interpreted asthe bin for each iteration.  You can also specify a keyword argument by name.You should only use direct assignment if you really have to, because it's moreerror prone, and it makes it more difficult to change the number of bins.It alsointroduces job division logic into your script which `cluf` was designed toprevent.## `cluf_options` and `.clufrc`For more extensive configuration, you can include a dictionary named`cluf_options` in your target script tocontrol the behavior of `cluf`.  This can be more convenient than the commandline if you have to set a lot of options, and helps to document the optionsyou used.  You can also set options globally in a file at `~/.clufrc`.All options that can be set on the command line can be set within `cluf_options`or `.clufrc`, plus a few extras.  There is one exception, however, which isthe option to force direct / dispatch mode (`--mode` or `-m`), which can onlybe set on the command line.`cluf_options` should be a dictionary whose keys are the long option names,and whose values are strings representing the option values as you would enterthem on the command line.  `.clufrc` should be a valid JSON object, with thesame key-value format.  See the following examples:**Using `cluf_options`**(in `my_script.py`)```pythoncluf_options = {'hash': (0,1,2,5),	# Or use '0-2,5'.  Applies to .clufrc as well'nodes': 12,'queue': True		# Flag options based on truthiness of value}```**Using `.clufrc`**(in `~/.clufrc`)```json{"hash": [0,1,2,5],"nodes": 12,"queue": true}```In some cases you may want to use `cluf_options` to simply modify (e.g. add to)the options in `.clufrc`, rather than overriding them.  You can access the`.clufrc` options by importing them, using `from clusterf_func import RC_PARAMS`.For the most part, any option that can be set on the command line can be set in`cluf_options` and `.clufrc`, and vice versa, but there are a few options thatcan *only* be set in `cluf_options` and `.clufrc`.  We cover those now alongwith some options that are just less convenient to set on the command line.See **<a href="#reference">Reference</a>** for all available options.### Environment variables(Note: This option can be set on the command line but is somewhat lessconvenient.)Let's suppose execution of your target script requires certain environmentvariables to be set.  If you run `cluf` in *direct mode*, there's nothing tothink about—your script will inheret the current environment.For example, if you did:```bashMYENV=1 cluf my_script.py```The value of `MYENV` would be seen by your script.However, if you run `cluf` in dispatch mode, then the job scripts willrun in a different environment.  Use the `env` option to specify anyenvironment variables that should be set when running the subjobs.  `env` shouldbe a dictionary within `cluf_options` or `.clufrc` that has variable namesas keys and values as, well, values:(in `my_script.py`)```pythoncluf_options = {'env': {'MYENV': 'foo','OTHER_ENV': 'bar',}}```(in `~/.clufrc`)```json{"env": {"MYENV": "foo","OTHER_ENV": "bar"}}```If provided on the commandline, the contents of `env` will be pasted as-isin front of the line that runs the subjob within the subjob script.  So theequivalent `env` specification would be:```bashcluf my_script.py --nodes=4 --env='MYENV=foo OTHER_ENV=bar'```Each of these methods will cause the invocation of subjobs within subjobscripts to look like this:```bashMYENV=foo OTHER_ENV=bar cluf my_script [...options]```Note that, regardless of where it is specified, if you want to provide avalue for an environment variable that contains a space or other characterneeding escaping, keep in mind that a round of interpretation by thecommand line, Python, or JSON will have occurred (depending on where it was set)So, e.g., this won't work:```bashcluf my_script.py --nodes=4 --env='MYENV=foo bar'```Nor will, this:```bashcluf my_script.py --nodes=4 --env=MYENV=foo\ bar```But This will work:```bashcluf my_script.py --nodes=4 --env='MYENV=foo\ bar'```And so will this:```bashcluf my_script.py --nodes=4 --env=MYENV=foo\\\ bar```### <a name="pbs-options">PBS options</a>(This option cannot be set on the command line.)Portable Batch System options control how the cluster scheduler schedules yourjob, and allows you to request specific compute resources and specify theamount of time that your job should run.In general PBS options should be set using a dictionary of key-value pairsusing the option name as the key.  For example, to request that yoursubjobs run on compute nodes with at least 4 cpu cores and 2 gpus, you can dosomething like this:(in `my_script.py`)```pythoncluf_options = {'pbs_options': {'ppn': 4, 'gpus': 2}}```(in `~/.clufrc`)```json{"pbs_options": {"ppn": 4, "gpus": 2}}```The `ppn` (processes per node) option should usually match the number of workerprocesses set by the  `processes` option (whether set on the command line,`cluf_options`, or in `.clufrc`).  So if `ppn` isn't explicitly set in yourPBS options, but `processes` is set, then it will default to the value of`processes`.  You can still set a different value for ppn, if e.g.  your targetfunction itself spawns proceses.There are three special options whose names differ from thePBS option names slightly, and these options are set to defaults unlessspecifically overridden.- `'name'`: the name of your subjobs as they appear in the job scheduler.This is also used to name your subjob scripts (by appending `'.pbs'`).The default is the format string `'{target}-{subjob}-{num_subjobs}'`,with the fields being interpolated by the target module's name, thesubjob number, and the total number of subjobs respectively.If you overridethis you can also use those format fields, and you must at least usethe `{subjob}` field to ensure that each of your subjobs gets aunique name (otherwise the subjob scripts will overwrite one another).- `'stdout'`: the path at which to place stdout captured from your subjobs,relative to the `jobs_dir` if set (if not set then it is relative to thecurrent working directory).This defaults to the subjob name plus `'.stdout'`As for the `'name'` option,if you override this, make sure that the paths for subjobsare unique by using the `{node_num}` field somewhere.- `'stderr'`: similar to `'stdout'`.  Defaults to the subjob name plus`'.stderr'`The combinations of PBS options that are available and/or required depends onthe setup of your cluster.  Usually a system is configured with smart defaultsso that you can queue simple jobs without setting any PBS options.### Additional statements(This option can be set on the command line but may be more convenient to setwithin your target script.)You can also specify any additional statements that you want to appear in yoursubjob scripts.  This gives you more flexibility than simply setting environmentvariables.   You can include an external script, whose statements get mergedinto the job script before the line that runs subjob using`prepend_script`.  Include an external script *after* the line that runsthe subjob using the `append_script` option.If you just want to add a statement or two, it maybe more convenient toput them in your target script (or `.clufrc`).The options `prepend_statements` and `append_statements` can be used provide alist of shell statements that get inserted right before or after the linethat actually runs the subjob within subjob scripts.  Each element of the listshould be a valid shell statement which will appear on its own line when mergedinto the jobscripts.  The options aren't available on the command line.# <a name="reference">Reference</a>The `cluf` command has lots of options, which can be specified in threedifferent places:1. as command line arguments, or2. within your target module, inside a dictionary called `cluf_options`, or3. in the `~/.clufrc` file, in the form of a JSON objectThese locations are in decreasing order of precedence, i.e. the command lineoverrides all other options, and the `.clufrc` file doesn't override the others.Options given in the `cluf_options` dictionary in the target module or in the`.clufrc` JSON object should be identified by the long option name, withoutthe leading '--'.### All `cluf` options<pre>usage: cluf [-h] [-j JOBS_DIR] [-t TARGET] [-a ARGS] [-q] [-p PROCESSES][-b BINS] [-e ENV] [-P PREPEND_SCRIPT] [-A APPEND_SCRIPT][-m {dispatch,direct}] [-x HASH | -k KEY][-n NODES | -i ITERATIONS]target_moduleRun a function many times on multiple processors and machines.positional arguments:target_module         path to the python module that contains the targetfunction to be run many times.optional arguments:-h, --help            show this help message and exit-j JOBS_DIR, --jobs-dir JOBS_DIRSpecify a directory in which to store job scripts andthe files generated from the stdout and stdin duringjob execution. This directory will be made if itdoesn't exist. This option only takes effect indispatch mode.-t TARGET, --target TARGETAlternate name for the target callable to be invokedrepeatedly. Default is "target".-a ARGS, --args ARGS  Alternate name for the arguments iterable. Default is"args".-q, --queue           Enqueue the generated scripts using qsub. This optiononly takes effect in dispatch mode.-p PROCESSES, --processes PROCESSESNumber of processors to use.-b BINS, --bins BINS  Optionally specify a portion of the work to be done.Should take the form "x/y" meaning "do the x-thsection out of y total sections. For example, "0/2"means divide the work into two halves, and do thefirst (0th) half. Note that x and y should beintegers, and x should be from 0 to y-1 inclusive.This option only takes effect in direct mode.-e ENV, --env ENV     Provide environment variables that should be set whenrunning sub-jobs. This is for use in dispatch mode,since job scripts will run in a different environment.In direct mode, the environment is inherited. Thevalue of this option should be an enquoted string ofspace-separated key=value pairs. For example: $ clufmy_script -n 12 -e 'FOO=bar BAZ="fizz bang"'will setFOO equal to "bar" and BAZ equal to "fizz bang". Thisoption only takes effect in dispatch mode.-P PREPEND_SCRIPT, --prepend-script PREPEND_SCRIPTPath to a script whose contents should be included atthe beginning of subjob scripts, being executed beforerunning the subjob. You can include multiple comma-separated paths. This option only takes effect indispatch mode.-A APPEND_SCRIPT, --append-script APPEND_SCRIPTPath to a script whose contents should be included atthe end of subjob scripts, being executed after thesubjob completes. You can include multiple comma-separated paths. This option only takes effect indispatch mode.-m {dispatch,direct}, --mode {dispatch,direct}Explicitly set the mode of operation. Can be set to"direct" or "dispatch". In direct mode the job is run,whereas in dispatch mode a script for the job(s) iscreated and optionally enqueued. Setting either -n or-i implicitly sets the mode of operation to"dispatch", unless specified otherwise.-x HASH, --hash HASH  Specify an argument or set of arguments to be used todetermine which bin an iteration belons in. Thesearguments should have a stable string representation(i.e. no unordered containers or memory addresses) andshould be unique over the argumetns iterable. Thisshould only be set if automatic binning won't work,i.e. if your argument iterable is not stable.-k KEY, --key KEY     Integer specifying the positional argument to use asthe bin for each iteration. That key argument shouldalways take on a value that is an integer between 0and num_bins-1. This should only be used if you reallyneed to control binning. Prefer to rely on automaticbinning (if your iterable is stable), or use the-xoption, which is more flexible and less error-prone.-n NODES, --nodes NODESNumber of compute nodes. This option causes thecommand to operate in dispatch mode, unless the modeis explicitly set-i ITERATIONS, --iterations ITERATIONSApproximate number of iterations per compute node.This option causes the command to operate in dispatchmode, unless the mode is explicitly set. Note thatusing this instead of --nodes (-n) can lead to delaybecause the total number of iterations has to becounted to determine the number of compute nodesneeded.</pre>