ipython/ipython — pull request #3465 by jabooth (2013-06-25 09:56:35+00:00)
Title: Add HTCondor bindings to IPython.parallel

This PR adds a new set of subclasses in `IPython/parallel/apps/launcher.py` to add support for the HTCondor batch management system.

I've tried to make a template that has sensible defaults that should work out of the box for most users.

**Help config setup**
I think I have added config/help support where required but I am still getting my head around the config/traitlets setup in IPython. Does everything look ok?

**BatchLauncher changes**
1. Adding job and queue templates to the job are now overridable methods (Condor has to place it's `job_array` at the end)
2. Added `job_id_regexp_group` to allow for subclasses to specify what group of the `job_id_regexp` needs to be matched. By default this is 0, which means all current implementations remain unchanged, whilst allowing for Condor to have a simple regexp setup. I'm not the worlds foremost regexp expert so there might be a way around this without introducing groups..

**Batch launch commands changes**
Condor destroys sys.executable for the jobs it farms out, ruining Python's module path checking. This is pretty problematic for the current batch setup, which calls something like
`/local/python/path/bin/python -c 'commands required to start the engine or controller' --profile-dir=...`
on the remote.

However, by instead calling
`/local/python/path/bin/ipengine --profile-dir=...` and
`/local/python/path/bin/ipcontroller --profile-dir=...`
we can skirt around the issue as on entry to these scripts the shebang causes `/local/python/path/bin/python` to be called, with the sys.executable path set correctly.

I can't think of any downside to this approach, other than you now demand that python is running from a folder containing `ipengine` and `ipcontroller`, which is of course the setup everyone has from install. My only concern is that I have no idea how these scripts (and Condor in general for that matter) behave in the Windows world.

**Documentation changes**
Finally, once accepted the docs probably want to be upgraded to mention the support. Is that something I should do in this PR?