arq5x/gemini — pull request #82 by roryk (2013-03-15 19:10:35+00:00)
Title: Added iPython interaction, speeded up merging/loading steps.

This implements what we talked about over email.

1) loading now pipes the grabix subset, it is quite a bit faster on my test data set
2) merge now happens recursively. it merges pairs of all of the chunks. then pairs of those pairs, etc until all of the pairs are merged.
3) The scheduler handling has been shifted over to using IPython. This allows for easier support of different schedulers. Cloudifying it will be easier that way too when the million genomes project gets done.
4) SGE support was added.

This needs a refactor, there is some cruft and some duplication.
The IPython step should not spin up a new cluster for the recursive merge, I'll add that.