PGFI/rum — Issue #11: Perhaps we can farm out the work of splitting up the input file
Author: mdelaurentis
Created: 2012-02-28 19:13:08+00:00
State: open

This is just an idea for a possible future enhancement. It looks like we're currently splitting up the input reads files in one process. It seems like this takes a couple hours for a large job. I'm not completely sure if this would work, but I wonder if we could instead break it up as follows:

1. Estimate the size of each chunk by dividing the size of the input file by the number of chunks.
2. Come up with an approximate region of the input file(s) for each worker to process, defined as a byte offset into the file and a length in bytes.
3. Submit a qsub job for each chunk that includes the file to split and the approximate offset and length to work on.
4. Each worker carves out its own section of the input file to work on. The byte offsets we calculated in step 2 will be approximate, so would have to make each worker seek to the offset in the file and then backtrack until it finds the start of a read.

I suppose we would need to make sure that corresponding forward and reverse reads make it into the same chunk. I'm not sure exactly how that would work. If the forward read and reverse read start at the same byte in their input files, then this would be easy.

We could also consider starting to process each chunk as it is created, rather than waiting until the whole file is split up to start processing any of the chunks.