vadimzalunin/crammer — pull request #29 by virajbdeshpande (2012-10-06 02:23:22+00:00)
Title: Sharding Cram files and inputting multiple datasets simultaneously

Hello Vadim,

Sorry for the delay, I was distracted by some other stuff. The patch I have added is a very small one.
The only external change to CramTools is that if the name for a read is not preserved, then the name(number) of the outputted read is now prefixed with shard number.

The modifications to Bam2Cram.java are aimed to uniformly shard the reads based on their position on the reference.
The reason I choose to shard the file independently on each work unit (processor) is so that when a downstream application wants to process this data in a parallel fashion, it does not have to do a RPC to retrieve the relevant shard and parallel disk copy to multiple nodes is easier.
There is no special modification for Cram2Bam, because this can achieved by simply concatenating the decompressed bam file.

Let me know you comments about the same and if you would like a different design for this. Also looking forward to the release of Cram C/Java API.

Viraj

PS: I hope I have added the copyright notice in the right fashion, though that should not be important as it will be reverted back to EBI with the next revisions of the files.

PS: Comments from the commit:
cramtools command:
bam:
New input arguments:
--input-bamlist-file (optional alternative for --input-bam-file)
--output-cramlist-file
--read-annolist-file
--stats-outlist-file
--tram-outlist-file
--work-unit
--num-work-units

cram:
New input argument:
--read-name-prefix (should be unique for different shards)

New requirement:
Dictionary for reference sequence is now required.
If ref.fa or ref.fasta is the reference-fasta-file, then ref.dict should be created.
Use picard tools library to create ref.dict file:
java -jar picard_tools/CreateSequenceDictionary.jar R=ref.fa O=ref.dict