bigdatagenomics/adam — pull request #9 by tdanford (2013-11-27 16:17:26+00:00)
Title: Adding support for a Sequence Dictionary from BAM files

One of the things that's missing from the current ADAM format and tools is the ability to represent and manipulate the "sequence dictionaries" (from the @SQ lines in the SAM / BAM header) which determine the ordering and indexing of the referenceName fields within the records of a BAM file.

We're going to need to represent and handle these dictionaries for anything that works with more than one ADAM file at a time -- two examples would be ADAM comparison tools, and multi-sample methods for analyzing read data.

This request includes an extension to the ADAM avdl file containing a schema for an ADAMSequenceRecord, and the definition of a class (SequenceDictionary) which handles multiple sequence records, and mapping between different dictionaries. I've extended the bam2adam command to extract and save a set of ADAMSequenceRecord objects to a parquet file, and I've included two new commands: ListDict (for listing the contents of one of these files) and CompareAdam, for counting the number of identically-aligned reads between two ADAM files even if they have different sequence dictionaries.

Places that _I_ know of, that probably need review and editing:
1. the Bam2Adam code that saves the dictionary -- we're just adding a ".dict" extension to the given ADAM output path, but that's probably fragile.
2. CompareAdam doesn't handle paired end reads, and is a little underpowered at the moment. We (GenomeBridge) are planning on building a fuller set of tools, using ADAM, to compare and validate the output of multiple pipelines on the same data, but that's not part of CompareAdam (yet). In the meantime, perhaps CompareAdam would be better as a test?
3. We're leaving undetermined, here, whether there should be a "master" SequenceDictionary which is maintained and saved across all ADAM files, and to which each ADAM file's dictionary can be mapped or whether every analysis should also take the sequence dictionary for each component ADAM file as an argument (what we call the "local dictionaries only" approach).
4. I've checked in example BAM and parquet files for testing CompareAdam, but I haven't incorporated them into a proper test yet.