samtools/htslib — pull request #132 by jkbonfield (2014-09-19 16:25:20+00:00)
Title: Cram3 updates

The big patch of CRAM v3.0 (and other CRAM io_lib) updates. Sorry it's munged together a bit, but this was inevitable due to the diversion between io_lib and htslib. I'll endeavour to do much much smaller syncs from now on. You may wish to accept the cram/* changes but be more scrutinising of the top-level htslib and samtools changes.

I can also do the white-space fixing of cram at some point, but assumed you'd want to merge this first so that the diff is smaller and easier to inspect.

Basic summary:

1) CRAM 3.0 - rANS codec; bzip2 & lzma codecs; CRC32; multi-base features; etc.

2) Major refactoring of handling data-series in CRAM. Gone are lots of custom named fields in the cram structures, replaced by arrays indexed by DS_?? enums. Most data series now go to external data blocks unless tiny (and arguably even those may not be best put there). Added a mechanism for working out which data series are dependent on which other data series, both a function of code paths and also data paths.

3) Added an hts.[ch] API for specifying arbitrary options in CRAM input and output. This includes the CRAM_OPT_REQUIRED_FIELDS option in point 2 above. NB: does this belong in hts.c, or in sam.c? It all depends on whether we'll ever envisage having a newer BCF equivalent that also uses columnar storage.

The test/test_view tool uses this with -i and -o flags for specifying input and output options.

4) Multi-threading for CRAM is enabled now, but still only for writing. The thread pool is used for this, but isn't yet shared with the decoder side (this is another issue here IIRC).

5) Added a stab at auto-detecting libbz2 and liblzma to the Makefile, but these are commented out as although it works fine in htslib it breaks samtools builds. We need an htslib-config script (a la curl-config) or a pkgconfig file so we can specify the library dependencies when using the htslib.a archive.

6) Removed the need for -DSAMTOOLS when compiling.