Minor speed increases to cram_byte_array_stop_decode_init and
GET_BIT_MSB macros.
Added a range coder (order 0 and 1) plus support for LZMA if compiled
in (no option for this at present).

Internally the choice of compression method is now a bit field
listing the various methods suitable.  To aid this, ZLIB_RLE is its
own method internally (although just ZLIB in the file format as it is
in the same ZLIB byte stream format), allowing better for auto-tuning
between fill ZLIB and RLE only variants.

This means we can specify as many encoding methods as we support and
get the code to auto-tune to the best methods available.
Tweaks for specifying version number. Now accepting version 3.0, and
also correctly setting the globals.

FIXME: shouldn't be using globals!
Updated the version checks to lop off major/minor components rather
than using an exact == comparison.

Also fixed the EOF block for version 3.0.
Fixed BETA codec so that it honours beta offset value for zero length
codes.
Bug fix to the external decoders. If there is an attempt to decode 0
bytes then it no longer matters if the block does not exist. (This
comes about when faced with silly CIGAR strings like "0S".)

Bug fix to cope with block searching during encoding where
slice->block_by_id is not defined.  Note this doesn't happen in normal
behaviour, but the change is for code validity.

Plus a couple minor spacing changes and commenting.
Added support for more codecs, as part of refactoring how data is
pushed.  These were in the V2.1 spec, but not used.

- EXTERNAL now access type char, instead of only for int.
- BYTE_ARRAY_LEN is now implemented fully for encoding allowing two
  distinct sub-codecs.
- BYTE_ARRAY_STOP is now fully implemented.

Both the _LEN and _STOP codecs can now be used as proper obj->encode()
style codecs, rather than assuming the calling code is manually
filling out the data in the appropriate format. (Although this is
still done by cram_encode.c.)
Major refactoring of the way CRAM handles external blocks.
We now prefer to output as many things as possible to their own
specific external block instead of utilising the CORE block more
often.  This has the impact that it is much easier to do a partial
decode.

- The C structs are now more array based, preferring ->block[DS_XX]
  instead of ->XX_blk and similar.

- Many more block types wil be external by default now. It
  auto-selects CORE vs external based on size.

- Selection of compression algorithm for external blocks is more
  advanced.  Every external block has a metrics array element. This
  now tracks more than 2 types of compression and it culls candidate
  methods if they repeatedly are unpromising.

  The upshot is the hard-coded selection of which method to use for
  which block is (mostly) removed in favour of auto detection.

- CRAM now has the option to ignore certain types of fields when
  decoding.  Combined with more external blocks this permits faster
  decoding for tools like scram_flagstat, potentially 2-3x faster.

- Auxiliary tags now get split up into multiple external blocks based
  on data type. We still aggregate many together, but tag strings,
  integers, sequences and quality strings now get their own blocks.

Also sped up the process_one_read() function when converting from BAM
packed sequence to base-calls.
Removal of a couple memory leaks added in the restructing of the last
patch.
Bug fix of use of TAG_ID macro and k vs key in hash for auxiliary
headers.

Updated the encoding of auxiliary data itself to agree with the
headers (and with the latest Scramble updates).

Bug fix to encoding the 'd' (double) aux data type.  We've never
tested
this as it's not even in the SAM spec, but I added it anyway as
samtools seems to include it.
Modified the thread pool to use as few threads as possible. The
intention is that when given, say, 16 threads but being I/O bound such
that 12 cores is enough to keep up with the I/O then we have 12
threads kept busy and 4 totally idle.

This is implemented by dispatching jobs on lower thread IDs in
preference and maintaining a queue of pending jobs in proportion to
the number of threads currently running, rather than instantly
dispatching.

Removed uninitialised memory behaviour (worse, freeing an
uninitialised pointer). This triggered crashes on SPARC/Solaris when
testing in multi-threading, but bizarrely never fired on intel/linux
not even under valgrind.
Bug fixed scramble -x as it was not correctly setting the RI data
series for some tests.
A mishmash of changes for CRAM v3.0 (not yet the default output
format). Sorry it's munged together.

The 'b' and 'q' feature types have been implemented, allowing multiple
bases and qualities to be stored as a single feature. This greatly
speeds up scramble -x mode (referenceless) while also reducing the
file size.  Portions of this snuck in with the earlier commits; an
artifact of merging bit by bit with io_lib.

Added support for the BYTE_ARRAY_LEN encoder. It already existed in
decoder, but it is optimal for some cases to use as encoder instead of
the usual BYTE_ARRAY_STOP.

Bug fix to the external decoders. If there is an attempt to decode 0
bytes then it no longer matters if the block does not exist. (This
comes about when faced with silly CIGAR strings like "0S".) Changed
scramble to no longer output empty blocks.

Cram_dump: this now outputs the compression types used for each
external block as g(gzip), b(bzip2), l(lzma), r(rANS0) and R(rANS1).
Minor gcc warnings fixups.

Small speed increase - only attempt to compress the CORE block when it
is of sufficient size to make it worth while.
Added support for compressed SAM headers.

This involved removing the various #ifdefs for different types of
padding and instead coding it up with a variable that changes
depending on CRAM v2.1 or v3.0. (Cramtools doesn't allow for a second
padding block in 2.1, although technically it could fit within the
existing spec.)

Bug in the EOF block writing code. It was outputting
0xff 0xff 0xff 0xff 0xff instead of 0xff 0xff 0xff 0xff 0x0f for -1.
This caused the CRC to be invalid if you checksummed the raw data
instead of re-encoding it as Scramble does.
Final part of Staden io_lib's commit r3686.

The BD:Z and BI:Z were already split up in this git version, but I
failed to initialise the BI:Z block.
Initialise refs_t ->ref_id in refs_load_fai().  This has no bearing on
Samtools/htslib, but this function is used within Gap5 and calling it
in this order gave arise to crashes unless this initialisation code is
here, so it is a good belt and braces approach.

In htslib for now this function is still static and unexported, which
is appropriate until we see a good use case claiming otherwise.
Added CRC32 to the blocks and containers.
Fixed the file format detection code so it handles CRAM v3.0 magic
number as a valid CRAM file.
Changed the hts_open code to support "C" as a format mode as well as
"c".  Both are CRAM, but "C" is version 3.0.  This is a temporary hack
while CRAM V3.0 is still undergoing work, but we will need a more
formal way of indicating output version numbers in the future.
Equally so the CRAM code itself for specifying versions also needs
improvements, to avoid global variables. Test_view has a -3 option to
use "C" instead of "c".

Also changed the compression formats (this is something else we cannot
yet expose to test_view) so that the rANS codec is used by default in
V3.0.