include master curve for frequency distribution of characters
covered by dictionary for use when encoding permutations.
add more stats output when building dictionaries.
nice to see permutation tables packing smaller as a result of
using a curve.
work on permutation coding.
Tried using probability of higher or lower than last rank to
reduce entropy -- but makes things worse than doing it naively
-- at least now that we are using a curve model and filtering out
used symbols.
whoops! forgot to commit this file previously.
implemented permutation decode function (but untested)
permutation encoding and decoding now reciprocal and work.
verification of nodes in gen_stats.c now works for non-duplicate
permutation records.
confirmed that compression performance is as it should be.
don't verify in gen_stats now that we know it works.
disabled up/down filtering again after proving conclusively that
it makes matters worse.
factor out permutation sub-list picking code.
prepare to make permutation handling code more general so that we
can also compress unicode frequency statistics.
permutation_decode now uses common sub-list service function.
permutation_encode given extra arguments in preparation for delta
coding of permutations.
wrote code to estimate savings that might arise from delta coding
permutations.  Returns look very limited.
improved reporting of delta savings potential.
purged delta coding of permutations since savings are <5%.
first cut at simplifying frequency distributions by reordering tail
of distribution, but performance is terrible.
replaced permutation simplification with the much easier option
of only encoding the first half of a permutation, since the remainder
will have fairly flat probabilities.
but something else is broken, because compression performance has been
broken.
backed out sublist handling in decode function that was broken.
only write the 1st half of a permutation, since the 2nd half will
have fairly flat statistics.  Reduces dictionary sizes by about
10% for small onces.
removed last traces of delta coding permutations.
removed superfluous function prototype.