update to unicode 6.2
MORE COMMENT regarding the careful format of the encoded UCD data
Incorporate some decomposition information in ucd table

Oh boy.  This one is quite intricate.  We have two bytes free in
the 8-byte entries for information about characters, so use one of
them to indicate if the character has a decomposition, and if so of
what kind it is.  Adapt the ucd.lisp tools-for-build code to
parse and preserve that information.

However, this causes there to be more than 256 distinct possible
classes of character known to the system: not a problem in principle,
but Teemu Kalvas' implementation of the double indirection depended on
having a one-byte index.  But since Unicode characters are limited to
21 bits, with a careful packing scheme we can in fact steal 3 more bits
for the index, at the cost of needing to do an extra memory reference
and some arithmetic to reconstruct the index.  (In the process, change
the endianness of the ucd.dat filesystem representation, because it's
easier that way).

But wait, there's more.  Before, there were only two kinds of
lower-case characters: those whose upper-case transformation
lowercase back to the original character, and those where there is
no round-trip.  (The former are cl:lower-case-p, the latter aren't).
This gave rise to straightforward implementations of lower-case-p
and friends; in the new world, where there are multiple different
kinds of lower-case characters (with various decomposition classes)
we need to adjust the implementations, still fairly straightforward,
of lower-case-p and related functions.

The extra information provided in the ucd table by this commit
is largely useless on its own; the next step is to incorporate
the actual decomposition data.  Stay tuned.
delete now-unused code from ucd.dat
beginnings of decomposition

Store enough information in output from ucd.lisp to be able to actually
decompose individual characters.  Include proof-of-concept implementation
of decomposition, not hooked into anything yet.
work-in-progress towards full normalization support
handle Hangul syllable decomposition

Entries for the codepoint range (#xac00 -- #xd7a3) have 1 for
their decomposition-info, a decomposition length of 2 or 3, but
a zero decomposition index (the decomposition is handled
algorithmically instead).
comment on LSTRING implementation
improve normalize-string

* now works on non-simple strings;
* more likely to be correct under #!-sb-unicode
fix test for Blocked condition in canonical normalization

Would most likely otherwise fail in Jamo with combining characters in
between.
apply recursive decomposition in DECOMPOSE-STRING

We should really precompute the result of the recursion during the build;
working on getting tests up and running so that we can check whether
we've done that correctly.
add a comment about one-basing the character tables
first cut at testing unicode normalization

Parts 0 and 1 from Unicode NormalizationTest.txt, fully tested for
NFD and NFKD.
finish handling NormalizationTest test vectors

NFC/NFKC still not hooked in, but otherwise complete.
better UCD treatment of characters not allocated by Unicode

fixes lp#1178038 (reported by Ken Harris)
actually run Part3 of Unicode Normalization tests
implement primary and canonical composition, and hence NFC/NFKC

Read in the non-algorithmically-specified composition exclusions from
Unicode's CompositionExclusions.txt file, and generate a hash table
using the concatenated 42 bits of code points.  This is a bit of a
sucky hash-table key, particularly on 32-bit platforms; I have a plan
to reduce the key to 24 bits (using some auxiliary information in ucd)
but the advantage of getting this try in is...

... hook in NFC/NFKC into normalization tests, and check that tests
pass.
NEWS entries for Unicode normalization work