From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.fonts
Subject: Re: Why Unicode is 16-bit
Date: 13 Sep 92 15:35:29 GMT
Organization: MIT Artificial Intelligence Laboratory


  In article <markg.16.716257747@county.lmt.mn.org> markg@county.lmt.mn.org (Mark Gilbert) writes:

  >Unicode does encode Han (Chinese) characters in the range 0x4000 to 0x8BFF.
  >This is space for 19456 characters.

In the process of merging with ISO10646, this range has changed to
0x4E00 - 0x9FFF, or 20,992 codepoints, of which 20,902 codepoints are
currently assigned to Ideographic characters.  In addition, a CJK
Compatibility Zone is present from 0xF900 - 0xFAFF, or 512 codepoints,
of which 302 codepoints are currently assigned.  That makes a total
of 21,204 ideographic characters currently assigned to codepoints in
Unicode, i.e., the ISO10646 Base Multilingual Plane (BMP).  Volume 2
of the Unicode Standard, Version 1.0, is now available in bookstores,
and contains all Ideographic character assignments along with mappings
to other popular CJK character sets.

  >Volume 1 of the Unicode Standard book describes a plan to merge Unicode
  >with a 32 bit ISO standard (ISO 10646).  This forward to the book was
  >written in late 1991, and I don't know what progress has been made on
  >this merger.  32 bits can encode over 4 billion characters, and that ought
  >to be enough to encode all languages living and dead.

The merger is complete.  However, Unicode did not change into a 32-bit
standard.  Instead, it became the base multilingual plane (BMP) of ISO10646.
The BMP represents the first 65,536 codepoints in the 10646 encoding space
(which actually is only 31 bits, the upper half of the 32-bit space is not
used).  No characters are currently defined in ISO10646 outside of the BMP.

10646 encodings come in two flavors, UCS-4 and UCS-2, for Universal Character
Set 4-octets, and 2-octets respectively.  Although this terminology suggests
that these encodings are interpreted as a sequence of bytes, they are not.
Rather, UCS-4 and UCS-2 encodings must be interpreted as 32-bit and 16-bit
unsigned integer values respectively, i.e., unsigned long and unsigned short,
to use C language terminology.  This means that byte order issues across
different machine architectures must be dealt with properly.  Also, 10646
provides escape sequences (ala ISO2022) to invoke both 10646 UCS4 and UCS2
forms.

The UCS-2 encoding form is provided in order to create implementations
that implement only the base multilingual plane, i.e., Unicode.  UCS-2
and Unicode are identical [note that some changes have been made from the
published Unicode Vol 1 in order to make this possible, see Unicode Vol 2
for more information on these changes].

I believe that many major vendors, such as Apple and Microsoft,
will only implement UCS-2 or Unicode.  It is currently felt from studying
the requirements of all scripts (past and present), that Unicode will
be able to incorporate them all [though it may have to employ compositional
techniques in order to do this - this is entirely consistent with current
Unicode coding principles].  This will allow *all* languages to be
represented in the 16-bit UCS-2 or Unicode encoding space.

I should note that 10646 has not been published in its final form.  It
is currently being edited for final publication and isn't expected to
be completed until sometime next year.  I advise implementors to obtain
the Unicode standard volumes, which are not only much cheaper than the 10646
standard will be, but also contain much additional information regarding
implemnetation and usage that is excluded from the ISO10646 standard.

Glenn Adams