From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.fonts
Subject: Re: Unicode and Fonts
Date: 27 Aug 92 20:47:06 GMT
Organization: MIT Artificial Intelligence Laboratory


  In article <1992Aug27.015805.9116@sq.sq.com> lee@sq.sq.com (Liam R. E. Quin) writes:

  >It looks to me as if Unicode is being thought of as a panacea solution that
  >it isn't trying to be.

Unicode solves some very serious problems; but it certainly isn't a panacea.
Anyone who represents it as such simply doesn't know much about it.  What does
it solve?

1. It can stop the proliferation of partial, incompatible character sets.
   Building software that works transparently with any language cannot cope
   with the multitude of character set encodings that predate Unicode.  As
   an example, in a recent Internet Draft, "X.400 use of extended character
   sets," Harald Alvestrand lists 138 languages (out of a total of over 5000
   in the World).  He finds that by using 13 different character sets, one
   can cover 46 of these languages; he has no idea about the remaining 92
   languages which he lists.  In contrast, Unicode covers 133+ of these
   languages, the remaining five being Amharic, Burmese, Khmer, Mongolian,
   and Sinhalese - the five scripts which cover these languages are planned
   for inclusion in Unicode in the near future [actually, most writing in
   Mongolian is with Cyrillic so it can be represented with Unicode; however,
   Mongolia is gradually returning to its use of the Mongolian script.]

   Furthermore, by representing all modern scripts, and by employing combining
   characters (non-spacing marks), Unicode allows *all* 5,000+ languages to be
   represented without adding additional characters [excepting those needed for
   the new scripts currently being formulated for inclusion].

2. Unicode encodes the symbols of a script only once, irrespective of its use
   in particular writing systems; so, the Han script is encoded once even
   though it is used by many writing systems, e.g., Chinese, Japanese, Korean,
   and Vietnamese (chu nom & chu han).  The consequence of this is very
   important for unifying the treatment of the same symbol across different
   uses.  Of over 120,000 Han symbols encoded by existing Asian character sets,
   only ~21,000 unique symbols were present and required encoding in Unicode.
   Having a unified encoding of the same script allows software to be
   developed around supporting particular scripts, rather than particular
   languages.  Of the 5,000+ languages in the world, fewer than 40 scripts
   need to be encoded and processed.  [Language dependent processing such as
   collation, spell checking, etc., will still be required; however, many
   important processes, such as display, can largely be built around script
   specific behavior rather than writing system specific behavior.]

3. Most large character set encodings such as JIS, GB, KSC, CNS, CCCII, BIG5,
   etc., employ variable width encodings which intersperse single and multiple
   byte encodings in an 8-bit byte (octet) stream.  Because of this use of
   variable width encodings, software cannot randomly index into a sequence
   of character encodings and expect to land at a character boundary.  Unicode
   solves this by employing a fixed width encoding model, like ASCII.  The
   following is a well defined program:

   typedef unsigned short UniChar;

   UniChar *
     nthchar ( UniChar *ustr, int n )
   {
     return & ustr [ n ];
   }

   On the other hand, the following will produce random results since it
   may point into the middle of a character encoding:

   typedef unsigned char MultiByteChar;

   MultiByteChar *
     nthchar ( MultiByteChar *mbstr, int n )
   {
     return & mbstr [ n ];
   }

   The only way to implement the above in a well defined fashion is as follows:

   MultiByteChar *
     nthchar ( MultiByteChar *mbstr, int n )
   {
     while ( n-- )
       mbstr += mblen ( mbstr, MB_CUR_MAX );
     return mbstr;
   }
   
   The problem with the above is that it converts indexing from O(1) to O(n).
   That can be awfully expensive.

4. The next major problem Unicode solves is uniqueness of encoding.  In
   Unicode, a single code element (encoding value) is *always* interpreted
   as the same character; in contrast, wide character implementations such
   as EUC explicitly make wchar_t encodings opaque, allowing, in fact, for the
   same encoding value to represent different characters depending on the
   setting of the locale.  Indeed, EUC doesn't even define wchar_t to be
   a consistent width:  it is implementation dependent.  I'd like to know
   how I'm going to create a spell checker if I can't even determine if some
   character encoding value is an <A>, <B>, or <C>.  [Can you imagine having
   is_latin_capital_a(), is_latin_capital_b(), ..., for determining the
   identity of every wchar_t?]

   In contrast, Unicode adopts a unique and transparent encoding which
   guarantees that the encoding value 0x0621 is an ARABIC LETTER HAMZA and
   not something else.  This is the model which made ASCII so useful; i.e.,
   that, when so desired, one could optimize on the basis that a particular
   code element value was always a particular character [actually this hasn't
   turned out true with ISO646 (the ISO version of ASCII) since various ASCII
   code element values have been reinterpreted depending on national usage,
   sigh....]  Such an optimization is impossible with wchar_t since it is
   (supposed to be) opaque, and even implementation dependent in width and
   interpretation.

5. Another problem Unicode addresses is bidirectional display behavior found
   in the use of Arabic and Hebrew scripts.  By using a logical representation
   order and by specifying an algorithm for translating this order into
   physical display order, Unicode solves the primary problem of display in
   the context of these two scripts.

6. Unicode solves the problem of representing the composition of symbols
   from smaller units by providing a collection of NON-SPACING MARKS, i.e.,
   floating diacritics, combining marks, etc.  The solution adopted in
   Unicode specifies an unambiguous and uniform treatment of such marks
   in such a way as to allow implementors and users to compose larger
   symbols from sequences of a BASE MARK character and one or more *following*
   NON-SPACING MARK characters.

7. Unicode provides a plain text model which represents the primary content
   of text irrespective of its final form.  This means that simple programs
   like VI, EMACS, and GREP can operate on Unicode text files treated simply
   as an array of Unicode characters.  [Of course these programs will need
   enhancments to support the full linguistic coverage provided by Unicode.]
   
Unicode solves the above problems and really no more.  I can't see how anyone
could claim that the above represents a panacea.  In fact, Unicode will cause
a variety of problems for existing systems.  But these problems are not
inherent in Unicode per se; instead they are inherent in the problem domain
that Unicode is trying to solve, namely, the representation of all languages
using efficient, uniform, unique encodings.  Any solution whatsoever that
has these design criteria will arrive at a similar result.

  >Here is a very simple model of how things work in most systems now:
  >
  > characters encoding ->
  > names               ->
  > font encoding
  >

In general, this model will not work for Unicode.  It must be replaced
by a more sophisticated model:

  string of unicode characters ->
  rendering & layout engine    ->
  positioned glyphs

In general, the rendering and layout engine must be able to do the following:

  - map 1 character to N (possibly discontiguous) glyphs*
  - map 1 character to 1 of N glyphs depending on context
  - map N characters to 1 glyph*
  - map M characters to N (possibly discontiguous) glyphs*
  - reorder resulting glyphs according to bidirectional display requirements
  - compute attachment points for glyphs which attach to other glyphs
  - deform glyphs for performing justifiction which doesn't use whitespace
    justification

  *may also require context sensitivity

  >It wouldn't take much to include ligatures, and I for one should like very
  >much to be able to preserve ct, st, tall-s and fi in certain hostorical
  >documents that I type.

I think you might underestimate the numbers of ligatures needed.  In the
Ruq`ah style of Arabic script, a style of medium-level complexity, there
are 14,674 possible ligatures if one includes points; if you also add
vowel diacritics [considering that they form part of a composite ligature
glyph], then the number of possible ligatures goes up to 3,161,928.  A more
complex style of Arabic such as Nastaliq used to write Urdu would have more
still.  [See T. F. Mitchell, "Writing Arabic : A Practical Introduction to
the Ruq'ah Script," Oxford University Press, 1953, and M. Majidi, "Geschichte
und Entwicklung der arabisch-persischen Schrift," Hamburg: Helmut Buske
Verlag, 1986, for more information.]

This problem is not only present in Arabic, but also in Korean Hangul
where each syllable is really a ligature; in Indic scripts, where sequences
of consonants form conjunct ligatures, and which combine with a variety of
superscript and subscript marks for vowels, nasalization, medial consonants,
etc.; and, in various writing systems based on the Latin script where one
can consider a combination of a base letter and one or more diacritics as
a ligature (I define a ligature as a single glyph which represents two or
more glyphs, where the single glyph may or may not look like a combination
of the individual glyphs).

Unicode cannot possibly directly encode all of these ligatures (combinations
of glyphs).  Nor should it do so.  Should a user be able to specify when or
when not to display a particular ligature or variant form, such as you
mention, <ct>, <st>, tall-s, <fi>, etc.?  Sure, but that is the job of rich
text, not of the primary text representation which underlies the rich text.
One can intersperse escape sequences in an Unicode string just like ASCII (or
ISO2022) strings.  In fact, ISO6429 gives us a way to explicitly specify
ligatures or variants.  This is simply a form of rich text (one that I am
not arguing to be used).  Many other forms of rich text are possible.  SGML
is another one.  Indeed, for representing historical or legal documents,
one might want to  represent the actual image of writing, or signatures,
etc.  Should this be represented in Unicode?  No way.  A character set
should encode only the unadorned, basic content of text; overall
appearance, formatting, stylistic variations, and secondary content should
not be part of the character set.  This is the domain for rich text.
Now all we need to do is define some really useful rich text interchange
standards [of course there are already some out there, e.g., ISO8613, Office
Document Architecture (ODA), ISO8879 Standard Generalized Markup Language
(SGML), etc.].  

  >but I cannot help but wonder if it wouldn't be better if Unicode were not
  >something it isn't.

Unicode has already been designed, published, and incorporated into an
international standard (ISO10646 UCS2).  It is not "something it isn't."
It is up to us to come up with systems that use it.  It isn't going to
change into something else, no matter how much we argue over it.

Glenn Adams
Cambridge, Massachusetts