From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.fonts
Subject: Re: Unicode and Fonts
Date: 27 Aug 92 18:32:05 GMT
Organization: MIT Artificial Intelligence Laboratory


  In article <5154@dove.nist.gov> gilligan@bldrdoc.gov writes:

  >In this context, I (ignorant of UNICODE beyond what I've read on this
  >newsgroup) would like to ask what UNICODE does about the ``ll'' and
  >``ch'' characters in Spanish that are generally written with two
  >glyphs. That is, ``ll'' is a distinct character in terms of indexing
  >from ``l''. Similarly, ``ch'' is distinct from ``c''. To sort
  >Spanish words alphabetically, one needs to compose ``ch'' and
  >``ll'' into distinct characters (opposite of the hypothetical
  >decomposition of ``ff''). Does UNICODE address this and have a codes
  >for ``ll'' and `ch'', or does it encode glyphs rather than characters
  >in this instance?

The short answer is no.  Unicode does not encode <ll> or <ch> as
distinct units.  The reason for this follows.

1. <ll> and <ch> are atomic orthographic units for the purpose of sorting
   (most) Spanish text.  In Unicode terminology, these are called TEXT
   ELEMENTS.

2. A TEXT ELEMENT is a unit of textual information according to a particular
   text process (e.g., display, search, sort, etc.); a particular writing
   system (i.e., a particular language, one or more scripts, and an ortho-
   graphy which maps the language to the symbols of its scripts); and, the
   textual information itself.  A text element is composed from one or more
   CODE ELEMENTs (units of character encoding), not necessarily contiguous.

3. In general, Unicode (like other character sets) encodes primarily the
   elements of scripts, irrespective of their use in particular writing
   systems.  Consequently, it does not generally encode language specific
   (i.e., writing system specific) units like <ll> and <ch>, which can be
   interpreted as larger scale text elements composed from smaller code
   elements.

4. There are exceptions to the above to the extent that Unicode inherited
   certain digraph encodings from other character sets; e.g., LATIN DZ HACEK,
   etc.  These can be mapped at import time to their componential
   representation if desired.

5. Perhaps the most important reason not to encode larger orthographic units
   is because their interpretation as an atomic unit is context sensitive or
   process sensitive.  For example, if I am writing a routine to do hyphenation
   of English text, and I didn't care about any other processes, I might argue 
   for encoding each English syllable as a separate code; while this would make
   line breaking very efficient, it would require a lot of effort to do
   display, assuming that I didn't have a font that had a separate glyph for
   each syllable.  As an example, this would make it easy to hyphenate
   "cathode" versus "cathouse" which would normally require a dictionary
   lookup.  [Also consider what consequence such a coding would have on
   compression algorithms, which value lower symbol counts.]

6. If you think a minute about English, you realize we have many digraphs and
   trigraphs that people don't think about.  For example, in the words "great"
   "crate", the digraphs "ea" and "a*e" appear, the latter being discontiguous.
   If I am building a text-to-speech system, or a phonemic transliterator,
   then these are indeed atomic text elements; but no one would ever consider
   encoding them as distinct units.

7. The best solution is a compromise representation which allows one to
   reconstruct text elements as needed.  ASCII uses it; so does Unicode.
   Also, if you consider that there are over 5,000 languages in the world,
   the vast majority written using the Latin script, then there is simply 
   an enormous collection of orthographic units one would need to encode
   if you started encoding <ll>, <ch>, and the like.  Just counting all
   such entities would be almost impossible; on the other hand, counting
   the script symbols used to compose these is quite possible.

Glenn Adams
Cambridge, Massachusetts
Newsgroups: comp.fonts