From: glenn@wheat-chex.ai.mit.edu (Glenn A. Adams)
Newsgroups: comp.fonts,comp.protocols.iso,comp.os.os2.programmer,comp.os.ms-windows.programmer.misc,comp.os.misc,sci.lang
Subject: Re: Character Sets
Keywords: character-set, charset, font
Date: 27 Sep 92 14:05:56 GMT
Article-I.D.: life.28418
Followup-To: comp.fonts
Organization: MIT Artificial Intelligence Laboratory

>  Date: Fri, 25 Sep 1992 21:45:11 +0200
>  From:         Jon Whalen <jon@hook.corp.mot.com>
>  Subject:      Character Sets
>
>  1. EBCDIC
>  2. ISO Latin-1 (ISO 8859-1)
>  3. ISO Latin-2 (ISO 8859-2)
>  4. Unicode [more than just the ascii part]
>  5. ISO 10646

ISO standard documents may purchased in the US from Omnicom, (703) 281-1135.
Unicode, and by extension ISO10646, is available in two volumes at your
local computer bookstore (published by Addison-Wesley).  As for EBCDIC,
see "EBCDIC Bibliographic Character Sets - Sources and Uses: A Brief Report,"
Journal of Library Automation, Vol. 12/4, December 1979.  This report
documents the extended EBCDIC used by many bibliographic services.  Also
see IBM Publication 1403-03, Order #GA 24-3073-7, p. 36-37.

>  1. A grapheme is logical unit of meaning used in the written representation
>     of a language.

In my recent text, "Introduction to Unicode," Proceedings of the Third
Unicode Implementor's Workshop, Ausust 1992, I defined it as follows:

  "A minimally distinctive unit of writing in the context of a particular
  writing system."

>  2. A glyph is a pictorial representation of a grapheme.

I had:

  "An abstract form which represents one or more glyph images, and which is
   used to visually depict encoded character data."

A "glyph image" is then:

  "The actual, concrete image of a glyph representation having been
   rasterized or otherwise imaged onto some display surface."

There is some distinction between "glyph" as "allograph" and "glyph image"
as a particular visual instance of a glyph, since one may take the same
allograph and display it under different transformations, linear and
non-linear.

>  3. A font is an ordered set of glyphs.
>     (i.e. a mapping between numeric (integer) values and glyphs.)

OK.  I had:

  "A collection of glyphs used for the visual depiction of character data.
   A font is often associated with a set of parameters, e.g., size, posture,
   weight, serifness, etc., which, when set to particular values, generate
   a collection of imagable glyphs."

>  4. A character set is an ordered set of graphemes.
>     (i.e. a mapping between numeric (integer) values and graphemes.)

Unfortunately, it doesn't quite work this way.  The reason being that
what constitutes a grapheme can only be determined in the context of
a particular writing system, i.e., a particular language, a particular
set of orthographic rules, and one or more collections of symbols (scripts)
used to depict writing.

Because character sets (generally) exclude language and orthographic
differences, the characters therein end up encoding the elements of
scripts independently of their status as graphemes.  Also, because
character sets in the past have often been confused with the enumeration
of the elements of a font, allographic symbols and arbitrary glyphic
elements have been encoded along with the more independent graphemic
symbols.

The best working definition for a character set is simply as "a collection
of elements used to organize, control, or represent information."  A character
is then simply "an element of a character set," i.e., an element which
"organizes, controls, or represents information."  Trying to read anything
else into "character," at least as currently implement by existing character
set standards, will soon lead you to great conceptual difficulties.

However, to go out on a conceptual limb, I would suggest that under ideal
circumstances a character represents "an abstract form which may solely
or jointly be used to represent the symbol of a script, which, in turn, may
represent one or more graphemes in the context of implicit or explicit
assumptions about the language and orthographic rules which apply to the
character encoding."  For example, the ASCII character 0x41 represents
the LATIN CAPITAL LETTER A symbol, which, in an American English text,
represents the graphemes <a> and <majuscule>.  How's that for clarity?

>  5. A font encoding is a mapping between a character set and a font.

No.  A font encoding is a mapping from glyph codes, to the glyphs of a font.
The mapping between character set(s) and font(s), is, in general, much more
complex.  I offer the following excerpt from a message I sent recently
on the newsgroup "comp.fonts."

  In general, this model will not work for universal character sets which
  cover large numbers of writing systems, e.g., Unicode.  It must be replaced
  by a more sophisticated model:

    string of unicode characters ->
    rendering & layout engine    ->
    positioned glyphs

  In general, the rendering and layout engine must be able to do the following:

    - map 1 character to N (possibly discontiguous) glyphs*
    - map 1 character to 1 of N glyphs depending on context
    - map N characters to 1 glyph*
    - map M characters to N (possibly discontiguous) glyphs*
    - reorder resulting glyphs according to bidirectional display requirements
    - compute attachment points for glyphs which attach to other glyphs
    - deform glyphs for performing justifiction which doesn't use whitespace
      justification

    *may also require context sensitivity

>  III. I'm looking for pointers to documentation on standard and ad-hoc
>  mappings between 7-bit ascii and 8-bit charsets.

The Unicode standard contains a large number of mapping tables to/from other
character sets and Unicode, e.g., all of the ISO8859 series, PC code pages,
Apple character sets, etc.

Glenn Adams
Cambridge, Massachusetts
Newsgroups: comp.fonts