From: jon@hook.corp.mot.com (Jon Whalen)
Subject: Character Sets: Summary (Part I)
Organization: MOTOROLA 
Date:  3 Oct 92 21:38:58 GMT
Keywords: character, code, font, glyph

To All:

 A week ago I posted an article asking for information on the
 definition of several character code sets, including ISO 10646/Unicode.
 Also I asked for a review of the definitions I gave for the terms "grapheme",
 "glyph", "font", "character set" and "font encoding". And thirdly, I asked
 for information pertaining to various encodings of non-ascii character
 sets into ascii. 

 I received many, many very detailed and thoughtful replies to my posting
 and I'd like to thank all those who repsonded, in particular:

 unicode-inc@hq.m4.metaphor.com  - Steven A. Greenfield, Unicode Office Manager
 andras@gatekeeper.calera.com    - Andra1s Kornai
 mark@kirk.retix.com             - Mark Hoy
 <00V0HORVATH@LEO.BSUVC.BSU.EDU> - Vera Horvath 
 cuong@haydn.stanford.edu        - Cuong T. Nguyen
 ath@linkoping.trab.se           - Anders Thulin       
 glenn@metis.com                 - Glenn Adams
 KRAFT@PENNDRLS.UPENN.EDU        - Bob Kraft
 Harald.Alvestrand@delab.sintef.no - Harald Tveit Alvestrand
 lou@vax.ox.ac.uk                - Lou Burnard 
 dominic@british-national-corpus.oxford.ac.uk - Dominic Dunlop 
 enag@ifi.uio.no                 - Erik Naggum
 connolly@memstvx1.memst.edu     - Leo Connolly
 churchh@emx.cc.utexas.edu       - Henry Churchyard
 keld@login.dkuug.dk             - Keld J|rn Simonsen


In the sections below, I've tried to summarize my requests and the responses
I received. In general, I tried to paraphrase, merge and otherwise 
condense the replies without distorting the meaning. In many cases, however, 
I didn't think I would do justice to the original and so I have included 
whole sections of text from various replies, with (I hope) appropriate 
accreditation. I apologize in advance for the length and for any typos
or omissions I may have made.

Some of what follows will be familiar to those of you who have read the
follow-up articles, especially on comp.fonts. However, in fairness to those
in other news groups or on various mailing lists, I have included information
from both public responses and those e-mailed directly to me.

Due to the number and length of responses, I have split this summary into
two pieces. Parts I covers questions 1) and 2) in my original posting.
Part II includes repsonses to question 3).

--jon


*Jon S. Whalen                                        Phone: (708) 576-0166*
*Staff Engineer, Motorola, Inc.                         Fax: (708) 576-0892*
*Corporate, Computer & Communications R&D                                  *
*Internet:  jon@hook.corp.mot.com / Compuserve: 76665,3043 / AOL:JonSWhalen*

[DISCLAIMER: Please note, I do _not_ speak for Motorola, Inc. Nor do I have any 
 commercial, financial or other interest in or affiliation with any other 
 company or organization mentioned herein.]
 
------------------------------------------------------------------------------

Summary of Responses (Part I)

   I asked:

>To All,
>
>I.  I'm looking for pointers to documents describing the encodings used 
>by several different character sets. In particular, I'm looking for the
>encodings for:
>
>1. EBCDIC
>2. ISO Latin-1 (ISO 8859-1)
>3. ISO Latin-2 (ISO 8859-2)
>4. Unicode [more than just the ascii part]
>5. ISO 10646

   Replies:

I. Standards References and Sources

A. ISO

   Unicode/ISO 10646
      
To obtain more information on Unicode or to order their printed material
and/or diskettes contact:

   Steven A. Greenfield
   Unicode Office Manager
   1965 Charleston Road
   Mountain View, CA 94043
   Tel. 415-966-4189
   Fax. 415-966-1637
   
In the information packet I received from him, the current prices on the
materials (in the US and all prices in US dollars) are:

   Volume One of the Unicode 1.0 Character Encoding Standard
      Paperback book                            -- $32.95
      Implementers Book with Mapping Diskette   -- $37.50
      
   Volume Two of the Unicode 1.0 Character Encoding Standard
      Paperback book                            -- $29.95
      Implementers Book with Mapping Diskette   -- $33.95
     
   Unicode Implementers Workshop #3 Proceedingd -- $15.00 
   
   Diskettes:
      Cross Reference Diskette for Volume 1     --  $5.00
      Cross Reference Diskette for Volume 2     --  $5.00
      Character Name List                       --  $5.00
      
    Shipping and handling are additional.

For other material...

   Glenn Adams writes:
   
>ISO standard documents may purchased in the US from Omnicom, (703) 281-1135.
>Unicode, and by extension ISO10646, is available in two volumes at your
>local computer bookstore (published by Addison-Wesley).  As for EBCDIC,
>see "EBCDIC Bibliographic Character Sets - Sources and Uses: A Brief Report,"
>Journal of Library Automation, Vol. 12/4, December 1979.  This report
>documents the extended EBCDIC used by many bibliographic services.  Also
>see IBM Publication 1403-03, Order #GA 24-3073-7, p. 36-37.

   and in a response to pfk@rz.uni-jena.de (Frank Klemm) Glenn says:

>No fonts exist yet that cover all of Unicode.  The Association for Font
>Information and Interchange (AFII) is in the process of creating a font
>which will be used to print the ISO10646 code charts.  I understand this
>font *may* become available when complete.  The Unicode Consortium is
>in the process of creating online mapping files from other character sets
>to Unicode, by means of which one could obtain a partial mapping in the
>reverse direction.  I believe they intend to make these available by FTP
>once they are verified for accuracy.


   Dominic Dunlop writes:
   
>For [ISO Latin-1] and [ISO 10646], use anonymous ftp from dkuug.dk,
>   subdirectory isp.
>---
>Dominic

   The files there are locally produced I believe, not authorized by ISO
   or Unicode, so I'm not sure what the restrictions on their use might
   be. At a glance the 8859-1 character set seems to be complete, but the
   10646 file only contains a subset of the 10646/Unicode characters.

B. CCITT
   
   Among the relevant CCITT standards are:
   
   T.50 - International Alphabet No. 5
   
      This is the same as ISO 646, the 7-bit invariant character set.
      
   T.51 - Coded Character Sets for Telematic Services
   
      This describes the use of the ISO 2022 escape sequence mechanism
      for specifying a desired code set in terms of control groups C0 and 
      C1 and graphic groups G0, G1, G2, and G3. It also contains several 
      tables in Annex A, sec. A.2 through A.5 which define a set of identifiers
      for the Latin (extended) character set, digits, and various symbols.
      Annex B contains a table cross-referencing the various T-series
      recommendations defining alternative C0, C1, G0, G1, G2, G3 groups.
      
      [The tables are quite usable. The description of ISO 2022 is
        impenetrable.]
      
   T.61 - Character Repertoire and Coded Character Sets for the
      International Teletex Service
      
      This recommendation is specifically applicable to teletex.
      It contains material from T.50, T.51, ISO 646, ISO 2022, ISO 6429 
      and ISO 6937, as well as new material and emmendations. It also, 
      includes a set of identifiers for the extended Latin character set
      (which appears to be the same as in T.51.)
      
   X.408 - Message Handling Systems: Encoded Information Type
      Conversion Rules
      
      "This Recommendation specifies the algorithms the MHS [Message
      Handling System] uses when converting between different types of
      encoded information.
      
      This recommendation offers conversion rules and tables for converting
      among Telex (F.1), IA5 (T.50), Teletex (T.60, T.61 and F.200), Group3 
      fax, Group4 class1 fax, Videotex, and mixed mode.
   
D. Internet RFC
 
   Keld Simonsen writes:
     
>I have written the RFC 1345 with information on the above,
>although only about 2000 characters of UNICODE and ISO 10646
>is covered. It is available by ftp via the normal RFC archives.

Harald Tveit Alvestrand adds:

>
>1345  Simonsen, K.  Character Mnemonics & Character Sets.  1992 June; 103 p.
>      (Format: TXT=249738 bytes)
>
><To avoid using up bandwidth: Erik Naggum does not like this document.
>He says he is working on an alternative document that he likes, but has not
>yet (as far as I know) published it.>

   Erik Naggum writes:
   
>I think it would be fair to include the reasons I don't like it, rather
>than imply that it's just a matter of "liking" a format or not: the
>tables are generally unreadable, they're full of errors, and they can't
>be debugged by inspection, so you can't even find the errors without
>doing very time-consuming comparisons with the original material.  I
>started doing this time-consuming work, but found it easier to go to the
>original sources myself, and start over.  That's why I don't "like" RFC
>1345.  All other users of it will also have to do this painstaking
>checking all over, because the RFC's content can't be trusted.  (The
>author has announced a new, improved edition, but again, we have to
>trust it, since its correctness and accuracy is extremely hard to
>inspect, even for character sets you know well by heart.)

   [By way of comparison, Erik included a sample encoding from RFC 1345
    and a sample of his encoding for the code set ISO 8859-1]

>Slightly more verbose :-), but also easily debuggable: it's parsable,
>and the character number is explicitly identified with the character
>name.  (Missing characters and resulting "shifts" account for about 400
>errors in RFC 1345.)  The names are drawn from ISO/IEC DIS 10646-1.2,
>and will be updated to include the official names from the published
>standard.  The fact that these are actually delimited strings also makes
>it possible to construct conversion tables by name lookup, instead of
>typing in (with errors) a pre-composed conversion table.
>
>If this peaks your interest, please drop me a line.
>
>Best regards,
></Erik>

   [There was further discussion on both sides, not included here]

Internet RFC's are available by anonymous ftp from nic.ddn.mil and many
of other places. There may also be e-mail based ftp servers available, check
with your local network administrator.
 
II. Definitions

   I asked:
   
>II. Are the following statements correct? If not, how far off the mark are they?

   And the replies were:
   
1. Grapheme

   I wrote:

>  1. A grapheme is logical unit of meaning used in the written representation
>     of a language.

   Glenn Adams replies:

>In my recent text, "Introduction to Unicode," Proceedings of the Third
>Unicode Implementor's Workshop, Ausust 1992, I defined it as follows:
>
>  "A minimally distinctive unit of writing in the context of a particular
>  writing system."

   Leo Connolly replies:

>No way.  A grapheme is a minimal unit of writing.  Example: <a> and
><b>, or even capital <A> and lower case <a>, are different graphemes
>because they are distinct and non-interchangeable in the English
>writing system.  (Graphemes are written between the angled brackets
><>.)  On the other hand, Roman _a_ (with curve extending up from the
>right side and over the top) and "manuscript" _a_ (o-like) are
>"allographs" of a single grapheme because they are interchangeable in
>our writing system; the choice depends on the writer or font designer.
>
>Graphemes have *NOTHING WHATSOEVER* to do with meaning.

   I totally munged it, sorry. I understand now that it is dangerous even
   to speak of a grapheme except in the context of a particular language
   and writing system.
   
2. Glyph

I wrote:

>  2. A glyph is a pictorial representation of a grapheme.

Glenn Adams replies:

>I had:
>
>  "An abstract form which represents one or more glyph images, and which is
>   used to visually depict encoded character data."
>
>A "glyph image" is then:
>
>  "The actual, concrete image of a glyph representation having been
>   rasterized or otherwise imaged onto some display surface."
>
>There is some distinction between "glyph" as "allograph" and "glyph image"
>as a particular visual instance of a glyph, since one may take the same
>allograph and display it under different transformations, linear and
>non-linear.

   I should not have tried to link glyphs and graphemes, it appears that
   there's no direct connection, in general.
   
3. Font

I wrote:

>  3. A font is an ordered set of glyphs.
>     (i.e. a mapping between numeric (integer) values and glyphs.)

Glenn Adams replies:

>OK.  I had:
>
>  "A collection of glyphs used for the visual depiction of character data.
>   A font is often associated with a set of parameters, e.g., size, posture,
>   weight, serifness, etc., which, when set to particular values, generate
>   a collection of imagable glyphs."

4. Character Set

I wrote:

>  4. A character set is an ordered set of graphemes.
>     (i.e. a mapping between numeric (integer) values and graphemes.)

   Glenn Adams replies:

>Unfortunately, it doesn't quite work this way.  The reason being that
>what constitutes a grapheme can only be determined in the context of
>a particular writing system, i.e., a particular language, a particular
>set of orthographic rules, and one or more collections of symbols (scripts)
>used to depict writing.
>
>Because character sets (generally) exclude language and orthographic
>differences, the characters therein end up encoding the elements of
>scripts independently of their status as graphemes.  Also, because
>character sets in the past have often been confused with the enumeration
>of the elements of a font, allographic symbols and arbitrary glyphic
>elements have been encoded along with the more independent graphemic
>symbols.
>
>The best working definition for a character set is simply as "a collection
>of elements used to organize, control, or represent information."  A character
>is then simply "an element of a character set," i.e., an element which
>"organizes, controls, or represents information."  Trying to read anything
>else into "character," at least as currently implement by existing character
>set standards, will soon lead you to great conceptual difficulties.
>
>However, to go out on a conceptual limb, I would suggest that under ideal
>circumstances a character represents "an abstract form which may solely
>or jointly be used to represent the symbol of a script, which, in turn, may
>represent one or more graphemes in the context of implicit or explicit
>assumptions about the language and orthographic rules which apply to the
>character encoding."  For example, the ASCII character 0x41 represents
>the LATIN CAPITAL LETTER A symbol, which, in an American English text,
>represents the graphemes <a> and <majuscule>.  How's that for clarity?

5. Font Encoding

   I wrote:

>  5. A font encoding is a mapping between a character set and a font.

   Glenn Adams replies:

>No.  A font encoding is a mapping from glyph codes, to the glyphs of a font.
>The mapping between character set(s) and font(s), is, in general, much more
>complex.  I offer the following excerpt from a message I sent recently
>on the newsgroup "comp.fonts."
>
>  In general, this model will not work for universal character sets which
>  cover large numbers of writing systems, e.g., Unicode.  It must be replaced
>  by a more sophisticated model:
>
>    string of unicode characters ->
>    rendering & layout engine    ->
>    positioned glyphs
>
>  In general, the rendering and layout engine must be able to do the following:
>
>    - map 1 character to N (possibly discontiguous) glyphs*
>    - map 1 character to 1 of N glyphs depending on context
>    - map N characters to 1 glyph*
>    - map M characters to N (possibly discontiguous) glyphs*
>    - reorder resulting glyphs according to bidirectional display requirements
>    - compute attachment points for glyphs which attach to other glyphs
>    - deform glyphs for performing justifiction which doesn't use whitespace
>      justification
>
>    *may also require context sensitivity

   I goofed. I believe what I was trying to describe was not a font encoding,
   but (at least a part of) a rendering. 

Reference definitions:

   CCITT Recommendation T.51:

character - A member of a set of elements used for the organization, control
   or representation of data.
   
coded character set; code - A set of unambiguous rules that establishes a 
   character set and the one-to-one relationship between the characters of
   the set and their bit combinations.

   The Unicode Standard 1.0, Volume 1:
   
character - (1) The smallest component of written language that has semantic
   value. Character refers to the abstract idea, rather than a specific 
   shape (see also glyph), though in code tables some form of visual
   representation is essential for the reader's understanding. (2) The basic
   unit of encoding for the Unicode character encoding, 16 bits of information.
   (3) Synonym for "code element." (4) The English name for the ideographic
   written elements of Chinese origin.
   
character encoding - association of a unique number with each character in
   a set of characters. The distinction between characters and glyphs is
   not absolutely clear in all cases, and so most large character
   encodings also encode some set of glyphs.
   
code set - A character encoding; this term is widely used by programmers.
  
glyph - (1) The actual shape (bit pattern, outline) of a character image.
   For example, an italic "a" and a roman "a" are two different glyphs
   representing the same underlying character. In this strict sense, any
   two images which differ in shape constitute different glyphs. In this
   usage, "glyph" is a synonym for "character image" or simply "image."
   (2) A kind of idealized surface form derived from some combination of
   underlying characters in some specific context, rather than an actual
   character image. In this broad usage, two images would constitute the
   same glyph whenever they have essentially the same topology (as in
   oblique "a" and roman "a"), but different glyphs when one is written
   with a hooked top and the other without (as in italic "a" and roman
   "a"). In this usage, "glyph" is a synonym for "glyph type," where
   glyph is defined as in sense 1.
   
[ This is the end of Part 1 -- Part 2 follows in a separate posting ]
--
*Jon S. Whalen                                        Phone: (708) 576-0166*
*Staff Engineer, Motorola, Inc.                         Fax: (708) 576-0892*
*Corporate, Computer & Communications R&D                                  *
*Internet:  jon@hook.corp.mot.com / Compuserve: 76665,3043 / AOL:JonSWhalen*