BrnoJune LATEX 2ε Encoding Interfaces
Purpose, Concepts, and Open Problems

Frank Mittelbach
Zedernweg 6255128 Mainz
Germany

Contents
1 Overview

22 What’s that?

33 And what’s that?

44 One Encoding Problem

55 Input Encodings

66 Input Encodings: Possible Realisations

7

7 Input Encodings: The LATEX 2ε solution

88 Another Encoding Problem

9

9 Output Encodings

1010 Output Encodings: Possible Realisations

1111 Output Encodings: The LATEX 2ε solution

1212 Internal Encoding

1313 Internal Encoding: Possible Realisations

1414 Internal Encoding: The LATEX 2ε solution

15

15 Encoding Specific Commands: Concepts

161

16 The \lowercase and the \lccode Table

17

17 Example: Using \lowercase on input

18

18 Example: Hyphenation and the \lccode Table

19

19 Example: Hyphenation and the \lccode Table

20

20 Example: Hyphenation in a Paragraph

21

21 Consequences

22

22 The LATEX 2ε Encoding Solution

23

23 Future tasks

2421Overview• Input encodings (keyboard encodings) introduced withLATEX 2ε 1994/12/01• Output encodings (font encodings) introduced with NFSS
• Standard internal representations (internal encoding)• Future tasks

This talk will describe the models used by LATEX 2ε to translate input characters in a source
document via internal representations to glyphs in a font. This is important for everybody
writing in a language other than English and/or using a TeX system that allows 8-bit input.
The talk will cover• input encodings (keyboard encodings) introduced with LATEX 2ε 1994/12/01• output encodings (font encodings) introduced with NFSS
• standard internal representations
Future issues discussed will be a proposal for 8-bit math encoding developed by the LATEX3
Project and a general mechanism for providing “short references” to ease coding tasks such as"a → \"{a} as they are already available in certain language files.
Short refs are clearly part of input encodings: indeed, inputenc could be looked on as making
all 128 upper input chars into short refs.
∗∗∗To set the stage . . .

3

2What’s that?
GrŤáeThat’s the German word “Größe” (height) typeset on a German PC and displayed using T1 fonts.

4

3

And what’s that?
$That’s somebody trying to typeset a pound sign but getting a dollar sign.
∗∗∗Let’s look at the encoding problems in some more detail . . .

5

4One Encoding Problemú←→\243 −−−−−−→ č£←→\243 −−−−−−→ čKey . . . In file (and on screen) . . . Typeset output: oops!
Top line is (IBM) code page cp850.
Second line is ISO-Latin1.
Both with no input or output translation: both with T1 fonts.
Needs
Input encoding: translates what is in the input file into:
Internal (LATEX) representation;
Output encoding: translates to correct code (for the font) in the dvi file.

65Input Encodings• The mapping between keyboard glyphs and character numbers in the .tex source file
• There are up to 256 character numbers available (8-bit)• It is possible that a source file is composed using several input encodings

The input encoding describes the relationship between the 8-bit characters inyour input file and their meaning, i.e. the glyphs you will see if you display thesource file with a browser (if that browser uses the same encoding.

7

6Input Encodings: Possible Realisations• Translate characters using hardwired tables inside TEX the program• Translate characters to a standard representation using external programs• Translate characters inside LATEX by declaring translation tables

The first solution will result in source files which are not portable and whose
non-portability will only be visible by looking at the typeset output.
The second solution produces portable documents if you consider the translated
source file as the major source.
Neither of the two allows to mix input encodings within one document.
Only the third solution will result in fully portable documents which will produce
identical output at different sites. However, it has the disadvantage of taking
up space within the LATEX format and also of needing extra processing time.

87Input Encodings: The LATEX 2ε solution• 7-bit printable ascii is considered to be essentially transparent• Input encodings are declared by name.
• Each named encoding defines the mapping for (a subset of) the character numbers between
128–255 to the internal encoding of LATEX 2ε• Input encodings are declared for a whole document or for parts of it

Exceptions are the standard escape characters for LATEX (since they are notpassed through) and the non-printable ascii characters at the moment.
This means, for example, that encodings like EBCDIC (in any variant are
not supported. This isn’t totally surprising because to be able to support aninput encoding it must at least have the characters that form TEX’s commandlanguage in well-defined positions.

98Another Encoding Problem\textrm{Hung\H{a}rian umlaut}−−−−−−−→
cmr10

Hunga̋rian umlaut\texttt{Hung\H{a}rian umlaut}−−−−−−−−→
cmtt10Hung}arian umlaut

This shows that the encoding of the original TEX fonts (OT1) is unfortunately
not completely fixed.

109Output Encodings• The mapping between glyphs and character numbers in the .dvi file
• There are up to 256 character numbers available for TEX fonts• Usually typesetting involves several fonts, not all with the same output encoding

The output encoding describes the relationship between the 8-bit characters put
into the .dvi by TEX and the glyphs that should be produced for them.

1110Output Encodings: Possible Realisations• Translate characters using hardwired tables inside TEX the program• Use a TEX extension: MLTEX’s \charsubdef, Omega, NTS . . .
• Translate characters inside LATEX by declaring translation tables

The first solution only works for characters written to plain files not to character
numbers written to the .dvi file and thus is not usable.
None of the TEX extensions is generally available. MLTEX doesn’t give a generalsolution. Omega might (?) do — but I haven’t studied it yet.
Only the third solution will right now result in fully portable documents whichwill produce identical output at different sites. However, it has the disadvantageof taking up space within the LATEX format and also of needing extra processing
time.

12

11Output Encodings: The LATEX 2ε solution• Provide a standard internal encoding• Translate characters inside LATEX by declaring translation tables from the internal encoding to any output encodings

So have a look at the internal encoding which will lead us eventually back to
the implementation of output encodings . . .

13

12Internal Encoding• Must mediate between the input and output encodings• Must be 7-bit to ensure system independence while reading and writing external files• Should be independent of the input encoding(s) used
• Should be independent of the font(s) used for typesetting

Internal representation:
Must be 7-bit printing-chars since LATEX uses external files to store information
between runs;
reading and writing to these files must be system-independent.

14

13

Internal Encoding: Possible Realisations• Transparent: the plain TEX & LATEX 2.09 solution• Mediating: the LATEX 2ε solution

The character number of the input character (placed into the source file) is
passed unchanged to the .dvi file; commands always generate the same output
character position, e.g. \it\$ →£ — in other words: essentially no internal
encoding.
This does work as long as input, internal, and output encoding are essentiallythe same, as happened with the early versions of TEX/LATEX with only a limited
number of fonts and only 7-bit input.

15

14

Internal Encoding: The LATEX 2ε solution• 7-bit printable ascii• Encoding specific commands– without arguments, e.g. \textquoteleft– with arguments containing 7-bit printable ascii or further commands, e.g. \"{a} or\^{\i}}7-bit printable ascii is considered to be essentially transparent
Encoding specific commands are commands that change their meaning when
the output encoding changes.
We decided to represent accented characters by commands with arguments a)
to avoid taking up an enormous name space, b) to allow for accented characterswhich are not in the font, and c) to allow for the use of \uppercase an issue
which will be discussed later on.
For the really dedicated I suggest having a look at the actual implementationin ltoutenc.dtx which contains quite complicated code to handle spacing, ligatures and kerns correctly.

16

15

Encoding Specific Commands: Concepts• Commands change their meanings if used with different output encodings• Change happens, when the command gets used not when the output encoding changes• Commands are robust, i.e. they stay unchanged during LATEX’s internal processing, including writing out to external files and reading back inLots to say here . . .
∗∗∗Let us now turn to the problem of changing the case of letters and its relation
to hyphenation within TEX . . .

17

16

The \lowercase and the \lccode TableThe \lowercase command:• Changes character code but keeps category codes (e.g. \active)• Doesn’t act on commands
The \lccode table:• Used by the \lowercase primitive• Used to translate words before applying hyphenation patterns• For hyphenation: table is consulted only at the end of the paragraphLet’s have a look at these issues in detail.

18

17

Example: Using \lowercase on input

Keybord encoding latin2:Ž←→−−−−→ "AE −−−−−−→"AE −−−\lowercase
Latin-1Ž←→−−−→"AE −−−
Latin-1\v{Z} −→ Ž−−−−→ \v{z} −→ ž
\v{Z} −−−\lowercase"AE is the character “ő” (hungarian umlaut over o) in the T1 encoding, theencoding which is used by default to set up the lower and upper case tables.
In other input encodings one might end up with some active character which
isn’t even defined.

19

18

Example: Hyphenation and the \lccode TableMANŒUVRESManœuvresmanœuvres• No hyphenation if any char has \lccode of zero• Incorrect hyphenation for first word if \lccode of “M” isn’t “m”, the one for “Œ” isn’t “œ”,etc.
• Incorrect hyphenation for second word if \lccode of “M” isn’t “m”, the one for “œ” isn’t“œ”, etc.
• Incorrect hyphenation for third word if \lccode of any character used is different fromthe character itself

Now in what encoding are these characters at this point when TEX is trying toapply hyphenation to them?
ANSWER: in the output encoding of the font in which they will be typeset.
They must be, since after hyphenation the char codes are simply written to thedvi file and if they aren’t the char codes that match the position of “Œ”, “œ”,“u”, “v”, “r”, “s”, etc. then you will find garbage on your printed page.
This is a slight oversimplification because TEX works harder by breaking up
ligatures and possibly reinserting new ones etc.
∗∗∗The characters must be in the output encoding of the current font otherwise
one will end with something . . . like this.

20

19

Example: Hyphenation and the \lccode TableMANŒUVRESMan œuvresman œuvres• No hyphenation if any qar has \lccode of zero
• Incorrect hyphenation for first word if \lccode of \M" isn't \m", the one for\Œ" isn't \ œ", etc.
• Incorrect hyphenation for second word if \lccode of \M" isn't \m", the one for\œ" isn't \ œ", etc.
• Incorrect hyphenation for third word if \lccode of any qaracter used is different from the qaracter icelf

This is what happens if the encoding used doesn’t match the font (output)
encoding.
The typewriter comes out correctly because \verb switches back to the typewriter font in the main document encoding.

21

20

Example: Hyphenation in a ParagraphSome text with a \russian{Russian phrase} in the middle.

TEX3 enables• Both parts of the text to be hyphenated with the correct set of hyphenation patterns forthat language
But• Both parts are changed to lowercase using the same \lccode table: the one that is current
at the end of the paragraphThus• Hyphenation will be wrong if output encodings are used which need different \lccodetables

22

21

Consequences• All output encodings used by LATEX 2ε must have the same \lccode table (Cork T1encoding)• Direct use of \uppercase and \lowercase (on the input encoding) is not supported• Instead, LATEX 2ε provides \MakeUppercase and \MakeLowercase which operate on theinternal encoding

Keep in mind that the fixing the \lccode tables restricts font encodings only
slightly. The T1 encoding is far from perfect but it is the agreed on TEX standard.
The alternative would be to make up to 600 assignments each time an encodingchanges (and that doesn’t solve the problem of multiple encodings within aparagraph).
But the \Make* commands will not always do exactly what you want: eg maths
letters, private commands, . . .

23

22

The LATEX 2ε Encoding Solution\usepackage[cp850]{inputenc}

ú

←→

\243

−−−−−−→
cp850

\’{u}

−−−−→
T1

\char250−→ú−−−−−→
OT1{\accent 19 u}−→ú\usepackage[latin1]{inputenc}

£

←→

\243

−−−−−−−→
Latin-1\textsterling

−−−−→
T1

\char191

−→

−−U−→

\Error...

−→

24

£

23

Future tasks• Math font encoding (8-bit)• All caps fonts (new NFSS axis?)• Short references, e.g.
"a → äor-> → \rightarrow25