Presentation given at the TUG 2019 Conference, Palo Alto

file version: August 25, 2019 0:02

1

Taming UTF-8 in pdfTEXFrank Mittelbach
Abstract
To understand the concepts in pdfLATEX for processing UTF-8 encoded files it is helpful to first take a lookat the models used by the TEX engine and earlier attempts made by LATEX on top of TEX. The talk provides
a short historical review of that area and explains• how it is possible in a TEX system that only understands 8-bit input to nevertheless interpret andprocess UTF-8 files successfully;• what the obstacles are and how they can be overcome and• what restrictions will remain if one doesn’t switch to a Unicode-aware engine such as LuaTEX or XETEX.
It will finish with an overview about the improvements with respect to UTF-8 handling that will be activatedin LATEX within 2019 and explains how they can already be tested today.
The slides have been retrospectively constructed from the mindmap used during the presentation.
 Frank Mittelbach
Mainz, Germany
https://www.latex-project.org

A short history
lessonNew - upcoming

LaTeX2e solution

RestrictionsTaming UTF-8
in pdfTeXFrank Mittelbach
TUG 2019, Palo Alto
A short history
lession part 2(UTF-8)

Slide #1

Taming UTF-8 in pdfTEX

2Presentation given at the TUG 2019 Conference, Palo Alto

file version: August 25, 2019 0:02

7bit

TeX79

Input „key code“ = Font slot

8bit

Input „key code“ = Font slot

TeX82

8 bit code pages differ country by country

Font slots 129-255 not really used
A short history
lesson\language
TeX 3
Cork font encoding!

Problems
1995

Slide #2

The German word „Größe“

Problems
1995

A „pound“ symbol in italics

Slide #3

Frank Mittelbach

Presentation given at the TUG 2019 Conference, Palo Altomaps toinputenc package

file version: August 25, 2019 0:02

3translated by

LICR (LaTeX Internal
Character Representation)

fontenc package

no dependencies on code pages

LICR surives the roundtrip through .aux .toc,
etc., since only ASCII chars are used andcommands are protected from expansion

LaTeX2e solution
Font encodings are (in theory) welldefined

no dependencies on font slots

Fonts with the same encoding map exactly thesame charactersno tofu!

or notSlide #4

1 byte = ascii

0xxxxxxx

2 bytes

110xxxxx 10xxxxxx

3 bytes

1110xxxx 10xxxxxx 10xxxxxx

Encoding

A short history
lession part 2(UTF-8)...

Approach (in pdfTeX)

FeaturesSlide #5

Taming UTF-8 in pdfTEX

4

file version: August 25, 2019 0:02

Presentation given at the TUG 2019 Conference, Palo Alto

Program reads only bytes

Start byte is made „active“This reads all necessary further bytes

Approach (in pdfTeX)
Determines the Unicode slot
maps toLICR (LaTeX Internal
Character Representation)translated by...

fontenc

we know that alreadyUTF8 characters are only supported if theglyph exists in the loaded fonts

Features

But those again without tofu!

Slide #6

Each UTF-8 document needs \usepackage[utf8]{inputenc}

no \Straßeand also no \label{Überblick}

Multi-Byte UTF-8 can’t be used as partof command names

Multi-Byte UTF-8 has problems in \typeout etc
Restrictions
Problems with \input, \include orgraphic files

In file names only restricted usage
possible --- if at allExample:

Slide #7

Frank Mittelbach

Presentation given at the TUG 2019 Conference, Palo Alto

\input{fürchterlich}\input{Straße}! LaTeX Error: File `f\unhbox \voidb@x \bgroup \let \unhbox \voidb@x \setbox@tempboxa \hbox {u\global \mathchardef \accent@spacefactor\spacefactor }\accent 127 u\egroup \spacefactor \accent@spacefactor rchterlich.tex' notfound.

file version: August 25, 2019 0:02

5InputOT1
Example:! LaTeX Error: File `Stra\OT1\ss e.tex' not found.
(only an error because thefile didn’t exist)! LaTeX Error: File `fürchterlich.tex' not found.
T1! LaTeX Error: File `Stra\T1\ss e.tex' not found.

Slide #8

UTF-8 is the new default
input encodingThis is a pdfTeX restriction which
can’t be realistically overcome

Each UTF-8 document needs\usepackage[utf8]{inputenc}
April 2018
Combining chars, e.g., „a“ + „U+0301“ = á
are not supported

character not in loaded fonts areallowed too!
\label, \cite,\typeout, etc

in file names

Multi-Byte UTF-8
works now in ...
Fall 2019

New - upcoming

but still not in command names

This is a pdfTeX restriction which
can’t be realistically overcome

Space in file names are allowed
without quoting the nameAvailable now for testing
LaTeX-dev format

Slide #9Taming UTF-8 in pdfTEX