expl3 and LATEX3
Will Robertson & Frank Mittelbach
And the LATEX3 Project

July 29, 2014, Portland

Outline

LATEX3
expl3
Case changing

Outline

LATEX3
expl3
Case changing

LATEX Project team members
Largely chronologically:— Frank Mittelbach,— Thomas Lotze,— Rainer Schöpf,— Morten Høgholm,— Chris Rowley,— David Carlisle,— Javier Bezos,— Michael Downes († 2003),— Will Robertson,— Johannes Braams,— Joseph Wright, and— Robin Fairbairns,— Bruno Le Floch— Alan Jeffrey,— Denys Duchier,LATEX3 stats

What is LATEX3?

— You know what LATEX 2ε is…(we assume)— So LATEX3 is the next version of LATEX, right?
— Not so fast.

LATEX 2ε status— LATEX 2ε must remain backwards compatible, warts and all.
— Many things that many people would change!
— Default document design:Some [many?] questionable/controversial aesthetics …
— Programming:
Not enough hooks, missing or unclean interfaces, separation of‘layers’, default font encodings, …

LATEX 2ε status— LATEX 2ε must remain backwards compatible, warts and all.
— Many things that many people would change!
— Default document design:Some [many?] questionable/controversial aesthetics …
— Programming:
Not enough hooks, missing or unclean interfaces, separation of‘layers’, default font encodings, …
Explosion of packages doing similar things but each slightly
differently and only parts of it…

LATEX 2ε improvements?

— We can/do ﬁx certain bugs in LATEX 2ε — but not aspects thatchange layout or bugs that we know people worked around.
— More drastic changes can occur in fixltx2e, but that doesn’treally work or solve the issue (see ‘explosion of packages’
earlier).
— But even seemingly ‘harmless’ changes have consequences.

LATEX 2ε improvements?

— We can/do ﬁx certain bugs in LATEX 2ε — but not aspects thatchange layout or bugs that we know people worked around.
— More drastic changes can occur in fixltx2e, but that doesn’treally work or solve the issue (see ‘explosion of packages’
earlier).
— But even seemingly ‘harmless’ changes have consequences.
Conclusion: In short, it just doesn’t work.

What is LATEX3?
— So we’re not going to get rid of latex the format, and its
interface is not going to change.
— That means whatever LATEX3 is, it will be an alternative.
— The package concept means some LATEX3 ideas can be layeredon top of LATEX 2ε .
— Not everything can be layered (e.g. galley).
— In time, we will have a latex3 format.

What is LATEX3?
— So we’re not going to get rid of latex the format, and its
interface is not going to change.
— That means whatever LATEX3 is, it will be an alternative.
— The package concept means some LATEX3 ideas can be layeredon top of LATEX 2ε .
— Not everything can be layered (e.g. galley).
— In time, we will have a latex3 format.
N.B. LATEX3 ̸= expl3

Outline

LATEX3expl3
Case changing

What is expl3?

— An interface to TEX programming, stabilised in the last ﬁve or
so years.
— (Invented 1992.)— It forms the programming/coding layer for LATEX3 but can beused independently:▶
▶

for package writing on top of LATEX 2ε ,for coding in other TEX formats; e.g., plain TEX, ConTEXt.

What is expl3?

Why not Lua?
— The ﬁrst versions of expl3 appeared around the same time as
Lua itself (1993).
— expl3 predates LuaTEX by some 20 years.
— expl3 supports pdfTEX, XETEX, and LuaTEX, consistently.
— Also note that Lua doesn’t always help.

What is expl3?

Why not Lua?
— The ﬁrst versions of expl3 appeared around the same time as
Lua itself (1993).
— expl3 predates LuaTEX by some 20 years.
— expl3 supports pdfTEX, XETEX, and LuaTEX, consistently.
— Also note that Lua doesn’t always help.
And how would we use JSBox?

expl3 in LATEX 2εThe goal is to make it easier to write LATEX packages:— We eat our own dog food with siunitx, fontspec, etc. (thishas formed the basis for iteration and solidiﬁcation).
— More comprehensive than etoolbox &c.

expl3 in LATEX 2εThe goal is to make it easier to write LATEX packages:— We eat our own dog food with siunitx, fontspec, etc. (thishas formed the basis for iteration and solidiﬁcation).
— More comprehensive than etoolbox &c.
All you plain users now in luck.
— expl3 now loadable in plain TEX and even ConTEXt.
— This was done specially for ‘generic’ packages; speciﬁcally,
Heiko Oberdiek asked us to provide this functionality tominimise variants of his packages.

expl3 is a success
acro
hobby
chemmacrosclassics
conteqctex
endiagram
enotezexsheets
lt3graph
newlfmfnpct
GS1
hobetekantlipsum
lualatex-mathmedia9pkgloader
substances
withargs
xecjk
xpatch, regexpatch
xpeek
xpinjinzhnumberzxjatype

Interface for creating (classes of) acronyms
Hobby’s algorithm in PGF/TiKZ for drawing optimally smooth curves.
Typesetting in the ﬁeld of chemistry.
Traditional-style citations for the classics.
Continued (in)equalities in mathematics.
A collection of macro packages and document classes for Chinese typesetting.
Draw potential energy curve diagrams.
Support for end-notes.
Question sheets and exams with metadata.
A graph data structure.
The venerable class for memos and letters.
Interaction between footnotes and punctuation.
Barcodes and so forth.
Beamer theme for the Univ. of Hohenheim.
Generate sentences in Kant’s style.
Extended support for mathematics in LuaLATEX.
Multimedia inclusion for Adobe Reader.
Managing the options and loading order of other packages.
Lists of chemicals, etc., in a document.
Ephemeral macro use.
Support for CJK documents in XELATEX.
Patch command deﬁnitions.
Commands that peek ahead in the input stream.
Automatically add pinyin to Chinese characters
Typeset Chinese representations of numbers
Standards-conforming typesetting of Japanese for XELATEX.

expl3 is a successacrohobby
chemmacrosclassics
conteqctex
endiagram
enotezexsheets
lt3graph
newlfmfnpctGS1
hobetekantlipsum
lualatex-mathmedia9pkgloader
substances
withargs
xecjk
xpatch, regexpatch
xpeek
xpinjinzhnumberzxjatype
copyeditting

Interface for creating (classes of) acronyms
Hobby’s algorithm in PGF/TiKZ for drawing optimally smooth curves.
Typesetting in the ﬁeld of chemistry.
Traditional-style citations for the classics.
Continued (in)equalities in mathematics.
A collection of macro packages and document classes for Chinese typesetting.
Draw potential energy curve diagrams.
Support for end-notes.
Question sheets and exams with metadata.
A graph data structure.
The venerable class for memos and letters.
Interaction between footnotes and punctuation.
Barcodes and so forth.
Beamer theme for the Univ. of Hohenheim.
Generate sentences in Kant’s style.
Extended support for mathematics in LuaLATEX.
Multimedia inclusion for Adobe Reader.
Managing the options and loading order of other packages.
Lists of chemicals, etc., in a document.
Ephemeral macro use.
Support for CJK documents in XELATEX.
Patch command deﬁnitions.
Commands that peek ahead in the input stream.
Automatically add pinyin to Chinese characters
Typeset Chinese representations of numbers
Standards-conforming typesetting of Japanese for XELATEX.
New!

What’s new in the last six months?

— Joseph wrote l3build, which Frank covered yesterday.
— (Already mentioned that expl3 now loads on plain.)— Joseph and Bruno implemented expandable case switching.
— Will played around with something and Frank complained
about it (auxiliary data).

Outline

LATEX3
expl3
Case changing

Case changing

1. There is more to case changing than meets the eye:▶

Uppercase, lowercase
Titlecase (with language-dependent rules)▶ Case folding▶

2. Simple \uppercase and \lowercase are not sufficient!
▶
▶

Can have one-to-many mappings (ß → SS).
Can have many-to-one mappings (i, ı → I but also i → İ)

3. Unicode provides data, but is not providing a solution.

Case changing in regular TEX
TEX provides \uppercase and \lowercase:\uppercase{%\def\mytitle{Some normal text}%}\mytitle→ SOME NORMAL TEXT
The characters are not uppercased until the stomach.
I.e., case changing is not expandable.
This is the basis for \MakeUppercase in LATEX 2ε , which has extra
LICR-related code.

Case changing in LATEX 2εFrom source2e:
These commands have some nasty features, such as
uppercasing mathematics, environment names, labels, etc.
A much better long-term solution is to use all-caps fonts,
but these aren’t generally available.** A problem for fontspec?

Case changing in LATEX 2εFrom source2e:
These commands have some nasty features, such as
uppercasing mathematics, environment names, labels, etc.
A much better long-term solution is to use all-caps fonts,
but these aren’t generally available.** A problem for fontspec?
For expl3, we’re not yet tackling this problem either.
The case-changing is intended to operate on ‘characters’ in token
lists without discrimination.

What else are \uppercase & \lowercase used for?
expl3 has long had \tl_to_(upper/lower)case:n and we needed
to deprecate them!
We need to distinguish three main features:
1. Text manipulation in section titles, running headers, &c.
2. Normalizing (folding) text for sorting or ﬁlename searching etc.
3. Doing tricks with TEX programming.
Only one of these relates to typesetting!
Case changing for ‘real’ text input is a hard problem; not yet
addressed.

Subsection 1
Case changing for programming

Case ‘folding’We’ll cover programming ﬁrst because it’s simplest. Quoting
unicode.org:
Case folding is primarily used for caseless comparisonof text, such as identiﬁers in a computer program, rather
than actual text transformation.
Case folding in Unicode is based on the lowercase
mapping, but includes additional changes to the source text
to help make it language-insensitive and consistent.
As a result, case-folded text should be used solely forinternal processing and generally should not be stored ordisplayed to the end user.

Case folding examples

ASCII:\str_fold_case:n { ABCdef } → abcdef
Greek sigma variants:\str_fold_case:n { σςΣ } → σσσ
Deprecated ligature glyphs:\str_fold_case:n { ﬁ ﬆ } → fi stImplementation detail
Can’t blindly compare for the 1000s of characters in Unicode.
From l3unicode-data.def:

Subsection 2
Case changing for typesetting

Expandable case changing

Currently ONLY catering for plain Unicode text
(i.e., more work is needed.)\tl_set:Nx \g_my_title_tl{ \tl_upper_case:n {Some~ normal~ text} }\g_my_title_tl→ SOME NORMAL TEXT

Expandable case changing

Braces ‘hide’ content:\tl_set:Nx \g_my_title_tl{ \tl_upper_case:n {Some~ {normal}~ text} }\g_my_title_tl→ SOME normal TEXTMultilingual in XELATEX/ LuaLATEX\tl_upper_case:n { åéîøдα } → ÅÉÎØДΑ\tl_lower_case:n { ὭƐ } → ὥɛLanguage support:\tl_upper_case:n { Ragıp Hulûsi Özdem }→ RAGIP HULÛSIÖZDEM
\tl_upper_case:nn {tr} { Ragıp Hulûsi Özdem }→ RAGIP HULÛSİ ÖZDEM

Mixed caseTowards automatic sentence formatting

Note this is not intended to iterate over words in a sentence.
Only the very ﬁrst ‘character’ (besides exceptions such as quotes) isuppercased.
— \tl_mixed_case:n {frank} →— \tl_mixed_case:n {``frank''} →— \tl_mixed_case:nn {ne} {ijsje} →— \tl_mixed_case:n {THIS IS AN UPPERCASE TITLE} →Mixed caseTowards automatic sentence formatting

Note this is not intended to iterate over words in a sentence.
Only the very ﬁrst ‘character’ (besides exceptions such as quotes) isuppercased.
— \tl_mixed_case:n {frank} → Frank— \tl_mixed_case:n {``frank''} →— \tl_mixed_case:nn {ne} {ijsje} →— \tl_mixed_case:n {THIS IS AN UPPERCASE TITLE} →Mixed caseTowards automatic sentence formatting

Note this is not intended to iterate over words in a sentence.
Only the very ﬁrst ‘character’ (besides exceptions such as quotes) isuppercased.
— \tl_mixed_case:n {frank} → Frank— \tl_mixed_case:n {``frank''} → “Frank”— \tl_mixed_case:nn {ne} {ijsje} →— \tl_mixed_case:n {THIS IS AN UPPERCASE TITLE} →Mixed caseTowards automatic sentence formatting

Note this is not intended to iterate over words in a sentence.
Only the very ﬁrst ‘character’ (besides exceptions such as quotes) isuppercased.
— \tl_mixed_case:n {frank} → Frank— \tl_mixed_case:n {``frank''} → “Frank”— \tl_mixed_case:nn {ne} {ijsje} → IJsje— \tl_mixed_case:n {THIS IS AN UPPERCASE TITLE} →Mixed caseTowards automatic sentence formatting

Note this is not intended to iterate over words in a sentence.
Only the very ﬁrst ‘character’ (besides exceptions such as quotes) isuppercased.
— \tl_mixed_case:n {frank} → Frank— \tl_mixed_case:n {``frank''} → “Frank”— \tl_mixed_case:nn {ne} {ijsje} → IJsje— \tl_mixed_case:n {THIS IS AN UPPERCASE TITLE} →This is an uppercase title

Extending mixed case to title case
Not a ‘token list’ function.
— THIS IS AN UPPERCASE TITLE→ This is an Uppercase Title
Lots of edge cases! Style guides differ:— Variable exception list:a an and as at but by en for if in of on or the to v via vs— Modern words like ‘iPhone’ and ‘eyeTV’— Always capitalise ﬁrst and last words regardless of other rules
Anyway, not impossible, but part of some future ‘text processing’
module.

Subsection 3
Using weird tokens

TEX programming tricks\begingroup\lccode`\~=`\_\lowercase{\endgroup\def~{\sb}}\mathcode`\_="8000\relax\catcode`\_=12\relax
x_2 \quad $x_2$→ x_2x2TEX programming tricks\begingroup\catcode`P=12\catcode`T=12\lowercase{\def\x{\def\rem@pt##1.##2PT{##1\ifnum##2>\z@.##2\fi}}}\expandafter\endgroup\x\def\strip@pt{\expandafter\rem@pt\the}Anything better with expl3?
— Potential wrapper around \lowercase.
— Not entirely decided upon yet.
\char_set_catcode_active:N \*\tl_transform:nn{ \char_transform:NN \* \_ }{ \cs_set:Npn * { \sb } }Of course, for something like this we also have candidate function\char_set_active:Npn.

Anything better with expl3?
\tl_transform:nn{\char_set_catcode_other:N \P\char_set_catcode_other:N \T\char_transform:NN \P \p\char_transform:NN \T \t}{\cs_set:Npn \__dim_to_decimal:w ##1.##2 PT{ ##1 \int_compare:nT { ##2 > 0 } { .##2 } }}\__dim_to_decimal:w used to deﬁne \dim_to_decimal:n .