TUGboat, Volume 43 (2022), No. 3
Mapping to individual characters in expl3
Joseph Wright
It is natural to think that separating a word up into
individual characters is an easy operation. It turns
out that for the computer this isn’t really the case.
If we look at a system that natively understands Unicode (like XƎTEX or LuaTEX), most of the time one‘character’ is stored as one codepoint. A codepointis a single character entity for a Unicode programme.
For example, if we take the input ‘café’ in a file saved
as UTF-8, it is made up of four codepoints:
U+0063 (LATIN SMALL LETTER C)
U+0061 (LATIN SMALL LETTER A)
U+0066 (LATIN SMALL LETTER F)
U+00E9 (LATIN SMALL LETTER E WITH ACUTE)
So we could, in XƎTEX/LuaTEX, use a simple
mapping to grab one character at a time from thisword and do stuff with it. However, that’s not always
the case. Take for example ‘Spın̈al Tap’: the dotless-i
is a single codepoint, but there is no codepoint foran umlauted-n. Instead, that is represented by two
codepoints: a normal n and a combining umlaut.
As a user, it’s clear that we’d want to get a single‘character’ here. So there’s clearly more work to do.
Luckily, this is not just a TEX problem andthe Unicode Consortium have thought about it forus. They provide a data file and rules that describe
how to divide input into graphemes: ‘user perceived
characters’. So ‘all’ that is needed is to examine theinput using these rules, and to divide it up so that‘characters’ stay together.
For pdfTEX, there’s an additional wrinkle: ituses bytes, not codepoints, and so if we use a naïveTEX mapping, we would divide up any codepoint outside the ASCII range into separate bytes: not good.
Luckily, the nature of codepoints is predictable: allthat is needed is to examine the first byte and collect
the right number of further bytes to re-combine intoa valid codepoint.
This work isn’t something the average end user
wants to do. Luckily, they don’t have to as the LATEX
team have worked on this and created a suitable setof expl3 functions to do it: \text_map_function:nNand \text_map_inline:nn.

279
For example, we can do (mapping each characterto printing itself in parentheses):\ExplSyntaxOn\text_map_inline:nn{ Spın̈al ~ Tap } { (#1) }\ExplSyntaxOffand get(S)(p)(ı)(n̈)(a)(l)( )(T)(a)(p)in any TEX engine — assuming we are set up to print
the characters, of course. Getting the right fonts is
an independent issue from parsing the input.
Taking a more ‘serious’ example (and one thatis going to use LuaTEX for font reasons), we mightwant to map over Bangla text: I’m going to useন্দ্রকিন্দ্র as my example. Our \text_map_inline:nnfunction divides up the characters correctly:(ন্)(দ্)(র)(কি)(ন্)(দ্)(র).
In contrast, the generic expl3 token-list function \tl_map_inline:nn gives:(ন)(্)(দ)(্)(র)(ক)(ি)(ন)(্)(দ)(্)(র),
which is a very odd result. In short: Unicode characters are neither bytes nor tokens.
(If you want to try that demo yourself, you’ll
need a document preamble that can work properlywith Bangla: I’m using\usepackage{fontspec}\newfontface\harfbangla{NotoSansBengali-VariableFont_wdth,wght.ttf}[Renderer = HarfBuzz, Script = Bengali]then using \harfbangla in a brace pair around mydemonstrations. Finding a monospaced font thatproperly renders Bangla is . . . tricky.)
So, as you can see, mapping to ‘real’ text is easy
with expl3: you just need to know that the tools are
there.
 Joseph WrightNorthampton, United Kingdom
joseph dot wright (at)morningstar2.co.uk

Mapping to individual characters in expl3
doi.org/10.47397/tb/43-3/tb135wright-grapheme