LATEX3 News
Issue 10, November 2016

There has been something of a gap since the last
LATEX3 News, but this does not mean that work has
not been going on. The Team have been working on anumber of areas, many of which reflect wider take-upof expl3. There have also been a number of significant
new developments in the LATEX3 “sphere” in the last
two years.

l3build: Testing LATEX packages
Testing has been an important part of the work of theteam since they assumed maintenance of LATEX overtwenty years ago. Various scripts have been used overthat time by the team for testing, but these have until
recently not been set up for wider use.
With the general availability of LuaTEX it is now
possible to be sure that every TEX user has a powerful general scripting language available: Lua. The
team have used this to create a new general testing
system for TEX code, l3build. This is designed to be
used beyond the team, so is now available in TEX Liveand MiKTEX and is fully documented. Testing using
l3build makes use of a normalised version of the .log
file, so can test any aspect of TEX output (e.g., by using \showbox) or its algorithms (by displaying resultsin the .log).
Part of the remit for creating l3build was to enable
the team to work truly cross-platform and to allow
testing using multiple TEX engines (earlier systems
were limited to a single engine, normally ε-TEX). The
new testing system means we are in a much stronger
position to support a variety of engines (see below). It
has also enabled us to give useful feedback on development of the LuaTEX engine.
As well as the core capability in testing, l3build also
provides a “one stop” script for creating release bundles. The script is sufficiently flexible that for many
common LATEX package structures, setting up for creating releases will require only a few lines of configuration.
In addition to the documentation distributed withl3build, the project website [1, publications in 2014]
contains some articles, videos and conference presentations that explain how to use l3build to manage andtest any type of (LATEX) package.

Automating expl3 testing
As well as developing l3build for local use, the teamhave also set up integration testing for expl3 using theTravis-CI system. This means that every commit to theLATEX3 code base now results in a full set of tests being run. This has allowed us to significantly reduce thenumber of occasions where expl3 needs attention before
being released to CTAN.
Automated testing has also enabled us to check that
expl3 updates do not break a number of key third-party
packages which use the programming environment.

Refining expl3
Work continues to improve expl3 both in scope and robustness. Increased use of the programming environment means that code which has to-date been underexplored is being used, and this sometimes requires
changes to the code.
The team have extended formal support in expl3 tocover the engines pTEX and upTEX, principally used
by Japanese TEX users. This has been possible in partdue to the l3build system discussed above. Enginedependent variations between pdfTEX, XETEX, LuaTEX
and (u)pTEX are now well-understood and documented.
As part of this process, the “low-level” part of expl3,
which saves all primitives, now covers essentially all
primitives found in all of these engines.
The code in expl3 is now entirely self-contained, loading no other third-party packages, and can also beloaded as a generic package with plain TEX, etc. Thesechanges make it much easier to diagnose problems andmake expl3 more useful. In particular it can be used asa programming language for generic packages, that thencan run without modifications under different formats!
The team have made a range of small refinements
to both internals and expl3 interfaces. Internal selfconsistency has also been improved, for example removing almost all use of nopar functions. Performanceenhancements to the l3keys part of expl3 are ongoingand should result in significantly faster key setting. Askeyval methods are increasingly widely used in defining
behaviours, this will have an impact on compile times
for end users.

Replacing \lowercase and \uppercase
As discussed in the last LATEX3 News, the team havefor some time been keen to provide new interfaces

LATEX3 News, and the LATEX software, are brought to you by the LATEX3 Project Team; Copyright 2016, all rights reserved.

which do not directly expose (or in some cases evenuse) the TEX primitives \lowercase and \uppercase.
We have now created a series of different interfaces thatprovide support for the different conceptual uses which
may flow from the primitives:• For case changing text, \tl_upper_case:n, \tl_
lower_case:n, \tl_mixed_case:n and related
language-aware functions. These are Unicodecapable and designed for working with text. They
also allow for accents, expansion of stored text andleaving math mode unchanged. At present some of
the interface decisions are not finalised so they aremarked as experimental, but the team expect thecore concept to be stable.
• For case changing programming strings, \str_upper_case:n, \str_lower_case:n and \str_fold_case:n. Again these are Unicode-aware,
but in contrast to the functions for text are notcontext-dependent. They are intended for caselesscomparisons, constructing command names on-thefly and so forth.
• For creating arbitrary character tokens, \char_generate:nn. This is based on the \Ucharcat
primitive introduced by XETEX, but with the ideas
extended to other engines. This function can beused to create almost any reasonable token.
• For defining active characters, \char_set_active_eq:NN and related functions. The concept here isthat active characters should be equivalent to somenamed function, so one does not directly define theactive character.

Extending xparse
After discussions at TUG2015 and some experimentation, the team have added a new argument type, e(“embellishment”), to xparse. This allows arguments
similar to TEX primitive sub- and superscripts to beaccepted. Thus

• Margins between the edges of the galley and theparagraph (for example an indented block);• Cut-out sections running over a fixed number oflines, to support “in place” figures and so forth;• Running or single-paragraph shape.
There are additional elements to consider here, forexample whether lines are the best way to model thelength of shaping, how to handle headings, cut-outs atpage breaks, etc.

Globally optimized pagination of documents
Throughout 2016 Frank Mittelbach has worked on
methods and algorithms for globally optimizing thepagination of documents including those that containfloats. Early research results have been presented atBachoTEX 2016, TUG 2016 in Toronto and later inthe year at DocEng’16, the ACM Symposium on Document Engineering in Vienna. A link to the ACM paper(that allows a download free of charge) can be found on
the project website [1]. The site also holds the speakernotes from Toronto and will host a link to a video ofthe presentation once it becomes available.
The framework developed by Frank is based on theextended functionality provided by LuaTEX, in particular its callback functions that allow interacting with thetypesetting process at various points. The algorithm
that determines the optimal pagination of a given document is implemented in Lua and its results are thenused to direct the formatting done by the TEX engine.
At the current point in time this a working prototype but not yet anywhere near a production-readysystem. However, the work so far shows great potential and Frank is fairly confident that it will eventually
become a generally usable solution.

Looking forward\DeclareDocumentCommand\foo{e{^_}}{\showtokens{"#1"}}\foo^{Hello} world

The LuaTEX engine has recently reached version 1.0.
This may presage a stable LuaTEX and is likely to result in wider use of this engine in production documents.If that happens we expect to implement some of
the more complex functionality (such as complex pagination requirements and models) only for LuaTEX.

will show

References"{Hello}{-NoValue-}".

[1] Links to various publications by members of theLATEX Project Team.
https://www.latex-project.org/publications.

At present, this argument type is experimental: there
are a number of models which may make sense for thisinterface.

A new \parshape model
As part of development of l3galley, Joseph Wright has
proposed a new model for splitting up the functions ofthe \parshape primitive into three logical elements: