TUGboat, Volume 0 (2001), No. 0

preliminary draft, September 15, 1999 0:05

Notes on Oldenburg etex/latex3/context
MeetingDavid Carlisle
1Present1etex Peter Breitenlohner, Philip Taylor, BerndRaichle, Jiri Zlatuska, Karel Skoupy.
latex3 Frank Mittelbach, Rainer Schöpf, Chris
Rowley, Matthias Clasen, David Carlisle.
context Hans Hagen, Taco Hoekwater.
2Categories

To start the discussion, Peter (initially) suggested
some basic ‘categories’ of problem that could be discussed. The following discussion did not always follow this categorisation but it proved a useful starting
point (in fact this list was modified during the discussion in th e first session, and ended up something
like this:
Syntax/Expansion/Programming Extensions
to the macro programming language, not directly affecting typesetting.
Line-breaking/hmodePage-breaking/vmode
Grid Typesetting
Characters and Fonts
Alignments
Math
Character Attributes (e.g., Color)
Translation (e.g., I/O)
List Manipulations Manipulations on token or
node lists.
3Expansion Control\undefhcsnamei Undefine hcsnamei. Like\let\xxx\@undefined but without relying on\@undefined being undefined. Some discussion
over whether this was really any improvement
over the \let form (given that you need toknow in practice that \undef has not been redefined) or whether it would be worth it if it did
not also remove hcsnamei from the hash table(which would be hard).
Real Register A data type that has the same behavior/arithmetic as hdimeni but without theneed to supply units. (This would save themany uses of \strip@pt in latex where units
are added just to do arithmetic, and then needto be removed.
Boolean Datatype Some question as to what if
anything was wanted here. Boolean variables,or boolean expressions, or. . . Also any extension would need to fit with current \if\fi nesting behavior.
\ifinsidebox Similar to \ifinner but different.
(See Hans’ email).
Local/Global Assignments Some mechanism to
specify that within the current scope global assignments are ‘global’ to nested groups but local
to the current group.2
4\expandedhgeneral texti Expandable command
returning the full expansion of the tokens inhgeneral texti.
\expandlaterhnumberi One level expand the nthtoken ahead. (\expandafter = \expandlater\tw@)preliminary draft, September 15, 1999 0:05

1. Extended hyphenation patterns withweighted hyphenation points.
2. Extended hyphenation algorithm thattries first with patterns designed to split
on compound word boundaries, then if
needed a second pass with a finer set ofpatterns.
Second version has possible advantages in keeping the current hyphenation file format. Not
clear what exactly is needed.

The following variants were discussed with nodefinite conclusion (?)

1 not everyone present all the timeLine Breaking (hmode)

Full typography in hmode A switch to allow
hboxes to be constructed with all the featuresthat currently are only active in outer hmode.
(Language nodes etc).
Discretionaries after - Switch to turn off automatic insertion of discretionaries after (ligatures ending in) -.
Hyphen desirability levels Two basic proposals.

This was more or less discussed by email before themeeting. The following new commands were agreed :\expandfromtokenhnumberi This would lookahead to token  (as in delimited argumentmatching) and then move forward (or back
if negative) hnumberi tokens and expand theresulting token one level.

1001

5Page Building (vmode)

Need for a vertical analogue of \discretionary
identified. (Probably just make \discretionary
2 Rainer sketched a possible interface on the board, which
didn’t make it to my notes — Rainer?

preliminary draft, September 15, 1999 0:05

1002

preliminary draft, September 15, 1999 0:05

work in vmode.) This would seem to solve several
problems with current lack of control over top-ofpage behavior.
Agreed new commands:\forcebreakpenaltylimit If set to ≥ −10000 thisis is ignored, but if set to < −10000 is taken as
the threshold at which penalties force a break.
\nthmarkhnumberihnumberihnumberi

TUGboat, Volume 0 (2001), No. 0

done in the context system) or whether a more general multiple token to multiple token system would
be needed, but that leads down the road to implementing regexp replace (or OTP) and it was notclear if that fitted the etex framework.
An alternative to having an explicit map prefixcommand would be to have a way of automatically
applying the map to input characters (but see later
discussion on encoding support).

\nummarkshnumberihnumberi
8 Mathematics\nummarkshmark classihbox numberi
returns
the number of marks of that class in the speciMatthias Clasen3 reported on an investigation offied box.
possible extensions that he had recently done with\nthmarkhmark classihmark numberihbox numberi Michael Downes. (See document Matthias posted.)
returns the nth mark of the specified class.
Briefly the main points were:(Generalizes \firstmark).
mathstyle The providing of an integer valuedThis pair of commands was thought better
variable that reveals and controls the current
than the original suggestion of an \allmarks
math mode (would work in a way similar tocommand returning a list of all marks.
\interactionmode). This would require theprovision of a prefix form for \overand friends.
Other items to be considered:\overwould still work but would be docu\outputpenalty10000 Look at removing themented as not setting the mathstyle variable
penalty node in the \outputpenalty10000
to the expected value if this infix form is used.
case.
\ifcramped Test for cramped style, together withholdinginserts \holdinginserts=2 (or different
switch to force entry to cramped style.
name for compatibility reasons) Hold but alsokerning
Between more kinds of math atoms.
copy inserts (ie a combination of current beSpacing table Access to the table of spaces inhavior for 0 and ≥ 1). Alternatively just use
serted between the math atoms.
\holdinginserts=1 but give access to individual inserts in some way similar to \nthmark.
math classes Extend the number of these to allow more varied automatic spacing possibilities.
buildpagehook Insert a token register at theAlso primitive to ask the math class of an atom.
point where ‘build page’ is called. The (ef(This would save bm taking the \meaning andfective) default contents of the register would
then parsing the information out of the hex
be \buildpage, a new primitive that actuallystring returned.)builds the page. cf \output/\shipout.
subformulas Avoid boxing subformulas.
6 Node Handlingglyph positions Remove dependence on weird
Perennial discussion about lack of \last* and \un*glyph positioning in the font, like radical hangfor most node types, and problems with revealing
ing below baseline, so math fonts could be usedmachine dependence and lack of register type correwith other systems.
sponding to some nodes, e.g., rules.
radicalextensions Right end for radical? Or
Agree to look at \removelastnode.
(harder) Both ends extending together.
7 Parsing General Texts
Horizontal extensibles Some discussion of whether
this would need an extension of tfm format.
There was some discussion of generalizing theOutcome seemed to be that it would not, just\uppercase mapping of character tokens, although
use existing tfm data structure, the TeX primany decision would have to wait to see the final
itive would know whether it was constructing
outcome of the proposals for symbolic charactera horizontal or vertical extension.
handling (see below)
General idea is \maphmap nameihgeneral texti.
overloading of font dimens Agreed to look at reThus \uppercasewould be \map\uccode(orducing this.
perhaps a special csname to refer to the huppercase mapi. 3 Matthias has since experimented with someSome discussion of whether a token-token mapchange files to implement some of these ideas, seewould be enough (especially for the kind of parsing
ftp://peano.mathematik.uni-freiburg.de/pub/etex
preliminary draft, September 15, 1999 0:05

preliminary draft, September 15, 1999 0:05

TUGboat, Volume 0 (2001), No. 0

preliminary draft, September 15, 1999 0:05

1003

After discussing several strategies for solving theproblem that anything non expandable breaks ligature and kerning, including the possibility of allowing assignments in the inner loop (finally thoughtto dangerous) the following proposal (from Bernd,initially) was agreed to be worthy of further consideration:
Redesign hpack routine and the linebreak algorithm so that kerning and ligature is separated.
First create the hlist with no lig/kern then make asecond pass adding the ligatures and kerns. (This isthe conceptual algorithm, an implementation may
want to interleave these processes when ‘safe’ e.g.,word by word.)
This would mean that fi always produced the filigature.
new primitive would be needed to break ligatures but not break hyphenation process also similarly \nokern.
Some questions on semantics of unhbox in thissetup, whether the list should be “re-ligatured”.
Modified algorithm could further split things
up separating ligatures first then kerning. (Currently tex just does whichever comes first in the fonts
lig/kern table if both are specified for a pair)
Also discussed mechanisms of overriding ligatures for a font e.g., the Portuguese no fl ligature
example. Also, simpler the possibility to turn off all
ligatures in a font, for verbatim type uses.

• total number of hyphens• maximum number of adjacent hyphens• maximum badness of a line
• minimum badness of a line
• total demerits• fit class of worst line
The plan implemented but not released in V2
broke the paragraph onto the current vertical list,
as normal but returned an hlist for further tries.
This means that you need to program the removalof the original paragraph if you need to retry, whichmeans not working on the main vertical list. Somealternatives were considered.
Basic scheme in tex is:construct hlist until reach $$ or \par or end-ofvmode-group. Then
• Modification at end (unskip, add parfillskipetc), leave hmode.
• (A)• look at legal breakpoints & choose optimal path(+/- \looseness)• line break decision (1st pass, hyph, 2nd pass,3rd pass)• (B)• postlinebreak (package, migration, append tovlist)• in vmode or display math (first point tex looksfor token)
Plan 1 copy list at (A) into one global variable (destroy previous contents). This is a private unpackaged hbox. New primitive to ‘unhbox’ thiswhere required.
Plan 2 At (B) call ‘output routine’. Has badness
information from linebreaking routine but notvertical size information. (This plan rejected,
as vertical size information seems vital for intended applications).
Plan 2b Append the new vertical material, but
don’t exercise the page builder.

1011

Fixed parameters e.g., clearance around rules.
Agreed to look at this as well.
under accents Use information that could be
coded in the ‘reverse’ kerning with the skewchar for underaccents. However full accent
solutions, including multiple accents requires
table driven lookup. or otp or . . .
16 families Increase?
tfm If format was extended, possible benefits?
9Inner loop — lig/kern problems

Reconsider reconsider paragraphIf the possibility is given to re-break an hlist, tex
needs to return more information about the resulting paragraph to enable a choice to be programmed.
(It was thought unlikely that one could algorithmically set the paragraph parameters to ‘correct’ anearlier try, but one could at least try several preprogrammed possibilities, and then use the returned
information to choose the ‘best’ outcome.
Possible info that might be useful• Information currently only available in display
math (\predisplaysize)
preliminary draft, September 15, 1999 0:05

Word Grabbing

The following proposal seemed the most plausible(after several other schemes were rejected)
New expandable primitive \grabword
Expands tokens up to the first non expandable
token that is not a hletter-other-spacei. Returns theresulting tokens surrounded by explicit {} tokens
and leaves the ‘terminating’ non expandable token
in place.
Typical use\def\foo#1{<pcdata>#1</pcdata>}\expandafter\foo\grabword user supplied stuff.
preliminary draft, September 15, 1999 0:05

1004

preliminary draft, September 15, 1999 0:05

Note that the ‘word’ that is picked up will be prematurely terminated by all the stuff that currently
breaks ligatures, but if the symbolic character proposals could be implemented these would rarely need
to appear mid word.
12

Named Reference Points

Named reference points on (h or v)boxes.
Like \mark\specialetc puts special named reference point into current (vh)list.
When hbox is packaged reference point node
has a position relative to reference point on box (integer arithmetic).
These positions could be made available similar
to current \wd and friends.
\xxxxxhnumberihnamei
takes a box number and a symbolic name andreturns the position (might need two commands toget x,y coordinates, or one that returns a pair)
When unbox a vertical list the information islost but the reference nodes are still in the list so
their positions in the new list will be made available
when the new list is packaged.
When packaging an outer box, inherit all labelsfrom inner ones.
Possibly (or more likely not) return some some
info about the nesting.
Would the linebreaking algorithm need to add
labels to each line???
13Special \specials

A \special for writing positional information to thedvi file?
A delayed \special.
(cf non-immediate\write).
14

Combining penalties\penalty-500 \penalty10000 currently still allowable break, so user has no way to overrule some automatically inserted penalties. Various algorithms
considered, e.g., some average, max, min, etc. In
practice probably doesn’t matter as long as it preserves ‘finiteness’.
Any algorithm that allows to overwrite existing
penalties that favor a break would be a very big
improvement and should be looked at closely.
15

Symbolic Characters

Some schematics of current and proposed flow of‘characters’ through tex. (Peter can write on theboard faster than I can draw ASCII art on this laptop. The diagrams in figure 1 probably are rubbish,

TUGboat, Volume 0 (2001), No. 0

but I keep them exactly for now. Need replacing by
something a bit more meaningful)4
for each language have subset ≤ 256 SC which
map to hyphenable ASCII codes
SC+language encoding+font encoding ——————–Hyphenation Patterns (always based on 8bit table)
Possibly the mapping from symbolic characters
to ASCII codes may be many to one?
SC + font enc — represented in a font (somehow)
possibility of using Unicode as ‘symbolic names’[This section needs filling in in a lot more detail]16

Parameter matching

Two sets of extensions to parameter matching wereconsidered.
16.1

Alternative delimited argument
tokens\def\foo#1\a#|\b#|\c#2{...}
The argument #1 would be all the tokens up tothe first occurrence at the current brace group levelof \a or \b or \c.
Within the definition the following forms would
be available#1 the tokens of the first argument.
#|1 the tokens delimiting the first argument,
thus one of \a,\b,\c in the example.
#2 the tokens of the second (non-delimited) argument.
#|2 As the second argument is not delimited it
is not clear what this should be. Probably empty,
but could be an error, either would be OK as a behavior.
Note that this extension is upwards compatible
as the #| forms are all error conditions in the current
tex.
16.2

A step towards regexp matching

Currently \def\foo#1#2#3\xx{...} always assigns
the smallest possible arguments to the the first andsecond arguments with the third argument taking up
the maximum number of tokens up to the delimiting
token.
Proposal to allow\def\foo#*1#2#?3\xx{...}#*1 argument may take non or more bracegroups or tokens.

4 Matthias has made some more extensive notes on thissubject

preliminary draft, September 15, 1999 0:05

preliminary draft, September 15, 1999 0:05

TUGboat, Volume 0 (2001), No. 0

input file --> char tokens|vpreliminary draft, September 15, 1999 0:05

1005

1:1 font encoding---> char nodes --> dvi nos
|v

write
want:
symbolic char-------------------------------input file --> char tokens|v

1:1 font encoding---> char nodes --> dvi nos
|v

write

Figure 1: diagrams#+1 argument may take one or more bracegroups or tokens.
#?1 argument may take at most one bracegroups or tokens.

preliminary draft, September 15, 1999 0:05

preliminary draft, September 15, 1999 0:05