TUGboat, Volume 0 (2001), No. 0

preliminary draft, September 15, 1999 0:07

A regression test suite for LATEX 2εFrank Mittelbach
Abstract
This paper describes the history of the developmentof a regression test suite for LATEX and its importance for the release of stable and reliable future
distributions of that software. A more detailed description of the concepts and the implementation of
the test suite will be given in [1].
As experience shows that there can’t be enoughtest files in such a suite, we make a plea to the TEX
community to help us in making LATEX distributions
even more reliable by joining a new volunteer group
working on the task of updating and adding to thissuite.
1Introduction

Back in 1992 when the LATEX3 team took over maintenance of LATEX and started to work on the current
LATEX version [3], also known as LATEX 2ε , I had theidea of producing a test environment for LATEX thatwould help us in providing stable and reliable distributions. My idea originated in the trip test for
the TEX program [2], a fixed test file which is run
through TEX, containing code that tries to exercise
as many boundary cases as Don Knuth could thinkof. The output of this run is then compared witha set of files certified by Don to contain the correct
information. Only if a new implementation of theTEX program produces the same output (with welldefined minor deviations in certain places) is it allowed1 to be called TEX. The idea behind this isto ensure that TEX behaves identically on all implementations and the trip test was the measureproving this.
With LATEX the idea was not to ensure that it isidentical on all platforms — this is automatically thecase if the standard installation is obtained and theinstallation procedures are applied — but to ensurethat that new releases of LATEX do not inadvertently
modify the behavior of commands. Since LATEX is alarge and complex system, this is definitely a nontrivial task: in ‘fixing’ one bug, it is often necessary
to modify the definitions of several ‘internal’ commands, and these may in turn affect many other1 An additional requirement according to the trip test

documentation is that the author of the TEX implementation
has to be satisfied with the product. In other words, a simple
program that throws away all its input and always output
the files needed to satisfy the trip test would be allowed tocall itself TEX as long as the author of that program is happywith it.

preliminary draft, September 15, 1999 0:07

1001

commands which have no obvious connection withthe original problem.
We have had some pretty disastrous experiencesof this type, often finding that harmless looking corrections had effects on what seemed, at first glance,completely unrelated areas. This is in part due tothe fact that LATEX is based on the macro languageof TEX, which allows reuse and redefinition of arbitrary code fragments.
For that reason we started working on a concept
for automated tests to detect such problems. Whenthat system was available, we asked for volunteers tohelp us in building up a suitable test suite for LATEX(which at that time was LATEX 2.09). Part of the rationale behind this work was to ensure that a futuretransition from LATEX 2.09 to LATEX 2ε (for whichdevelopment was under way) would become as painless as possible, i.e., these tests were also supposed
to ensure that the new code for LATEX 2ε would notchange the user interface behavior without detection.
This approach seems to us to have been very
successful; this is in large parts due to the quality
and quantity of the work of the volunteers helping
us at that time, in particular Daniel Flipo and Chris
Martin. Figure 1 shows an excerpt from the volunteer task list from 1993 describing this task (and myrather optimistic time requirements for it).
When LATEX 2ε was released for the first time in1994 we updated the regression test support macros
and tried to improve the test suite by adding new
test files when we fixed bugs or when we added newfunctionality to LATEX. However, being human, we
have not followed this practice as rigorously as weshould have: especially since the first releases it has
become more and more common for us to fix a small
bug without spending the additional time necessary
to also write a test file that exhibits the correct behavior.
Today our test suite has about 300 test files
which are automatically executed and compared before a new release hits the streets. And indeed, thesetest files have saved us from embarrassment manytimes already.
2This year’s boo-boos!

However, results show that such a suite can neverbe large enough to avoid the need for a patch release once in a while. It is particularly importantthat new features, such as the release of additional
files or the correction of recently found bugs, gettested and frozen within this suite so that there isno unexpected change later on. For example, with
the December 1997 release we added the packages
preliminary draft, September 15, 1999 0:07

1002

preliminary draft, September 15, 1999 0:07

TUGboat, Volume 0 (2001), No. 0

Validating LATEX2.09
Writing test files for regression testing: checking bug fixes and improvements to verify that they don’t
have undesirable side effects; making sure that bug fixes really correct the problem they were intended
to correct; testing interaction with various document styles, style options, and environments.
We would like three kinds of validation files:
1. General documents.
2. Exhaustive tests of special environments/modules such as tables, displayed equations, theorems,
floating figures, pictures, etc.
3. Bug files containing tests of all bugs that are supposed to be fixed (as well as those that are not
fixed, with comments about their status).
A procedure for processing validation files has been devised; details will be furnished to anyone
interested in this task.
Estimated time required: 2 to 3 weeks, could be divided up.
Coordinator [25 August 1992]:
Daniel Flipo flipo@citil.citilille.fr
Other volunteers:
Chris Martin cs1cwm@sunc.sheffield.ac.uk
Figure 1: An excerpt from the volunteer task list 1993
calc and textcomp to the distribution but, due totime constraints, did not add to the suite additional
test files designed to exercise these packages; and,by Murphy’s law, textcomp did not contain a necessary \ProvidesPackage command, with the resultthat it claimed to be written for a future release2 —something that would have been caught by any test
file exercising the package.
Another embarrassing example of a missing test
file in that release was the \t error. To better support language files from the Babel suite, some ofwhich make the " character active, we changed allinternal definitions of characters and accents fromhexadecimal notation, such as "7F, to decimal, i.e.,
to 127 in that case. Unfortunately in the definitionfor \t we did this wrong and "7F became 79, givingvery strange effects when the accent was used.3 An
error like this would have been automatically caught
if we had, for each output encoding, a test file tocheck that each definition in the encoding results inthe ‘right’ glyph or glyphs.

2 The technical reason for this behavior, for those whowonder, was that the release date of the package, which
is an optional argument to the (missing) \ProvidesPackagecommand, was there but was mistakenly picked up the\NeedsTeXFormat which then produced a warning as the release date of textcomp was later than the nominal release
date (of 1997/12/01) for the format of the distribution.
3 Both errors got found and reported several times within
two days after the release, so the patch release came out quite
quickly this time.

preliminary draft, September 15, 1999 0:07

3Call for volunteers

Thus to make the LATEX system even more reliable
we call on you for help! What we hope to find is a
new group of volunteers that is interested in working on an extension of the LATEX test suite system.
There is no need to be an expert TEX or LATEX programmer for this task though some experience withLATEX and its inner workings will be necessary.
If you are interested in joining this effort,
please contact Daniel Flipo atDaniel.Flipo@univ-lille1.fr
who kindly agreed to act as a coordinatorbetween the individual volunteers.
There are a number of areas in which further
test files would improve the system enormously. They
are outlined in the following sections.
3.1

Testing existing interfaces

Testing existing interfaces is a very important task,
one not so far, for several reasons, adequately covered by the test suite. This will not only help us
to detect problems when fixing errors in LATEX but,
more importantly, it will help one day in the transition to a new system since these test files willthen clearly identify which interfaces are compromised (deliberately or by mistake) by the new system. This in turn will then help to produce, if necessary, procedures to automatically translate source
documents from LATEX to its successor.
preliminary draft, September 15, 1999 0:07

TUGboat, Volume 0 (2001), No. 0

preliminary draft, September 15, 1999 0:07

What we are looking for are test files that describe and test the current interfaces on all levels.
This is certainly an ambitious task, but perhaps
also one of the most interesting and rewarding ones
within this list.
3.2

Testing corrected bugs

As described above, several of the bugs reported tous have been fixed and a test file showing the correct
behavior has been added. But for many this is notthe case.
What we are looking for is the provision of test
files for all bugs reported and fixed, so that future
releases will not by mistake revert any of these fixes
without alerting the maintainers. This means working through our bug database and devising test files
showing the correct behavior. As we ask submitters of bug reports to send in a test file that shows
the incorrect behavior, and they usually do so, it isoften possible to start from the submitted file andmodify it slightly so that it fits into the regression
suite concepts.
3.3

Testing new extensions

What is important for the kernel interfaces is also
important for the core packages and extensions: theseinterfaces should be exercised in such a way thatany future changes will be automatically detected.
Again this provides interesting mental exercise since
it isn’t always easy to decide what is pertinent for
the interface and how to exercise it so that enough(but not too much) information ends up in the .log
file.
3.4

1003

ing used up in chasing after errors introduced in thisway (being neither our fault nor being correctable
by us), it would be necessary to develop clear protocols for how this part of the test suite should bemaintained, e.g., what requirements a package must
fulfill to be included into it, what obligations an author of such a package agrees to, etc. This is not yet
done and so it is part of the volunteer effort.
We close our plea for help with a quote takenfrom [2] which shows how the Grand Wizard sees
the task of writing such test files (which does notmean you have to follow his advice):
To write such a fiendish test routine, one
simply gets into a nasty frame of mind andtries to do everything in the unexpected way.
Parameters that are normally positive are set
negative or zero; borderline cases are pushed
to the limit; deliberate errors are made inhopes that the compiler will not be able torecover properly from them.
Donald Knuth 1984
References[1] David Carlisle and Frank Mittelbach. The LATEX
regression test suite: concepts and implementation. TUGboat; to appear.
[2] Donald E. Knuth. A torture test for TEX. Report
STAN-CS-84-1027, Stanford University, Department of Computer Science, Stanford, CA, USA,1984.
[3] Leslie Lamport. LATEX: A Document Preparation System. Addison-Wesley, Reading, Massachusetts, second edition, 1994.

Testing contributed packages

A final area which is important is the testing of packages which lie outside the control of the LATEX maintainers. Although we cannot in all cases guaranteethat corrections to the kernel software will not harm
any such package, we are, of course, very much concerned to avoid making any change that makes third
party packages invalid. In the past, whenever wenoticed (or even suspected) such a problem we tried
either to avoid it, by choosing a different solution,or, if that was not possible for some reason, to findthe maintainers of the package and give them noticeof a possible clash so that such problems could be
avoided.
There is a problem with testing the interfacesof third party packages: changes by the package author, to either the interface or the implementation of
the package, can upset the test suite as easily as canchanges to the LATEX kernel by the LATEX3 project
team. Thus, to avoid our limited time resources bepreliminary draft, September 15, 1999 0:07

 Frank Mittelbach
LATEX3 projectFrank.Mittelbach@eds.com

preliminary draft, September 15, 1999 0:07