170

TUGboat, Volume 42 (2021), No. 2
layoutcontrolOn the road to Tagged PDF:
About StructElem, Marked Content,
PDF/A and Squeezed Bärs
Ulrike Fischer

goo
PDF

Abstract
In this article I present two packages as part of the
LATEX Project’s “Tagged PDF” effort:
tagpdf which contains the core code to create atagged PDF and is used by the LATEX team totest new code.
pdfmanagement-testphase which contains a large
number of PDF-related commands and tools andinstalls a new management command for central
PDF dictionaries.
I will show how to use these packages and the benefits they will bring for the average user, while alsomentioning resulting incompatibilities and requiredchanges in documents.
12

Some technical details

In the PDF format, every page is stored in a pagestream. This stream contains various operators which
select fonts, move around on the page, and insert
images and text.
As described in a previous article [1], tagging
requires first the writing of markers called Marked
Content (MC) markers into the page stream whichlabel and number all chunks of meaningful content.
The next step is to add various objects to thePDF. These objects contain references to each otheras parent and kid objects and so form a tree. The
numbered Marked Content chunks are leaf nodes
in this tree. Figure 2 shows a simple tree createdUlrike Fischer
doi.org/10.47397/tb/42-2/tb131fischer-tagpdf

dstru
ct
lots ure inof t fo w
agg ithingOK

ut clots ontrolof C withSSOKHTML

go odstructure
meaningFigure 1: PDF vs. HTML, layout vs. structureroot (obj 1)
document (obj 4)
section (obj 5)

paragraph (obj 6)MCID 0

MCID 1Figure 2: Simple structure tree

Introduction

At its core, PDF is a page and graphic-related format: it allows describing, very accurately, a pagelayout, the font sizes, font types, colors, placementof characters and graphical elements, but essentially
it simply says where to put ink on the page. It ispossible to extract text but one doesn’t get the semantic meaning of such a text, the reading order canbe unclear, and — with PDF produced by TEX — itcan even be unclear where words begin and end.
However, PDF is also quite an extensible format. In the same way you can enhance HTML with
Cascading Style Sheets (CSS) and JavaScript, youcan enhance PDF in various ways: you can embed
files with source code or other material, you can add
alternative text to objects. And, most importantly
for our purposes, you can tag it: with this you areadding a structural layer over the content (figure 1).

layowith such objects. The last step is to create various
objects for cross-referencing and housekeeping, andto register them in the catalog so that a PDF viewer
can find and use the structure.
3Inspecting a tagged PDF

Checking the tags of a PDF can be done with Adobe
Acrobat Pro; some free options are also slowly appearing. One of them is the online editor at ngpdf.com.
It allows you to upload a tagged PDF, view the tag
structure and view and also export an HTML representation of the PDF. If the math has an embedded
associated file with the MathML representation, thenthe HTML will use MathJax to render it.
Unlike the copy & paste heuristic of PDF viewers, the site doesn’t try to guess the reading orderbut shows you what you get from the tags and thestructure tree. The HTML export demonstrates thattagging isn’t only for screen readers. It gives you
more options to reuse the PDF. The old saying thatPDF is like scrambled eggs and that you can’t getback an egg (a structured LATEX document) from it
is no longer true: You still don’t get all the details
back, but depending on the amount of tagging that’s
been added, you can restore quite a lot.
4

Amount of tagging:
About (im-)perfection and standards

A document must be tagged in its entirety. You can’t
tag only a specific section or some table or some math

TUGboat, Volume 42 (2021), No. 2
to improve its export or copy & paste, even if the restof the document already works satisfactorily.
But this still leaves you with quite a lot of freedom about the amount of tagging. If a paragraphcontains some emphasized text you can tag it as anEm-structure, but you don’t have to. This naturallymeans that some information will be lost, but quite
apart from the fact that is often impossible anyway
to mirror the intention of every detail of a layout
into such a structure tree, tagging doesn’t have to be
perfect to be useful; it improves accessibility significantly even if merely headings are tagged for betternavigation and paragraphs are shown in the correct
reading order.
The same goes for standards: There exist various
guides and standards which describe requirements
for accessible and well-tagged PDF documents. Theyand the validators checking such standards are useful
to set goals which we should aim to reach, but that
it is often not (yet) possible to fulfill all aspects ofsuch a standard should not stop you from creating
more usable documents.
The end users of the documents we produce are
not validators but intelligent humans, and intelligent
humans can often handle imperfect documents surprisingly well, and in the end, their feedback is what
will really matter.
5Tagging with the tagpdf package

I wrote the tagpdf package (ctan.org/pkg/tagpdf)
around four years ago. At that time I was alreadyconvinced that tagging at a reasonable scale can’t
be done in an external package, but that changes
would be required in LATEX itself. If we want a large
number of documents to be tagged, “normal” tagging
shouldn’t need expert skills and fragile patches but
should be supported by LATEX directly.
So the goals of tagpdf were• to develop the basic tagging code for the kernel,• to provide examples and tests,• to identify LATEX problems, and• to develop stable solutions which can be eventually integrated into LATEX.
This means that tagpdf wasn’t written as astandard user package, but rather primarily as a
research and development tool and will eventuallyvanish again. As such
• it requires the newest LATEX code, sometimes
even LATEX-dev,• it can still change,• it concentrates on basic commands, and• it doesn’t patch other packages.

171

Listing 1: Tagging commands\tagstructbegin{tag=P}%\tagmcbegin{tag=P}...text...\tagmcend\tagstructbegin{tag=Link}\tagmcbegin{tag=Link}...text...\tagmcend\tagstructend\tagmcbegin{tag=P}...text...\tagmcend%pagebreak: new mc-chunk\tagmcbegin{tag=P}...text...\tagmcend\tagstructend

tagpdf defines a number of commands, but thetwo core command pairs are these: First, the commands to mark the MC-chunks. As they split the text
into small labeled chunks these commands should be
used linearly: \tagmcbegin ... \tagmcend.
Secondly, there are the commands to mark thestructure. They are typically nested:\tagstructbegin ... \tagstructend.
Listing 1 shows a simple example of their use. It
is important to note that if there is a page break inthe middle of a text chunk, the MC-marker must be
closed before the break and reopened after the break.
This is easy to do manually but rather challenging
if it has to be done automatically, at least withother engines than LuaLATEX. Frank Mittelbach’s
companion talk discusses this.
Listing 2 shows the source code of a small but
complete document using tagpdf. In general, tagging works best with LuaLATEX. To use other engines,
you need the code mentioned in Frank’s talk, or markup all page breaks manually. Neither LATEX & dvips
nor XELATEX support real space characters.
There are a few important points to note here:• The document should be compiled twice, sometimes more. Be aware that you don’t necessarily
get a rerun message yet, even if one is needed.
• tagpdf requires the pdfmanagement-testphasepackage. I will say more about this below.
• tagpdf is still a normal package and can beloaded with \usepackage, but as a first step
towards full integration into LATEX itself it canalso be loaded with the testphase key.
• Tagging must be activated explicitly. This canbe done as in the example with the activate
key, and also with the \tagpdfsetup commandfrom the package (see the documentation).
• The example shows lots of tagging commands
around the section, but none at all around theparagraph or the link.
This last point reflects the state of the project:
With the help of the work done over the last months

On the road to Tagged PDF: About StructElem, Marked Content, PDF/A and Squeezed Bärs

172

TUGboat, Volume 42 (2021), No. 2

Listing 2: A full example\RequirePackage{pdfmanagement-testphase}\DeclareDocumentMetadata{pdfversion=2.0,testphase={tagpdf}, activate=tagging}\documentclass{article}\usepackage{hyperref}\pagestyle{empty}\begin{document}\tagpdfparaOff\tagstructbegin{tag=H1}\tagmcbegin{tag=H1}\section{A section}\tagmcend\tagstructend\tagpdfparaOn
A paragraph with some text and a link to\url{https://www.latex-project.org}
Another paragraph \ldots\end{document}

add associated files to a structure), to create links
which have suitable hooks for the automatic tagging,commands to create form fields with built-in tagging
support, and more.
But the PDF management does more: In aPDF there are a number of central dictionaries towhich packages shouldn’t write directly, to avoid
clashes with other packages. Until now, packages
had no way to avoid such problems apart from testing for potentially problematic packages, but thenew PDF management will resolve this problem: Itoffers a command which allows to write to the dictionaries in a managed way. The command exists
in two versions, for LATEX 2ε code and for expl3 code:\PDFManagementAdd and \pdfmanagement_add:nnn,
respectively, which together we’ll call pdfmanagement
for short. Details on how to use the command canbe found in the documentation l3pdfmanagement.
6.1

on the new paragraph hooks and PDF management,
paragraphs and links can be automatically tagged,
but sectioning commands still need either manual
tagging or patches. It will be the next task to automate this too, which will require some months’ work.
You may wonder why it takes so long to insert whatamounts to two lines of code. The first reason is thatthe code must be added to commands which are used,
patched and redefined by many classes and packages.
If we simply change them to support tagging, thiswould break these classes and packages. So we needa proper change strategy here. The second reason isthat as we are obliged to change the sectioning commands, we want to use the opportunity to modernize
and improve them.
6

PDF management

Tagging writes many objects and other code intothe PDF. This requires proper tools. “Proper tools”
means, above all, abstracted, backend-independent
tools provided directly by LATEX. When I started
with tagpdf there was nearly nothing available. Youeither had to use some external package or a primitive command. By now the situation has changed
dramatically, and for the better: The LATEX format
now includes the L3 programming layer (expl3) codeand loads backend files from l3backend. This meansthat even in small documents, commands to write
PDF objects, to set the PDF version, and to use
colors are available.
The new PDF management support, currently
in the external package pdfmanagement-testphase,extends this set of tools. It offers commands towrite the MC-markers, to embed files (needed toUlrike FischerIncompatibilities

This new pdfmanagement support replaces the fiveprimitives \pdfinfo, \pdfcatalog, \pdfpageattr,\pdfpagesattr and \pdfpageresources and theanalogous commands of the other engines and backends. Replace truly means replace here: every package which uses those primitives is incompatible with
the new pdfmanagement. For this reason we built asafety net around its use:
• The code has not been added directly to the kernel, but is offered now as an external testphasepackage.
• For the central management command, there isan explicit activation trigger command:\DeclareDocumentMetadata.
• We already wrote a number of replacements forincompatible packages and drivers, for example
hyperref, transparent, pdflscape.
• We wrote a number of “firstaid” patches for
packages which will stay until the packages adapt
their code, e.g., pgf and hyperxmp.
• We notified various package and class maintainers and asked them to adjust their code.
But we can see problems only with packages
that are on CTAN and in the TEX distributions. For
the various house classes and packages of universities
and journals we need the help of users to reportproblems. So please test and check!
6.2Benefits

The new pdfmanagement is not only good for tagging.
The rewriting and checking of PDF related code andpackages was also used to modernize, correct andextend various other features. I would like to mentiona few improvements.

TUGboat, Volume 42 (2021), No. 2

173

Listing 3: New hyperref options\hypersetup{href/protocol=https://,href/urlencode}\hrefurl{www.köln.de}{Köln}\url[format=\textsc]{www.köln.de}

Listing 5: Specifying a PDF standard\RequirePackage{pdfmanagement-testphase}\DeclareDocumentMetadata{pdfstandard=A-2b}\documentclass{article}\usepackage{hyperxmp}\usepackage{hyperref}....

Listing 4: Rotating of float pages\begin{figure}[p]\PDFManagementAdd{ThisPage}{Rotate}{90}... landscape picture ...
\end{figure}

6.2.1

Colors and link options for hyperref

When the new pdfmanagement is used, hyperref(ctan.org/pkg/hyperref) uses a new, generic driver.
This driver uses better default colors, has keys tochange to other color schemes, and styles like linkcolors can now be changed at any time in the document.
The support for non-ASCII links has been extended:
links can be input in UTF-8 and hyperref will convert them internally into the needed “percent” encoding. It is possible to preset a protocol, most likely
https://, which is then prefixed to all links. Thereis also a hook system which allows packages to add
support to differentiate internal links by concepts
such as citation, acronym, glossary and similar. Seelisting 3.
6.2.2

Rotations of float pages

With the pdfmanagement command it is possible
to rotate float pages; depending on the engine thisrequires one or two compilations. See listing 4.
6.2.3

6.2.5

New form field code

Finally I would like to mention the code for form
fields, which has been completely rewritten.
The new code is currently in a testphase package
called l3pdffield-testphase, which is part of thepdfmanagement-testphase bundle. It is not yet
decided if it will be part of hyperref like the oldcode or stay in a separate package.
One important change in this code is that it has
been adapted to PDF 2.0. This means that it uses
“appearances” in various places for the look of thefields (appearances are small graphics called “formXObjects”, stored in the PDF). Using appearancesmeans that radio buttons or checkbox fields are nolonger restricted to characters but can contain arbitrary images, e.g., animals from the tikzlings
package (ctan.org/pkg/tikzlings).
But be careful! Appearances are squeezed by
the PDF viewer into the dimensions of the form field,
and this can hurt the Bär!

Support for PDF standards

The pdfmanagement bundle also contains a module
for PDF standards. Currently it mostly handles Astandards. When such a standard is requested thecode will, for example, include a color profile andregister it in the PDF catalog. It will also enableverification tests that other packages can use in theircode; for example, the new hyperref driver uses
this to suppress JavaScript actions. This replaces
the key pdfa which is no longer used by the newdriver. XMP-metadata can be added by loading thehyperxmp package (ctan.org/pkg/hyperxmp). See
listing 5.
6.2.4(ctan.org/pkg/newpax), and it supports including
PDFs inside another PDF without losing the links. Itworks with more backends and engines and, unlike
pax, it doesn’t require an external Java application,
instead using LuaTEX.

Preserving links

As part of the work, a modernized version of the pax
package (ctan.org/pkg/pax) from Heiko Oberdiek
has been written. The new package is called newpax

Ouch
Ouch

References[1] U. Fischer. Creating accessible pdfs with LATEX.
TUGboat 41(1):26–28, 2020. https://tug.org/TUGboat/tb41-1/tb127fischer-accessible.pdf⋄ Ulrike Fischer
LATEX project team
Mönchengladbach
Germanyulrike.fischer (at)latex-project.org

On the road to Tagged PDF: About StructElem, Marked Content, PDF/A and Squeezed Bärs