XML LONDON 2013
CONFERENCE PROCEEDINGSUNIVERSITY COLLEGE LONDON,LONDON, UNITED KINGDOMJUNE 15–16, 2013

XML London 2013 – Conference Proceedings
Published by XML London
Copyright © 2013 Charles Foster
ISBN 978-0-9926471-0-0

Table of Contents
General Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Sponsors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Building Rich Web Applications using XForms 2.0 - Nick van den Bleeken. . . . . . . . . . . . . . . . . . . . . 9
When MVC becomes a burden for XForms - Eric van der Vlist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
XML on the Web: is it still relevant? - O'Neil Delpratt. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Practice what we Preach - Tomos Hillman and Richard Pineger. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
Optimizing XML for Comparison and Change - Nigel Whitaker and Robin La Fontaine. . . . . . . . . . . . 57
What you need to know about the Maths Stack - Ms. Autumn Cuellar and Mr. Paul Topping. . . . . . . . 63
Small Data in the large with Oppidum - Stéphane Sire and Christine Vanoirbeek. . . . . . . . . . . . . . . . . . 69
Extremes of XML - Philip Fennell. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
The National Archives Digital Records Infrastructure Catalogue:
First Steps to Creating a Semantic Digital Archive - Rob Walpole. . . . . . . . . . . . . . . . . . . . . . . . . . . 87
From trees to graphs: creating Linked Data from XML - Catherine Dolbear and Shaun McDonald. . . 106
xproc.xq - Architecture of an XProc processor - James Fuller. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Lazy processing of XML in XSLT for big data - Abel Braaksma. . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Using Distributed Version Control Systems
Enabling enterprise scale, XML based information development - Dr. Adrian R. Warman. . . . . . . . . 145
A complete schema definition language for the Text Encoding Initiative -Lou Burnard and Sebastian Rahtz. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

General Information
Date

Saturday, June 15th, 2013
Sunday, June 16th, 2013

Location
University College London, London – Roberts Engineering Building, Torrington Place, London, WC1E 7JE
Organising Committee
Kate Foster, Socionics Limited
Dr. Stephen Foster, Socionics Limited
Charles Foster, XQJ.net & Socionics Limited
Programme Committee
Abel Braaksma, AbraSoft
Adam Retter, FreelanceCharles Foster (chair), XQJ.net
Dr. Christian Grün, BaseX
Eric van der Vlist, Dyomedea
Jim Fuller, MarkLogic
John Snelson, MarkLogic
Lars Windauer, BetterFORM
Mohamed Zergaoui, Innovimax
Philip Fennell, MarkLogic
Produced By
XML London (http://xmllondon.com)

Sponsors
Gold Sponsor• OverStory - http://www.overstory.co.uk

Silver Sponsor• oXygen - http://www.oxygenxml.com

Bronze Sponsor• Mercator IT Solutions - http://www.mercatorit.com

Preface
This publication contains the papers presented during the XML London 2013 conference.
This was the first international XML conference held in London for XML Developers – Worldwide, Semantic Web& Linked Data enthusiasts, Managers / Decision Makers and Markup Enthusiasts.
This 2 day conference covered everything XML, both academic as well as the applied use of XML in industries suchas finance and publishing.
The conference took place on the 15th and 16th June 2013 at the Faculty of Engineering Sciences (Roberts
Building) which is part of University College London (UCL). The conference dinner and the XML London 2013
DemoJam were held in the Jeremy Bentham Room at UCL, London.
The conference will be held annually using the same format in subsequent years with XML London 2014 taking
place in June 2014.
— Charles Foster
Chairman, XML London

Building Rich Web Applications using
XForms 2.0
Nick van den Bleeken
Inventive Designers<nick.van.den.bleeken@inventivegroup.com>

Abstract
XForms is a cross device, host-language independent markup
language for declaratively defining a data processing modelof XML data and its User Interface. It reduces the amountof markup that has to be written for creating rich webapplications dramatically. There is no need to write any
code to keep the UI in sync with the model, this is completely
handled by the XForms processor.
XForms 2.0 is the next huge step forward for XForms,
making it an easy to use framework for creating powerful
web applications.
This paper will highlight the power of these newfeatures, and show how they can be used to create real life
web-applications. It will also discuss a possible framework
for building custom components, which is currently still
missing in XForms.

1. Introduction
Over the last 2 years there is a trend of moving awayfrom browser plug-in frameworks (Adobe Flash, JavaFX,
and Microsoft silverlight) in favor of HTML5/Javascriptfor building rich web-applications. This shift is driven by
the recent advances in technology (HTML5 [HTML5],
CSS [CSS] and Javascript APIs) and the vibrant browser
market on one hand, and the recent security problems inthose plug-in frameworks on the other hand.
Javascript is a powerful dynamic language, but apotential maintenance nightmare if one is not extremely
diligent. Creating rich web-applications using javascript
requires a lot of code. There are a lot of frameworks (likeDojo [DOJO] and jQuery [JQUERY]) that try tominimize the effort of creating user interfaces. Dojo evengoes one step further by allowing you to create modelview-controller applications, but you still have to write a
lot of javascript to glue everything together.

doi:10.14337/XMLLondon13.Bleeken01

XForms is a cross device, host-language independent
markup language for declaratively defining a dataprocessing model of XML data and its User Interface. It
uses a model-view-controller approach. The model
consists of one or more XForms models describing thedata, constraints and calculations based upon that data,and submissions. The view describes what controls
appear in the UI, how they are grouped together, and towhat data they are bound.
XForms reduces the amount of markup that has to be
written for creating rich web-applications dramatically.
There is no need to write any code to keep the UI in
sync with the model, this is completely handled by the
XForms processor.
XForms 2.0 is the next huge step forward for
XForms, making it an easy to use framework for creating
powerful web applications. This paper will first discuss
the most important improvements in this new version of
the specification, followed by an analysis of possible
improvements.

2. XForm 2.0
This section will discuss the most important
improvements of XForms compared to its previous
version. Those improvements make it easier to create
powerful web applications that integrate with data
available on the web.

2.1. XPath 2.0
XPath 2.0 [XPATH-20] adds a much richer type system,
greatly expands the set of functions and adds additional
language constructs like 'for' and 'if'. These new
language features make it much easier to specify
constraints and calculations. At the same time it makes it
easier to display the data the way you want in the UI.

Page 9 of 162

Building Rich Web Applications using XForms 2.0

Example 3. Variables
Example 1. XPath 2.0: Calculate order price
The folowing XPath expression calculates the sum of the<xf:var name="paging" value="instance('paging')"/>
multiplication of the price and quantity of each item in<xf:group>
the order:<xf:var name="total"sum(
for $n in order/item
return $n/price * $n/quantity)

2.2. Attribute Value Templates
Attribute Value Templates [AVT] allow you the use
dynamic expressions virtually everywhere in the markup.
They are not limited to the XForms elements, but aresupported on most host language attributes. Attribute
Value Templates enable even more powerful styling ofyour form based on the data. As an example, a form
author can now easily highlight rows in a table based on
certain data conditions (overdue, negative values, orcomplex conditions). In HTML5, this feature enables
the form author to declaratively specify when certain cssclasses apply to an element.
Example 2. AVT: Higlight overdue jobs
<xf:repeat ref="job"><tr class="{
if(current-dateTime() > xs:dateTime(@due))then 'over-due' else ''
}">
...
</tr>
</xf:repeat>

2.3. Variables
Variables [VAR] make it possible to break down complex
expressions into pieces and make it easier to understandthe relationship of those pieces, by using expressive
variable names and documenting those individual pieces
and their relationships.
Variables also facilitate in de-duplication of XPathexpressions in your form. In typical forms the same
expression is used multiple times (e.g.: XPath expressionthat calculates the selected language in a multi-lingual
UI).

Page 10 of 162

value="$paging/@total"/><xf:var name="page-size"value="$paging/@page-size"/><xf:var name="page-count"value="($total + $page-size - 1)
idiv $page-size"/><xf:output value="$page-count"><xf:label>Number of pages</xf:label></xf:output></xf:group>

2.4. Custom Functions
Custom functions [CUST_FUNC] like variables allow
form authors to simplify expressions and prevent code
duplication without using extensions.
Example 4. Custom Functions: Fibonacci<function signature="
my:fibonacci($n as xs:integer) as xs:integer"><var name="sqrt5" value="math:sqrt(5)"<result value="(
math:power(1+$sqrt5, $n) math:power(1-$sqrt5, $n)) div(math:power(2, $n) * $sqrt5)" /></function>

2.5. Non-XML data
Because XForms' data model is XML it can consumedata from a lot of sources with little effort (SOAP
services, XML data bases using XQuery, REST XML
services, ...). Starting from XForms 2.0, XForms cannatively consume JSON data [JSON]. As more and more
services on the web are starting to deliver JSON today
this is an important feature. XForms implementations
may support other non-XML file formats like CSV,
vCard, ...
The form author can use all normal XForms
constructs (binds, UI controls, actions,...) independantfrom the data format of the external source. The XForms
processor will build an XPath data model from therecieved data.

Building Rich Web Applications using XForms 2.0

2.6. Miscellaneous improvements

4.1. Using Custom Components

Other interesting new features are:
• Model based switching/repeats allows form authors to
capture the state of the switch/repeat in the model,
which makes it possible to save and restore the actual
runtime state of the form.
• The iterate attribute makes it much easier to executeactions for all nodes matching a certain condition.
• Ability to specify the presentation context for the
result of a replace-all submission. This feature makes
it possible to display the result of a submission inanother frame or tab, when you using HTML as yourhost language.
• MIP functions (e.g.: valid()) which can be used inactions and the user interface. They can for example
be used to conditionally show content based on thevalue of a MIP on a data node.

This section will discuss the aspects of a custom
component that are relevant to the user of thecomponent and demonstrate how custom controls canbe used by the form author in an XForms document.
In general there are two different categories ofcustom components:
1. Components that are just a different appearance of an
existing XForms control. An example of this is agraphical state selection component.

3. Possible features for a futureversion of the specification

Example 5. Custom appearance: Graphical state
selection component<xf:select1 ref="state" appearance="cc:us-states"><xf:itemset ref="instance('states')"><xf:label ref="@name"/><xf:value ref="@code"/></xf:itemset/><xf:select1>

There are a lot of possible enhancements that could bemade to the current specification. Ranging from
improving UI events, better expression of conditional
default values, structural constraints to better integration
with other web platform technologies (Javascript, geolocation, Web Storage, Crypto, ...).
But in my opinion the most important thing that isstill missing in the current version of the specification, isa framework for defining custom components that canbe re-used in multiple forms/applications. This section
will discuss possible requirements for such a framework
and a proposal of a possible custom components
framework which is heavily based on the framework thatis currently implemented by Orbeon Forms[ORBEON].

4. Custom Components
Custom components should be easily re-usable overmultiple forms/applications. They should feel andbehave the same way as native XForms UI controls. It
should also be possible to strongly encapsulate theirinternal data and logic. The following section will go
into more depth about these requirements and how they
could be implemented.

Page 11 of 162

Building Rich Web Applications using XForms 2.0

2. Advanced and/or complex user interface controls that Example 7. Custom Container Control: Tab vieware not an appearance of an existing XForms User
<cc:tabview>
Interface control. An example of such a control is a
<cc:tab>donut chart:<xf:input ref="foo">...</xf:input>
Example 6. Custom element: Donut chart<cc:chart><cc:slice-serie
ref="instance('accounts')/account"><cc:label value="@label"/><cc:name value="@id"/><cc:value value="@amount"/></cc:slice-serie></cc:chart></cc:tab><cc:tab>...</cc:tab><cc:tabview>

4.1.2. Events
XForms uses XML events and XForms actions to execute
various tasks during the form execution. The appropriate
xforms events are dispatched to the custom control (e.g.:
the data node related events like xforms-value-changed
and xforms-readonly are sent to the control if the control
has a data binding). The form author can attach event
listeners to all custom controls just like he does for native
XForms controls.
The following example attaches a value change
listener to the custom pie chart control:
Example 8. Events: Attach handler to Custom Control<cc:pie-chart ref="account"><cc:slices ref="instance('accounts')/account"><cc:label value="@label"/><cc:name value="@id"/><cc:value value="@amount"/></cc:slices>

As shown in the above example, the markup to use
custom components is similar to the markup for native
XForms controls. To use the first category of controls the<xf:action ev:event="xforms-value-changed">
form author just has to specify a custom appearance. For...
the second category new element names should be used,</xf:action>
but the same constructs as for native form controls are</cc:pie-chart>
used (ref-attribute for specifying the repeat sequence,value-attribute for the values, a structure similar to
Custom events, which can be handled by the custom
xf:itemset is used for cc:slices).
control, can be dispatched to the custom control using
the xf:dispatch action.
4.1.1. Container control
The following example dispatches an event my-event
Some custom components, such as tabview, act like a to the control with id foo-bar when the trigger iscontainer controls (xf:group, xf:switch, xf:repeat). Those activated:
controls have typically one or multiple insertion points,
to which the custom control host’s children are Example 9. Events: Dispatch event to Custom Control
transposed. The transposed children can contain any<cc:foo-bar id="foo-bar">
host language and XForms content, which will be visible ...
from the “outside” (e.g.: IDs are also visible to actions in <xf:trigger>
the host document outside of this custom control).
<xf:label>Do it</xf:label>
The following example creates a tab view with two<xf:dispatchtabs (Host language and xforms markup can be used
ev:event="DOMActivate"
under the tab elements):
name="my-event"targeted="foo-bar"/></xf:trigger>

Page 12 of 162

Building Rich Web Applications using XForms 2.0

4.1.3. Responsive Design

4.2. Creating Custom Components

When targeting a wide variety of devices with different
capabilities (screen size/resolution, mouse/touch,…) andusages, it might be desirable to change the appearance ofa control depending on the used device and or
environment in which it is used. Examples of this aredesktop versus mobile, but also landscape versus portrait
on a tablet. This is currently not supported but it issomething that should be considered for the next version
of XForms, and might be related to the custom controls
framework.

Implementing a custom component is typically doneusing a mixture of XForms and host language specificmarkup. There are a lot of possibilities on how to specify
this implementation. A possibility is to extend the work
done for XBL 2.0, but because this specification is nolonger maintained it is probably better to specify
something that is a bit more tailored to the XForms
requirements.
A simple custom component that just wraps an input
control, has a data binding and supports LHHA (label,hint, help and alert) might look like this:Example 10. Responsive Design: different appearence
depending on orientation
Example 11. Custom Control: Implementation<xf:component
xmlns:xhtml="http://www.w3.org/1999/xhtml"
xmlns:xforms="http://www.w3.org/2002/xforms"
xmlns:ev="http://www.w3.org/2001/xml-events"
xmlns:cc="http://www.sample.com/custom-controls"
id="foo-bar-id"
element="cc:foo-bar"
mode="data-binding lhha"><xf:template><xf:input ref="xf:binding('foo-bar-id')"/></xf:template></xf:component>

In the above example the mode attribute on thecomponent element ensures that the custom control will
support the data binding attributes (ref, context andbind) and supports the LHHA-elements.

4.3. Data and logic encapsulation
A custom component should be able to have private
models to abstract and encapsulate their internal data
and logic. This implies that a component can define its
own instances, binds and submissions.

4.4. Event encapsulation
Events that are sent from and are targeted to elements
inside the component should not be visible to the user ofthat component. But it should be possible to send events
to, and receive events from, the user of the component.
To fulfill these requirements the elements of thecustom control will reside in its own ‘shadow’ tree. But
events dispatched to, and listener attached to, the root
element of the custom control will be re-targeted to thehost element.

Page 13 of 162

Building Rich Web Applications using XForms 2.0

4.5. Component lifecycle

5. Conclusion

Standard XForms initialization events (xforms-modelconstruct and xforms-model-construct-done and xformsready) and destruction events (xforms-model-destruct)will be sent to the models in the custom control when
the custom control is created and destroyed respectively.
A custom control can be created either when the form isloaded or when a new iteration is added to an xf:repeat.
A custom control is destroyed when the XForms
Processor is shutdown (e.g.: result of load action or
submission with replace all) or if an iteration in an
xf:repeat is removed.
The events are sent to the implementation of thecustom control and therefore, not traverse any of thehost document elements.

The new features in XForms 2.0 like XPath 2.0,
Attribute Value Templates and variables make it easier tocreate web applications with XForms. The support ofnon-XML data sources ensures that the technology canbe used to consume data from a variety of sources. Oneof the biggest strengths of XForms is its abstraction bydeclaratively defining its data processing model(dependencies , validations, calculations and databinding). But it is currently missing a standardized way
for abstracting re-usable high level components, that canbe used to build rich forms/applications. Hopefully sucha frame is something that can be added in the next
version of XForms.

4.6. Styling
By default, styling should not cross the boundary
between the host document and the component. In otherwords, the styling rules from the host document should
not impact with the styling rules from the component
and vice versa. But it should be possible to style parts ofthe custom control from within the host document thatare explicitly specified as being style able by the custom
controls’ implementation. When using CSS as a stylinglanguage it is recommended to use custom pseudo
elements, just like defined in Shadow DOM[SHADOW_DOM].

References[AVT]
http://www.w3.org/TR/xforms20/#avts[CSS]
http://www.w3.org/TR/CSS/[CUST_FUNC]
http://www.w3.org/TR/xforms20/#The_function_Element[DOJO]
http://dojotoolkit.org/[DOJO_DECL]
http://dojotoolkit.org/documentation/tutorials/1.8/declarative/[HTML5]
http://www.w3.org/TR/html51/[JQUERY]
http://jquery.com/[JSON]
http://www.json.org/[ORBEON]
http://www.orbeon.com/[SHADOW_DOM] http://www.w3.org/TR/shadow-dom/[VAR]
http://www.w3.org/TR/xforms20/#structure-var-element[XFORMS-20]
http://www.w3.org/TR/xforms20/[XPATH-20]
http://www.w3.org/TR/2012/WD-xforms-xpath-20120807/

Page 14 of 162

When MVC becomes a burden for XForms
Eric van der Vlist
Dyomedea
Abstract

1.1.2. Answer

XForms is gaining traction and is being used to develop Model:
complex forms, revealing its strengths but also its weaknesses. <xf:model>
One of the latest is not specific to XForms but inherent<xf:instance>
to the MVC (Model View Controller) architecture which is<figures><line>
one of the bases of XForms.
<length>
In this talk we see how the MVC architecture<value>10</value>
dramatically affect the modularity and reusabilty of XForms<unit>in</unit>
developments and some of the solutions used to work around</length>this flaw.
</line>1. Practice: a quiz</figures></xf:instance></xf:model>

Let's start with a quiz...

1.1. Basic XForms
1.1.1. Question
Given the following instance:<xf:instance><figures><line><length><value>10</value><unit>in</unit></length></line></figures></xf:instance></xf:instance>

implement a standard XForms 1.1 form displaying thefollowinguserinterface:doi:10.14337/XMLLondon13.Vlist01

Page 15 of 162

When MVC becomes a burden for XForms

View:

1.2.2. Answer<xf:group ref="line/length"><xf:input ref="value"><xf:label>Length: </xf:label></xf:input><xf:select1 ref="unit"><xf:label></xf:label><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item><xf:item><xf:label>font size</xf:label><xf:value>em</xf:value></xf:item><xf:item><xf:label>font height</xf:label><xf:value>ex</xf:value></xf:item><xf:item><xf:label>inches</xf:label><xf:value>in</xf:value></xf:item><xf:item><xf:label>centimeters</xf:label><xf:value>cm</xf:value></xf:item><xf:item><xf:label>millimeters</xf:label><xf:value>mm</xf:value></xf:item><xf:item><xf:label>points</xf:label><xf:value>pt</xf:value></xf:item><xf:item><xf:label>picas</xf:label><xf:value>pc</xf:value></xf:item><xf:item><xf:label>%</xf:label><xf:value>%</xf:value></xf:item></xf:select1></xf:group>

Model:

1.2. Using instances and actions
1.2.1. Question
Implement the same user interface if the instance uses
the CSS2 / SVG 1.1 conventions for sizes:<xf:instance id="main"><figures><line length="10in"/></figures></xf:instance>

Page 16 of 162<xf:model><xf:instance id="main"><figures><line length="10in"/></figures></xf:instance><xf:instance id="split"><line><length><value/><unit/></length></line></xf:instance>.../...
</xf:model>

View:<xf:group ref="instance('split')/length"><xf:input ref="value" id="length-control"><xf:label>Length: </xf:label></xf:input><xf:select1 ref="unit" id="unit-control"><xf:label/><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item>.../...
<xf:item><xf:label>%</xf:label><xf:value>%</xf:value></xf:item></xf:select1></xf:group>When MVC becomes a burden for XForms

Controller:

1.3.2. Answer<xf:model>.../...
<xf:action ev:event="xforms-ready"><xf:setvalue
ref="instance('split')/length/value"
value="translate(instance('main')/line/@length,'%incmptxe','')" /><xf:setvalue ref="instance('split')/length/unit"
value="translate(instance('main')/line/@length,'0123456789','')" /></xf:action><xf:action ev:event="xforms-value-changed"
ev:observer="length-control"><xf:setvalue
ref="instance('main')/line/@length"
value="concat(
instance('split')/length/value,
instance('split')/length/unit)" /></xf:action><xf:action ev:event="xforms-value-changed"
ev:observer="unit-control"><xf:setvalue
ref="instance('main')/line/@length"
value="concat(
instance('split')/length/value,
instance('split')/length/unit)" /></xf:action></xf:model>

Model:

1.3. Modularity
1.3.1. Question
Still using XForms 1.1 standard features, extend this userinterface to edit the height and width of a rectangle:<xf:instance id="main"><figures><rectangle height="10in" width="4em"/></figures></xf:instance>

Hint: copy/paste is your friend!

<xf:model><xf:instance id="main"><figures><rectangle height="10in" width="4em"/></figures></xf:instance><xf:instance id="height"><height><value/><unit/></height></xf:instance>.../...
<xf:instance id="width"><width><value/><unit/></width></xf:instance>.../...
</xf:model>

View:<xf:group ref="instance('height')"><xf:input ref="value" id="height-value-control"><xf:label>Height: </xf:label></xf:input><xf:select1 ref="unit" id="height-unit-control"><xf:label/><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item>.../...
</xf:select1></xf:group><xh:br/><xf:group ref="instance('width')"><xf:input ref="value" id="width-value-control"><xf:label>Width: </xf:label></xf:input><xf:select1 ref="unit" id="width-unit-control"><xf:label/><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item>.../...
</xf:select1></xf:group>

Page 17 of 162

When MVC becomes a burden for XForms

Controller:<xf:model>.../...
<xf:action ev:event="xforms-ready"><xf:setvalue ref="instance('height')/value"
value="translate(instance('main')/rectangle/@height,'%incmptxe', '')" /><xf:setvalue ref="instance('height')/unit"
value="translate(instance('main')/rectangle/@height,'0123456789', '')"/></xf:action><xf:action ev:event="xforms-value-changed"
ev:observer="height-value-control"><xf:setvalue
ref="instance('main')/rectangle/@height"
value="concat(instance('height')/value,
instance('height')/unit)" /></xf:action><xf:action ev:event="xforms-value-changed"
ev:observer="height-unit-control"><xf:setvalue
ref="instance('main')/rectangle/@height"
value="concat(
instance('height')/value,
instance('height')/unit)" /></xf:action>.../...
<xf:action ev:event="xforms-ready"><xf:setvalue ref="instance('width')/value"
value="translate(instance('main')/rectangle/@width,'%incmptxe', '')"/><xf:setvalue ref="instance('width')/unit"
value="translate(instance('main')/rectangle/@width,'0123456789', '')"/></xf:action><xf:action ev:event="xforms-value-changed"
ev:observer="width-value-control"><xf:setvalue
ref="instance('main')/rectangle/@width"
value="concat(instance('width')/value,
instance('width')/unit)" /></xf:action><xf:action ev:event="xforms-value-changed"
ev:observer="width-unit-control"><xf:setvalue
ref="instance('main')/rectangle/@width"
value="concat(instance('width')/value,
instance('width')/unit)"/></xf:action></xf:model>

1.4. Homework: repeated content
Still using standard XForms features, extend this form tosupport any number of rectangles in the instance.

Page 18 of 162

Hint: you will not be able to stick to atomic instances forthe width and height but act more globally and maintaininstances with a set of dimensions which you'll have tokeep synchronized with the main instance whenrectangles are inserted or deleted.

1.5. What's the problem?
XForms lacks a feature to define and use "components"that would package a group of controls together with
their associated model and actions.

2. Theory: the MVC design pattern
XForms describes itself as a MVC architecture:
An XForm allows processing of data tooccur using three mechanisms:• a declarative model composed offormulae for data calculations andconstraints, data type and other property
declarations, and data submission
parameters• a view layer composed of intent-baseduser interface controls• animperative
controller
fororchestrating
data
manipulations,interactions between the model and viewlayers, and data submissions.
Micah Dubinko argues that the mapping is more
obvious with Model-view-presenter (MVP), a derivativeof the MVC software pattern but that’s not the point I’dlike to make and I’ll stick to the MVC terminology
where:• The model is composed of XForms instances andbinds• The view is composed of the XForms controls
together with the HTML elements and CSS
stylesheets• The controller is composed of the actions

When MVC becomes a burden for XForms

MODEL

UPDATESMANIPULATES

VIEWCONTROLLER
Hierarchical model-view-controller in JavaWorld
U

S

SE
S

E
SE

USER

3. Solutions
A number of solutions are being used to work aroundthis issue with XForms.

3.1. Copy/Paste
Mode-view-controler on wikimedia
Orbeon Form Builder/Form Runner go one step forward
and add a fourth concern for localization and we get amodel/view/localization/controller pattern.
This separation of concerns is great to differentiate
different roles and split work between specialists but
doesn’t play well with modularity and reusability.
I am currently working on a project to develop big
and complex forms and this is becoming one of thebiggest issues: these forms share a number of common
fields and group of fields and, not even speaking of
sharing these definitions, this separation of concerns adds
a significant burden when copying these definitions fromone form to another.
To copy a field from one form to another you needto copy definitions from the model, the view, thelocalization and the controller and can’t just copy a“component”.
And of course, there is no easy way to reuse common
components instead of copying them.
This kind of issue is common with the MVC design
pattern and the Hierarchical model–view–controller(HMVC) has been introduced for this purpose, but how
can we use such a pattern with XForms?

This is what we've done for our quiz and we've seen that
this is easy -but very verbose and hard to maintain- until
we start to deal with repeated content.
I would guess that this is the most common practicewhen fields (or group of fields) are being reused in
XForms though!

3.2. XForms generation or templating
We're XML developers, aren't we? When something isverbose we can use XSLT or any other tool to generate it
and XForms is no exception.
XForms can be generated from any kind of model
including annotated schemas or other vocabularies suchas DDI (we'll be presenting this option at the Balisage
International Symposium on Native XML User
Interfaces in August.
Projects without any obvious model formats in mind
often chose to transform simplified versions of XForms
into plain XForms. In that case the approach may tends
toward a templating system where placeholders areinserted into XForms documents to be transformed into
proper XForms.
We may want for instance to define <my:dimension/>placeholders which would look like XForms controls andgenerate the whole model, view and controller XForms
definitions.

Page 19 of 162

When MVC becomes a burden for XForms

The source form would then be something as simple as:

And the instances and actions can be generated similarly:<xh:html xmlns:xh="http://www.w3.org/1999/xhtml"
xmlns:xf="http://www.w3.org/2002/xforms"
xmlns:my="http://ns.dyomedea.com/my-components/"><xh:head><xh:title>Template</xh:title><xf:model><xf:instance id="main"><figures><rectangle height="10in" width="4em"/></figures></xf:instance></xf:model></xh:head><xh:body><my:dimension ref="rectangle/@height"><xf:label>Height</xf:label></my:dimension><br/><my:dimension ref="rectangle/@width"><xf:label>Width</xf:label></my:dimension></xh:body></xh:html><xsl:template match="my:dimension" mode="model"><xsl:variable name="id"select="if (@id) then @id else generate-id()"/><xf:instance id="{$id}-instance"><height><value/><unit/></height></xf:instance><xf:action ev:event="xforms-ready"><xf:setvalue
ref="instance('{$id}-instance')/value"
value="translate(instance('main')/{@ref},'%incmptxe', '')"/><xf:setvalue
ref="instance('{$id}-instance')/unit"
value="translate(instance('main')/{@ref},'0123456789', '')"/></xf:action><xf:action ev:event="xforms-value-changed"
ev:observer="{$id}-value-control"><xf:setvalue ref="instance('main')/{@ref}"value="concat(
instance('{$id}-instance')/value,
instance('{$id}-instance')/unit)" /></xf:action><xf:action ev:event="xforms-value-changed"
ev:observer="{$id}-unit-control"><xf:setvalue ref="instance('main')/{@ref}"value="concat(
instance('{$id}-instance')/value,
instance('{$id}-instance')/unit)" /></xf:action></xsl:template>

A simplistic version of a transformation to process thisexample is not overly complex. The controls are quite
easy to generate from the placeholders:<xsl:template match="my:dimension"><xsl:variable name="id"select="if (@id) then @id else generate-id()"/><xf:group ref="instance('{$id}-instance')"><xf:input ref="value" id="{$id}-value-control"><xsl:apply-templates/></xf:input><xf:select1 ref="unit" id="{$id}-unit-control"><xf:label/><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item>.../...
</xf:select1></xf:group></xsl:template>

A model can be appended to the <xh:head/> element:<xsl:template match="xh:head"><xsl:copy><xsl:apply-templates select="@*|node()"mode="#current" /><xf:model><xsl:apply-templates select="//my:dimension"
mode="model" /></xf:model></xsl:copy></xsl:template>

Page 20 of 162

As always, the devil is in details and this would be far
from perfect:• In actions, references to the main instance do not take
into account the context node under which the<my:dimension/> placeholder is defined (paths are
therefore expected to be relative to the default
instance). Mimicking the behavior of an XForms
control and its support of the context node would bemuch more challenging.
• Supporting repetitions would be another challenge.

3.3. Orbeon Forms' XBL implementation
Orbeon's component architecture is inpired by XBL 2.0
which describes itself as:When MVC becomes a burden for XForms

XBL (the Xenogamous Binding Language) describes
the ability to associate elements in a document withscript, event handlers, CSS, and more complex
content models, which can be stored in anotherdocument. This can be used to re-order and wrap
content so that, for instance, simple HTML or
XHTML markup can have complex CSS styles
applied without requiring that the markup be
polluted with multiple semantically neutral div
elements.
It can also be used to implement new DOM
interfaces, and, in conjunction with otherspecifications, enables arbitrary tag sets to be
implemented as widgets. For example, XBL could be
used to implement the form controls in XForms or
HTML.
--XBL 2.0
Even if this specification is no longer maintained by the
W3C Web Applications Working Group, the concepts
described in XBL 2.0 fit very nicely in the context of
XForms documents even though the syntax may
sometimes look strange, such as when CSS selectors areused where XPath patterns would look more natural in
XForms documents.

Note
The syntax of XBL declarations has been changed
between Orbeon Forms version 3 and 4. The syntax
shown in this paper is the syntax of version 4.

The XForms component's model goes into theimplementation:<xbl:implementation><xf:model id="my-dimension-model"><xf:instance id="my-dimension-instance"><dimension><value/><unit/></dimension></xf:instance>.../...
</xbl:implementation>

The XForms component's controls are defined into thetemplate:<xbl:template><xf:input ref="value"
id="my-dimension-value-control"/><xf:select1 ref="unit"
id="my-dimension-unit-control"><xf:label/><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item>.../...
</xf:select1></xbl:template>

The XForms actions are split between the handlers andthe implementation (or the template): handlers are used
to define actions triggered by events which are external tothe component (such as in our case xforms-ready) while
traditional XForms actions are used to handle events"internal" to the component such as user actions.
The handlers would thus be:

The definition of an XBL component to implement ourdimension widget would be composed of three parts:<xbl:handlers>
handlers, implementation and template:<xbl:binding id="my-dimension"
element="my|dimension"
xxbl:mode="lhha binding value"><xbl:handlers>.../...
</xbl:handlers><xbl:implementation>.../...
</xbl:implementation><xbl:template>.../...
</xbl:template></xbl:binding>

A fourth element could be added to define component
specific resources such as CSS stylesheets.

<xbl:handler
event="xforms-enabled xforms-value-changed"><xf:setvalue
ref="instance('my-dimension-instance')/value"
value="translate(
xxf:binding('my-dimension'),'%incmptxe', '')" /><xf:setvalue
ref="instance('my-dimension-instance')/unit"
value="translate(
xxf:binding('my-dimension'),'0123456789', '')" /></xbl:handler></xbl:handlers>Page 21 of 162

When MVC becomes a burden for XForms

And the remaining actions:<xbl:implementation><xf:model id="my-dimension-model">.../...
<xf:setvalue ev:event="xforms-value-changed"
ev:observer="my-dimension-value-control"
ref="xxf:binding('my-dimension')"value="
concat(instance('my-dimension-instance')/value,
instance('my-dimension-instance')/unit)" /><xf:setvalue ev:event="xforms-value-changed"
ev:observer="my-dimension-unit-control"
ref="xxf:binding('my-dimension')"value="
concat(instance('my-dimension-instance')/value,
instance('my-dimension-instance')/unit)"/></xf:model></xbl:implementation>

I won't go into the details which are described in
Orbeon's XBL - Guide to Using and Writing XBL
Components but it is worth noting that there is a strict
encapsulation of both the model, the view and thecontroller of this component that seen from the outsideacts as a standard XForms control.
Of course, this component can be used as a standard
XForms control:<xh:body><my:dimension ref="rectangle/@height"><xf:label>Height</xf:label></my:dimension><br/><my:dimension ref="rectangle/@width"><xf:label>Width</xf:label></my:dimension></xh:body>

Page 22 of 162

When MVC becomes a burden for XForms

The complete form with the component definition
would be:<?xml-stylesheet href="xsltforms/xsltforms.xsl" type="text/xsl"?><?xsltforms-options debug="yes"?><xh:html xmlns:xh="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"
xmlns:xxf="http://orbeon.org/oxf/xml/xforms" xmlns:ev="http://www.w3.org/2001/xml-events"
xmlns:xbl="http://www.w3.org/ns/xbl" xmlns:xxbl="http://orbeon.org/oxf/xml/xbl"
xmlns:fr="http://orbeon.org/oxf/xml/form-runner" xmlns:my="http://ns.dyomedea.com/my-components/"><xh:head><xh:title>Simple XBL Component</xh:title><xbl:xbl script-type="application/xhtml+xml"><xbl:binding id="my-dimension" element="my|dimension" xxbl:mode="lhha binding value"><xbl:handlers><xbl:handler event="xforms-enabled xforms-value-changed"><xf:setvalue ref="instance('my-dimension-instance')/value"
value="translate(xxf:binding('my-dimension'), '%incmptxe', '')"/><xf:setvalue ref="instance('my-dimension-instance')/unit"
value="translate(xxf:binding('my-dimension'), '0123456789', '')"/></xbl:handler></xbl:handlers><xbl:implementation><xf:model id="my-dimension-model"><xf:instance id="my-dimension-instance"><dimension><value/><unit/></dimension></xf:instance><xf:setvalue ev:event="xforms-value-changed"
ev:observer="my-dimension-value-control"
ref="xxf:binding('my-dimension')"value="concat(instance('my-dimension-instance')/value,
instance('my-dimension-instance')/unit)"/><xf:setvalue ev:event="xforms-value-changed"
ev:observer="my-dimension-unit-control"
ref="xxf:binding('my-dimension')"value="concat(instance('my-dimension-instance')/value,
instance('my-dimension-instance')/unit)"/></xf:model></xbl:implementation><xbl:template><xf:input ref="value" id="my-dimension-value-control"/><xf:select1 ref="unit" id="my-dimension-unit-control"><xf:label/><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item><xf:item><xf:label>font size</xf:label><xf:value>em</xf:value></xf:item><xf:item><xf:label>font height</xf:label><xf:value>ex</xf:value></xf:item><xf:item><xf:label>inches</xf:label><xf:value>in</xf:value></xf:item>

Page 23 of 162

When MVC becomes a burden for XForms<xf:item><xf:label>centimeters</xf:label><xf:value>cm</xf:value></xf:item><xf:item><xf:label>millimeters</xf:label><xf:value>mm</xf:value></xf:item><xf:item><xf:label>points</xf:label><xf:value>pt</xf:value></xf:item><xf:item><xf:label>picas</xf:label><xf:value>pc</xf:value></xf:item><xf:item><xf:label>%</xf:label><xf:value>%</xf:value></xf:item></xf:select1></xbl:template></xbl:binding></xbl:xbl><xf:model><xf:instance id="main"><figures><rectangle height="10in" width="4em"/></figures></xf:instance></xf:model></xh:head><xh:body><my:dimension ref="rectangle/@height"><xf:label>Height</xf:label></my:dimension><br/><my:dimension ref="rectangle/@width"><xf:label>Width</xf:label></my:dimension><fr:xforms-inspector/></xh:body></xh:html>

3.4. Subforms
Subforms are implemented by XSLTForms andbetterFORM. They have been considered for inclusion
in XForms 2.0 but no consensus have been reached andthey won't be included in 2.0.
There are a number of differences between the
XSLTForms and Betterform implementations but the
principle -and the shortcomings- are the same.

Page 24 of 162

The basic principle behind subforms is to embed (orload) a form within another one. This embedding mustbe specifically performed using an <xf:load> action witha @show="embed" attribute. Subforms can also be
unloaded.
The fact that subforms are explicitly loaded andunloaded in their "master" form is a key feature for big
forms where this mechanism reduces the consumption of
resources and leads to important performance
improvements.

When MVC becomes a burden for XForms

3.4.1. Subforms, betterFORM flavor
Subforms are described, in the betterFORM
documentation, as “a way to avoid redundancies andkeep the documents maintainable”:
As XForms follows a MVC architecture the XForms
model is the first logical candidate when
decomposing larger forms into smaller pieces.
Aggregating more complex forms from little snippets
of UI (or snippets of a model) is a limited approachas the interesting parts are located on the bind
Elements. This is where controls learn about theirconstraints, states, calculations and data types.
Instead of just glueing pieces of markup together the
inclusion of complete models allow the reuse of all
the semantics defined within them.
--betterFORM "Modularizing forms"
Joern Turner, founder of Chiba and co-founder ofbetterFORM, makes it clear that subforms haven't been
introduced to implement components, though:Sorry i need to get a bit philosophic here but
subforms are called subforms as they are *not*components ;) I don't want to dive into academic
discussions here but the main difference for us is thatfrom a component you would expect to use it as apart of a form as a means to edit one or probably
several values in your form and encapsulate theediting logic inside of it. A subform on the other
hand should be designed to be completely
standalone. Typically we build our subforms ascomplete xhtml documents which can be run andtested standalone without being a part of a host
document.
--Joern Turner on the betterform-users mailing list
A proper solution for components, based on Web
Components) should be implemented in betterFORM 6:
We have also considered to implement this [XBL]
but decided against it due to the acceptance andfuture of XBL and due to the fact that we found it
overly complex and academic. We will come up with
our own component model in betterFORM 6 which
will orient at more modern approaches (WebComponents).
--Joern Turner on the betterform-users mailing list
In the meantime it is still possible to use subforms todesign component like features assuming we take into
account the following constraints:• Communications between the master form and thesubform are done using either in memory submissions(ContextSubmissionHandler identified by a model:
pseudo protocol), the instanceOfModel() function
which gives access to instances from other models or
custom events passing context information.
• There is no id collision between the main form andthe subforms which are loaded simultaneously.
This second constraint should be released in the future
but the current version of the processor doesn't address
it. In practice it means that in our sample we cannot load
simultaneously an instance of the subform to edit thewidth and a second instance to edit the height but wecan still take a "master/slave approach" where a single
instance of the subform will be used to edit the width
and the height separately or mimic an "update in place
feature" where an instance of the subform will replace
the display of the width or height.
A way to implement our example using theseprinciples could be:• In the master form:• Define an instance used as an interface with thesubform to carry the value to edit and identify theactive subform instance.
• Include triggers to load and unload subforms.
• Define actions to load and unload the subforms
and maintain the "interface" instance.
• Control when to display the triggers to avoid thatsimultaneous loads of the subform.
• In the subforms:• Synchronize the local model with the instanceused as an interface.
• Perform all the logic attached to the component.

Page 25 of 162

When MVC becomes a burden for XForms

The master form would then be:<xh:html xmlns:xh="http://www.w3.org/1999/xhtml" xmlns:ev="http://www.w3.org/2001/xml-events"
xmlns:xf="http://www.w3.org/2002/xforms"><xh:head><xh:title>Subforms</xh:title><xf:model id="master"><xf:instance id="main"><figures><rectangle height="10in" width="4em"/></figures></xf:instance><!-- Instance used as an "interface" with the subform --><xf:instance id="dimension-interface"><dimension active=""/></xf:instance></xf:model><!-- Dirty hack to style controls inline --><xh:style type="text/css">.xfContainer div {display: inline !important;}.xfContainer span {display: inline !important;}</xh:style></xh:head><xh:body><xf:group ref="rectangle"><!-- Height --><xf:group ref="@height"><xf:label>Height: </xf:label><!-- This should be displayed when the subform is not editing the height --><xf:group ref=".[instance('dimension-interface')/@active!='height']"><xf:output ref="."/><!-- Display the trigger when the subform is not loaded anywhere --><xf:trigger ref=".[instance('dimension-interface')/@active = '']"><xf:label>Edit</xf:label><xf:action ev:event="DOMActivate"><!-- Set the value of the interface instance --><xf:setvalue ref="instance('dimension-interface')"value="instance('main')/rectangle/@height"/><!-- Remember that we are editing the height --><xf:setvalue ref="instance('dimension-interface')/@active">height</xf:setvalue><!-- Load the subform --><xf:load show="embed" targetid="height" resource="subform-embedded.xhtml"/></xf:action></xf:trigger></xf:group><xh:div id="height"/><!-- This should be displayed only when we're editing the height --><xf:group ref=".[instance('dimension-interface')/@active='height']"><xf:trigger><xf:label>Done</xf:label><xf:action ev:event="DOMActivate"><!-- Copy the value from the interface instance --><xf:setvalue value="instance('dimension-interface')"

Page 26 of 162

When MVC becomes a burden for XForms

ref="instance('main')/rectangle/@height"/><!-- We're no longer editing any dimension --><xf:setvalue ref="instance('dimension-interface')/@active"/><!-- Unload the subform --><xf:load show="none" targetid="height"/></xf:action></xf:trigger></xf:group></xf:group><br/><!-- Width --><xf:group ref="@width"><xf:label>Width: </xf:label><xf:group ref=".[instance('dimension-interface')/@active!='width']"><xf:output ref="."/><xf:trigger ref=".[instance('dimension-interface')/@active = '']"><xf:label>Edit</xf:label><xf:action ev:event="DOMActivate"><xf:setvalue ref="instance('dimension-interface')"value="instance('main')/rectangle/@width"/><xf:setvalue ref="instance('dimension-interface')/@active">width</xf:setvalue><xf:load show="embed" targetid="width" resource="subform-embedded.xhtml"/></xf:action></xf:trigger></xf:group><xh:div id="width"/><xf:group ref=".[instance('dimension-interface')/@active='width']"><xf:trigger><xf:label>Done</xf:label><xf:action ev:event="DOMActivate"><xf:setvalue value="instance('dimension-interface')"
ref="instance('main')/rectangle/@width"/><xf:setvalue ref="instance('dimension-interface')/@active"/><xf:load show="none" targetid="width"/></xf:action></xf:trigger></xf:group></xf:group></xf:group></xh:body></xh:html>

Page 27 of 162

When MVC becomes a burden for XForms

And the subform:<xh:div xmlns:xh="http://www.w3.org/1999/xhtml" xmlns:ev="http://www.w3.org/2001/xml-events"
xmlns:xf="http://www.w3.org/2002/xforms"><xf:model id="dimension-model"><xf:instance id="concat"><data/></xf:instance><xf:instance id="split"><height><value/><unit/></height></xf:instance><!-- Get the value from the "interface" instance and initialize the--><xf:submission id="get-dimension-value"
resource="model:master#instance('dimension-interface')/*"
replace="instance" method="get"><xf:action ev:event="xforms-submit-done"><!--<xf:message level="ephemeral">Subform has updated itself.</xf:message>--><xf:setvalue ref="instance('split')/value"
value="translate(instance('concat'), '%incmptxe', '')"/><xf:setvalue ref="instance('split')/unit"
value="translate(instance('concat'), '0123456789', '')"/></xf:action><xf:message ev:event="xforms-submit-error" level="ephemeral">Error while subform update.
</xf:message></xf:submission><xf:send ev:event="xforms-ready" submission="get-dimension-value"/><xf:submission id="set-dimension-value" resource="model:master#instance('dimension-interface')/*"
replace="none" method="post"><xf:action ev:event="xforms-submit-done"><!--<xf:message level="ephemeral">Main form has been updated</xf:message>--></xf:action><xf:message ev:event="xforms-submit-error" level="ephemeral">Error while main form update.
</xf:message></xf:submission></xf:model><xf:group ref="instance('split')"><xf:input ref="value"><xf:action ev:event="xforms-value-changed"><xf:setvalue ref="instance('concat')"value="concat(instance('split')/value, instance('split')/unit)"/><xf:send submission="set-dimension-value"/></xf:action></xf:input><xf:select1 ref="unit"><xf:action ev:event="xforms-value-changed"><xf:setvalue ref="instance('concat')"value="concat(instance('split')/value, instance('split')/unit)"/><xf:send submission="set-dimension-value"/></xf:action><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item><xf:item><xf:label>font size</xf:label><xf:value>em</xf:value></xf:item>

Page 28 of 162

When MVC becomes a burden for XForms<xf:item><xf:label>font height</xf:label><xf:value>ex</xf:value></xf:item><xf:item><xf:label>inches</xf:label><xf:value>in</xf:value></xf:item><xf:item><xf:label>centimeters</xf:label><xf:value>cm</xf:value></xf:item><xf:item><xf:label>millimeters</xf:label><xf:value>mm</xf:value></xf:item><xf:item><xf:label>points</xf:label><xf:value>pt</xf:value></xf:item><xf:item><xf:label>picas</xf:label><xf:value>pc</xf:value></xf:item><xf:item><xf:label>%</xf:label><xf:value>%</xf:value></xf:item></xf:select1></xf:group></xh:div>

The code for defining the subform has the same level of When a user clicks on one of the "Edit" buttons, thecomplexity than the definition of the XBL in Orbeon corresponding subform is loaded (note that all the "Edit"
Forms but a lot of geeky stuff needs to be added around buttons
have
disappeared):the invocation of the form which becomes tricky.
From a user perspective, the page would initially look
like:

Page 29 of 162

When MVC becomes a burden for XForms

Once the user is done editing the values in this subform, 3.4.2. Subforms, XSLTForms flavor(s)he can click on "Done" to come back to a state where
both the height and width are displayed and can be The support of subforms in XSLTForms is illustrated bya sample: a writers.xhtml master form embeds aedited:books.xhtml subform.
The main principle behind this subform
implementation appears to be the same than for
betterFORM but there are some important differences
between these two implementations:• XSLTForms doesn't isolate the models from themaster form and its subform and it is possible toaccess directly to any instance of the master form
from the subforms.
• The features to communicate between models
implemented by betterFORM are thus not necessary
and do not exist in XSLTForms.
• The context node is not isolated and is available
directly from the controls in the subform (see thewriters/books example for an illustration.
•A
specific action (xf:unload) is used to unload
The presentation can be improved replacing for instancesubforms
in XSLTForms while an xf:load action
the buttons by trendy icons but we had to bend our
withan@show="none" attribute is used inrequirements to get something that can be implemented
betterFORM
for the same purpose.
with subforms.
Of course here we are misusing subforms to With these differences, the code developed for
implement components, something which was not a betterFORM could be adapted to work withdesign goal, and it's not surprising that the resulting code XSLTForms as:
is more verbose and that we've had to accept a different
user interface. The future component feature announced
by Joern Turner should solve these glitches.
<?xml-stylesheet href="xsltforms/xsltforms.xsl" type="text/xsl"?><?xsltforms-options debug="yes"?><xh:html xmlns:xh="http://www.w3.org/1999/xhtml"
xmlns:ev="http://www.w3.org/2001/xml-events"
xmlns:xf="http://www.w3.org/2002/xforms"><xh:head><xh:title>Subforms</xh:title><xf:model id="master"><xf:instance id="main"><figures><rectangle height="10in" width="4em"/></figures></xf:instance><!-- Instance used as an "interface" with the subform --><xf:instance id="dimension-interface"><dimension active=""/></xf:instance></xf:model><!-- Dirty hack to style controls inline --><xh:style type="text/css"><![CDATA[.xforms-group-content, .xforms-group, span.xforms-control, .xforms-label {display:inline;}

Page 30 of 162

When MVC becomes a burden for XForms]]></xh:style></xh:head><xh:body><xf:group ref="rectangle"><!-- Height --><xf:group ref="@height"><xf:label>Height: </xf:label><!-- This should be displayed when the subform is not editing the height --><xf:group ref=".[instance('dimension-interface')/@active!='height']"><xf:output ref="."/><!-- Display the trigger when the subform is not loaded anywhere --><xf:trigger ref=".[instance('dimension-interface')/@active = '']"><xf:label>Edit</xf:label><xf:action ev:event="DOMActivate"><!-- Set the value of the interface instance --><xf:setvalue ref="instance('dimension-interface')"value="instance('main')/rectangle/@height"/><!-- Remember that we are editing the height --><xf:setvalue ref="instance('dimension-interface')/@active">height</xf:setvalue><!-- Load the subform --><xf:load show="embed" targetid="height" resource="subform-embedded.xml"/></xf:action></xf:trigger></xf:group><xh:span id="height"/><!-- This should be displayed only when we're editing the height --><xf:group ref=".[instance('dimension-interface')/@active='height']"><xf:trigger><xf:label>Done</xf:label><xf:action ev:event="DOMActivate"><!-- Copy the value from the interface instance --><xf:setvalue value="instance('dimension-interface')"
ref="instance('main')/rectangle/@height"/><!-- We're no longer editing any dimension --><xf:setvalue ref="instance('dimension-interface')/@active"/><!-- Unload the subform --><xf:unload targetid="height"/></xf:action></xf:trigger></xf:group></xf:group><br/><!-- Width --><xf:group ref="@width"><xf:label>Width: </xf:label><xf:group ref=".[instance('dimension-interface')/@active!='width']"><xf:output ref="."/><xf:trigger ref=".[instance('dimension-interface')/@active = '']"><xf:label>Edit</xf:label><xf:action ev:event="DOMActivate"><xf:setvalue ref="instance('dimension-interface')"value="instance('main')/rectangle/@width"/><xf:setvalue ref="instance('dimension-interface')/@active">width</xf:setvalue><xf:load show="embed" targetid="width" resource="subform-embedded.xml"/></xf:action></xf:trigger></xf:group><xh:span id="width"/><xf:group ref=".[instance('dimension-interface')/@active='width']"><xf:trigger>

Page 31 of 162

When MVC becomes a burden for XForms<xf:label>Done</xf:label><xf:action ev:event="DOMActivate"><xf:setvalue value="instance('dimension-interface')"
ref="instance('main')/rectangle/@width"/><xf:setvalue ref="instance('dimension-interface')/@active"/><xf:unload targetid="width"/></xf:action></xf:trigger></xf:group></xf:group></xf:group></xh:body></xh:html>

for the main form and:

<xf:item><xf:label>font size</xf:label><?xml-stylesheet href="xsltforms/xsltforms.xsl"<xf:value>em</xf:value>
type="text/xsl"?></xf:item><?xsltforms-options debug="yes"?><xf:item><xh:html xmlns:xh="http://www.w3.org/1999/xhtml"<xf:label>font height</xf:label>
xmlns:xf="http://www.w3.org/2002/xforms"<xf:value>ex</xf:value>
xmlns:ev="http://www.w3.org/2001/xml-events"></xf:item><xh:head><xf:item><xh:title>A subform</xh:title><xf:label>inches</xf:label><xf:model id="subform-model"><xf:value>in</xf:value><xf:instance id="split"></xf:item><height><xf:item><value/><xf:label>centimeters</xf:label><unit/><xf:value>cm</xf:value></height></xf:item></xf:instance><xf:item><xf:action ev:event="xforms-subform-ready"><xf:label>millimeters</xf:label><xf:setvalue ref="instance('split')/value"<xf:value>mm</xf:value>
value="translate(</xf:item>
instance('dimension-interface'),<xf:item>'%incmptxe', '')"/><xf:label>points</xf:label><xf:setvalue ref="instance('split')/unit"<xf:value>pt</xf:value>
value="translate(</xf:item>
instance('dimension-interface'),<xf:item>'0123456789', '')"/><xf:label>picas</xf:label></xf:action><xf:value>pc</xf:value></xf:model></xf:item></xh:head><xf:item><xh:body><xf:label>%</xf:label><xf:group ref="instance('split')"><xf:value>%</xf:value><xf:input ref="value"></xf:item><xf:label/></xf:select1><xf:setvalue ev:event="xforms-value-changed"</xf:group>
ref="instance('dimension-interface')"</xh:body>value="concat(instance('split')/value,</xh:html>
instance('split')/unit)"/>for the subform.
</xf:input><xf:select1 ref="unit">
Acknowledging that things could be easier,<xf:label/>
XSLTForms has introduced a new experimental feature,<xf:setvalue ev:event="xforms-value-changed" derived from subforms, to implement simple
ref="instance('dimension-interface')"components:
value="concat(instance('split')/value,
instance('split')/unit)"/><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item>

Page 32 of 162

When MVC becomes a burden for XForms

and the subform (or component):
I have implemented a new component control in
XSLTForms. It is named "xf:component" and has<?xml-stylesheet href="xsltforms/xsltforms.xsl"
two attributes named "@ref" and "@resource". Theretype="text/xsl"?>are still restrictions within a component: ids cannot<xh:html xmlns:xh="http://www.w3.org/1999/xhtml"
xmlns:xf="http://www.w3.org/2002/xforms"
be used if the component is to be instantiated more
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
than once. The default instance is local to eachxmlns:ev="http://www.w3.org/2001/xml-events">
instantiated component and the subform-instance()<xh:head>function can be used to get the document element of<xh:title>Size</xh:title>
it. From the main form to the component, a binding<xf:model>
with a special mip named "changed" is defined. The<xf:instance><size>
subform-context() allows to reference the node<value xsi:type="xsd:decimal">2</value>
bound to the component control in the main form.
<unit>cm</unit>
The corresponding build has been committed to</size>repositories:
http://sourceforge.net/p/xsltforms/</xf:instance>code/ci/master/tree/build/<xf:bind ref="subform-instance()/value"changed="translate(subform-context(), '%incmptxe', '')"/>--Alain Couthures on the Xsltforms-support mailing<xf:bind ref="subform-instance()/unit"listchanged="translate(subform-context(), '0123456789', '')"/>
With this new experimental feature and another</xf:model>
extension (the @changed MIP implemented in</xh:head><xh:body>
XSLTForms), the master form would be:<?xml-stylesheet href="xsltforms/xsltforms.xsl"
type="text/xsl"?><?xsltforms-options debug="yes"?><xh:html xmlns:xh="http://www.w3.org/1999/xhtml"
xmlns:ev="http://www.w3.org/2001/xml-events"
xmlns:xf="http://www.w3.org/2002/xforms"><xh:head><xh:title>Subforms</xh:title><xf:model><xf:instance id="main"><figures><rectangle height="10in" width="4em"/></figures></xf:instance></xf:model></xh:head><xh:body><xf:group ref="rectangle"><!-- Height --><xf:group ref="@height"><xf:label>Height: </xf:label><xf:component ref="."resource="component-subform.xml"/></xf:group><br/><!-- Width --><xf:group ref="@width"><xf:label>Width: </xf:label><xf:component ref="."resource="component-subform.xml"/></xf:group></xf:group></xh:body></xh:html><xf:input ref="subform-instance()/value"><xf:label/><xf:setvalue ev:event="xforms-value-changed"
ref="subform-context()"
value="concat(subform-instance()/value,
subform-instance()/unit)"/></xf:input><xf:select1 ref="subform-instance()/unit"><xf:label/><xf:item><xf:label>pixels</xf:label><xf:value>px</xf:value></xf:item><xf:item><xf:label>font size</xf:label><xf:value>em</xf:value></xf:item><xf:item><xf:label>font height</xf:label><xf:value>ex</xf:value></xf:item><xf:item><xf:label>inches</xf:label><xf:value>in</xf:value></xf:item><xf:item><xf:label>centimeters</xf:label><xf:value>cm</xf:value></xf:item><xf:item><xf:label>millimeters</xf:label><xf:value>mm</xf:value></xf:item><xf:item><xf:label>points</xf:label><xf:value>pt</xf:value></xf:item><xf:item>

Page 33 of 162

When MVC becomes a burden for XForms<xf:label>picas</xf:label><xf:value>pc</xf:value></xf:item><xf:item><xf:label>%</xf:label><xf:value>%</xf:value></xf:item><xf:setvalue
ev:event="xforms-value-changed"
ref="subform-context()"
value="concat(subform-instance()/value,
subform-instance()/unit)"/></xf:select1></xh:body></xh:html>

The level of complexity of both the definition of the
subform component and its invocation are similar towhat we've seen with Orbeon's XBL feature. The maindifference is the encapsulation (no encapsulation in
XSLTForms and a controlled encapsulation in Orbeon
Forms which handles the issue of id collisions).
Note that we are escaping the issue caused by id
collision because we are accessing the instance from themaster form directly from the subform using the
subform-context() function. This feature allows us touse only one local instance in the subform and we takecare of not defining any id for this instance and access itusing the subform-instance() function. This trick
wouldn't work if we needed several instances or if we had
to define ids on other elements in the subform.

4. Conclusion
The lack of modularity has been one of the serious
weaknesses in the XForms recommendations so far.
A common solution is to generate or "template" XForms
but this can be tricky when dealing with "components"
used multiple times in a form and especially within
xf:repeat controls.
Different implementation have come up with different
solutions to address this issue (XBL for Orbeon,
subforms for betterFORM and XSLTForms).
The main differences between these solutions are:
• The syntax:• XBL + XForms for Orbeon Forms• XForms with minor extensions for betterFORM
and XSLTForms)

Page 34 of 162• The encapsulation or isolation and features tocommunicate between the component and othermodels:• complete for betterFORM with extensions tocommunicate between models• either complete or partial for Orbeon Forms withextension to communicate between models• no isolation for XSLTForms with extensions toaccess to the context node and default instancefrom a component• The support of id collisions between components andthe main form:• Id collisions are handled by Orbeon Forms• They are forbidden by betterFORM andXSLTForms
The lack of interoperability between these
implementations will probably not be addressed by the
W3C XForms Working Group and it would be very
useful if XForms implementers could work together todefine interoperable solutions to define reusable
components in XForms.
In this paper, generation (or templating) has beenpresented as an alternative to XML or subforms but they
are by no mean exclusive. In real world projects, hybrid
approaches mixing XForms generation (or templating)and components (XBL or subforms) are on the contrary
very valuable. They have been demonstrated in a numberof talks during the pre-conference day at XML Prague.
These hybrid approaches are easy to implement with
common XML toolkits. The generation/templating can
be static (using tools such as XProc, Ant or classical makefiles) or dynamic (using XProc or XPL pipelines or plain
XQuery or XSLT) and Orbeon Forms XBL
implementation even provides a feature to dynamically
invoke a transformation on the content of the boundelement).

4.1. Acknowledgments
I would like to thank Erik Bruchez (Orbeon), Joern
Turner(betterFORM)
and
Alain
Couthures(XSLTForms) for the time they've spent to answer myquestions and review this paper.

XML on the Web: is it still relevant?
O'Neil Delpratt
Saxonica<oneil@saxonica.com>

Abstract
In this paper we discuss what it means by the term XML on
the Web and how this relates to the browser. The success of
XSLT in the browser has so far been underwhelming, andwe examine the reasons for this and consider whether thesituation might change. We describe the capabilities of thefirst XSLT 2.0 processor designed to run within web
browsers, bringing not just the extra capabilities of a newversion of XSLT, but also a new way of thinking about how
XSLT can be used to create interactive client-side
applications. Using this processor we demonstrate as a usecase a technical documentation application, which permits
browsing and searching in a intuitive way. We show its
internals to illustrate how it works.

1. Introduction
The W3C introduced Extensible Markup Language(XML) as a multi-purpose and platform-neutral textbased format, used for storage, transmission andmanipulation of data. Fifteen years later, it has matured
and developers and users use it to represent theircomplex and hierarchically structured data in a variety oftechnologies. Its usage has reached much further than its
creators may have anticipated.
In popular parlance 'XML on the Web' means 'XMLin the browser'. There's a great deal of XML on the web,
but most of it never reaches a browser: it's converted
server-side to HTML using a variety of technologies
ranging from XSLT and XQuery to languages such Java,
C#, PHP and Perl. But since the early days, XML has
been seen as a powerful complement to HTML and as areplacement in the form of XHTML. But why did thisnot take off and revolutionise the web? And could thisyet happen?

doi:10.14337/XMLLondon13.Delpratt01

XML has been very successful, and it's useful to remindourselves why:
• XML can handle both data and documents.
• XML is human-readable (which makes it easy todevelop applications).
• XML handles Unicode.
• XML was supported by all the major industry players
and available on all platforms.
• XML was cheap to implement: lots of free software,
fast learning curve.
• There was a rich selection of complementary
technologies.
• The standard acquired critical mass very quickly, andonce this happens, any technical deficiencies become
unimportant.
However, this success has not been gained in thebrowser. Again, it's a good idea to look at the reasons:• HTML already established as a defacto standard forweb development• The combination of HTML, CSS, and Javascript was
becoming ever more powerful.
• It took a long while before XSLT 1.0 was available ona sufficient range of browsers.
• When XSLT 1.0 did eventually become sufficiently
ubiquitous, the web had moved on ("Web 2.0").
• XML rejected the "be liberal in what you accept"
culture of the browser.
One could look for more specific technical reasons, butthey aren't convincing. Some programmers find the
XSLT learning curve a burden, for example, but there areplenty of technologies with an equally daunting learning
curve that prove successful, provided developers have the
incentive and motivation to put the effort in. Or one
could cite the number of people who encounter
problems with ill-formed or mis-encoded XML, but thatproblem is hardly unique to XML. Debugging Javascriptin the browser, after all, is hardly child's play.

Page 35 of 162

XML on the Web: is it still relevant?

XSLT 1.0 was published in 1999 [1]. The original aim
was that it should be possible to use the language toconvert XML documents to HTML for rendering on thebrowser 'client-side'. This aim has largely been achieved.
Before the specification was finished Microsoft
implemented XSLT 1.0 as an add-on to Internet
Explorer (IE) 4, which became an integral part of IE5.
(Microsoft made a false start by implementing a draft of
the W3C spec that proved significantly different fromthe final Recommendation, which didn't help.) It thentook a long time before XSLT processors with asufficient level of conformance and performance wereavailable across all common browsers. In the first coupleof years the problem was old browsers that didn't haveXSLT support; then the problem became new browsersthat didn't have XSLT support. In the heady days while
Firefox market share was growing exponentially, its
XSLT support was very weak. More recently, somemobile browsers have appeared on the scene with similar
problems.
By the time XSLT 1.0 conformance across browsers
was substiantially achieved (say around 2009), othertechnologies had changed the goals for browser vendors.
The emergence of XSLT 2.0 [2], which made big strides
over XSLT 1.0 in terms of developer productivity, neverattracted any enthusiasm from the browser vendors - andthe browser platforms were sufficiently closed that there
appeared to be little scope for third-party
implementations.
The "Web 2.0" movement was all about changing
the web from supporting read-only documents to
supporting interactive applications. The key component
was AJAX: the X stood for "XML", but Javascript andXML never worked all that well together. DOM
programming is tedious. AJAX suffers from "Impedencemismatch" - it's a bad idea to use programming
languages whose type system doesn't match your data.
That led to what we might call AJAJ - Javascript
programs processing JSON data. Which is fine if yourdata fits the JSON model. But not all data does,
especially documents. JSON has made enormous
headway in making it easier for Javascript programmers
to handle structured data, simply because the data
doesn't need to be converted from one data model toanother. But for many of the things where XML has had
most success - for example, authoring scientific papers
like this one, or capturing narrative and semi-structured
information about people, places, projects, plants, or
poisons - JSON is simply a non-starter.

Page 36 of 162

So the alternative is AXAX - instead of replacing XML
with JSON, replace Javascript with XSLT or XQuery.
The acronym that has caught on is XRX, but AXAX
better captures the relationship with its alternatives. The
key principle of XRX is to use the XML data model andXML-based processing languages end-to-end, and the
key benefit is the same as the "AJAJ" or Javascript-JSONmodel - the data never needs to be converted from one
data model to another. The difference is that this time,
we are dealing with a data model that can handle
narrative text.
A few years ago it seemed likely that XML would go
no further in the browser. The browser vendors had nointerest in developing it further, and the browser
platform was so tightly closed that it wasn't feasible for athird party to tackle. Plug-ins and applets as extension
technologies were largely discredited. But paradoxically,
the browser vendors' investment in Javascript provided
the platform that could change this. Javascript was neverdesigned as a system programming language, or as atarget language for compilers to translate into, but that iswhat it has become, and it does the job surprisingly well.
Above all else, it is astoundingly fast.
Google were one of the first to realise this, andresponded by developing Google Web Toolkit (GWT)[3] as a Java-to-Javascript bridge technology. GWT
allows web applications to be developed in Java (alanguage which in many ways is much better suited forthe task than Javascript) and then cross-compiled toJavascript for execution on the browser. It provides mostof the APIs familiar to Java programmers in otherenvironments, and supplements these with APIs offering
access to the more specific services available in thebrowser world, for example access to the HTML DOM,
the Window object, and user interface events.
Because the Saxon XSLT 2.0 processor is written in
Java, this gave us the opportunity to create a browserbased XSLT 2.0 processor by cutting down Saxon to its
essentials and cross-compiling using GWT.

XML on the Web: is it still relevant?

We realized early on that simply offering XSLT 2.0 was
not enough. Sure, there was a core of people using XSLT
1.0 who would benefit from the extra capability andproductivity of the 2.0 version of the language. But it
was never going to succeed using the old architecturalmodel: generate an HTML page, display it, and walkaway, leaving all the interesting interactive parts of theapplication to be written in Javascript. XRX (or AXAX, if
you prefer) requires XML technologies to be usedthroughout, and that means replacing Javascript not only
for content rendition (much of which can be done withCSS anyway), but more importantly for user interaction.
And it just so happens that the processing model for
handling user interaction is event-based programming,and XSLT is an event-based programming language, so
the opportunities are obvious.
In this paper we examine the first implementation of
XSLT 2.0 on the browser, Saxon-CE [4]. We show how
Saxon-CE can be used as a complement to Javascript,
given its advancements in performance and ease of use.
We also show that Saxon-CE can be used as areplacement of JavaScript. This we show with an
example of a browsing and searching technicaldocumentation.
This is classic XSLT territory, and the requirement istraditionally met by server-side HTML generation, either
in advance at publishing time, or on demand throughservlets or equivalent server-side processing that invoke
XSLT transformations, perhaps with some caching.
While this is good enough for many purposes, it falls
short of what users had already learned to expect fromdesktop help systems, most obviously in the absence of awell-integrated search capability. Even this kind of
application can benefit from Web 2.0 thinking, and wewill show how the user experience can be improved bymoving the XSLT processing to the client side andtaking advantage of some of the new facilities to handle
user interaction.
In our conference paper and talk we will explain theprinciples outlined above, and then illustrate how theseprinciples have been achieved in practice by reference toa live application: we will demonstrate the applicationand show its internals to illustrate how it works.

2. XSLT 2.0 on the browser
In this section we begin with some discussion on theusability of Saxon-CE before we give an overview of its
internals. Saxon-CE has matured significantly since its
first production release (1.0) in June 2012, following on
from two earlier public beta releases. The current release(1.1) is dated February 2013, and the main change isthat the product is now released under an open source
license (Mozilla Public License 2.0).

2.1. Saxon-CE Key Features
Beyond being a conformant and fast implementation of
XSLT 2.0, Saxon-CE has a number of features speciallydesigned for the browser, which we now discuss:
1. Handling JavaScript Events in XSLT: Saxon-CE is not
simply an XSLT 2.0 processor running in thebrowser, doing the kind of things that an XSLT 1.0
processor did, but with more language features(though that in itself is a great step forward). It also
takes XSLT into the world of interactive
programming. With Saxon-CE it's not just a questionof translating XML into HTML-plus-JavaScript andthen doing all the interesting user interaction in the
JavaScript; instead, user input and interaction ishandled directly within the XSLT code. The XSLT
code snippet illustrates the use of event handling:<xsl:template match="p[@class eq 'arrowNone']"mode="ixsl:onclick"><xsl:if test="$usesclick"><xsl:for-each select="$navlist/ul/li"><ixsl:set-attribute name="class"select="'closed'"/></xsl:for-each></xsl:if></xsl:template>XSLT is ideally suited for handling events. It's alanguage whose basic approach is to define rules that
respond to events by constructing XML or HTML
content. It's a natural extension of the language tomake template rules respond to input events rather
than only to parsing events. The functional anddeclarative nature of the language makes it ideallysuited to this role, eliminating many of the bugs that
plague JavaScript development.

Page 37 of 162

XML on the Web: is it still relevant?

2. Working with JavaScript Functions: The code snippets
below illustrates a JavaScript function, which getsdata from an external feed:
var getTwitterTimeline = function(userName){try {return makeRequest(timelineUri + userName);}catch(e) {console.log("Error in getTwitterTimeline: " + e );return "";}};Here is some XSLT code showing how the JavaScript
function can be used; this is a call to the
getTwitterTimeline function in XSLT 2.0 using
Saxon-CE. The XML document returned is thenpassed as a parameter to the a JavaScript API function
ixsl:parse-xml:<xsl:variable name="tw-response"
as="document-node()"select="ixsl:parse-xml(
js:getTwitterTimeline($username))" />3. Animation: The extension instruction ixsl:scheduleaction may be used to achieve animation. The bodyof the instruction must be a single call on<xsl:call-template/>, which is done asynchronously.
If an action is to take place repeatedly, then eachaction should trigger the next by making another callon <ixsl:schedule-action />4. Interactive XSLT: There are a number of Saxon-CE
defined functions and instructions which areavailable. One indispensable useful function is theixsl:page(), which returns the document node of theHTML DOM document. An example of thisfunction's usage is given as follows. Here we retrieve a
div element with a given predicate and bind it to an
XSLT variable:<xsl:variable name="movePiece" as="element(div)"
select="if (exists($piece)) then $pieceelse
id('board',ixsl:page())/div[$moveFrom]/div"/>

In the example below, we show how to set the styleproperty using the extension intruction ixsl:setattribute for a current node in the HTML page.
Here we are changing the display property to 'none',

Page 38 of 162

which hides an element, causing it not to take up any
space on the page:<xsl:if test="position() &gt; $row-size"><ixsl:set-attribute name="style:display"
select="'none'"/></xsl:if>

In the example below we show how we can get theproperty of a JavaScript object by using the ixsl:getfunction:<xsl:variable name="piece" as="element(div)"
select="ixsl:get(ixsl:window(),'dragController.piece')"/>

The full list of the extension functions and extension
instructions in Saxon-CE can be found at thefollowing location: http://www.saxonica.com/ce/userdoc/1.1/index.html#!coding/extensions
andhttp://
www.saxonica.com/ce/user-doc/1.1/index.html#!coding/extension-instructions

2.2. Saxon-CE Internals
In this section we discuss how we build the client-side
XSLT 2.0 processor and how we can invoke it fromJavaScript, XML or HTML. The Java code base wasinherited from Saxon-HE, the successful XSLT 2.0
processor for Java. The product was produced by crosscompiling the Java into optimized, stand-alone
JavaScript files using the GWT 5.2. Although no
detailed performance data is available here, all deliver a
responsiveness which feels perfectly adequate for
production use. The JavaScript runs on all majorbrowsers, as well as on mobile browsers, where JavaScript
can run.
The key achievements in the development of SaxonCE are given below:• The size of the Java source was cut down to around76K lines of Java code. This was mainly achieved by
cutting out unwanted functionality such as XQuery,
updates, serialization, support for JAXP, support forexternal object models such as JDOM and DOM4J,
Java extension functions, and unnecessary options likethe choice between TinyTree and Linked Tree, or the
choice (never in practice exercised) of different sorting
algorithms. Some internal hanges to the code basewere also made to reduce size. Examples include
changes to the XPath parser to use a hybrid
precedence-parsing approach in place of the purerecursive-descent parser used previously; offloading
the data tables used by the normalize-unicode()function into an XML data file to be loaded from theXML on the Web: is it still relevant?

server on the rare occasions that this function is
actually used.
• GWT creates a slightly different JavaScript file foreach major browser, to accommodate browser
variations. Only one of these files is downloaded,
which is based on the browser that is in use. The sizeof the JavaScript file is around 900KB.
• The key benefits of the server-side XSLT 2.0
processor were retained and delivered on the browser.
Saxon has a reputation for conformance, usability,and performance, and it was important to retain this,as well as delivering the much-needed functionality
offered by the language specification. Creating
automated test suites suitable for running in thebrowser environment was a significant challenge.
• Support of JavaScript events. The handling of
JavaScript events changes the scope of Saxon-CE
greatly, meaning it can be used for interactive
application development. Our first attempts tointegrate event handling proved the design of thelanguage extensions was sound, but did not perform
adequately, and the event handling is the final
product was a complete rewrite. The Events arisingfrom the HTML DOM and the client system, which
are understood by GWT, are handled via Saxon-CE.
This proxying of event handling in the Java code
makes it possible for template rules which have amode matching the event to overide the default
behavour of the browser. Events are only collected at
the document node (thus there's only one listener foreach type of event). As a result, the events are bubbled
up from the event target. This mechanism handles the
majority of browser events. There are a few specialist
events like onfocus and onblur which do not operate
at the document node, and these events are best
handled in JavaScript first. GWT provides relatively
poor support for these events because their behaviour
is not consistent across different browsers.
• Interoperability with JavaScript. Many simple
applications can be developed with no user-written
Javascript. Equally, where Javascript skills orcomponents are available, it is possible to make use ofthem, and when external services are avalable only via
Javascript interfaces, Saxon-CE stylesheets can still
make use of them.
Figure 1 illustrates the input and output components
involved in building the XSLT 2.0 processor, Saxon-CE:

Figure 1. Saxon-CE Development

Static view of the Saxon-CE product and components
involved in the build process
As shown in Figure 1 we use GWT to cross-compile the
XSLT 2.0 processor. Saxon-HE and GWT libraries are
input to this process. In addition, we write the JavaScript
API methods in Java using the JavaScript Native
Interface (JSNI), which is a GWT feature. This feature
proved useful because it provided access to the low-levelbrowser functionality not exposed by the standard GWTAPIs. This in effect provides the interface for passing andreturning JavaScript objects to and from the XSLT
processor.
The output from this process is Saxon-CE, which
comprises of the XSLT 2.0 processor and the stub file,both in highly compressed and obfuscated JavaScript.
GWT provides separate JavaScript files for each majorbrowser. User JavaScript code can happily run alongside
the XSLT processor.
The invoking of Saxon-CE is achieved in several ways.
The first method employs a standard<?xml-stylesheet?> processing-instruction in the prologof an XML document. This cannot be used to invoke
Saxon-CE directly, because the browser knows nothingof Saxon-CE's existence. Instead, however, it can be used
to load an XSLT 1.0 bootstrap stylesheet, which in turncauses Saxon-CE to be loaded. This provides the easiest
upgrade from existing XSLT 1.0 applications. The code

Page 39 of 162

XML on the Web: is it still relevant?

We discussed earlier that the JavaScript API provides an
API with a rich set of features for interfacing andinvoking the XSLT processor when developing Saxon<?xml-stylesheet type="text/xsl"
CE applications. There are three JavaScript API Sections
href="sample.boot.xsl"?><dt:data-set xmlns:dt="urn:system.logging.data.xml"> available: The Command, which is designed to be used asa JavaScript literal object and effectively wraps the<dt:rows name="test-data">
Saxon-CE API with a set of properties so you can run an...
</dt:data-set>
XSLT transform on an HTML page in a more
The XSLT 1.0 bootstrap stylesheet is given below. It declarative way; the Saxon object, which is a static object,generates an HTML page containing instructions to load providing a set of utility functions for working with the
XSLT processor, initiating a simple XSLT function, andSaxon-CE and execute the real XSLT 2.0 stylesheet:working with XML resources and configuration; and the<xsl:transformXSLT20Processor, which is modeled on the JavaScript
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
XSLTProcessor API as implemented by the major
version="1.0">
browsers. It provides a set of methods used to initiate
XSLT transforms on XML or direct XSLT-based HTML
<xsl:output method="html" indent="no"/>updates.
<xsl:template match="/">
The code snippet below shows a Command API call
<html><head>
to run a XSLT transform. Here the stylesheet is declared<meta http-equiv="Content-Type"as ChessGame.xsl and the initial template is defined ascontent="text/html" />main. We observed the logLevel as been set to<script type="text/javascript"'SEVERE'. Saxon-CE provides a debug version which
language="javascript"
ouputsuseful log messages to the JavaScript console,
src="../Saxonce/Saxonce.nocache.js"/>
accessable in the browser development tools:
snippet below illustrates the bootstrap process of the
XSLT 2.0 processor:<script>var onSaxonLoad = function() {Saxon.run( {source:location.href,
logLevel:"SEVERE",
stylesheet: "sample.xsl"});}</script></head><!-- these elements are required also --><body><p></p></body></html></xsl:template></xsl:transform>

The second method involves use of the script element in
HTML. In fact there are two script elements: one with
type="text/javascript" which causes the Saxon-CE engine
to be loaded, and the other with type="application/xslt+xml" which loads the stylesheet itself, as shown here:<script type="application/xslt+xml"
language="xslt2.0" src="books.xsl"
data-source="books.xml"></script>

The third method is to use an API from Javascript. The
API is modelled on the XSLTProcessor API provided by
the major browsers for XSLT 1.0.

Page 40 of 162

var onSaxonLoad = function() {proc = Saxon.run( {stylesheet:'ChessGame.xsl',
initialTemplate: 'main',logLevel:'SEVERE'} );};XML on the Web: is it still relevant?

3. Use Case: Technicaldocumentation application
We now examine a Saxon-CE driven application used forbrowsing technical documentation in a intuative
manner: specifically, it is used for display of the Saxon
9.5 documentation on the Saxonica web site. The
application is designed to operate as a desktop
application, but on the web.
The documentation for Saxon 9.5 can be found at:• http://www.saxonica.com/documentation/index.html
When you click on this link for the first time, there will
be a delay of a few seconds, with a comfort message
telling you that Saxon is loading the documentation.
This is not strictly accurate; what is actually happening isthat Saxon-CE itself is being downloaded from the website. This only happens once; thereafter it will be pickedup from the browser cache. However, it is remarkable
how fast this happens even the first time, consideringthat the browser is downloading the entire Saxon-CE
product (900Kb of Javascript source code generated fromaround 76K lines of Java), compiling this, and thenexecuting it before it can even start compiling andexecuting the XSLT code.

The application consists of a number of XML documents
representing the content data, ten XSLT 2.0 modules, aJavascript file, several other files (CSS file, icon andimage files) and a single skeleton HTML webpage; theinvariant parts of the display are recorded directly inHTML markup, and the variable parts are marked byempty <div> elements whose content is controlled fromthe XSLT stylesheets. Development with Saxon-CE
often eliminates the need for Javascript, but at the sametime it happily can be mixed with calls from XSLT. In
this case it was useful to abstract certain JavaScript
functions used by the Saxon-CE XSLT transforms.
Key to this application is that the documentation
content data are all stored as XML files. Even the linkage
of files to the application is achieved by a XML file called
catalog.xml: this is a special file used by the XSLT torender the table of contents. The separation of thecontent data from the user interface means that changes
to the design can be done seamlessly without modifiying
the content, and vice versa.

3.1. Architecture
The architecture of the technical documentation
application is shown in Figure 2:
Figure 2. Architecture of Technical Documentation application

Architectural view of a Saxon-CE application

Page 41 of 162

XML on the Web: is it still relevant?

The documentation is presented in the form of a singlepage web site. The screenshot in Figure 3 shows its
appearance.
Figure 3. Technical documentation application in the browser

Screen-shot of the Technical documentation in the browser using Saxon-CE
Note the following features, called out on the diagram.
We will discuss below how these are implemented in
Saxon-CE.
1. The fragment identifier in the URL
2. Table of contents
3. Search box
4. Breadcrumbs
5. Links to Javadoc definitions
6. Links to other pages in the documentation
7. The up/down buttons

3.2. XML on the ServerThis application has no server-side logic; everything on
the server is static content.
On the server, the content is held as a set of XML
files. Because the content is fairly substantial (2Mb of
XML, excluding the Javadoc, which is discussed later),it's not held as a single XML document, but as a set of a
20 or so documents, one per chapter. On initial loading,
we load only the first chapter, plus a small catalogue
document listing the other chapters; subsequent chapters
are fetched on demand, when first referenced, or when
the user does a search.
Our first idea was to hold the XML in DocBook form,
and use a customization of the DocBook stylesheets toPage 42 of 162

present the content in Saxon-CE. This proved infeasible:
the DocBook stylesheets are so large that downloadingthem and compiling them gave unacceptableperformance. In fact, when we looked at the XML
vocabulary we were actually using for the
documentation, it needed only a tiny subset of what
DocBook offered. We thought of defining a DocBook
subset, but then we realised that all the elements we wereusing could be easily represented in HTML5 without
any serious tag abuse (the content that appears inhighlighted boxes, for example, is tagged as an <aside>).
So the format we are using for the XML is in fact
XHTML 5. This has a couple of immediate benefits: itmeans we can use the HTML DOM in the browser tohold the information (rather than the XML DOM), andit means that every element in our source content has adefault rendition in the browser, which in many cases(with a little help from CSS) is quite adequate for our
purposes.

XML on the Web: is it still relevant?

Although XHTML 5 is used for the narrative part of the
documentation, more specialized formats are used for theparts that have more structure. In particular, there is an
XML document containing a catalog of XPath functions(both standard W3C functions, and Saxon-specificextension functions) which is held in a custom XML
vocabulary; and the documentation also includes full
Javadoc API specifications for the Saxon code base. Thiswas produced from the Java source code using thestandard Javadoc utility along with a custom "doclet"(user hook) causing it to generate XML rather thanHTML. The Javadoc in XML format is then rendered by
the client-side stylesheets in a similar way to the rest of
the documentation, allowing functionality such as
searching to be fully integrated. For the .NET API, wewrote our own equivalent to Javadoc to produce classand method specifications in the same XML format.
The fact that XHTML is used as the delivered
documentation format does not mean, of course, that theclient-side stylesheet has no work to do. This will
become clear when we look at the implementation of thevarious features of the user interaction. A typical XML
file fragment is shown below:<article id="changes"
title="Changes in this Release"><h1>Version 9.4 (2011-12-09)</h1><p>Details of changes in Saxon 9.4 are detailedon the following pages:</p>
<nav>
<ul/>
</nav><section id="bytecode-94"
title="Bytecode generation"><h1>Bytecode generation</h1><p>Saxon-EE 9.4 selectively compiles
stylesheets and queries into Java bytecode
before execution.</p>...

For the most part, the content of the site is authoreddirectly in the form in which it is held on the site, using
an XML editor. The work carried out at publishing time
consists largely of validation. There are a couple ofexceptions to this: the Javadoc content is generated by atool from the Java source code, and we also generate an
HTML copy of the site as a fallback for use from devices
that are not Javascript-enabled. There appears to be little
call for this, however: the client-side Saxon-CE version of
the site appears to give acceptable results to the vast
majority of users, over a wide range of devices. Authoring
the site in its final delivered format greatly simplifies theprocess of making quick corrections when errors arefound, something we have generally not attempted to doin the past, when republishing the site was a major
undertaking.

3.3. The User Interface
In this section we discuss the user interface of the
documentation application. The rendition of the
webpages is done dynamically, almost entirely in XSLT
2.0. There are a few instances were we rely on helper
functions (amounting to about 50 lines) of JavaScript.
The XSLT is in 8 modules totalling around 2500 lines ofcode. The Javascript code is mainly concerned withscrolling a page to a selected position, which in turn isused mainly in support of the search function, discussed
in more detail below.
3.3.1. The URI and Fragment Identifier
URIs follow the "hashbang" convention: a page might
appear in the browser as:• http://www.saxonica.com/documentation/index.html#!
configuration
For some background on the hashbang convention, andan analysis of its benefits and drawbacks, see Jeni
Tennison's article at [5]. From our point of view, themain characteristics are:
• Navigation within the site (that is, between pages of
the Saxon documentation) doesn't require going backto the server on every click.
• Each sub-page of the site has a distinct URI that canbe used externally; for example it can be bookmarked,
it can be copied from the browser address bar into an
email message, and so on. When a URI containing
such a fragment identifier is loaded into the browser
address bar, the containing HTML page is loaded,
Saxon-CE is activated, and the stylesheet logic thenensures that the requested sub-page is displayed.
• It becomes possible to search within the site, without
installing specialized software on the server.

Page 43 of 162

XML on the Web: is it still relevant?

• The hashbang convention is understood by search Some of this logic can be seen in the following templateengines, allowing the content of a sub-page to be rule:indexed and reflected in search results as if it were an<xsl:template match="*" mode="handle-itemclick">ordinary static HTML page.
<xsl:variable name="ids"
The XSLT stylesheet supports use of hashbang URIs in
select="(., ancestor::li)/@id"
two main ways: when a URI is entered in the address
as="xs:string*"/><xsl:variable name="new-hash"bar, the stylesheet navigates to the selected sub-page; andselect="string-join($ids, '/')"/>
when a sub-page is selected in any other way (for<xsl:variable name="isSpan"
example by following a link or performing a search), theselect="@class eq 'item'"relevant hashbang URI is constructed and displayed in
as="xs:boolean"/>the address bar.
<xsl:for-each select="if ($isSpan) then ..
The fragment identifiers used for the Saxon
else ."><xsl:choose>
documentation are hierarchic; an example is<xsl:when test="@class eq 'open'• #!schema-processing/validation-api/schema-jaxp
and not($isSpan)">
The first component is the name of the chapter, and<ixsl:set-attribute name="class"corresponds to the name of one of the XML files on theselect="'closed'"/></xsl:when>server, in this case schema-processing.xml. The<xsl:otherwise>
subsequent components are the values of id attributes of<xsl:sequence select="js:disableScroll()"/>
nested XHTML 5 <section> elements within that XML<xsl:choose>
file. Parsing the URI and finding the relevant subsection<xsl:when test="f:get-hash() eq
is therefore a simple task for the stylesheet.
$new-hash">
3.3.2. The Table of Contents
The table of contents shown in the left-hand column ofthe browser screen is constructed automatically, and thecurrently displayed section is automatically expanded andcontracted to show its subsections. Clicking on an entry
in the table of contents causes the relevant content toappear in the right-hand section of the displayed page,
and also causes the subsections of that section (if any) toappear in the table of contents. Further side-effects arethat the URI displayed in the address bar changes, andthe list of breadcrumbs is updated.

Page 44 of 162<xsl:variable name="new-class"select="f:get-open-class(@class)"/><ixsl:set-attribute name="class"select="$new-class"/><xsl:if test="empty(ul)"><xsl:call-template
name="process-hashchange"/></xsl:if></xsl:when><xsl:otherwise><xsl:sequence
select="f:set-hash($new-hash)"/></xsl:otherwise></xsl:choose></xsl:otherwise></xsl:choose></xsl:for-each></xsl:template>XML on the Web: is it still relevant?

Most of this code is standard XSLT 2.0. A feature
particular to Saxon-CE is the ixsl:set-attributeinstruction, which modifies the value of an attribute in
the HTML DOM. To preserve the functional nature of
the XSLT language, this works in the same way as the
XQuery Update Facility: changes are written to apending update list, and updates on this list are applied
to the HTML DOM at the end of a transformation
phase. Each transformation phase therefore remains, to adegree, side-effect free. Like the xsl:result-document
instruction, however, ixsl:set-attribute delivers noresult and is executed only for its external effects; ittherefore needs some special attention by the optimizer.
In this example, which is not untypical, the instruction isused to change the classattribute of an element in theHTML DOM, which has the effect of changing its
appearance on the screen.
The code invokes a function f:set-hash which looks likethis:
<xsl:function name="f:set-hash"><xsl:param name="hash"/><ixsl:set-property name="location.hash"select="concat('!',$hash)"/></xsl:function>

This has the side-effect of changing the contents of thelocation.hash property of the browser window, that is,

the fragment identifier of the displayed URI. Changing
this property also causes the browser to automatically
update the browsing history, which means that the backand forward buttons in the browser do the right thing
without any special effort by the application.
3.3.3. The Search Box

The search box provides a simple facility to search theentire documentation for keywords. Linguistically it iscrude (there is no intelligent stemming or searching forsynonyms or related terms), but nevertheless it can behighly effective. Again this is implemented entirely inclient-side XSLT.

The initial event handling for a search request is
performed by the following XSLT template rules:<xsl:template match="p[@class eq 'search']"mode="ixsl:onclick"><xsl:if test="$usesclick"><xsl:call-template name="run-search"/></xsl:if></xsl:template><xsl:template match="p[@class eq 'search']"mode="ixsl:ontouchend"><xsl:call-template name="run-search"/></xsl:template><xsl:template name="run-search"><xsl:variable name="text"
select="normalize-space(ixsl:get($navlist/div/input,'value'))"/><xsl:if test="string-length($text) gt 0"><xsl:for-eachselect="$navlist/../div[@class eq 'found']"><ixsl:set-attribute name="style:display"
select="'block'"/></xsl:for-each><xsl:result-document href="#findstatus"
method="replace-content">
searching...
</xsl:result-document><ixsl:schedule-action wait="16"><xsl:call-template name="check-text"/></ixsl:schedule-action></xsl:if></xsl:template>

The existence of two template rules, one responding to
an onclick event, and one to ontouchend, is due todifferences between browsers and devices; the Javascript
event model, which Saxon-CE inherits, does not always
abstract away all the details, and this is becoming
particularly true as the variety of mobile devices
increases.
The use of ixsl:schedule-action here is not so much toforce a delay, as to cause the search to proceedasynchronously. This ensures that the browser remains
responsive to user input while the search is inprogress.The template check-text, which is called fromthis code, performs various actions, one of which is toinitiate the actual search. This is done by means of aPage 45 of 162

XML on the Web: is it still relevant?

recursive template, shown below, which returns a list ofpaths to locations containing the search term:

3.3.5. Javadoc Definitions

As mentioned earlier, the Javadoc content is handled alittle differently from the rest of the site.
This section actually accounts for the largest part ofthe content: some 11Mb, compared with under 2Mb forthe narrative text. It is organized on the server as one
XML document per Java package; within the package theXML vocabulary reflects the contents of a package interms of classes, which contains constructors andmethods, which in turn contain multiple arguments. The
XML vocabulary reflects this logical structure rather thanbeing pre-rendered into HTML. The conversion toHTML is all handled by one of the Saxon-CE stylesheet
modules.
Links to Java classes from the narrative part of the
documentation are marked up with a special classattribute,forexample<aclass="javalink"
href="net.sf.saxon.Configuration">Configuration</a>.
A special template rule detects the onclick event for such
links, and constructs the appropriate hashbang fragment
identifier from its knowledge of the content hierarchy;
the display of the content then largely uses the same logic
This list of paths is then used in various ways: the as the display of any other page.
sections containing selected terms are highlighted in thetable of contents, and a browsable list of hits is available, 3.3.6. Links between Sub-Pages in theallowing the user to scroll through all the hits. Within Documentation
the page text, search terms are highlighted, and the page
scrolls automatically to a position where the hits are Within the XML content representing narrative text,
visible (this part of the logic is performed with the aid of links are represented using conventional relative URIs in
the form <a class="bodylink" href="../../
small Javascript functions).
extensions11/saxon.message">saxon:message</a>. This
"relative URI" applies, of course, to the hierarchic
3.3.4. Breadcrumbs
identifiers used in the hashbang fragment identifier used
In a horizontal bar above the table of contents and the to identify the subpages within the site, and the clickcurrent page display, the application displays a list of events for these links are therefore handled by the Saxon"breadcrumbs", representing the titles of the chapters/ CE application.
sections in the hierarchy of the current page. (The name The Saxon-CE stylesheet contains a built-in link checker.
derives from the story told by Jerome K. Jerome of how There is a variant of the HTML page used to gain accessthe Three Men in a Boat laid a trail of breadcrumbs to to the site for use by site administrators; this displays a
avoid getting lost in the Hampton Court maze; the idea button which activates a check that all internal links haveis to help the user know how to get back to a known a defined target. The check runs in about 30 seconds,
place.)and displays a list of all dangling references.
Maintaining this list is a very simple task for thestylesheet; whenever a new page is displayed, the list can 3.3.7. The Up/Down buttons
be reconstructed by searching the ancestor sections anddisplaying their titles. Each entry in the breadcrumb list These two buttons allow sequential reading of theis a clickable link, which although it is displayed narrative text: clicking the down arrow navigates to thedifferently from other links, is processed in exactly the next page in sequence, regardless of the hierarchic
structure, while the up button navigates to the previous
same way when a click event occurs.
page.
<xsl:template match="section|article"
mode="check-text"><xsl:param name="search"/><xsl:param name="path" as="xs:string"
select="''"/><xsl:variable name="newpath"select="concat($path, '/', @id)"/><xsl:variable name="text" select="lower-case(string-join(*[not(local-name() = ('section','article'))],'!'))"/><xsl:sequence
select="if (contains($text, $search)) thensubstring($newpath,2)
else ()"/><xsl:apply-templates mode="check-text"
select="section|article"><xsl:with-param name="search"select="$search"/><xsl:with-param name="path"select="$newpath"/></xsl:apply-templates></xsl:template>

Page 46 of 162

XML on the Web: is it still relevant?

Ignoring complications caused when navigating in the Can this transform the fortunes of XML on the Web?
sections of the site that handle functions and Javadoc It's hard to say. We are in an industry that is surprisingly
specifications, the logic for these buttons is:
influenced by fashion, and that is nowhere more true
than among the community of so-called "web<xsl:template name="navpage">developers". The culture of this community is in many
<xsl:param name="class" as="xs:string"/>
ways
more akin to the culture of television and film
<xsl:variable name="ids"production than the culture of software engineering, andselect="tokenize(f:get-hash(),'/')"/>
the delivered effect is more important than the<xsl:variable name="c" as="node()"select="f:get-item(
technology used to achieve it. The costs related to$ids, f:get-first-item($ids[1]), 1
content creation may in some cases swamp the software)"/>development costs, and many of the developers may<xsl:variable name="new-li"
regardthemselves as artists rather than engineers. This isselect="
not therefore fertile territory for XML and XSLT withif ($class eq 'arrowUp') then
their engineering focus.
($c/preceding::li[1] union$c/parent::ul/parent::li)[last()]Nevertheless, there is a vast amount of XML inelse ( $c/ul/li union $c/following::li)[1]"/>existence and more being created all the time, and there<xsl:variable name="push"
are projects putting a great deal of effort into rendering
select="that XML for display in browsers. In many cases, thestring-join(($new-li/ancestor::li union
developers on those projects are already enthusiastic$new-li)/@id,'/')"/>
about XSLT (and sometimes very negative about<xsl:sequence select="f:set-hash($push)"/></xsl:template>learning to write in Javascript). It is perhaps to thiscommunitythat we should look for leadership. They
Here, the first step is to tokenize the path contained in
won't
conviceeveryone, but a few conspicuous successes
the fragment identifier of the current URL (variable
will
go
a
long
way. And perhaps this will also remind$ids). Then the variable $c is computed, as the relevant
peoplethat
there
is a vast amount of XML on the web;entry in the table of contents, which is structured as a
it's
just
that
most
of it never finds its way off the web
nested hierarchy of ul and li elements. The variable$new-li is set to the previous or following li element in and into the browser.
the table of contents, depending on which button waspressed, and $push is set to a path containing the 5. Acknowledgement
identifier of this item concatenated with the identifiers of
its ancestors. Finally f:set-hash() is called to reset the Many thanks to Michael Kay who initiated the Saxonbrowser URL to select the subpage with this fragment CE project and reviewed this paper. Also thanks to
identifier.
Philip Fearon for his contribution in the development of
the Saxon-CE project, most notably the design of theJavascript event handling and the automated test harness.
4. Conclusion
He also wrote most of the Saxon-CE documentation
In this paper we have shown how Saxon-CE was viewer described in this paper.

constructed, with the help of Google's GWT technology,
as a cross-browser implementation of XSLT 2.0,
performing not just XML-to-HTML rendition, but also
supporting the development of richly interactive clientside applications. Looking at the example application
shown, there are clear benefits to writing this in XSLT
rather than Javascript.

Page 47 of 162

XML on the Web: is it still relevant?

References[1] XSL Transformations (XSLT) Version 1.0. W3C Recommendation. 16 November 1999. James Clark. W3C.
http://www.w3.org/TR/xslt[2] XSL Transformations (XSLT) Version 2.0. W3C Recommendation. 23 January 2007. Michael Kay. W3C.
http://www.w3.org/TR/xslt20[3] Google Web Toolkit (GWT). Google.
http://code.google.com/webtoolkit/[4] The Saxon XSLT and XQuery Processor. Michael Kay. Saxonica.
http://www.saxonica.com/[5] Hash URIs. Jeni Tennison.
http://www.jenitennison.com/blog/node/154

Page 48 of 162

Practice what we PreachTomos Hillman
Oxford University Press<tomos.hillman@oup.com>

Richard Pineger
Tech Doc Direct Limited<richard.pineger@techdocdirect.com>

2. Challenges caused by theThis case study describes how we use in-house XML skills instructions
Abstract

and the DITA XML semantic architecture to present highvolumes of many-layered technical instructions to the To reiterate, we had multiple documents with a
confusing document hierarchy and precedence oftypesetters who capture our XML for publishing.
instructions which typesetters needed to relate to a
separately maintained hierarchy of DTDs before they
1. Capturing variety in XML
could press the first key on a manuscript XML file.
Our maintenance of the instructions was complex
OUP content types include academic monographs, with multiple Data Engineers accessing several storage
textbooks, encyclopediae, journals, reported law cases locations in MS SharePoint to retrieve,version, and trackand legislation. They also include bilingual and the documents. So the web of documents lacked
monolingual dictionaries, but we're not covering those consistency, contained conflicting instructions, andtoday.
where instructions were replicated in more than one
We capture each of these content types to a suite of place, caused partial updates to instructions.
our own modularised DTD data models. The content
It's fair to say that the audience, the typesetters, werearchitecture used for the DTDs allows for abstraction confused. The documents didn't always even target themand re-use of content and processes that use XML for as the audience. Many instructions carried descriptive
publishing.
passages that were only useful for Data Engineers andContent is captured to XML from manuscripts and DTD maintenance, leaving the capturer scratching their(retrospectively) from our back catalogue by typesetters, head.
and these typesetters need instructions.
Also the language used had started out as passive and
OUPs advantage in the publishing world is the uncertain, "sometimes, normally, generally, should,
quality of our academic authors; however, this means would, might" were used in virtually every sentence.
that we must accept what is sometimes bespoke content Ironically, this use of language was at least reasonably
with a disorganised structure with, for instance, special consistent as generations of Data Engineers fitted withfeatures like custom marginalia and themed boxes.
each other's style. But it didn't help the typesetter to
The original instructions, called Text Capture know that a structure "should be" captured andInstructions (TCIs) were written as a suite of 20-30 sometimes left them thinking, "who by?"Word documents, some specific to products and titles,
Worse still, for Data Engineers used to precision andothers specific to core DTD structures that are certainty, there really was no effective way of automatingreferenced from the product specific documents. In QA of updates to the instructions.
short, complex, confusing and difficult to maintain andA team of XML engineers using MS Word? It nevernavigate.
made sense... it was time to fix this so we hired an XML
technical author... Richard

doi:10.14337/XMLLondon13.Hillman01

Page 49 of 162

Practice what we Preach

3. The DITA XML solution
Technical writing changed in the last 30 years. The web
and related technologies like online help means that we
no longer want to lock our content into long linear
documents as promoted by MS Word or even, dare I say
it, DocBook XML. We want the freedom to collaborateand have team members each create small chunks of
information like you see in help systems and on websites.
Sure, many customers still expect a nice manual with
their product or instrument, but we can build those outof chunks. The new kid in town is called “component
content management” and when you combine it with“single-sourcing” - the ability to publish to multiple
channels, you have solutions that are infinitely more
flexible and open to automation. Practically speaking,this means keeping a database of XML topics, mixing
and matching them into books, websites, help systems
and publishing them through XSL.
Conveniently, and separately, “single-sourcing” isalso the track that the publishing industry has been
pursuing over the same time frame.
In the technical communications world, one solution
came out of IBM 10 or 12 years ago in the form of a setof DTDs called the Darwinian Information Typing
Architecture (DITA) [1].
IBM, at one time the largest publisher in the world,
had made great progress with the study of what
constituted quality technical information, and thecapture of semantic information with the content. Whatthey needed was a new scheme for capturing thismodular, semantic content, tying it together andpublishing to multiple channels. They also needed to do
this in an extensible way that meant they could capture
new semantic content in a dynamic way, that is, capture
the semantic immediately and configure the global teams
and publishing engines later. Capture the right content
and the rest can follow. An approach that saved themmillions - I'm sure Tom can verify the effort required tochange a single attribute on a traditional publishing
DTD. And that effort obviously translates to cost; andfor global organizations, that can represent great deal of
cost.

Page 50 of 162

But DITA is more than just a single-sourcing solution, it
is a methodology, a paradigm, an ethos, that promotes
quality in technical communications. It focusses on theaudience and their goals. It promotes minimalism andclear thinking (and therefore clear writing) by removing;the “shoulds”, the “woulds”, unnecessary tense changes,
passive constructions, and superfluous clutter. It
structures information around tasks and tells the reader
what to do instead of building knowledge so that they canwork it out for themselves. It provides the “minimum
effective dose” and no more, which makes it efficient touse although it can take longer to write. As Blaise Pascal
explained (translated from the original French), “I'msorry for the length of this letter, I didn't have time towrite a shorter one.”
So, hopefully you're now sold on the idea of DITA
XML and want to investigate further. Fortunately, youcan. IBM open-sourced the whole scheme in about
2001, including DTDs and an Open Toolkit publishingengine, DITA-OT [2].
My project at the OUP can be described simply: I
converted their instructions into DITA XML in threepasses over 4 months.
• First I rewrote the headings in the Word documents
and tagged all the semantic content using MS Word
styles• Next I used Eliot Kimber's excellent word2dita
conversion from the DITA4Publishers Open Toolkit
plug-in [3]. The word2dita XSL converts MS OfficeOpen XML (OOXML) into a ditamap and dita
topics by referencing an XML mapping file. It createsthe semantic nesting that is absent in office
documents and can even capture complex constructs
such as definition lists and images with captions
within figures• Finally, I worked through in the DITA XML
semantic markup and combined and rewrote the2000+ raw topics into 1300 focussed, definite
instructions for typesetters.
Each topic is stored in an XML file with theextension .dita and these are tied together into apublication using a ditamap XML file. The topic XML
structure is very simple and contains only semantic
elements with no layout or style information. The map
XML structure is very simple allowing relatively easy
manipulation of maps and referenced topics with XSL.
Obviously I worked with Tom through the project toautomate global changes and incorporate as many data
integration features as we could. Tom will tell you aboutsome of those now.

Practice what we Preach

4. Further benefits from XML
technologies
Writing and validating XML examples in XML
Since there are so many potential ways in which we canleverage our team's XML expertise with DITA, we leavecompiling a complete list (if such a thing is possible!) as
an exercise for the reader. Examples range from
improvements at quite a low level (e.g. version control
on topics through SVN keywords) to sophisticated XML
processing (e.g. XSLT transforms or delivery options for
global changes).
Instead we will concentrate on one particular issue:
example instances of XML.
The particular problem is that an example will
normally include contextual mark-up that isn't directly
illustrating the main subject. If the data-model or
capture rules change, it's difficult or impossible to findevery instance of the change in all examples across theentire data set.
A secondary issue was identified in the way in which thecode examples are written. Styling the examples byadding @outputclass attribute values to every angled
bracket, attribute name or value, processing instruction
etc. was an obvious time sink.

Our proposed solution was to write the examples in
XML, validate them using NVDL [4], then convert
them into a DITA shared resource where each example
can be referenced and reused dynamically.

4.1. Combining example document types with
NVDL
I created a schema for a wrapper XML format to containmultiple examples from the different OUP datamodels.
Creating the schema had some challenges because of
OUP's current use of DTDs for our datamodels:
1. OUP DTDs have limited namespace support
2. The editor's implementation of NVDL doesn't
support DTDs
These were overcome:1. By use of NVDL context to change mode based onelement name in the wrapper XML format
2. By converting DTDs to schema.
NVDL allows separate validation at the level of thewrapper format as well as the contained OUP XML
examples:• Validation in the wrapper layer ensures higher levelaspects such as the uniqueness of id attributes.
• Validation in the contained layer ensures that eachexample conforms to the relevant data-model.

Example 1. XML Examples Sample
1 <?xml version="1.0" encoding="UTF-8"?>
2 <?xml-model href="Examples/examples.nvdl" type="application/xml" 1
3
schematypens="http://purl.oclc.org/dsdl/nvdl/ns/structure/1.0"?>
4 <x:examples xmlns:x="http://schema.oup.com/dita/examples">5<x:example id="OUP_Law_Reports_431_singapore01"> 2
6<x:OxDTD><disclaimer> 3
7<p>Decision © Singapore Academy of Law under exclusive licence from the Government of Singapore.
8
The Academy reserves all rights in the content, which may not be reproduced without the9
Academy's written permission.</p>
10 </disclaimer></x:OxDTD>
11</x:example>
12 </x:examples>Line 2: associates the NVDL schema
2 Line 5: includes the unique ID which will be used by DITA as a key.
3 Line 6: the OxDTD element is used by the NVDL engine to choose which validation to apply.
1

Page 51 of 162

Practice what we Preach

Example 2. NVDL Sample
1 <rules2
xmlns="http://purl.oclc.org/dsdl/nvdl/ns/structure/1.0"
3
xmlns:a="http://relaxng.org/ns/compatibility/annotations/1.0"
4
xmlns:x="http://schema.oup.com/dita/examples"
xmlns:sch="http://www.ascc.net/xml/schematron"
5
6
startMode="examples"
7><mode name="examples">89<namespace ns="http://schema.oup.com/dita/examples"><validate schema="examples.rng"> 11011<context path="OxDTD"> 2
12<mode><anyNamespace>
13
14<validate schema="schema/OxDTD/OxDTD.xsd"/> 3
15<attach useMode="unwrap"/>
16</anyNamespace></mode>17
18</context></validate>19
20<validate schema="examples.sch"> 121<context path="OxDTD" useMode="allow"/> 4
22</validate>23</namespace>24<anyNamespace><reject/>
25</anyNamespace>
26
27</mode>28<mode name="unwrap"><namespace ns="http://schema.oup.com/dita/examples">29<unwrap/>30</namespace>31
32</mode>33<mode name="allow"><anyNamespace>34<allow/>
35
36</anyNamespace>
37</mode>38 </rules>

Lines 10, 20: The wrapper format is defined as the RelaxNG schema 'examples.rng' and a schematron file
'examples.sch'. The former defines the model, the latter enforces ID uniqueness.
2 Line 11: defines a change of context when the OxDTD element is encountered
3 Line 14: defines which data-model to use to validate the contained XML
4 Line 21: excludes the contained XML from the schematron validation

1

4.2. Code highlighting in DITA XML using
XSLT
DITA doesn't contain semantic elements for all theconstituent parts of an XML example although it does
include phrase ph elements and an outputclass attributethat transforms to a CSS class in all the HTML-based
outputs.
A (remarkably simple) XSL script converts fromwrapper format to a dita topic with shared codehighlighted code blocks.

Page 52 of 162

The DITA XML architecture provides a reusemechanism using a system of content references to otherfiles using keys (stored in the map) and element id
attributes. To include an example in a topic, the Data
Engineer simply adds the codeblock element with aconkeyref attribute that points via the key to thecodeblock/@id in the examples topic.
We also added the classes and output classes to a CSS
file and used an @import CSS statement in the editor
CSS to style the examples consistently in the Oxygen [5]author view and HTML output formats.

Practice what we PreachFigure 1. Generated syntax highlighting in the editor:

Figure 2. Generated syntax highlighting in the output format:

Example 3. XSL Listing1 <?xml version="1.0" encoding="UTF-8"?>
2 <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"3
xmlns:xd="http://www.oxygenxml.com/ns/doc/xsl"
45
xmlns:x="http://schema.oup.com/dita/examples"
6
exclude-result-prefixes="xs xd x"
version="2.0">
7<xd:doc scope="stylesheet">
89<xd:desc><xd:p><xd:b>Created on:</xd:b> Feb 19, 2013</xd:p>
10<xd:p><xd:b>Author:</xd:b> TFJH</xd:p>
11<xd:p><xd:b>Last Modified:</xd:b> TFJH, 2013-02-19</xd:p>
12<xd:p>This stylesheet creates a shared repository of dita code
13
14
examples from a validated input source.</xd:p></xd:desc>
15</xd:doc>
16
17<xd:doc scope="component">
18
19<xd:desc><xd:p><xd:b>Last Modified:</xd:b> TFJH, 2013-02-19</xd:p>
20<xd:p>output parameters specify DTD declarations etc</xd:p>21</xd:desc>
22
23</xd:doc>
24<xsl:output doctype-public="-//OASIS//DTD DITA Topic//EN"
doctype-system="topic.dtd" encoding="UTF-8"/>25
26
27<xd:doc scope="component">
28<xd:desc>29<xd:p><xd:b>Last Modified:</xd:b> TFJH, 2013-02-19</xd:p><xd:p>The root template creates the DITA superstructure.</xd:p>30</xd:desc>
31
32</xd:doc>
33<xsl:template match="/"><topic id="topic_jys_5gr_fj">
34<title>Common code examples</title>
35
36<body>
37<xsl:apply-templates/>

Page 53 of 162

Practice what we Preach

38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

</body></topic></xsl:template><xd:doc scope="component"><xd:desc><xd:p><xd:b>Last Modified:</xd:b> TFJH, 2013-02-19</xd:p><xd:p>Each x:example element represents a new codeblock.</xd:p></xd:desc></xd:doc><xsl:template match="x:example"><codeblock id="{@id}"><xsl:apply-templates/></codeblock></xsl:template><xd:doc scope="component"><xd:desc><xd:p><xd:b>Last Modified:</xd:b> TFJH, 2013-02-19</xd:p><xd:p>Kick off code conversion when encountering a wrapper element.</xd:p></xd:desc></xd:doc><xsl:template match="x:OxDTD"><xsl:apply-templates mode="code"/></xsl:template><xd:doc scope="component"><xd:desc><xd:p><xd:b>Last Modified:</xd:b> TFJH, 2013-02-19</xd:p><xd:p>General handling of elements within code blocks</xd:p></xd:desc></xd:doc><xsl:template match="*" mode="code"><ph outputclass="XmlFurniture">&lt;</ph><ph outputclass="ElementName"><xsl:value-of select="name()"/></ph><xsl:apply-templates select="@*" mode="code"/>72<ph outputclass="XmlFurniture">&gt;</ph>73
74<xsl:apply-templates select="node() except @*" mode="code"/>75<ph outputclass="XmlFurniture">&lt;/</ph><ph outputclass="ElementName"><xsl:value-of select="name()"/></ph><ph outputclass="XmlFurniture">&gt;</ph></xsl:template>76
77
78<xd:doc scope="component">
79<xd:desc><xd:p><xd:b>Last Modified:</xd:b> TJFH, 2013-02-19</xd:p>
80<xd:p>General handling of attributes on elements</xd:p>
81
82</xd:desc>
83</xd:doc>
84<xsl:template match="@*" mode="code"><xsl:text> </xsl:text><ph outputclass="AttributeName"><xsl:value-of select="name()"/></ph><ph outp
85
utclass="XmlFurniture">=&quot;</ph><ph outputclass="AttributeValue"><xsl:value-of select="."/></ph><phoutputclass="XmlFurniture">&quot;</ph></xsl:template>
8687<xd:doc scope="component">
88<xd:desc>89<xd:p><xd:b>Last Modified:</xd:b> TFJH, 2013-02-19</xd:p>
90<xd:p>General handling of text nodes in code</xd:p>
91</xd:desc>
92</xd:doc>
93<xsl:template match="text()" mode="code">
94<xsl:value-of select="."/>
95

Page 54 of 162

Practice what we Preach

96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115

</xsl:template><xd:doc scope="component"><xd:desc><xd:p><xd:b>Last Modified:</xd:b> TFJH, 2013-02-19</xd:p><xd:p>General handling of comments in code</xd:p></xd:desc></xd:doc><xsl:template match="comment()" mode="code"><ph outputclass="Comment">&lt;--<xsl:value-of select="."/>--&gt;</ph></xsl:template><xd:doc scope="component"><xd:desc><xd:p><xd:b>Last Modified:</xd:b> TFJH, 2013-02-19</xd:p><xd:p>General handling of PIs in code</xd:p></xd:desc></xd:doc><xsl:template match="processing-instruction()" mode="code"><ph outputclass="PI">&lt;?<xsl:value-of select="name()"/><xsl:text> </xsl:text><xsl:value-of select="."/>?&gt;</ph>116</xsl:template>
117
118<xd:doc scope="component">
119<xd:desc>120<xd:p><xd:b>Last Modified:</xd:b> TJFH, 2013-02-19</xd:p><xd:p>Recursion templates to ensure all elements and attributes match</xd:p>121</xd:desc>
122
123</xd:doc>
124<xsl:template match="node()|@*" mode="#default"><xsl:apply-templates select="node()|@*" mode="#current"/>
125</xsl:template>126
127
128 </xsl:stylesheet>

4.3. Remaining challenges and developments
DTD/Schema support. The implementation of NVDL
[4] in Oxygen [5] lacks support for DTD validation so
we used trang to produce schema based versions. Any
updates to the DTDs will require a re-transformation to
update the schemas.
In the short term, we can mitigate by adding this as
an automated part of our release process. In the longterm we hope to migrate our datamodels to a compliant
schema format.
Example excerpts and highlighting. Each example
must be a complete, valid XML instance. This means
examples will sometimes contain elements and structuresthat do not directly relate to the example have to be
captured. Consider as an example an IDREF reference toa different part of the document structure!
One potential approach would be to mix elementsfrom the wrapper format among the example datamodel. These would be stripped out for validation, but
in place to highlight, selectively include or exclude rangesof the example.

XML and Doctype declarations. XML and Doctype
declarations would invalidate the data if included
verbatim in examples. However it should bestraightforward to add options to the wrapper format
which can be used to recreate them.
Rule based validation. At the moment OUP use aproprietary format for our rule based validation (XML
Probe) [6]. When we move to an NVDL compliant
format (viz. Schematron) [7], the system will become far
more powerful.

Page 55 of 162

Practice what we Preach

5. Conclusion
One of the major selling points of XML is that it
promises reuse not only of data but also of skills.
Sometimes it's easy to forget that we can apply theseadvantages to our own day to day work, such as
documentation, as well as the XML which is our
product.

We found that any technology that's available for thewider use of XML can introduce opportunities when you
decide to use XML with your own processes.
Those principles of reuse are mirrored on aconceptual level with the guidelines introduced by
DITA.
We hope that we've shown that there areopportunities to not only produce or process XML, but
to use XML ourselves: to practice what we preach.

References[1] Darwin Information Typing Architecture (DITA)
http://docs.oasis-open.org/dita/v1.2/os/spec/DITA1.2-spec.html[2] DITA Open Toolkit
http://dita-ot.sourceforge.net/[3] DITA for publishers
http://dita4publishers.sourceforge.net/[4] Namespace-based Validation Dispatching Language (NVDL)
http://www.nvdl.org/[5] Oxygen XML Editor by SyncRO Soft SRL
http://www.oxygenxml.com/[6] XMLProbe by Griffin Brown Digital Publishing Ltd.
http://www.xmlprobe.com[7] Schematron
http://www.schematron.com

Page 56 of 162

Optimizing XML for Comparison and Change
Nigel Whitaker
DeltaXML Ltd.

Robin La Fontaine
DeltaXML Ltd.
Abstract

We consider comparison to be a problem of identifying
similarity, aligning information, and then represent thechanges so that they can be processed in various ways.
Some data formats have built in mechanisms to describechange, for example the status attribute in DITA,
similarly @revisionflag in DocBook and the <ins> and<del> elements in HTML. Where these facilities areprovided it often makes sense to make a comparison toolthat takes two inputs and produce a result which uses thefacilities and is also a valid XML result. This goal
sometimes conflicts with the design of the schema andsome of these issues will be addressed later.

Almost every user of XML will, at some stage, need to dosome form of comparison between different versions of their
XML data or document. This could be because it is necessary
to review the changes that have been made, or to check thatthe output from one version of some software is the same as
the previous version, or to find changes so that they can thenbe processed in some other way.
Designers of XML formats often do not consider thisrequirement for comparison when designing an XML
format. However, if this is taken into account at the design
stage then it can make the usefulness of the XML greater andat the same time reduce the cost of developing software. This 2. Use a DTD or Schema for
paper outlines some of the issues that can usefully beconsidered by XML designers to make comparison and Whitespace Handlingchange easier to handle. The design issues will also be ofinterest to any user of XML to provide a better Consider Figure 1, “Added data” where a new (phone)element has been added to some contact data. The
understanding of processing change in XML.
inexperienced user says “I've added an element, it has
three children.”. We check if a DTD is used and when
1. Introduction
not finding one, we reply (perhaps a little pedantically)
with: “No you haven't. You've added two nodes, an
It is possible to achieve better comparison of XML if you element and some text (which is a newline and spaces).
have some knowledge of the data or format being The element contains seven children - three elements
processed. At DeltaXML we have produced specific and four text nodes.”.
comparison products for formats such as DocBook and
DITA and we often help our customers using their own Figure 1. Added datain-house data and document formats. When doing this<contact>
work we've often said to ourselves “if only they would...
have done this... it would make their life so much easier”.
<phone type="work">
This paper is a review of some of these issues and<countryCode>44</countryCode><areaCode>020</areaCode>recommendations - it could be called a 'wishlist' or<local>7234 5678</local>'manifesto' for comparison. Some of this is just 'good</phone>advice' that extends beyond comparison per-se. Taking</contact>some or all of this advice when designing XML formats
will make comparison and other XML processing tools
work better out of the box and with less configuration.

doi:10.14337/XMLLondon13.Whitaker01

Page 57 of 162

Optimizing XML for Comparison and Change

Of course, it is possible to make comparison software
remove the extra indentation whitespace and provide
some configuration options. But this is delegating
responsibility to the user, who may not have as good anunderstanding of the data as the developers.
As a user of a comparison tool - you may see changes
that you don't care about and are a nuisance - the four
added text nodes in the example above. There are otherless obvious implications that we should also mention
including performance and alignment.
Most comparison algorithm implementations are
similar to those of XSLT transformation, in that you
generally need to store the inputs in memory to navigate
around them. Storing those extra tree nodes for theunnecessary whitespace affects both performance (it takes
time to load and process them), but more importantly
the heap space requirements. If you are processing datalike that above, its quite possible to halve the heap
memory size needed for comparison with proper
whitespace handling.
Many comparison algorithms use Longest Common
SubSequence (or SubString) optimization techniques [1].
These work best (give good results and have good
performance) when there are reasonably long sequencesof similar or identical content. When they are faced withXML data like that above, but where one input has an
extra indentation whitespace node for each element andthe other file does not (perhaps its a single line of XML
without whitespace, which when loaded in an editor
makes you go to immediately to the indent button), it isalmost a nightmare scenario. The whitespace nodes
break-up the sequences of element nodes that can match,furthermore the whitespace nodes are all identical (thesame newline and number of spaces giving the same levelof indentation) and match each other. This kind of data
will often mismatch and is also slow to process.
So what is proper whitespace handling? It is possible
to remove the indentation whitespace above by preprocessing the data, or having a post-process which
ignores or removes the purely whitespace changes thatare identified. But by far the best way of dealing withwhitespace is to use a DTD or schema so that the parser
can differentiate between element-only and mixed
content [4]. When they are used (and associated with thedata using a DOCTYPE or @xsi:schemaLocation), parsers
such as Apache Xerces can use the ignoreableWhitespace
callback method; comparison and other tools then know
they can safely ignore that type of content.

Page 58 of 162

3. Using Schemas and DTDs
Consistently
We hope the previous section has provided a convincing
argument to use a DTD or schema. However, if you go
down this route it is worth remembering that using a
DTD or schema has a number of implications. In
addition to controlling whitespace and use for validity
checking they are also used for 'Infoset Augmentation'.
Infoset Augmentation means adding data from DTD
or schema to the resulting parsed representation. It is
often used to specify values of attributes, for example
that a table by default will have a 1 pixel border. It isalso, more controversially, used to provide a default
namespace to xhtml data. While it is possible in somecases to determine if data was provided by augmentation,
we would encourage instead that DTD DocTypes andschema association be used consistently. This will avoid
spurious attribute change that is often confusing to theuser (“I can't see that in either of my inputs”) and in thecase of xhtml, avoid the issues around an body element
not matching or aligning with an xhtml:body element.
We have recently been working on a comparison tool
for a documentation standard. That standard included
reference material for each element, in terms of a DTD
grammar. It also included, as a download, a large set ofmodular DTD/entity files and a related set of W3C
XML Schema files (.xsd), but nowhere in the standard
did it mention how XML instance files were meant to
use the DTD or xsd files; no mention of DOCTYPE
requirements or use of schemaInstance, or statements
about processing expectations. Implementers then chose
whether to use a DOCTYPE or not. We are then facedwith comparing mixed files and have to deal with thedifferences due to augmentation between the two inputs.
If you provide DTD or xsd files as part of a standard orformat definition and they are used for augmentation,then a little guidance to implementers on how you
expect them to be used would sometimes be appreciated!

Optimizing XML for Comparison and Change

4. Use Appropriate Data Types
There's another benefit to using a DTD or schema,
particularly for data-orientated XML. A DTD provides alimited form of data typing; attributes can be declared oftype ID which constrains uniqueness, and whitespacenormalization for CDATA attributes is different to that forother types of attribute. A richer form of data-typing isprovided by W3C and RelaxNG schema languages.
These use the XML Schema datatypes [2] to describe thetypes of elements and attributes. This information canthen be used for normalization purposes or postprocessing to remove unexpected change. For example,
you may consider these timestamped elements to beequivalent: <element time='2013-03-14T14:35:00Z'> and<element time="Thu 14 Mar 2013 14:35:00 GMT">.
When using floating point (or 'double') numbers
most developers programming in Java or C# are warned
about doing direct comparison operations and told to use
epsilon or 'units of least precision' (ULPs). Similar
consideration should be given to XML data values,
whether serialized as characters in an XML file or when
loaded into an in-memory tree.
It's often simpler to think of all of the attribute andother datatypes as strings particularly when developing
both reader and writer software. However, proper
definition of the datatypes has benefits when forcomparison and also for other more general forms of
XML processing such as XSLT 2.0 transformation.

5. Consider using xml:space
We have discussed how an XML parser's whitespace
handling is affected by the use of a DTD or schema. The
xml:space attribute also has an effect on whitespace
handling, but not within the parser. Instead it is used tocontrol how downstream applications handle whitespace.
Think of xml:space="preserve" as a hands off mywhitespace instruction!
It is often needed because many text processing
applications of XML will normalize text prior topresentation, collapsing sequences of spaces to a single
space, normalizing between line breaks and spaces andfitting or breaking lines to fit the page or display width.
There is an expectation that comparison tools do
likewise, for example, not reporting where two spaces in
one document correspond to three spaces in another.
Therefore our default processing of textual data involves
a whitespace normalization process. It is good practice tothen provide xml:space support to users who wish to
avoid this normalization.

The examples given in the XML spec for xml:space areof the poem and pre element. Many XML specifications
then allow use of xml:space on various elements and thisgives the user the opportunity to turn-off the default
behaviour described above. We would suggest thatgrammar designers sprinkle the xml:space attributearound their text or mixed content whenever possible
independently of whether they fix or default its
behaviour on elements such as pre. This allows the user
to refine the space handling on a per instance basis.

6. Consider Using xml:langUsers of comparison tools like to see changes at finegranularity, they generally don't want to see which text
nodes in an XDM tree have changed, but rather which'words' have changed. In order to do this text is usuallysegmented or broken into smaller chunks for alignmentand comparison. This process is harder than it initiallyappears. The concept of a word varies significantly in
different languages and character systems. In Latin/
European alphabets words are often delimited by spaces
or punctuation whereas in eastern scripts 'words' areoften represented by a single glyph or Unicode character.
Software is available to do this kind of text
segmentation (the Java BreakIterator classes, or the
widely used ICU [3] library which supports betterinternationalization). However in order to do its job
properly it needs to be told which language is being used.
This is where xml:lang is very useful. It is a small part of
the XML specification [4], but often bypassed. Pleaseconsider adding it to your DTD or schema even if you
don't envisage having multiple languages in a single
XML file. If you add xml:lang to the root element, thenit is possible for software to provide a default value, even
if the user does not. This could perhaps be based onsome user preferences, or computer locale settings.
Using xml:lang has benefits beyond comparison.
Consider the case of single, double or triple clicking a
range of text in an editor. The behaviour of these actions
is also something that is typically language dependantand could utilise this attribute. Another useful generalbenefit, discovered when preparing this paper, is that
xml:lang can be used to control spell checking
dictionaries.

Page 59 of 162

Optimizing XML for Comparison and Change

7. Data Ordering Issues
Some forms of data have an associated order and others
do not. We will use a contact record as an example, as
shown in Figure 2, “Mixed ordered and orderless contact
data”. In this example the order of the addressLine
elements is significant for postal delivery. However users
may not care about which order the various phoneelements appear.
Figure 2. Mixed ordered and orderless contact data<contact><name>John Smith</name><addressLine>25 Green Lane</addressline><addressLine>Bloomsbury</addressline><addressLine>London</addressline><addressLine>UK</addressline><postcode>W1 2AA</postcode><phone type="office">+44 20 1234 5678</phone><phone type="fax">+44 20 1234 5680</phone><phone type="mobile">+44 7123 123456</phone></contact></contact

Our comparison algorithms, and we suspect most others,
are suited to either ordered/list based processing, perhaps
optimizing a metric such as edit distance, or are suited to
set-based or 'orderless' processing perhaps using hashingtechniques and Map based data structures. Mixing thesealgorithms together so as to process the example above
increases complexity enormously. Instead we prefer totreat all of the children of an element as either ordered ororderless and use the algorithms separately.
So, for the above data, rather than having a DTD
like this :<!ELEMENT contact(name, addressLine*, postcode, phone*)>we prefer to use something like this:<!ELEMENT contact (name, address, phone*)><!ELEMENT address (addressLine*, postcode)>or perhaps this:<!ELEMENT contact(name, addressLine*, postcode, phoneNumbers)><!ELEMENT phoneNumbers (phone*)>Page 60 of 162

Introducing these grouping elements to the contact data
makes it possible to subdivide the ordered and orderlesscontent. It also allows us to attach attributes to them tocontrol how the comparison is then performed at a given
level of the tree. These attributes are easy to add using
XSLT if there is an XPath that corresponds to anorderless container. It is also possible to specify, and tosome extent document, the orderless nature of theelement in the DTD, for example:<!ATTLIST phoneNumbers deltaxml:ordered(true|false) "false")>8. Ids and Uniqueness in XML
The XML specification allows attributes to be declared oftype ID or IDREF. One use-case of this facility is for crossreferencing within a document. For example a user
inserts a table into a document and gives it an id, forexample <table xml:id="forecast">, and when talking
about the table would later write: ... In <xreflinkend="forecast"/> we describe the .... The
assumption being that the processing system would put
replace the xref with an appropriate reference.
The XML specification requires that such ids are
unique and this is supported by the editing andprocessing applications. For comparison and change
control we would like to recommend another property
that such ids should have, is that of persistence. The
cross reference is in some respect a cross-tree pointer inXML. That's fine, but it can also change and when it
does we are faced with the problem of working out if theuser has changed what is being pointed to, or if the thing
being pointed to has changed. Perhaps it is the samething, but it has also changed slightly? Working out
what's happened gets very difficult in these cases. We
would recommend that if you write software to read andwrite such content then as well as considering id
uniqueness please also consider persistence through read/
write cycles.

Optimizing XML for Comparison and Change

We've seen some examples of content which is converted
from a word-processor format into XML markup wheresections, tables etc are automatically numbered with ascheme such as "sect1_1, sect1_2 .. ". From acomparison perspective this is almost next to being
useless, the same information is usually extractable using
xsl:numberorcount(preceding-sibling::*)type
operations. When the user creates a cross-reference please
don't change the user-provided text. Adding artificial ids
is usually a hindrence for comparison, particularly whena new chapter or section is interspersed into the content
and causes subsequent elements to be renumbereddifferently. Finally we would suggest that schema
designers do not make id attributes REQUIRED so thatapplication developers and users do not have to come up
with, probably artificial, numbering schemes.
The example we have shown above is a good use-casefor XML ids. We don't recall many others. There is adanger of id/xml:id mis-use. Consider the case of a data
file with National Insurance (NI) numbers. Don't be
tempted to declare these of type ID/xs:ID because NI
numbers are considered unique. Sooner or later you may
need to add vehicle registration numbers of some othersimilar unique identifier and then face the problem that
ID/xs:ID provides a single global namespace. Consider
using schematron or perhaps XSD 1.1 assertions to
specify uniqueness, they tend to be more flexible and also
extensible for future needs.

9. Grammar Design Issues
When generating a comparison output, making use of
the facilities provided by the data or document format
being processed to describe the changes is useful as itallows users to make use of existing publication andother software available for that format. If that software
validates its input then the comparison result should
ideally also be valid. However there are some design
issues that make this quite hard. We will illustrate thistype of issue using DITA.
The DITA task specialization has a DTD model
essentially similar to:<!ELEMENT taskbody((steps | steps-unordered), result)>

We can see why the designers thought that there should
only be a single sequence of steps in a task. But from ourperspective its difficult to represent this type of change instandard DITA. We currently provide a parameter in our
DITA products which provides control over how theoutput is represented using either of these elements andinclude a comment which describes the details of thechange.
From a comparison perspective grammars whichallow singleton choice are harder to deal with, adding arepetition qualifier such as * or + makes representingchange easier.

10. Formatting Element Patterns
The process of marking up mixed content withformatting information (by wrapping text in element
such as: b, i, span or emphasis) is something which users(document editors or authors) typically do not see orappreciate in XML terms. Explaining that they havedeleted some text and replaced it with an element
containing some text rarely enlightens the situation.
They are generally interested in the words that havechanged and as a secondary concern what formatting
may have changed. However, they do like to see theadded or deleted text represented as it was formatted in
the two input files.
To meet this requirement we tend to flatten
formatting information. The result is a paragraph orother container of mixed content where all of the words
are siblings at the same level of the XML tree. When
compared in this re-organized form changes to the text
are easier to align and identify. There are a number offlattening techniques that can be used, including using
start and end markers, so that for example
some <b>bold</b> text becomessome <b-start/>bold<b-end/> text or by moving
formatting information to some out-of-band location.
When trying to generate a result we need toreconstruct as much of the original flattened hierarchies
around the added and deleted content and in some cases
those hierarchies can be in conflict.
We have found two properties of grammars that arevery useful when reconstructing a good representation of
the input hierarchies in the comparison result which we
informally call removability and nestability:

Using the status attributes we normally say whether anelement has been added or deleted and this can then bestyled appropriately with red or green colouring or strike- Removability
through. When one element is changed to another we
Can the wrapping element be removed still leaving
can normally represent this as one being deleted anda valid result? This is true for example, if theanother being added. However the design of the DITA
elements in the content model of a span or othergrammar precludes this when steps is changed to stepsformatting element, are the same as the content
unordered or vice-versa.

Page 61 of 162

Optimizing XML for Comparison and Change

model at the point where the span is used in thecontaining element.
Nestability
Can the formatting element contain an immediatechild of the same type? This is usually true whenrecursive reuse is used in a content model, forexample allowing a span element directly
containing another span element.
These properties are often true for established document
formats such as DocBook, DITA and (x)html, however
for simpler document formats they are not always trueand cause difficulty for our algorithms which try toreconstruct the formatting hierarchies. As well as forcomparison, we suspect these would also be good
properties if the document format were to be supported
by a WYSIWIG or authoring editor.

However, it is difficult to describe changes to thembecause they lack structure. PIs are often used torepresent changes using accept/reject or review systems
used in editors, and thus are a possible way of describing
comparison results. However, when used in this way its
almost impossible for accept/reject PI mechanisms todescribe changes to PIs. Similarly CSS styling using
attributes cannot be used for highlighting commentchange.
We would therefore caution their use as an ad-hoc
extensibility mechanism. It may work in limited
scenarios with writer/reader agreement, but a general
purpose comparison has a hard time describing thechanges without resorting to similar ad-hoc mechanisms
to do so.

11. Processing Instructions (PIs)and Comments
Comments and Processing Instructions (PIs) are often
used by developers as a simple or ad-hoc extensibility
mechanism: “The grammar doesn't allow me to put thisinformation here.... I know, I'll write it in a PI andchange the reading code to understand those PIs.”

Bibliography[1] Binary codes capable of correcting deletions, insertions, and reversals. Vladimir I. Levenshtein. Soviet Physics
Doklady, 1966.
[2] XML Schema Part 2: Datatypes Second Edition. W3C Recommendation. 28 October 2004.
http://www.w3.org/TR/xmlschema-2/[3] International Components for Unicode
http://sites.google.com/site/icusite/[4] Extensible Markup Language (XML) 1.0 (Fifth Edition). W3C Recommendation. 26 November 2008.
http://www.w3.org/TR/REC-xml/

Page 62 of 162

What you need to know about the Maths Stack
MathML, MathJax, HTML5, and EPUB 3
Ms. Autumn Cuellar
Design Science
Mr. Paul Topping
Design Science
Abstract
MathML is a well-known and widely-used standard for
encoding mathematics within XML workflows, but what
you may not know is that MathML is not just a standard
that affects your internal workflow, used only for storage andconverted to images when you need to present your content
to your audience. MathML is a key part of the digitalpublishing revolution towards enriched content. Its recent
inclusion into the HTML5 and EPUB 3 standards is
helping to bring to fruition the promise of interactive
content for maths-based industries around the world.
All of the major browser vendors have pledged supportfor HTML5, and many EPUB readers are built on browser
engines, thus full browser and e-reader support of MathML
is expected in the future. Gecko-based browsers including
Firefox already do include native MathML support.
WebKit, the engine behind Safari and Chrome, also hassome MathML support, though only Safari has included thisfeature in releases. Until universal support for MathML isavailable, MathJax fills in the gaps. MathJax is an opensource Javascript library for displaying MathML in allmodern browsers and has been included with success in
EPUB readers, such as Readium.
MathJax helps both content creators and content
consumers realize the full advantages of MathML. One
challenge that producers of digital content are faced with is
the range of devices, screen sizes, and screen resolutions thatmust be supported. Unlike with bitmap images, MathJax
easily adjusts equations to match the surrounding text
without losing any quality. MathJax also leaves the
MathML content available for re-use. This is particularly
important to audiences who'd like to play with the equations
in your content by loading (such as through copy and paste)
the equations into Microsoft Excel or other calculation ormodeling software, most of which accept MathML as an
input format. Another feature of MathML that MathJax
accommodates is its accessibility to audiences with visionand learning disabilities. MathML has been embraced bythese communities and most accessible technology software

doi:10.14337/XMLLondon13.Cuellar01

includes support of MathML by reading the equation aloud,highlighting parts of the expression, and allowing control of
the navigation of the equation. Finally, by leaving the
MathML in the content, the equations remain searchable,
allowing your content to be found.
The key elements to moving your maths forward into
this bright new interactive publication era are a basedoctype that allows MathML, tools to create MathML, XSL,and MathJax. Many standard doctypes already include
MathML or are on their way to adding MathML support.
OASIS, for example, keeps a DocBook MathML Document
Type, and the next version of DITA (1.3) will also include
MathML. All other doctypes, since XML is the eXtensible
Markup Language, can be extended to include MathML.
Though MathML can be manually written, it's a verbose
language that was never intended to be written by humans.
To create MathML, you will need a conversion tool toconvert existing maths content to MathML and/or a
MathML editor, such as MathFlow. To publish the XML toits digital output format, HTML5 or EPUB 3, a wide
range of publishing tools are available on the market, mostof which use XSL in some part of the process. The XSL or
publishing tool can usually be easily modified such that theoutput includes the MathJax libraries. Lastly, you will need
the MathJax libraries.
We are only just now starting to get a taste of theexciting future of digital content thanks to the HTML5 andEPUB 3 standards, and MathJax is filling the need of
publishers with maths in their content until the browser ande-reader technology catches up to the standards. The benefitsof MathML can only be fully realized when MathML is
supported not just as an internal storage format but also as adelivery format, and with the Maths Stack, consisting of
MathML, MathJax, HTML5, and EPUB 3, the maths inyour digital publications are interoperable, accessible, andsearchable.

Page 63 of 162

What you need to know about the Maths Stack

1. Introduction

MathML 3 was finalized in late 2010. With MathML 3,
MathML shows a new maturity by taking into
consideration the needs of various communities with amore specialized interest in mathematics. For example,
the publishing industry is now served by taking line
breaks to the next level: for long equations that may
extend past the width of pages or columns, content
providers can now specify both a maximum width (sothe equation automatically wraps) as well as how toindent or align subsequent lines. For the education
community, MathML 3 adds new features to support
elementary math, such as stacked addition andsubstraction, long division, and repeating decimals.
MathML 3 also became a truly international standard byadding support for right-to-left languages and fornotations that vary from region to region.

The digital publishing era promises to revolutionize how
we consume content. Plain text is now being enhanced
not only with images but also with, for example, video,
music, and interactive applications. At the heart of thisrevolution in publishing is XML. Recent developments
in XML-based standards, such as HTML5 and EPUB 3,
are facilitating the trend towards enhanced content.
The great news for those of us with any interest inscientific, technical, engineering, or mathematical(STEM) content is that mathematics, often anafterthought in web technologies, are a key part of therecent advancements toward enriched content. In thispaper we'll discuss the latest developments in the XML
standards for mathematics, web pages, and e-books; how
MathML brings to fruition the promise of enhanced
2.2. HTML5
content for maths-based industries; the current status of
support for these standards; and, finally, what you need While MathML has been developed and revised, it has
to do to prepare your content for the bright new future established itself as a standard for encoding mathematics.
of digital publishing.
It has been incorporated into a number of otherstandards from biological modeling languages (CellML
2. Recent Standards Developments and SBML) to medical journal archives (NLM). Tool
support for MathML has advanced, with MathML
import and/or export available from a variety of software
2.1. MathML
applications ranging from word processors to computer
MathML is the XML standard for encoding algebra systems. Most importantly, from the perspective
mathematical information. It's maintained by W3C, the of the HTML Working Group, MathML renderingsame organization that maintains the XML and HTML engines have arrived on the scene to provide expert
standards. MathML was conceived in the mid-1990's layout of complex equations.
Mathematics did not make the cut for inclusion into
during a discussion of requirements for HTML 3.
HTML, the language behind the World Wide Web (a HTML3 due to the challenges of rendering a
network built by scientists for scientists) at this time had complicated and varied language. However, throughno way to represent mathematics -- a failing that some MathML these challenges have been overcome. Thus,
felt needed to be addressed. However, the challenges for HTML5, which as of December 2012 is a W3C
representing maths on the web seemed too mountainous Candidate Recommendation, now includes MathML.
HTML5 has been praised for its inclusion of
to be tackled in an update to HTML, so the MathML
working group was formed to address the problems different content types. In previous versions, different
types of media (such as video, audio, math, and evensurrounding maths in XML/HTML.
The MathML 1 spec was finalized in 1998. Five years images) were treated as external objects, many of which
later, in 2003, MathML 2 was unveiled to correct flaws required plug-ins to the browser for the visitor toin the initial specification. For instance, in version 1, experience. The benefit of including media in themulti-line equations could only be expressed through the HTML is that browsers will consistently and correctly
use of tables. MathML 2 also favors certain features of display the content without requiring external software.
Cascading Stylesheets (CSS), which had grown more This means, theoretically, a given page with its included
popular for applying styles to content, over its former media will display the same on any platform or device.
[Fernandes, 2012]attributes for specifying key properties.

Page 64 of 162

What you need to know about the Maths Stack

2.3. EPUB 3
EPUB, the open standard for e-books, is maintained by
the International Digital Publishing Forum (IDPF). In
version 2 of the EPUB standard, the content of the ebook could be expressed in either of two varieties:DAISY or XHTML. DAISY is an independently
managed standard for specifying Digital Talking Books,
mostly used for accessibility purposes. Because theDAISY standard included MathML for specifying
mathematical content, one could create a valid EPUB filewith MathML.
In the most recent version of the EPUB standard,
IDPF eliminated the DAISY variant, based the XHTML
variant on HTML5, and endorsed inclusion of MathML
as an important aspect of e-books. Like HTML5, EPUB
3 has been lauded for taking digital content to the nextlevel with its support for media.

3. How MathML Fits in with theHTML/EPUB Promise of Enriched
Content
Maths content has long been included in e-books andweb pages as images, plain text (often referred to as
ASCII Math), or, for the enthusiasts, as TeX/LaTeX.
However, by delivering the mathematics in your content
as MathML, you can enable your audience new levels ofinteraction with the information presented. MathML is
integral to HTML5's and EPUB 3's promises ofenhanced publications. MathML offers the following
advantages in usability and presentation over alternative
formats.

3.1. Searchability
Annually, EMC, a maker of storage and big data systems,
publishes a study called "Digital Universe" measuring theamount of digital data available. By the end of 2012,
EMC measured 2.5 zettabytes of data available
worldwide. The study showed the amount of data
available worldwide had doubled in just two years. The
same study predicts that by 2020, the total data countwill exceed 40 zettabytes. "To put it in perspective, 40
zettabytes is 40 trillion gigabytes -- estimated to be 57
times the amount of all the grains of sand on all thebeaches on earth." [Mearian, 2012]

With so much data available at our fingertips, search is
becoming ever more important. An ambition in the fieldof mathematics is how best to search mathematicalnotation. Maths search could have several applications
including aiding research in scientific communities,
helping identify common patterns between remote
disciplines, and serving as an educational tool. [Miner,
2004] Most of the studies that have had success in thisarea, such as MathDex, EgoMath, and MathWebWorld,
have used alorithms relying on MathML. [Misutka,
2008]

3.2. Localization
While maths is a universal language and semantically one
expression might be the same as the next, the notation
used to express a given equation might vary significantly
from region to region. For instance, notation for
expressing long division differs distinctly depending on
where you are in the world. With XML stylesheets, one
can target the display of mathematical content to theregion where it's being observed.

3.3. Flexibility of Display
Since MathML is a part of the document and is rendered
at load time, environmental settings can be factored into
the display of the maths. Therefore, if a visitor or reader
has his browser or e-reader font size set to a larger fontsize than the default setting, the equations will also
appear at the larger base font size. The audience will also
see high quality equations no matter the resolution or
pixel density of the device loading the web page or ebook. This is a special shortcoming of using images formaths display, as images intended for lower pixel density
monitors display poorly on high pixel density monitors
and vice versa.
MathML, like XML, gives you power ofpresentation, allowing content providers to apply acertain style to equations along with the rest of the
document without having to go through and change
each equation individually. For example, equations intables can be displayed at a smaller font size than the restof the document or maths in change tracking systems canbe highlighted with a foreground (or background) color
to indicate changes to an expression.

Page 65 of 162

What you need to know about the Maths Stack

4. Status of MathML Support inContent providers may be required by government HTML5 and EPUB 3
3.4. Accessibility

statute to supply accessible materials to readers withvision and learning disabilities, however, even if
government regulations don't apply, there is no reason tolimit the available audience. The accessibility community
favors MathML because MathML's encoding of amathematical expression is precise enough that it may be
used to translate the equation to Braille or speech text
and can also be used to navigate a complex equation sothat parts of the equation can be repeated in asemantically relevant way. Most standards intended tosupport the accessibility community, including DAISY
for Digital Talking Books and NIMAS for instructional
materials, require maths to be encoded in MathML.

3.5. Interoperability
Perhaps the most exciting feature of MathML is that it isreceiving growing support in a wide variety of
applications: computer algebra systems, graphingapplications, calculators, modeling software, assessment
creation systems, educational whiteboards, etc. Even
today one can copy and paste (or import and export)
MathML between applications to make better use ofdata. It isn't difficult to envision the future of MathML:
Imagine a doctor is reading a journal article describing
mathematical functions behind chemical pathways in the
body. To help him visualize how changes in the level of
sodium affect the pathways, he sends the equations in thejournal article to a modeling system. Within seconds heis able to start adjustments until a life-saving solution isfound. Alternatively, the journal article may include a
simple modeling system that allows the reader to adjust
the parameters of the mathematical functions in thearticle for hands-on reinforcement of the concepts being
described. The potential for interoperability is only
limited by the imagination.

Page 66 of 162

The vendors of the major browsers have been vocal intheir support of HTML5. In fact, all of the browser
vendors have representatives on the HTML Working
Group. Support for MathML is already available in the
Gecko rendering engine, which is used by Firefox.
WebKit, the engine behind Safari, can render a subset of
MathML. Opera currently uses CSS to provide support
for MathML. As part of HTML5, browser support for
MathML will no doubt only improve in the coming
years.
Since EPUB 3 is based on HTML5, and since manye-readers are built on browser technology, some e-readers
have inherited MathML support. Apple's iBooks e-reader
is a primary example. Based on WebKit, iBooks has
limited support of MathML. With the enthusiasm that's
been shown for EPUB 3, the backing of all of its major
features, including MathML, is expected in futureversions of all e-readers.
However, knowing that support is coming is of little
use to those who want to start taking advantage of all of
the benefits of MathML today. With the gaps in browser
and e-reader display of MathML today, this is an
obstacle. No one wants to produce content so that theintended audience has to jump through hoops to view it.
Without a doubt, every website visitor has been on thewrong end of content delivery requirements: a website
might display a message stating that you do not have therequired version of Flash or that the website is optimized
for a certain version of browser, which is (of course) not
the browser you are using. You are left to contemplate
whether the content you have come to consume is worth
the effort of updating software or switching browsers,
and sometimes it's not.
Luckily, a Javascript project called MathJax has beendeveloped to eliminate the gaps in browser and e-reader
support of MathML.

What you need to know about the Maths Stack

4.1. MathJax
MathJax is an open-source Javascript library for
rendering MathML in all modern browsers. It isdeveloped by a consortium consisting of Design Science,
American Mathematical Society, and the Society for
Industrial and Applied Mathematics. Internet Explorer
and Google Chrome, two browsers that have no native
support of MathML, using MathJax, can now display
HTML5 pages with MathML in them without the use ofa client-installed plug-in. It also adds more complete
support of the MathML specification than Firefox,
Safari, and Opera have alone.
Because MathJax improves browser display of
MathML, all e-readers built on browser technology can
improve their EPUB 3 support by implementing the
MathJax Javascript library. Calibre is a popular e-reader
application that has already included MathJax to display
e-books with mathematical content.

5. Enriching Content with MathML
Your excitement to get started with your maths stack ispalpable. Without delving into too much technicaldetail, the four areas requiring consideration whensetting up your maths stack are your doctype, MathML
editors, the conversion process, and MathJax.
Assuming you are working in a source format otherthan HTML5 or EPUB 3, first you'll want to make sure
your XML doctype includes MathML. Some standard
doctypes, such as Docbook and NLM, do alreadyinclude MathML. If your doctype does not already,you'll need to modify your schema to include the
MathML schema.
Once your schema has been set up to include
MathML, the next step is to start creating MathML.
While MathML is human-readable, it's a verbose
language that doesn't lend itself to manual creation.
Quite possibly your XML editor of choice alreadyincludes a MathML editor or has a plug-in available fordownload. If not, Design Science offers a suite of
MathML editors for integration with your workflow.
Most MathML editors have a WYSIWYG interface sothat you never have to see the MathML.

After creating your content, you might run it through aconversion process, usually involving XSLT, to produce
HTML5 and EPUB 3 output. The conversion processneeds to be modified to include the calls to MathJax inthe header. Also, the MathML will need to be passed
through the process untouched. By default, XSLT
processors will remove the MathML tags and only pass
the CDATA through; thus, you will need to add a few
lines to your XSLT library to make sure the MathML
tags make it to the output.
Finally, you may need to set up MathJax on yourserver. The MathJax Consortium does host a version of
MathJax to which you can link your pages, but if you
have heavy traffic, like to have control of the libraries
used on your site, or want to customize MathJax at all,you will need to download and install MathJax.

6. Conclusion
Recent developments in digital publishing standards arechanging the way we present and consume information.
Mathematics are a key part of this revolution with
MathML's inclusion in HTML5 and EPUB 3. MathML
brings to fruition the promise of enriched content for
STEM industries by providing the following advantages:
MathML is searchable and accessible, it allows easy
localization of content, it is flexible in its display acrossmultiple devices and in its support of stylesheets, andfinally MathML's use as an exchange format provides
endless possibilities in the way mathematical content in
web and e-book publications can be enhanced for youraudience.
MathML is already supported by a number ofbrowsers and e-readers. To fill the gaps, an open-source
Javascript library by the name of MathJax provides
MathML rendering for all modern browsers. MathJax is
also being used to successfully add proper display of
MathML in e-readers.
The maths stack consisting of MathML, MathJax,
HTML5 and EPUB 3 can be implemented by modifying
your XML schema to include MathML, adding a
MathML editor to your XML editor, making sure yourconversion pipeline passes the MathML throughuntouched, and setting up MathJax on your server.

Page 67 of 162

What you need to know about the Maths Stack

Bibliography[Mearian, 2012]

Lucas Mearian. 11 December 2012. By 2020, there will be 5,200 GB of data for every person on
Earth. Computerworld Inc.
[Miner, 2004]
Robert Miner. 8 June 2004. Enhancing the Searching of Mathematics. Design Science.
[Misutka, 2008] J. Mišutka. Copyright © 2008. Indexing Mathematical Content Using Full Text Search Engine.
240-244. WDS '08 Proceedings of Contributed Papers . Part I.
[Fernandes, 2012] Rossi Fernandes. 15 March 2012. What HTML5 means to you. Tech2.com India.

Page 68 of 162

Small Data in the large with Oppidum
Stéphane Sire
Oppidoc<s.sire@oppidoc.fr>

Christine Vanoirbeek
EPFL<christine.vanoirbeek@epfl.ch>

Abstract
The paper addresses the topic of frameworks intended to
speed up the development of web applications using the
XML stack (XQuery, XSLT and native XML databases).
These frameworks must offer the ability to produce
exploitable XML content by web users - without technicalskills – and must be simple enough to lower the barrier entry
cost for developers. This is particularly true for a low-budget
class of applications that we call Small Data applications.
This article presents Oppidum, a lightweight open source
framework to build web applications relying on a RESTful
approach, sustained by intuitive authoring facilities topopulate an XML database. This is illustrated with a simple
application created for editing this article on the web.
Keywords: XML, web development, framework,
XQuery, XSLT, RESTful approach, Oppidum

1. Introduction
There are still millions of individually operated web
applications that contain only a few "megabytes" of data.
These are not Big Data applications although theiraddition still constitutes a remarkable part of the web. As
a matter of fact, it currently exists an amazing number of
web applications that run small to medium size corporate
web sites or aim at integrating the work of associations invarious domains.
This category of applications is often based on a
PHP/MySQL stack to deal with factual data and requires
aside use of an office suite (Word, Excel, etc.) to deal
with document-oriented information. Functionalities
range from publishing news, blogs, agenda, andcatalogue of members to provision of a set of services
such as registration, on-line shopping or more specificprocesses.

doi:10.14337/XMLLondon13.Sire01

We call this class of applications Small Data application.
Characteristics of those applications are the following
ones : mostly publication oriented, asymmetrical (fewcontributors and few to many readers), evolutive(frequent updates) and extensible (through modules, e.g.
image galleries, shopping cart, registration, e-bookgeneration, etc.).
Because semi-structured data models procure thepossibility to encompass both data and documentoriented representation of information [1], there aremany reasons why such applications would benefit froman XML technology stack. For instance, to build custom
schema aware search engines for better information
retrieval, for single-source and cross-media publishing, or
most importantly for data portability.
There is always the alternative to build Small Data
applications on a hosting platform with an embeddedsite authoring tool (e.g. Weebly or Google site).
However it does not easily support extensibility and datareuse.
For other Small Data applications, teams ofdevelopers are using classical (aka non-XML) longstanding technologies (aka relational databases). This ismost probably because there are now dozens of popularserver-side frameworks to lower the barrier entry cost fordevelopers, when not totally commodifying development
to customizing a set of configuration files in the so-called
Content Management Systems (CMS).
We believe there are two reasons preventing the
adoption of XML technologies for Small Data
applications. The first one is the lack of adequatebrowser's based-editing facilities. Most of the CMS use
rich-text-editors and HTML forms. Thus they miss thecapability to really structure the input space. The second
one is the complexity of building XML applications :teams developing Small Data applications work undersmall budget constraints (usually a few thousands Euro/
Dollar per project) and they can't afford a long learning
curve and/or development cycle.

Page 69 of 162

Small Data in the large with Oppidum

To some point XForms could have solved the first issue.
However it has been designed for form-based data and isless powerful for handling document-oriented semistructured data (see for instance these developer's
question on Stack Overflow [3] [4]).
The blossoming of many XML development
environments based on XQuery (BaseX, eXist-DB,
MarkLogic, Sausalito) could solve the second issue.
However each one comes with its own extensions andconventions to build applications, often still at very low
abstraction level compared to over MVC frameworks
available for other platforms (e.g. Ruby on Rails).
Despite these obstacles we have started to develop
Small Data applications with XML during the last three
years. We have solved the first issue by developing a
Javascript library for XML authoring in the browser
called AXEL [13]. It uses the concept of template toallow developers to create customized editing userinterfaces guaranteeing the validity of data [6]. We
resolve the second issue by providing a lightweightframework called Oppidum that is described in thispaper.
Oppidum is an open source XML-oriented
framework written in XQuery / XSLT. It is designed tocreate custom Content Management Solutions (CMS)
involving lightweight XML authoring chains. It iscurrently available as a Github repository to be deployedinside an eXist-DB host environment.
The paper is organized as follows. The first section
describes the architecture of Oppidum and its two-steps
execution model. The second section presents the way it
manages some common web application design patterns.
The third section presents some example applications
done with it. Finally the fourth section discusses somedesign choices of Oppidum compared with over XML
technologies.

Page 70 of 162

2. Oppidum architecture
2.1. Application model
Oppidum provides developers with a simple application
model that allows them to rapidly and efficiently deploy
a RESTful application; it relies on a few fundamental
principles:• the application must be entirely defined in terms ofactions on resources;• each action is defined declaratively in a mapping file
as a sequential pipeline composed of three steps :• the first step, called the model, is always an XQuery
script;• the second step, called the view, is always an XSLTtransformation;• the third step, called the epilogue, is always an
XQuery script that must be called epilogue.xql;• a pipeline may have only the first step, the first andsecond steps, the first and third steps, or the threesteps.
It is always possible to extend a pipeline by invoking one
or more XSLT transformations from XQuery in the first
and the third steps.

Small Data in the large with Oppidum

Figure 1. The application mapping defines a pipeline for each resource / action

The application mapping describes the mapping of the
URL space with the pipeline definitions.
An action is either an HTTP verb (e.g. GET, POST,
PUT, DELETE) or a custom name. In the first case the
HTTP request corresponding to the action is the request
using the same verb. In the second case thecorresponding HTTP request is any request with arequest URI path ending by the action name.
In application mapping terms the GET articles/xmllondon HTTP request is matched as calling the GET
action on the xml-london resource. The Figure 2,“Example of a pipeline to view this article using
Oppidum” shows an example of a pipeline that
implements the rendering of the resource as an HTML
page : the 1st step calls a models/read.xql script that
returns the raw XML data of the article. It could also
perform side effects such as checking user's access rightor updating access logs. The 2nd step calls an XSLTtransformation views/article2html.xsl that generates
an HTML representation. Finally the 3rd step is an
epilogue.xql script. It inserts the HTML representationof the article into an application page template with extra
decorations such as a navigation menu and/or somebuttons to edit or publish the article.

The previous example is equivalent to a RESTful
operation to get a representation for a resource [2]. It isalso possible to consider extended RESTful operations byconsidering the resource as a controller and to define
custom verbs. In that case the verb must be appended tothe end of the resource-as-a-controller URL. For
instance, the GET articles/xml-london/publish request
could be matched as calling a custom publish action onto
the xml-london resource. Currently custom actions
cannot be targeted at a specific HTTP verb, this is alimitation. As a consequence it is up to the developer toenforce specific semantics for different HTTP verbs if
required.

2.2. Execution model
Oppidum execution model is a two steps process. The
first step takes the client's HTTP request and theapplication mapping as inputs. It analyses the request
against the mapping and generates a pipeline to executein the host environment. The second step executes thepipeline and returns its output into the HTTP response.
Figure 3. The execution model

Figure 2. Example of a pipeline to view this article
using Oppidum

In the eXist-DB host environment, the first step isinvoked in a controller.xql script file. That scripts calls
an Oppidum gen:process method that returns an XML
structure specifying a pipeline to be executed by the
URLRewriter servlet filter of eXist-DB. The second stepis entirely left to that filter, that executes the pipeline.
The gen:process method executes a sequential algorithmwith three steps : the first step produces an XML

Page 71 of 162

Small Data in the large with Oppidum

structure that we call the command, the second steptransforms the command into an internal abstract
pipeline definition, finally the third step generates theexecutable pipeline for the host environment.
The command is persisted into the current request as an
attribute. Thus it is available for introspection to the
XQuery and XSLT scripts composing the pipeline. Thisis useful to write more generic scripts that can access data
copied into the command from the target resource or thetarget action mapping entry.

3.1. Template system

The pipeline generator generates the epilogue step of thepipeline if and only if the target mapping entry has an
epilogue attribute. The epilogue.xql script can interpretthe value of this attribute as the name of a page template
file defining common page elements such as anapplication header, footer and navigation menu. We callthis template a mesh.
This is summarized on Figure 4, “Conventional
pipeline for using the template system”. The template2.3. Conventions
system also relies on a pipeline where the view step must
output an XML document with a site:view root
Oppidum uses a few conventions, although it is very element containing children in the site namespaceoften possible to bypass them writing more code.
defined for that purpose.
As per eXist-DB there is only one way to invoke
Oppidum: it is to create a controller.xql file at the Figure 4. Conventional pipeline for using theapplication root. The script will be invoked by the eXist template system
servlet with the HTTP request. It must call thegen:process method with the mapping and a few
environment variables as parameters. Similarly the
epilogue must be called epilogue.xql and placed at theapplication root. The same controller.xql file can becopy / pasted from a project to another.
All the code in production should be stored in the
database inside a /db/www/:app collection (where :app is
the application name). Configuration files (error
messages, mapping resource, skin resource, etc.) must be A typical epilogue.xql script applies a typeswitch
placed inside the /db/www/:app/config collection. It is transformation to the mesh [14]. The transformation
recommended to store all application's user generated copies every XHTML element. When it finds an element
content inside a /db/sites/:app collection (and sub- in the site namespace, called an extension point, it
collections). In our current practice those conventions replaces it with the content of the children of theease up multiple applications hosting within the same <site:view> input stream that has the same local name.
If not available, it calls an XQuery function from thedatabase.
The library enforces conventions onto the URL epilogue file named after the element and replaces theelement with the function's result. So if an element frommappings mostly to offer debug services:themesh is <site:menu>, and there is no <site:menu>• adding .xml to a URL executes only the 1st step of
element
in the input stream, it will be replaced with thethe pipeline and returns its result;resultofthe site:menu function call.
• adding .raw to a URL executes only the 2nd step ofThetypeswitch
function at the heart of the template
the pipeline and returns its result;• adding .debug (or a debug=true request parameter) system can be copied / pasted between applications. It
prevents the pipeline execution, instead it returns a relies on a variable part for the extension points that is
dump of the command and of the generated customized for each application.
pipelines.
Consequently, developers should not use the epilogue toapply state changes or side effects to the database.

3. Oppidum design patterns
Oppidum architecture and execution model support
common web application design patterns. In some cases
we have extended the mapping language and/or the
Oppidum API to support more of them.

Page 72 of 162

Small Data in the large with Oppidum

3.2. Skinning applications

3.4. Data mapping

The template system can be used to decouple theselection of the CSS and JS files to be included in a pagefrom the direct rendering of that page.
A typical epilogue.xql script defines a site:skinfunction to be called in place of a <site:skin> elementfrom the head section of a mesh file. That function
dynamically inserts the links to the CSS and JS files.
Oppidum provides a skin module for managing theassociation between string tokens and sets of CSS and JS
files, called profiles.
The module supports several profile levels:• profiles associated with a given mesh name;• profiles associated with keywords generated andpassed to the epilogue script as a skin attribute of the<site:view> root element of the pipeline input
stream;• a catch-all * profile applying to every page renderedthrough the epilogue together with some exceptions
declared using predicates based on a micro-language.
The keyword profiles are useful for a fine-grain control
over the selection of CSS and JS files to be included bygenerating the appropriate keywords from the XQuerymodel scripts or XSLT views. The catch-all profile isuseful to insert a favicon or web analytics tracker code.
The profile definitions are stored in a skin.xml resource
in a conventional location inside the database.

It is common to generate pages from content stored in
the database. Thus it is very frequent to write code thatlocates that content in terms of a collection and a
resource and that simply returns the whole resource
content or that extracts some parts of it. In most cases,
the collection and resource paths are inferred fromsegments of the HTTP request path.
For instance, in a blog, the posts/123 entry could be
resolved as a resource 123.xml stored in the posts
collection.
The mapping file allows to associate a referencecollection and a reference resource with each URL. They
are accessible from the model script with the
oppidum:path-to-ref-coland oppidum:path-to-ref
methods that return respectively the path to the referencecollection and the path to the reference resource. Thismechanism fosters the writing of generic model scriptsthat adapt to different data mapping.
In addition, the reference collection and resource canbe declared using variables that will be replaced withspecific segments of the HTTP request path. For
instance, if a reference resource is declared as resource=$3, the $3 variable will be replaced by the third segmentof the request path.

3.3. Error management

It is frequent to restrict access to some actions to specificusers or groups of users. Thus, a common pattern is tocheck users' rights before doing any further action andask for user identification. Using the Oppidum
constrained pipeline that would mean to always invoke
the same kind of code in the model part of a pipeline.
Oppidum alleviates this constraint with someextensions to the application mapping syntax that allows
to declare access control rules. When rules have been
defined, they are checked directly by the gen:processfunction, before generating the pipeline. An alternative
pipeline is generated in case of a refusal. It redirects
clients to a login page with an explanation.
The access to each resource or action can be restrictedindividually by defining a list of role definitions. The
roles are defined relatively to the native database user
definitions and resource permissions : the u:name role
restricts access to the user named name; the g:name role
restricts access to users belonging to the group named
name; finally the owner role restricts access to the ownerof the reference resource declared in the data mapping.

Another common pattern is to signal errors from thescripts composing the rendering pipeline, and to display
these errors to the user. Oppidum API provides afunction to signal an error and another function torender an error message in the epilogue.xql script.
A typical epilogue.xql script defines a site:error
function to be called in place of a <site:error> element
placed anywhere inside a mesh file. That function calls
Oppidum error rendering function. The skinningmechanism is also aware of the error API since it allows
to define a specific skin to render for the displaying anddisposal of the error messages.
The error management code is robust to page
redirections, so that if a pipeline execution ends by aredirection, the error message is stored in a session
parameter to be available to the rendering of theredirected target page.
There is an identical mechanism to display messages
to the users. The messages are stored in an errors.xml(resp. messages.xml) resource in a conventional location
inside the database for internationalization purposes.

3.5. Form-based access control

Page 73 of 162

Small Data in the large with Oppidum

3.6. Development life cycle
It is common to have different runtime environments
when developing, testing or deploying an application(e.g. dev, test and prod environments). For instance theapplication may be developed with eXist-DB instandalone mode, while the final application may be
deployed under Tomcat. This may impose a few
constraints on some portions of the code. For that
purpose the application mapping defines a mode attributethat remains accessible with the Oppidum API. It can beused to adapt functionalities to the environment. For
instance it is possible to apply different strategies forserving static resources while in dev or in prod, or toenable different debug functionalities.
It is also recommended to share the application codewith other developers using a code repository systemsuch as Git. For that purpose we have found useful todevelop applications directly from the file system. Thisway it is possible to commit code to the repository at any
time and to move all of the application code to the
database only when in prod. Oppidum supports the turnover with an installation module. It provides a declarative
language for the application components and theirlocalization into the database. This module supports thecreation of an installer screen to automate installation.
The installer may also be used to copy someconfiguration files and/or some initial data or some test
sets to the database while in dev or in test modes.
Since the release of eXist-DB 2.0, that comes with acomplete IDE for in-browser development, the life-cycle
based on source code written in the file system may nolonger be the most efficient way to work. Thus we areconsidering to update the installation module so that it ispossible to work directly from the database and to
package the library and the applications as XAR archives.
However, at the moment, we are still unsure how thiswould integrate seamlessly with code versioning systems
such as Git.

Page 74 of 162

4. Example applications
4.1. Illustrative example
This article has been edited with an application writtenwith Oppidum. Its mapping and its mesh are shown inthe code extracts below. The article editor itself is an
XTiger XML document template [12] as explained in [6][13].
The mapping defines the following URLs:• GET /articles/xml-london returns the article asHTML for screen display• GET /articles/xml-london/edit returns an HTML
page containing an editor based on the AXEL library,that page loads two additional resources :• GET /templates/article returns the XTiger XML
template• GET /articles/xml-london.xml returns the article as
raw XML• POST /articles/xml-london.xml saves the article backto the database• some other URLs manages image upload andretrieval :• POST /articles/xml-london/images/ saves a newimage to the database• GET /articles/xml-london/images/:name returns theimage stored in the resource name
The raw XML version of the article shares the same
pipeline as the HTML version of the article. This
exploits the .xml suffix convention that short-circuits any
pipeline to return the output of the first step.
Four of the application pipelines are implemented asa single step pipeline executing an XQuery script. This isbecause either they directly returns a resource from thedatabase (this is the case for the XTiger XML template or
to serve an image previously stored into the database), or
because they are called from an XHR request
implementing a custom Ajax protocol where theexpected result is coded as raw XML. For instance, theprotocol to save the article only requires the POST
request to return an HTTP success status (201) and to
set a location header. It then redirects the window to thelocation header address which is the representation of thearticle as HTML. These Ajax protocols depend on the
AXEL-FORMS javascript library that we are using to
generate the editor [11], which is out of the scope of thisarticle.

Small Data in the large with Oppidum

Figure 5. Pipelines for the application used to edit this article

The code extract below shows most of the application Without entering into too much details of the mapping
mapping defining the previous pipelines.
language, the target resources are defined either by acollection element if they are supposed to contain an
Example 1. Mapping extract for the application used indefinite number of resources, or by an item element if
to edit this article
they are supposed to contain a finite number ofresources. Actions are defined by an action element. The<collection name="articles" collection="articles"
hierarchical structure of the mapping file follows theepilogue="standard">
hierarchical structure of the URL input space : the name<item collection="articles/$2"
resource="article.xml"
attribute matches the corresponding HTTP request path
supported="edit" method="POST"
segment in the hierarchy and anonymous item elements
template="templates/article"(ie. without a name attribute) match any segment string
epilogue="oppidocs">
at the corresponding level.
<access>
The use of anonymous item elements to define<rule action="edit POST" role="g:authors"
resources allows to create generic mappings that work
message="author"/></access>
with collections of resources such as collection of articles<model src="oppidum:actions/read.xql"/>
or collection of images inside articles. As such the xml<view src="article/article2html.xsl">
london resource illustrating this article is mapped with<param name="resource" value="$2"/>
the anonymous item element on the second line.
</view>
Some notations are supported to inject segment strings<action name="edit" epilogue="oppidocs">
from the request path into the mapping using positional<model src="actions/edit.xql"/><view src="views/edit.xsl">$ variables. For instance resolving $2 against the<param name="skin"/articles/xml-london URL returns the xml-london
value="article axel-1.3-with-photo"/>string.
</view>
The mapping language also makes use of annotations to</action>
support some of the design patterns or specific features:<action name="POST">• the epilogue attribute selects a mesh to render the<model src="actions/write.xql"/></action>
page in the epilogue as explained in the template<collection name="images"
system design pattern;collection="articles/$2"• the access element supports the access control design
method="POST">
pattern;<model src="models/forbidden.xql"/>• the collection and resource attributes support the<action name="POST">
data mapping design pattern;<model src="images/upload.xql"></model>• the template attribute indicates the URL of the</action>
XTiger XML template for editing a resource;<item resource="$4"• the param element implements mapping level
collection="articles/$2/images">parameters transmitted to the model or view scripts.
<model src="images/image.xql"/></item></collection></item></collection>

Page 75 of 162

Small Data in the large with Oppidum

The application screen design is quite simple as it displays a menu bar at the top with either an Edit button whenviewing the article, as shown on Figure 6, “Screen shots of the article editing application with the shared menu bar”,
or a Save and a Preview buttons when editing it. Both screen are generated with the mesh shown below.
Figure 6. Screen shots of the article editing application with the shared menu bar

Example 2. mesh to display the article or to edit thearticle<html xmlns:site="http://oppidoc.com/oppidum/site"
xmlns="http://www.w3.org/1999/xhtml"><head><site:skin/></head><body><div id="menu"><site:commands/></div><div id="article"><site:content/></div></body></html>The mesh defines three extension points in the sitenamespace. The <site:skin> extension point calls asite:skin XQuery function as explained in the skinning
design pattern. The <site:commands> extension point
calls a site:commands XQuery function that generates
some buttons to edit (when not editing), or to save(when editing) the article. Finally the <site:content>
extension point is a place-holder for the homonym
element's content to be pulled from the input stream andthat contains either the article when viewing or an
HTML fragment that defines an editor using AXELFORMS when editing.

4.2. Other applications
During the last three years, several Small Data
applications have been developed using Oppidum. We
mention a few of them, emphasizing the features theypresent accordingly to our definition of Small Data
applications: mostly publication oriented, asymmetrical,
evolutive and extensible.

Page 76 of 162

The first one is a publication-oriented application for theediting and publication of a 4 pages newsletter provided
by Platinn (an organization that offers coaching support
to Swiss enterprises). The newsletter data model has been
derived from a set of legacy newsletters with two goals :first the ability to define an XSLT transformation thatgenerates a CSS/XHTML representation suitable forconversion to PDF ready to be sent to the print shop,
second the ability to export the newsletter to a content
management system. The application design is quite
close to the illustrative example above, some additional
resources have been defined to manage authors profiles
and a log mechanism to track and display the editing
history, mainly to prevent concurrent editing of the samenewsletter. Until now 10 newsletters have been written
by a redaction team of up to 5 redactors, with more than
30000 printed copies.
The second one is an application for maintaining andpublishing a multilingual (french and english) databaseof startup companies members of the science park(societes.parc-scienfique.ch)at
EPFL(ÉcolePolytechnique Fédérale de Lausanne). A contact person
and the staff of the science park can edit a publiccompany profile with several modules. It illustrates the“asymmetrical” and “evolutive” characteristics since thegoal of the application is to encourage editors tofrequently and accurately update their presentation. For
that purpose we are using statistics about the company
profiles to generate some recommendations to improvetheir presentation, and to advertise most recently
updated companies. There are currently about two
hundreds companies using this system. The semistructured document-oriented approach has beenvalidated when we have been asked to open a web service
to feed an internal advertising TV screen network in the
different buildings of the park with some company
profile extracts to avoid multiple inputs.

Small Data in the large with Oppidum

The third and fourth applications are web sites ofassociations. One is a craftsmen's association of Belgium
called Union des Artisans du Patrimoine de Belgique(uniondesartisansdupatrimoine.be), and the other one is
the Alliance association (alliance-tt.ch) that provides
assistance in building partnerships between industry andacademic research in Switzerland. Both are centred on
traditional content management oriented features tomaintain a set of pages and to publish articles about
events and/or members that could have been done with
non-XML frameworks. However we have been able toextend them with custom modules such as a moderated
classified ads services reserved for members in the case of
the craftsmen's association, and an event registration
module for the Alliance association. This last module has
benefited from XML technologies in that it has been
turned into an editorial chain with the ability to edit
custom registration forms for each event, and to generate
different types of documents such as a list of badges forprinting and participants' list to distribute. Moreoverthese extensions, thanks to the application model
enforced by Oppidum, are clearly separated from theother parts of their original application. Thus they can beported to other projects by grouping and copying thepipeline files and by cut-and-paste of the corresponding
application mapping parts.

5. Discussion
We do a quick comparison between Oppidum andOrbeon Forms, RESTXQ, Servlex and XProc.
The Orbeon Forms page flow controller dispatches
incoming user requests to individual pages built out ofmodels and views, following the model / view /controller (MVC) architecture [5]. This is very close to
the Oppidum architecture presented in Section 2,“Oppidum architecture” however there are a few
differences. Orbeon page flow controller usually
integrates page flows definitions stored within the folders
that make up the application code. With Oppidum theapplication mapping, which is equivalent to the pageflows, is a monolithic resource, although we have been
experimenting with modularization techniques forimporting definitions not described in this article. Oneof the reasons is that we see the application mapping as afirst order object, and hence as a document which can be
stored in the database as the other user-generated
content. It could ultimately be dynamically generated
and or edited by end-users.
The syntax is also quite different : the page flows
determines the correspondence between URLs and theirimplementation with the implicit directory structure of
the source code and with regular expressions for less
implicit associations; in Oppidum this is the implicit
structure of the application mapping tree. The principal
reason is that the application mapping in Oppidum aims
at decoupling the RESTful hierarchy from the code
hierarchy which is difficult to achieve with Orbeon
Forms. Another side reason is that this allows to definesome kind of cascading rules to inherit the referencecollection and resource which have no equivalent with
Orbeon Forms.
Like RESTXQ [10] Oppidum proposes a complete
RESTful mapping of an application. However it diverges
in the granularity of this mapping. While RESTful
XQuery proposes a very elegant syntax to bind individual
server-side XQuery functions with RESTful web services,
Oppidum granularity is set at a coarser pipeline grain
level. As a consequence Oppidum mapping is also
decoupled from its implementation and is maintained ina single document which can be further processed using
XML technologies as explained above. On the opposite
RESTXQ mappings are defined as XQuery 3.0
annotations intertwined with function definitions only
available to the developers of the application.

Page 77 of 162

Small Data in the large with Oppidum

Both solutions provide different means to select thetarget mapping entry, and to parse and to communicate
parameters from the request to the code generating the
response. They also use syntactic artefacts to selectdifferent targets in the mapping based on different
request's properties (URL path segments, HTTP verbs,
HTTP headers, etc.). In this regards Oppidum is much
more limited than RESTXQ since it only discriminates
the targets from the path segments and the HTTP verbs.
We could envision to extend Oppidum mapping
language with further conditional expressions, however it
seems a better solution could be to mix both approaches :in a first step, Oppidum mapping could be used to do acoarse grain target selection, then in a second step,
RESTXQ could be used inside the model part of eachpipeline to select between a set of functions to call.
We can see this need emerging in Oppidum as forlimiting the number of XQuery files we found useful
under some circumstances to group together related
functionalities inside a single XQuery script shared as amodel step between several pipelines. For instance, it is
tempting to group reading and writing code of a given
type of resource, together with some satellite actions, tocreate a consistent code unit (e.g. use a single script tocreate, update and read a participant's registration in an
application). As a consequence the file starts by checking
more properties of the request to select which function tocall, which seem to be one of the reason that led Adam
Retter to propose RESTXQ in replacement of theverbish circuitry inside former eXist-DB controller.xql
files [9]. To our current perception, parts of these efforts
tend to address the same kind of issues as object
relational mapping in non-XML frameworks, without
yet a clear solution for XML applications.
To some extent Oppidum shares goals similar to theServlex EXPath Webapp framework [7]. Like it it defines
how to write web applications on server-side, using XML
technologies (currently XSLT and XQuery) and it
defines their execution context, as well as some functions
they can use [8]. However Oppidum is more restrictive
as it imposes a RESTful application model and someconstraints on the generated pipeline composition, thus
it looks like a restriction of this framework. It will beinteresting, in the future, to check if applications writtenwith Oppidum could be automatically converted and/or
packaged as EXPath Webapp applications and what
would be the benefits. In particular we see a strong
interest in accessing the host environment functionalities
which are database dependent today (like accessing to therequest or response objects) and that could be abstracted
into common APIs with the EXPath framework.

Page 78 of 162

Finally Oppidum does not currently makes use of XProc
although it's execution model is based on simple
pipelines. The first reason is historical since we havestarted to work with Orbeon Forms XML pipeline
language (XPL) before Oppidum. The lack of design
patterns and good practices led us to overcomplexify
simple developments, and thus led us to invent the
Oppidum self-limiting three steps pipeline model inreaction. But now with the return on experience, we are
more confident in opening up Oppidum to support
direct inclusion of XProc definitions within ourapplication model. A quick solution could be to support
pipeline files as models or views (as it is the case inOrbeon Forms) or eventually as epilogue. For that
purpose, it would be feasible to rewrite the Oppidum
pipeline generator to directly generate pipelines written
in XProc instead of the URLRewrite filter pipeline
format currently imposed by eXist-DB.

6. Conclusion
The XML Stack clearly proposes a valuable technologicaloption to deal with data manipulated by Small Data
applications. It offers the great potential to adopt a
uniform representation of information that bridges twooften separated paradigms: document and databasesystems. Adopting so-called semi-structured data models
allows capturing in a homogeneous way the structure of
information, at a very fine granularity level, if needed. It
significantly enhances reuse of content in different
purposes either for developing new services or delivering
information through many channels.
The resulting benefits are particularly important evenfor low-budget class of applications. For end-users it
avoids spending time in potentially manipulating thesame pieces of information through different userinterfaces or copying/pasting information, for instance,from a document to a web form. For developers existingsemi-structured data may be reused to cost effectively
add a wide range of new functionalities to a web
application as, for instance, the generation of badges andlist of participants who registered to an event. Finally, it
is obvious, the XML format is especially well adapted togenerate publishable documents or to deliver content on
cross-media platforms.

Small Data in the large with Oppidum

Two main challenges need to be taken up to promote the In this perspective, our further research work is targeting
adoption of the XML Stack in Small Data applications: two main inter-connected issues:
the capability offered to end-users to easily provide valid • the first one is addressing the modeling ofcontent on the web and the provision to developers withcooperative/communication processes to be supported
a framework that can be rapidly mastered. The paper
in order to make efficient the production, the sharing
presented Oppidum, a lightweight framework to build
and the exploitation of XML information by asuch applications. It integrates the use of AXEL, a
community of users. It covers, amongst othertemplate-driven editing library that guarantees thepreoccupations, the following issues: awareness,
validity of data provided by end-users.
notification and role controls;
The past and ongoing developments made us • the second one is addressing the collaborative editing
confident that Oppidum may evolve, while keeping itof content itself. For the time being, it does not exist
simple, to address another concern of Small Data
too many XML web-based collaborative authoring
applications: the cooperation aspects. Relying on aenvironments. We are interested in investigating
declarative approach, we believe that, with proper data
about models and specification of an appropriate
modelling, it would be easy to develop simple workflows
language that help to sustain the collaborative
to support cooperative processes among stakeholders of a
production task as well as associated actions.
web application.

Bibliography[1] Serge Abiteboul, Ioana Manolescu, Philippe Rigaux, Marie-Christine Rousset and Pierre Senellart: Web DataManagement. Cambridge University Press 2011.
[2] Subbu Allamaraju: RESTful Web Services Cookbook. O'Reilly Yahoo! Press.
[3] Anonymous (user1887755): Web-based structured document authoring solution (Stack Overflow).
http://stackoverflow.com/questions/13777777/web-based-structured-document-authoring-solution[4] Anonymous (user1887755): Is Drupal suitable as a CMS for complex structured content? (Stack Overflow).
http://stackoverflow.com/questions/14214439/is-drupal-suitable-as-a-cms-for-complex-structured-content[5] Erik Bruchez: Orbeon Developer and Administrator Guide : Page Flow Controller.
http://wiki.orbeon.com/forms/doc/developer-guide/page-flow-controller[6] Francesc Campoy-Flores and Vincent Quint and Irène Vatton: Templates, Microformats and Structured
Editing. Proceedings of the 2006 ACM Symposium on Document Engineering, DocEng 2006.
doi:10.1145/1166160.1166211.
[7] Florent Georges: Servlex.
https://github.com/fgeorges/servlex[8] Florent Georges: Web Application EXPath Candidate Module 9 March 2013 (in progress).
http://expath.org/spec/webapp/editor[9] Adam Retter: RESTful XQuery Standardised XQuery 3.0 Annotations for REST. XML Prague 2012.
http://archive.xmlprague.cz/2012/files/xmlprague-2012-proceedings.pdf[10] Adam Retter: RESTXQ 1.0: RESTful Annotations for XQuery 3.0.
http://exquery.github.com/exquery/exquery-restxq-specification/restxq-1.0-specification.html[11] Stéphane Sire: AXEL-FORMS Web Site.
http://ssire.github.com/axel-forms/[12] Stéphane Sire: XTiger XML Language Specification.
http://ssire.github.com/xtiger-xml-spec/[13] Stéphane Sire and Christine Vanoirbeek and Vincent Quint and Cécile Roisin:
Authoring XML all the Time, Everywhere and by Everyone. Proceedings of XML Prague 2010.
http://archive.xmlprague.cz/2010/files/XMLPrague_2010_Proceedings.pdf[14] Priscilla Walmsley: XQuery, Search Across a Variety of XML Data. O'Reilly

Page 79 of 162

KEYNOTE ADDRESS

Extremes of XML
Philip Fennell<philip.fennell@gmail.com>

1. Introduction
The Extensible Markup Language (XML) is a meta
language, it is used to describe other languages and assuch it has been phenomenally successful, it has beendrawn into just about every information domain
imaginable, not to mention some you really can't.
Although it might be unfair to term the extremes aslurching from the sublime to the ridiculous it is,however, fair to say that some applications of XML are
incredibly useful whilst others are insanely convoluted
and frustrating to work with.
XML, originally envisaged as a lingua franca for theWeb, was born of the Standard Generalized Markup
Language (SGML) but with rather neater, tighter rules tomake it more concise. Some of the earliest applications of
XML hinted at the diversity of uses to which it would be
put. However, of more recent times the XML
community has been reflecting upon the very nature ofXML, its processing rules and serialisation, and has been
looking for ways to simplify it in order that it might finda more amicable position along side, for example, the
JavaScript Object Notation (JSON).
As hinted at above, XML has been applied,sometimes ill advisedly, throughout the IT industry andbeyond. Over the last 15 years, looking at theapplications of XML as a whole, they can be broadly
boiled-down to the following categories:• Describing• Modeling• Processing• Publishing• Interacting• Presenting

Whilst periods of introspection are often fruitful, we
should also, on occassion, turn our attention outwards
towards the edges of this envelope and see where XML
has pushed itself into some new or unusual applications.

2. Describing
Describing things is not an edge-case for XML, it is at
the centre because it is at the heart of what we do withXML. By description we actually mean information
about things, metadata which, we capture as annotations
within or in reference to these structures. We'll pushoutwards towards the edges of our envelope and workaround the categories.

3. Modeling

We use XML to help us capture how we model a specificinformation domain using XML Schema (XSD) andAt the heart of what we do with XML is describe things, Relax NG, however, at the very edge of current data
essentially it is used for marking-up information to be modeling interest lies the Semantic Web where thepublished unless it is metadata about those published boundary between XML and the Semantic Web has been
entities or how the marked-up information is to be and still is somewhat blurred.
modelled, processed,interacted with or presented.
When looking at the extremes of XML, what I findfascinating is not how much or how fast but the breadthof applications to which XML has been applied, and thisis what one could call the 'XML Envelope'.

Page 80 of 162

doi:10.14337/XMLLondon13.Fennell01

Extremes of XML

XML and the Semantic Web have something of a mixed
and, it should be said, confused relationship. XML has
become, for many applications, a singular standard for
information representation and interchange whilst the
Semantic Web has managed to adopt at least four, andrising, representations of the Resource Description
Format (RDF) and for many people the existence of oneof them, RDF/XML, has posed a problem. A very real
problem because it was seen as an information markup
language rather that an XML representation of a graph.
There is a subtle but important difference.
Whilst it can still be advocated that there's is nothing
wrong with using XML technologies, like XSLT fortransform RDF/XML, the querying of it with non-graphbased languages, like XQuery, is analogous to querying
XML with text based techniques like Regular
Expressions. Whilst you can do it, in their respective
cases you fail to see the underlying structures to theinformation without a lot of additional work.

When we stand at the edge, the interface between theDocument and Semantic Web, we see that XML's usage
within Semantic Web technologies is somewhat on theslide. Compared to Notation 3 (N3), N-Triples, Turtle
and the upcoming JSON-LD. RDF/XML is increasingly
becoming a legacy format and with RDF 1.1 it will nolonger be the 'required' interchange format for RDF
implementations.
So, leaving aside questions of graph serialisation,
XML content does still intersect with the Semantic Web
because much of the information it marks-up has, or is,
metadata that could, and should, be modelled as RDF
Linked Data. The question is how do the two sides meet
and to what extent should they overlap. By way of a
related example, mappings between Relational Data andRDF already exist in the form of the W3C's 'RDB to
RDF Mapping Language' (R2RML) and the 'Direct
Mapping of Relational Data to RDF'. These twoconcepts provide a means to create RDF 'views' of
relational data that could be queried with the SPARQL
query language. This is a solution for structured data butwhat of semi-structured data?
Schema Lifting and Lowering are terms coined by the
Semantic Web community to describe the process of
mapping XML Schemas to RDF Ontologies and backagain. The idea being that you can enhance the basic
semantics of XML fragments by converting them to
RDF. Buried deep within the myriad of W3C Web
Services recommendations lies the 'Semantic Web
Annotations for WSDL and XML Schema' (SAWSDL)
which uses foreign attributes to embed references
between schema types and ontology classes andproperties.

<xs:complexType xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:sawsdl="http://www.w3.org/ns/sawsdl" name="entryType"
sawsdl:modelReference="http://bblfish.net/work/atom-owl/2006-06-06/#Entry"><xs:annotation><xs:documentation> The Atom entry construct... </xs:documentation></xs:annotation>...
</xs:complexType>

Page 81 of 162

Extremes of XML

To bring SAWSDL into line with the next part of this we'll translate the model references into proper schema
appinfo annotations.
<xs:complexType xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:sawsdl="http://www.w3.org/ns/sawsdl" name="entryType"><xs:annotation><xs:appinfo><sawsdl:modelReference
href="http://bblfish.net/work/atom-owl/2006-06-06/#Entry"/></xs:appinfo><xs:documentation> The Atom entry construct... </xs:documentation></xs:annotation>...
</xs:complexType>The next question is, how do you utilise theseannotations to build a mapping from a document tree toa graph? An XSLT transform is a good option but thevariability in schema construction has always made thistask more complicated than we would care for.
Alternatively, Type Introspection in XQuery, as
described by Mary Holstege in her paper 2012 Balisage
paper of the same title, is a useful tool to aid this process.
The ability to access schema annotations, through an
XQuery API, whilst processing an instance of that
schema provides a quick and easy root for interpretingthe schema to ontology references and thus extracting an
RDF graph derived from information within an XML
document that has not been otherwise explicitly markedup as such in the first place.

This technique may have uses beyond the domain of
Web Services as it has the potential to describe mappings
between semi-structured XML data and RDF ontologies
that enable hybrid Content / Graph stores to index
content for both tree and graph based querying from thesame source XML. To be able to almost seamlessly mix
XQuery and SPARQL querying over the same content
would be a powerful toolset in deed. Initial experiments
have gone some way to proving that this technique works
by generating an RDF dataset that can be loaded into agraph store and queried separately, but to fully realise
this concept would require an indexer to be able to build
the 'shadow' graph index via the these rules rather thancreate an entire graph and thus duplicate much, if not allof the original information.
In a world that's throwing itself<!-- OWL Class definition of an Atom Entry. -->at the notion of schemaless<owl:Class xmlns:owl="http://www.w3.org/2002/07/owl#"
databases it seems to fly-in-thexmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
face of the this current vogue toxmlns:rdfschema="http://www.w3.org/2000/01/rdf-schema#"
rdf:about="http://bblfish.net/work/atom-owl/2006-06-06/#Entry"> suggest using schemas and<rdfs:label xml:lang="en">Entry Class</rdfschema:label>
ontologies to define these<rdfs:comment xml:lang="en">see 4.1.2 of the rfc 4287 spec</rdfs:comment> mappings up-front. But, where...
better should we do such a thing</owl:Class>than in declarative languages that are independent of<!-- Source Atom XML Fragment. -->implementation.
<entry xmlns="http://www.w3.org/2005/Atom"><title>Atom-Powered Robots Run Amok</title>...
</entry>

4. Processing
There are many processes that can been seen

# Triple from the shadow graph (Turtle).
as a series of transformations of static@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix awol: <http://bblfish.net/work/atom-owl/2006-06-06/#> . information from one form or structure to[ a awol:Entry ] .

Page 82 of 162

another but the same can also be said for
application state. As surely as a document's structure is asnapshot of its current revision so is an application's data
structures a snapshot of its current state. XML has often
been used to describe an application's initialconfiguration or to maintain a set of preferences butwhat about using it in the actual manipulation of thatapplication's state. Crossing the line from passive state
representation to active state manipulation.

Extremes of XML

The XML Pipeline Language (XProc) succeeds as amechanism for orchestrating sequences of XML
processing steps but what is potentially a moreinteresting and useful aspect of XProc is its ability tocreate a Domain-specific Language (DSL) for a particular
set of steps. You can create, what could be call 'naked'
pipelines that use the underlying XProc grammar or,
alternatively, you can abstract that away behind typed
step descriptions.
The new state processing grammar that you create
becomes your DSL for this particular task and which ishosted within the XProc framework. A practical exampleof this being the ability to manipulate the configurationof an application; the consumption and transformationof application state and the execution of update
instructions to the application.
<admin:get-configuration/><admin:forest-create forest-name="forest1"
host-id="hp6910-772" data-directory=""/><admin:database-create ml-database-name="data"
security-db="Security" schema-db="Schemas"/><admin:save-configuration-without-restart/>

5. Publishing
For the most part, publishing appears to be a reasonably
straightforward process of marking-up textual content
with semantically meaningful elements that tell us more
about the author's intent than just the plain words, it
gives structure. There are extensive grammars for thisprocess, DocBook and XHTML being the more notable
but markup is only the first part.
The separation of concerns tells us to keep themarkup of content free from presentation and layout.
Whilst keeping them separate is the goal, at some point
they meet in order to present a finished result.
Taking the publishing of information in an
intriguing direction is the Document Description
Format (DDF), a framework that looks at documents as
Functions over a dataset and as such presents newopportunities for the merging of data and documents. As
we attempt to tackle the problems of handling vast
amounts of data and getting it to the point ofconsumption in a manner tailored to the recipient, DDF
is, quite possibly unique, as a publishing format in that it
can consume many and varied sources of information,transform them to a common structure and then apply
presentation constraints that can build flexible pagelayouts that adapt to variability in the source data.
DDF defines three layers:• Data Binding• Common Structure• Presentation

Application configuration can be viewed as a pipeline of
instructions that involve a source input as the current
state, a set of actions that may be ordered due todependencies, and an instruction to commit the changes.
Such a set of steps, that make a specific configuration
change, can be regarded as compound action and canthemselves be composed into large sets of changes.
<doc><admin:get-configuration/><mla:create-database database="data"
schema-db="Schemas" security-db="Security"><p:input port="forests"><p:inline><mla:forests><mla:forest name="forest1" host="hp6910-772"
data-dir=""/></mla:forests></p:inline></p:input></mla:create-database><data>
Source data for the document.
</data><struct>
Structure and Transforms.
</struct><pres>
Presentation and Transforms.
</pres></doc>

Through these layers the separation of concerns inaggregation, structuring and presenting information are
maintained, within a portable document structure.
DDF's main design decision was to 'consider the
The composability of XProc processing steps is anincredibly powerful tool that, allied with the ability to document as a function of some variable application
define your own steps and abstractions, has applications data'. When the document is evaluated, data (may befrom a number of sources), is transformed into aoutside of its originally intended domains.
common structure from which a view can be generated
for presentation. Embedding XSLT syntax and semantics
into the document provides the mechanism fortransforming within and between the layers.

Page 83 of 162

Extremes of XML

A value of XProc has been to lift the orchestration of
XML transformations out of programming language
dependency and into the domain of declarative
programming. However, any additional sequencing ofpipeline executions is still controlled outside of thepipeline. XProc step execution can be seen as inherently
event driven due to the fact that steps start when an
input is received at a source input port and end when aresult appears on a result output port.
The sequencing is implementation dependant
therefore allowing the possibility of asynchronous step
processing. Now, imagine an XProc Pipeline that
aggregates information from one or more sources.
6. Interaction
Through the use of SMIL Timesheets it is quite
An area where XML applications have found mixed conceivable that you can orchestrate the execution of
success is in user interaction. XForms is our most notable steps in a pipeline to explicitly execute steps
success which, despite having a long history of partial concurrently, like requesting information from multiple
implementations, which have come and gone over the web services:
years, XForms has managed to retain and has even grown <?xml version="1.0" encoding="UTF-8"?>enthusiasm for it with the more recent rise of Native <p:pipeline xmlns:p="http://www.w3.org/ns/xproc"
xmlns:terms="http://example.org/service/terms/"
XML databases.
name="term-aggregation"
That said, there's another area of interaction, thatversion="1.0">
goes beyond conventional form filling, and that involves
animation. The Synchronised Multimedia Integration<p:pipeinfo>
Language (SMIL) has a whole subset of its extensive<smil:timesheet
specification set aside for animation and it is a
xmlns:smil="http://www.w3.org/ns/SMIL30"><smil:par>specification that is shared by Scalable Vector Graphics<smil:item select="#service1"(SVG).
begin="term-aggregation.begin" dur="30s"/>
Whilst there is a lot of effort that goes into building<smil:item select="#service2"
scripting libraries for every form of user interaction andbegin="term-aggregation.begin" dur="30s"/>
animation these are mostly reinventions of the same basic<smil:item select="#service3"
principles of user interface or temporal event driven
begin="term-aggregation.begin" dur="30s"/></smil:par>alterations to the properties of some object in the page.
</smil:timesheet>
SMIL, like XForms has been there, in the background,</p:pipeinfo>
for many years and SMIL 1.0 is the oldest of theapplications of XML, becoming a recommendation in<terms:get xml:id="service1" name="OpenCalais"
June 1998 (four months after XML 1.0 itself).
href="http://opencalais.com/"/>SMIL Animation has, historically, been tied to its<p:sink/><terms:get xml:id="service2" name="MetaCarta"
host document types: SMIL, HTML+TIME and SVG
href="http://www.metacarta.com/"/>but with the advent of SMIL Timesheets that<p:sink/>
dependecmcy was broken, enabling the principles of<terms:get xml:id="service3" name="Yahoo"
event sequencing and orchestration being applied, outhref="http://search.yahooapis.com/"/>of-line. Normally we might consider animation<p:sink/>techniques being applied to presentation, timing of slide<p:wrap-sequence xml:id="aggregate-terms"
decks, linking steps in a configuration wizard or just
name="aggregate"
simple page display effects. But, if you can orchestratewrapper="terms:group">
the display processes of information in a web page, why<p:input port="source">
not orchestrate the aggregation and transformation of<p:pipe step="OpenCalais" port="result"/>
information in a workflow?
<p:pipe step="MetaCarta" port="result"/>
Partial evaluation is also possible where processing resultsin new transformations to be applied in subsequent
document workflow steps.
Along with the idea of 'document as a function',
another aspect of DDF that is worth noting is thepresentation system that extends the notions of layout
beyond both the conventional flow and and copy-hole
concepts. Although supporting flows, as can be found inHTML viewers, the introduction of constraints between
components that ensure their spatial relationship ismaintained within a variable data environment.

<p:pipe step="Yahoo" port="result"/></p:input></p:wrap-sequence></p:pipeline>

Page 84 of 162

Extremes of XML

The above example illustrates how three calls to separate
web services, that would conventionally run sequentially,
could orchestrated to run explicitly in parallel. Each stepis defined to begin when the pipeline starts and to have amaximum buration (timeout) of 30 seconds.
This is another example where bringing together
seemingly unrelated applications of XML can extend
their respective usefulness and push the boundaries ofwhat can be achieved with XML.

But what has this to do with XML? Another boundary,at the edge of the XML envelope is that between text andbinary data formats. XSLT is designed for generating
text-based representations of information. When XSLT
2.0 came along it introduced the concept of sequences. A
sequence can be made of nodes or atomic values,
including integers. A rasterized image is just one longsequence of integer values with some header info to tellyou, amongst other things, the format of a pixel's colour
values. To push the boundaries of the XML envelope still
further, using XSLT 2.0 and SVG, it is possible to
7. Presentation
develope a constrained implementation of the Reyes
The final step from information to presentation is the rendering pipeline and a TIFF encoder that proves theconversion of text, values and graphics into a visual point that XSLT is not just limited to text and markup
representation, a rasterization of shapes for display or output but can deliver, in effect, binary data too.
printed media.
When it comes to vector graphics on the web, SVG ison the ascendant again which is primarily due toincreased browser support. As a vector format, SVG has
had the lion's share of people's attention whilst X3D, theXML representation of the Virtual Reality Modeling
Language (VRML) has never really taken the web bystorm. Alongside these public facing vector graphics
markup grammars there are 'behind the scenes' formats
like Collada which is squarely aimed at 3D digital assetsinterchange and archiving. But, in all these cases theobjects or scenes they represent end-up being rendered
into raster image formats for presentation on some formof device or media.
Experts at Pixar, the Computer Generated Imagery(CGI) film studio that brought us Toy Story and the like The image you see here was created from a simple SVG
developed a 3D image rendering architecture called graphic where all the rectangles are sub-divided down
Reyes. The principle behind the Reyes rendering into pixel sized micro-polygons upon which deptharchitecture is to take complex 3D graphics primitives sorting, transparency, colour and final sampling
and break them down into smaller, simpler components calculations are applied to generate a sequence of pixelthat can be more easily processed for visibility, depth, value integers before finally encoded according to thecolour and texture. Reyes defines a processing pipeline TIFF format and then serialised as a Base64 encoded file,that applies a series of transforms to the source primitives to make it possible to output the resultant image data.
until they are finally at the point where sampling can The icing on the cake would be to create a new serializer
take place to generate the resultant pixels.
for the XSLT processor so that a binary file could be
generated directly as the output.
This example is somewhat extreme as it is highly
questionable as to whether XSLT is the right technology
for rendering simple images let alone complex photo
realistic ones. Certainly it is not fast and it is not very
efficient either but it illustrates that the boundaries are
not hard and fast. In fact the main components of thepipeline processing were relatively straight forward as
XSLT provides the tools to transform and XML themeans to represent the graphic primitive information.

Page 85 of 162

Extremes of XML

8. Overlapping Edges

9. Conclusion

In the XML Envelope there is no clear dividing line
between the categories:
1. All the categories require description.
2. Modeling relies upon processing to enable mappings
between data models.
3. Information and application state can be transformed
through processing.
4. Publishing utilises processing at every point of its lifecycle.
5. Interaction can be applied to the orchestration
processing.
6. Processes of presentation can be enhanced bymechanisms of interaction.

This walk around the edges of this so called XML
Envelope has illustrated, I believe, that whilst so muchhas been accomplished over the last 15 years and fromwhich we have very robust tools, techniques and design
patterns that we can apply, I'm convinced that XML
does not really have any hard and fast limits to its
application and whilst we may dwell upon issues ofserialisation and simplification the essence of what has
been created is right and good and that there are many
more applications to which XML and its attendant
technologies can be put than we might have originally
imagined...

Page 86 of 162

The National Archives Digital Records
Infrastructure Catalogue: First Steps toCreating a Semantic Digital Archive
Rob Walpole
Devexe Limited / The National Archives<rob.walpole@devexe.co.uk>

Abstract

1.2. The Digital Records InfrastructureThis paper describes the context and rationale for developinga catalogue based on Semantic Web technologies for The
National Archives of the United Kingdom as part of theDigital Records Infrastructure project, currently in progress
at Kew in London. It describes the original problem that
had to be resolved and provides an overview of the design
process and the decisions made. It will go on to summarise
some of the key implementation steps and the lessons learned
from this process. Finally, it will look at some of the possible
future uses of the catalogue and the opportunities presented
by the use of Semantic Web technologies within archives
generally.

TNA have been at the forefront of meeting this digitalpreservation challenge and have made great strides infinding solutions to many of the issues along withcolleagues from other national archives, libraries andacademia. In 2006, they deployed the Digital Repository
System (DRS) which provided terabyte scale storage.
Unfortunately DRS can no longer meet the vastly
increased volumes of information produced by the Big
Data era or the “keep everything” philosophy that cheap
storage allows.
Now a new and far more extensible archive system,
the Digital Records Infrastructure (DRI), is being built
on the foundations of DRS to provide a quantum leap inarchive capacity. This new system will allow long term
controlled storage of a huge variety of documents andmedia. Digitised Home Guard records from the Second
World War were used for the proof of concept and many
more record collections, such as the Leveson Enquiry and2012 Olympic Games (LOCOG), are now awaiting
accession into the new system. At its core DRI provides
its massive storage using a robot tape library. Althoughtapes provide highly resilient storage if treated andmonitored carefully, they are not suited to frequent
access. Therefore, the archive is designed to be a “darkarchive”. In other words, it is powered down until anaccess request is received.

1. Background1.1. The National Archives
The National Archives (TNA) are the official archives of
the UK Government. TNA holds over 11 million
historical government and public records [1] in the formof documents, files and images covering a thousand yearsof history. The vast majority of the documents currently
held are on paper. However, as the digital revolution
continues, this will soon be overtaken by a tsunami ofdigital files and documents for which a new andpermanent home must be found that will allow
controlled access for future generations.
These documents can take many forms including
standard office documents, emails, images, videos andsometimes unusual items such as virtual reality models.
Digital preservation brings a myriad of challenges
including issues such as format recognition, softwarepreservation and compatibility, degradation of digital
media and more. Some of these issues were clearly
demonstrated by the problems encountered by the BBC
Domesday Project [2].

doi:10.14337/XMLLondon13.Walpole01

Page 87 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

Although there will be frequent demands for access tothe data in the archive, many of these requests can be
met by substitutes from a disk cache. For example,
scanned documents can be substituted with a lower
quality JPEG file from disk, instead of the original JPEG
2000 held on tape. Whenever data is retrieved it will becached on disk for the next time so that frequently
requested items are always promptly available.

1.3. The DRI Catalogue
The DRI Catalogue is perhaps best described as aninventory of the items held within the repository. It isdistinct from the TNA Catalogue. The latter is acomprehensive accessioning, editorial management andpublic access system spanning both paper and digital
documents.
As the tape archive cannot easily be searched, it isvital that rich metadata is readily available to tellarchivists and other users what data is being held. Someof this metadata comes from the records' providers
themselves, usually a government department. Some isgenerated as part of the archiving process while some isobtained by inspecting or transcribing the documents.
With each collection of data stored, a comprehensive
metadata document is built up. A copy of this metadata
is placed in the archive and another copy is sent to
Discovery [3], TNA’s public access search portal,
provided the record is open to the public.
Controlled release of material from the archive is ofparamount importance. Although the majority of data in
the archive is open to the public, some is not. This may
be for reasons of national security, commercial interest or
simply because it would breach someone's privacy. For
example, a service and medical record is held for eachmember of the Home Guard. Service records are opened
to the public when the soldier in question is known to be
deceased. Medical records on the other hand are only
released some time later, usually not until the record
itself is 100 years old. Because some members of theHome Guard were very young when they served, it ispossible they would still be alive today.
This crucial need to provide fine-grained control overrecord closure lies at the heart of the DRI Catalogue's
requirements and has provided some of the key
challenges during implementation, which will be
discussed in more detail further on.

Page 88 of 162

2. Requirements
Archiving records is only of value if those records can beaccessed and viewed by researchers and members of the
public. TNA's search portal for the archives is Discovery
which holds over 20 million descriptions of records.
Once a user has located a record of interest fromsearching Discovery they can request to see the originalitem. In cases where the record has been digitised a builtin image viewer allows the record to be viewed on-line.
Until recently, the majority of records were on paper andpainstaking work by the cataloguing team provided thismetadata which was stored in the original electronic
catalogue system (PROCAT) which has now been
replaced by Discovery. In future the majority of records
will come via DRI. DRI has two fundamental
responsibilities with regard to Discovery which we canclassify as closure and transfer which are explained in
more detail below.

2.1. Closure
Until recently, most public records were closed for 30
years, however the government is now progressively
reducing the closure period to 20 years. Some records,
particularly those containing personal data, are closed for
longer periods - up to 100 years. However thejustifications for closing records for longer periods are
scrutinised by a panel of academics and other experts [4].
Closure of records has two possible forms: the record
itself can be closed but the description may be open or,alternatively, the record and the description may both beclosed. In either situation it is very important that DRI
does not release any closed records to the public.
Closure can apply at various levels. In one case adocument may be open whereas in another only the
metadata could be open. In some cases, even themetadata could be closed or possibly a whole collectionof data, depending on the content and the reasons forclosure.

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

2.2. Transfer

2.3.1. Results

DRI transfers information on all of the records it holds
to Discovery for public access. In the case of a closed
record what the public sees depends on whether just therecord, or the record and the description are closed. If
the record is closed but there is a public description thiswill be shown, albeit with no possibility to see the actual
record. In the case of a closed record and description
they will be able to see that there is a record but not what
the record is about. In other words, whilst providing as
much public access to the records as possible, closed
information must be filtered from the public view at all
times.

Relational

2.3. Initial Approach
In order to establish the most effective way of dealing
with the closure problem, three different approaches
were prototyped simultaneously. These approaches werebased on three fundamentally different models of thecatalogue data. These models can be categorised as therelational, hierarchical and graph approach.
• Relational – this approach was to retain the existing
relational database management system for storing
catalogue data but to rewrite the SQL queries used toestablish record closure status. On the face of it thiswould seem to be the most obvious and expedientsolution.
• Graph – the second approach was to re-structure thecatalogue as a graph using RDF and then query it
using SPARQL. This was the least well understood
approach of the three but its increasingly common
usage in large scale applications suggested it was a
viable solution.
• Hierarchical - the third approach was to use XML to
describe the DRI catalogue, store the catalogue data
in a native XML database and query the data using
XQuery. The nature of the catalogue is strongly
hierarchical and so this seemed a good solution.
Before any of these approaches could be tested extra dataneeded to be added the existing DRI catalogue. It wasessential that items in the catalogue knew their ancestry,or at least the item that represented their parent. To
achieve this a simple Scala program was written whichconnected to the database, built up a map of catalogue
entries and their descendants and then added a parent
reference column to the main catalogue table by looking
up the entry from the map.

Rewriting the SQL used to extract catalogue data led to adramatic improvement in query response times. Whereas
the original query took hours to complete, the new queryusing the extra parent column information completed inminutes. Optimising this query may well have further
reduced the query response times.
Graph

Considerable effort had to be made to create anenvironment where the graph approach could be tested.
These steps will briefly be described here and covered in
more detail later on.
1. Create a mapping from the relational databasecolumn entries to triples using D2RQ [5]2. Export the data from the relevant relational tables
into triples using D2RQ.
3. Load these new triples into a triple-store (Jena TDB[6]) which could be accessed via a SPARQL endpoint(Jena Fuseki[7]).
4. Write SPARQL queries using SPARQL 1.1 property
paths [8] to establish closure status.
Once all of this was done the results of this experimentwere stunning. It was possible to establish the closure
status of any catalogue item based on its ancestors anddescendants in seconds or split-seconds. The
performance far outstripped that of the relational
database queries. It was also possible to write queries thatshowed the ancestors and descendants of any item andverify beyond doubt that the results were correct.
Hierarchical

Testing of the hierarchical approach was abandoned; thereasons for abandoning this approach were threefold:
1. It was felt that the graph approach offered a goodsolution to closure problem.
2. The graph tests had led to a better understanding of
this approach and, with this understanding, a numberof new and interesting possibilities had arisen in termsof what could be done with the catalogue. It was feltthat the hierarchical approach did not offer thesesame possibilities.
3. Finally, and sadly, project deadlines and cost
overheads meant that, although it would have beeninteresting to complete the hierarchical test, the factthat a good solution had been found obliged the
project to move on.

Page 89 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

2.3.2. Conclusion
The issue of closure had meant that it was necessary for
the DRI project team to fundamentally question thenature of the DRI catalogue. Which of the available
models best represented the catalogue? While relational
tables may be very good at representing tabular data suchas you find in a financial institution they were found to
be less suited to describing the complex relationships
within the catalogue.
Because TNA accessions data from the full range ofgovernment activities it is difficult to predict what
structure this data will have and what information willneed to go in the catalogue. The hierarchical model
offers a good solution for documents and publications
but falls down when attempting to describe the polyhierarchical structures that you find in archives. For
example a scanned document may contain data about
many people. How do you nest this information in ahierarchy without repeating yourself many times over?
Fundamentally the archive holds information about
people, their relationships and activities over the courseof time. These things are complex and varied – they areafter all the nature of the world around us. The
conclusion of the experiment was that the graphapproach not only solved the immediate problem ofclosure but also most closely modelled our complex
world and would in the future provide a powerful toolfor discovering information held within the archive.

3. Design
3.1. Technology
The catalogue trials had provided valuable experience ina number of technologies. This included tools forworking with RDF such as D2RQ and Apache Jena plus
experience of new standards-based formats and languages
such as Turtle [9] and SPARQL. An important factor in
the technology choices made was the preference for using
open source and open standards specified by the UK
Government in the Government Services Design
Manual:“..it remains the policy of the government that,
where there is no significant overall cost difference
between open and non-open source products that fulfil
minimum and essential capabilities, open source will beselected on the basis of its inherent flexibility.” [10]And also for using open standards as stipulated by
criteria 16:“Use open standards and common Government
platforms (e.g. Identity Assurance) where available” [11]Page 90 of 162

All of these technologies met this criteria as being either
open source (D2RQ, Jena) or open standards (Turtle,
SPARQL).
Furthermore the trials had given developers a headstart with these particular tools and there was felt there
was no particular benefit to be gained by switching to analternative solution at this stage. Having said that, theuse of open standards means that, should the existing
open source technology cease to meet TNA's
requirements, the overhead in moving to a new tool-set
should be kept to a minimum.
Another significant reason for choosing the Apache
Jena framework was the excellent Java API provided.
DRI is predominantly a Java based system. Java waschosen originally because the underlying technology of
DRI (Tessella's Safety Deposit Box – SDB [12]) was
written in Java and therefore Java was the natural choicefor extending SDB functionality. The DRI development
team naturally had strong Java skills and Jena's API
provided a straightforward way for developers familiar
with Java to start working with RDF.

3.2. The Catalogue Services
The DRI catalogue is required to provide a number ofkey services:
Accessioning

Accessioning Firstly, it must accept new entries in the
catalogue when data is initially accessioned into DRI.
For each item accessioned it must provide a unique
identifier which is persisted with the entry. In fact theidentifiers generated must be globally unique identifiers[13].
Currently the catalogue recognises a number of item
types. These are:
• Collection – this describes a large related group of
documents, for example the Durham Home Guard
records are represented by a single collection.
• Batch – this represents a batch of records on disk.
This is how records are initially received by TNA andusually represents a substantial volume but it may or
may not be a whole collection.
• Deliverable Unit – this represents a single item of
information. It is a term coined from the paperarchive world and represents something that can behanded to someone else. This may be a box orrecords, a folder or a single document. Similar criteria
are used to for digital records.

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive• Manifestation – there are different types ofmanifestation. For example images have preservationand presentation manifestations. Preservation
manifestations of these would be the highest quality
images while presentation ones are a lower quality fordisplay purposes.
• File – these are the actual digital files held within thearchive.
Each of these types comes with a set of properties whichmust be retained in the catalogue, including things like
TNA Catalogue references and closure information.
Closure Updates

The catalogue must provide the functionality to alter the
closure status of a record (subject to review by an humanoperator).
Export

The catalogue must provide the functionality forexporting records. This is normally done in order totransfer the record to Discovery. The export process itself
involves numerous steps in a controlled work-flow. The
catalogue's responsibility is to allow a review of items forexport and to maintain a record of the status of theexport work-flow.
Lists

The catalogue must also provide the ability to persist listsof records. These are generic lists of records which may
be used for a variety of purposes. Currently they are used
to group records due for opening and export but there
are likely to be other uses in the future.
From these requirements it can be gathered that thecatalogue must provide read, write, update and delete
access to the triple-store. The Apache Jena frameworkprovides a straightforward approach to theserequirements.
• Reading data can be done using the SPARQL Query
Language [14].
• Writing data can be done by creating and persisting
new triples• Updating and deleting can be done using the SPARQL
1.1 Update Language [15].
There are a number of different ways of performing thesetasks including:
1. Using the Jena command line utilities
2. Using the Jena API to work directly with the triplestore.
3. Using a SPARQL server such as Fuseki to performthese tasks

The SPARQL server provides an attractive solution
which allows queries to be executed quickly andconveniently over HTTP as well as allowing new sets of
triples to be posted to the triple-store. The Jena Fuseki
SPARQL server also includes a built in version of the
TDB triple-store. As TDB can only be accessed safely
from within one Java Virtual Machine [16] it makes
sense to use this built-in version with the SPARQL server
approach. This server has a number of endpoints built inincluding a SPARQL query endpoint which can beaccessed via a web browser. This not only provides auseful tool for developers but could in the future be
exposed to staff within TNA, or even to the generalpublic, who could then query the data for themselves
given a little SPARQL knowledge.
Jena Fuseki with embedded TDB was chosen as thesolution.
One hurdle to acceptance of this semantic technology
stack within TNA however was the need to develop skills
around semantic search and in particular in terms oflearning RDF syntax and the SPARQL query language.
One solution to this problem is the Linked Data API[17]. This API offers a way for SPARQL queries to be
pre-configured and then accessed via RESTful URLs.
For example you can configure a SPARQL query thatlocates a catalogue entry's descendants and then accessthis via a pre-set URL structure e.g.
http://{server-name}/{catalogue-identitifer}/descendant

Elda [18] is an open source implementation of the API
written in Java. The SPARQL queries within Elda are
configured using the Turtle format so in this case theconfiguration for this specific URL would looksomething like this:spec:catalogueItemDescendant a apivc:ListEndpoint; apivc:uriTemplate "/catalogue/{uuid}/descendant"; apivc:variable [apivc:name "uuid";
apivc:type xsd:string]; apivc:selector [apivc:where """?catalogueItem dcterms:identifier ?uuid .
{?item dri:parent+ ?catalogueItem .
}
"""
];
.

Once this endpoint is registered (also within theconfiguration file) any requests that match this URI
template will execute a SPARQL SELECT statement
returning any matching catalogue items represented by
the variable ?item. The string value of the UUID passed
in via the URI is allocated to the ?uuid variable when thestatement is executed.

Page 91 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

Within this statement you will notice the dri:parent+
property. The + signifies a SPARQL 1.1 property path,in other words it will find the parent and the parent's
parent and so on until there are no more matches. The
dri prefix indicates that this parent property is part of the
DRI vocabulary which is discussed in more detail later.
Elda was chosen as the read API in front of Jena
Fuseki. This meant that all reads would go via Elda andall writes, updates and deletes would go directly to
Fuseki.
One final problem remained with regards to theacceptance of this solution within TNA. It was perceived
that there was some risk involved in using such a(relatively) new technology stack and therefore theimpact on other systems, in particular SDB, had to be
kept to a minimum. To solve this problem it was decided
to develop a simple bespoke Catalogue Service API
between SDB and the DRI Catalogue. Having SDB talkto this API meant that if the new DRI Catalogue failed
to deliver the expected results for some reason then it
could be swapped for another solution with only theCatalogue Service API needing to be re-engineered.

Page 92 of 162

Both Elda and the Catalogue Service API would be run
within an Apache Tomcat web container in common
with other components of DRI and SDB. Jena Fuseki
however would need to be run as a standalone
application as there is currently no mechanism for
deploying Fuseki as a web archive within Tomcat,
although it is a feature that has been discussed in depth[19].
The final design of the catalogue system is shown
below.

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

3.3. DRI VocabularyIn
thisexample
a
row
from
the
TNADRI.COLLECTION table is mapped to an
instance of the dri:Collection class and assigned a URI
based on the UUID column of the table. In means that
we end up with a resource described by the following
RDF triple (in Turtle format).

During the initial testing of a graphbased catalogue it was quickly
discovered that a vocabulary forcatalogue terms would need to be
developed in order to be able todescribe catalogue specific entities and <http://nationalarchives.gov.uk/dri/catalogue/collection/example1234>
rdf:type dri:Collection .
relationships. This vocabulary would
In other words a resource of type collection.
need to be established, as far as
Using this mapping technique the contents of thepossible, in advance of the initial import of catalogue
data. Although it would be possible to reclassify terms RDBMS catalogue were dumped into text files in theonce the data was converted into RDF, it is more Turtle format, which totalled approximate 1Gb of data.
expedient to resolve these terms up front and, as far as Approximately 8 million triples.
The second stage was to load these triples into TDB
possible, avoid renaming later.
In keeping with the W3C's guidelines laid out in the using the tdbloader command line tool which is part of
Cookbook for Open Government Linked Data[20] the Apache Jena framework. However this raw data wasexisting vocabularies are re-used as much as possible. still not in the desired format for use within the new
Extensive use is therefore made of OWL [21], Dublin catalogue. For starters the closure information was notCore [22], RDF Schema [23] and XML Schema [24] linked to the resources it referred to. Closure information
however quite a few terms are very specific to the is comprised of five elements:
Catalogue. Although catalogue items have “parents” • Closure period – how long the record must remainsuggesting use of the FOAF [25] vocabulary it wascloseddecided that catalogue items are emphatically not people • Description status – whether the description is open or
and the rules around people's parents (one mother andclosedone father) do not apply in this case. Use of the FOAF • Document status – whether the document is open or
vocabulary could therefore cause confusion at a later
closeddate. A DRI parent term was therefore created.
• Review Date - when the closure is due for review or
The full vocabulary is described in Appendix A, The
when the record was opened
DRI Vocabulary.
• Closure type – gives more information about the typeof closure
3.4. Implementation
However closure cannot be said to be a resource in its
own right as it only exists in the context of a catalogue
The first stage of the implementation was to extract the item. RDF has a concept for this type of information
existing catalogue data from the RDBMS where it was which seemed highly appropriate, the blank node orheld. D2RQ was to be used for this as had been done for bNode. Creating these blank nodes would require somethe initial trial. The difference now was that we had transformation of the data however. While there is noestablished a vocabulary the terms to be used in the new equivalent of XML the transformation language XSLT
catalogue. With this in place it was possible to map the for RDF the SPARQL language itself allows acolumns in the database to the terms that would be used transformation through use of CONSTRUCT queries.
in the RDF. This was done using the D2RQ mapping In this case new triples can be created based on existing
file, a sample of which is shown below.
triples.
# Table TNADRI.COLLECTION
map:collection a d2rq:ClassMap;
d2rq:dataStorage map:database;d2rq:uriPattern "http://nationalarchives.gov.uk/dri/catalogue/collection/@@TNADRI.COLLECTION.UUID@@";
d2rq:class dri:Collection;
d2rq:classDefinitionLabel "TNADRI.COLLECTION";.

Page 93 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

By loading the data into TDB and then using theSPARQL query endpoint in Fuseki to run constructqueries, it was possible to generate new triples in thedesired format that could be downloaded in Turtle
format from Fuseki and then reloaded into a new
instance of TDB. The following CONSTRUCT queryshows how the new triples could be created. In this caseby combining file and closure information into a new setof triples relating to a single resource with the closure
information held in a blank node signified by the square
brackets in the construct query:

3.5. Catalogue Service API

The Catalogue Service API is a RESTful Jersey JAX-RS[26] application that reads and writes data to the TDB
triple-store. As per the design, reading data is done via
Elda and writing data is done via Fuseki. The actual API
itself is extremely simple and returns data in an XML or
JSON format. For example, in the case of creating a new
catalogue entry it simply takes in the TNA cataloguereference as a request parameterand generates an entry in the DRI
catalogue of the appropriate type
PREFIX closure: <http://nationalarchives.gov.uk/dri/catalogue/closure#>(this depends on the URI called)
PREFIX dcterms: <http://purl.org/dc/terms/>
and, if successful, returns the
PREFIX dri: <http://nationalarchives.gov.uk/terms/dri#>unique identifier in a snippet of
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
XML as follows:PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>

CONSTRUCT{?file rdf:type dri:File ;rdfs:label ?name ;dri:directory ?directory ;dcterms:identifier ?identifier ;dri:closure [rdf:type dri:Closure ;dri:closurePeriod ?closureCode ;dri:descriptionStatus ?descriptionStatus ;dri:documentStatus ?documentStatus ;dri:reviewDate ?openingDate ;dri:closureType ?newClosureType
] ;
.
}
WHERE
{
?file rdf:type dri:File ;rdfs:label ?name ;dri:directory ?directory ;dcterms:identifier ?identifier .
?closure rdfs:label ?identifier ;dcterms:creator ?creator ;dri:closureCode ?closureCode ;dri:closureType ?closureType ;dri:descriptionStatus ?descriptionStatus ;dri:documentStatus ?documentStatus ;dri:openingDate ?openingDate ;BIND(IF(?closureType = 1, closure:A, closure:U)
AS ?newClosureType).
}With the data cleaned, refactored and accessible via
Fuseki, development of the Catalogue Service API could
begin.

Page 94 of 162<result
xmlns=”http://nationalarchives.gov.uk/dri/catalogue”><uuid>e9d8f987-5d49-40f2-869b-a2172e3d362c</uuid></result>

In the process it generates a new unique ID, writes out
the triple to file using the Jena API to then it to Fuseki
over HTTP and, if all goes well, returns the UUID in
the response as shown above.

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

To create the new triples it first creates an ontology In the case of a SPARQL update it writes out a SPARQL
model using the Jena API which is populated with the file and posts this to Fuseki.
required classes from our vocabulary:protected Model createModel() {OntModel model =ModelFactory.createOntologyModel(
OntModelSpec.RDFS_MEM);
collectionClass = model.createClass(
DRI_TERMS_URI + "Collection");batchClass = model.createClass(
DRI_TERMS_URI + "Batch");deliverableUnitClass = model.createClass(
DRI_TERMS_URI + "DeliverableUnit");preservationManifestationClass =model.createClass(
DRI_TERMS_URI + "PreservationManifestation");exportClass = model.createClass(
DRI_TERMS_URI + "Export");recordListClass = model.createClass(
DRI_TERMS_URI + "RecordList");model.setNsPrefix("dri", DRI_TERMS_URI);
model.setNsPrefix("dcterms", DCTerms.getURI());
return model;}

public ResultWrapper createRecordListAddItemFile(
String recordListUuid,
String itemUuid) {Model model = createModel();Resource recordList = model.createResource(
RECORD_LIST_URI + recordListUuid);
Literal itemUuidLiteral =model.createTypedLiteral(itemUuid);
QuerySolutionMap parameters =new QuerySolutionMap();
parameters.add("recordList", recordList);parameters.add("itemUuid", itemUuidLiteral);ParameterizedSparqlString paramString =new ParameterizedSparqlString(
getQueryProlog() +getRecordListItemAddString(), parameters);
UpdateRequest update = paramString.asUpdate();File queryFile = getFileHandler().writeUpdateFile(update, "insert" + "_" +getDtf().print(new DateTime()) + ".rq");
ResultWrapper rw =new ResultWrapper(queryFile, null);

It then creates the necessary resources and literals which
return rw;}
are added to the model.
private String addCollection(Model model,
String collectionRef,
String label) {UUID uuid = UUID.randomUUID();
Resource collection = model.createResource(
COLLECTION_URI + uuid.toString());
collection.addLiteral(RDFS.label, collectionRef);
collection.addProperty(RDF.type,collectionClass);collection.addLiteral(DCTerms.created,
new XSDDateTime(Calendar.getInstance()));
collection.addLiteral(DCTerms.identifier,uuid.toString());
collection.addLiteral(DCTerms.description,label);
return uuid.toString();}

The model is then written to a file in Turtle format
which is posted to Fuseki via HTTP.

In the above example the getQueryProlog() and
getRecordListItemAddString() methods generated thenecessary text for the SPARQL update as follows:
protected String getQueryProlog() {String prologString ="PREFIX dcterms: <"+DCTerms.getURI() + "> \n" +"PREFIX dri: <" + DRI_TERMS_URI + "> \n" +"PREFIX rdf: <" + RDF.getURI() + "> \n" +"PREFIX rdfs: <" + RDFS.getURI() + "> \n" +"PREFIX owl: <" + OWL.getURI() + "> \n" +"PREFIX xsd: <" + XSD.getURI() + "> \n";return prologString;}private String getRecordListItemAddString() {StringBuilder deleteBuilder =new StringBuilder();deleteBuilder.append("INSERT { ?recordList dri:recordListMember ?item . }");
deleteBuilder.append("WHERE { ?item dcterms:identifier ?itemUuid . }");
return deleteBuilder.toString();}

Page 95 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

In the case of reading the data it accesses Elda, requesting
XML format (for which a schema has been developed)
and unmarshalls this into JAXB objects from which it
extracts the required information. The results are thenmarshalled into the much simpler XML format describedabove.

3.6. Insights, Issues and Limitations
3.6.1. Elda
Whilst Elda and the Linked Data API provide enormous
benefits for users in terms of simplifying access to triplestores it has provided some challenges to the developers
wanting to implement SPARQL queries and make use of
the XML result format.
Elda extension

Early on in the development process it came to light that
Elda had a significant limitation in the type of SPARQL
queries it could run. Although Elda provides read accessto the underlying triple-store it was found to be
impossible to create new triples through the use of
CONSTRUCT queries. There was an early requirement
to know whether a record was open, closed or partiallyclosed. This information is not held within theknowledge-base but has to be generated as the result of aquery. Although you can embed SPARQL SELECT
queries within Elda there was no working mechanism fora CONSTRUCT query. As Elda is open source andwritten in Java, it was feasible for TNA to add thisfunctionality as an extension, which was subsequently
done.
As the project has continued however, we have foundthis functionality to be of limited benefit. Although there
is a temptation to use CONSTRUCT queries for all
kinds of things, quite often there is a more efficient way
to achieve the desired result, for example by configuring
a viewer within the API to control the properties
returned.
Furthermore it was found that complex constructqueries that relied on a number of SELECT sub-queries
became difficult to debug as there was limited visibilityof what was happening within the query. This led to arethink whereby more complex queries were built up
gradually using the Jena API calling a combination of
simpler sub-queries from Elda. This enabled us to embedlogging within the Java which could show the results of
sub-queries and this also gave us a structure for providing
much better error handling within the application.
Whilst the Elda construct extension is still in use, it islikely to be gradually phased out in the future.

Page 96 of 162

Elda caching

Elda is very efficient at caching query results. Whilst thisreduces query times for frequently called queries it cancause some confusion and frustration when the data inthe underlying triple-store is being frequently updated.
The cache can however be cleared by calling...
http://{server-name}/catalogue/control/clear-cache...and so the catalogue API calls this before any query
which is likely to be affected by updates.
Elda XMLAn early design decision was to use the XML results from
Elda and then unmarshall the results as JAXB objects
with the Catalogue Service from where the required
values could be extracted and returned. This meantcreating our own schema for the XML as there is no
publicly available one and, in any case, the XML
returned is dependent on the underlying vocabularies
being used. Because we had created our own vocabulary
we had no choice but to create our own schema.
An important point to note with Elda is that in thecase of item endpoints (i.e. where only one result will
ever be returned) a primaryTopic element is used tocontain the result. In the case of a list endpoint, which
can return zero to many results, an items element isreturned containing one item element for each result.
Understanding this enabled us to write generic services
for reading and writing these two types of result.
Elda Turtle

It is proposed that in future the DRI Catalogue Service
will make use of Turtle format results from Elda insteadof XML. Turtle response times seem to be considerably
faster than the equivalent XML (presumably because thisis closer to the native format of the data) and Jenaprovides a well developed API for reading RDF, meaningthat Plain Old Java Objects could be used within theCatalogue Service rather than specifically JAXB objects.
3.6.2. TDB
One point that was established early on in the design
process was that TDB could only be safely accessed fromone Java Virtual Machine. We had therefore chosen touse Fuseki. This presented us with a problem however
when it came to performing backups. We wanted the
backups to be controlled by a CRON job run on theserver itself. How could the server safely carry out abackup if TDB was being controlled by Fueski?

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

The answer was provided by the Fuseki management
console. A little documented but extremely useful featureof Fuseki which meant that the backup could becontrolled safely via the management console by calling
the relevant URL from the server shell.
wget -O - --post-data='cmd=backup&dataset=/
catalogue' http://{server-address}/mgt

3.6.3. Xturtle
Working with a semantic web technology stack meansthat you frequently have to manually edit RDF files in
Turtle format. For example the mapping files for D2RQ
and configuration files for Elda are written in Turtle.
Without a syntax highlighter, errors in these files can bedifficult to spot, especially as they get larger.
Xturtle [27], which comes as a plug-in for Eclipse,
provided us with a very useful tool for editing these files.
Especially as the majority of our developers were alreadyusing Eclipse as a Java IDE.

Secondly we are confident that the existing cataloguearchitecture can be scaled horizontally. This has alreadybeen done with other components. Additional servers
could be added, running further instances of the triplestore, possibly containing completely separate graphs.
Indeed there is a certain logic to one graph per catalogue
collection. With a SPARQL endpoint configured oneach collection it would be possible to have a “catalogueof catalogues” which would provide us with a panarchive search capability.
Finally, if the existing open source framework fails tomeet expectation the standards-based approach meansthat we can move to an alternative framework. For
example, TNA has previously worked with Ontotext inthe development of the UK Government Web Archive[29] which uses an OWLIM [30] triple-store containing
billions of triples.
We are confident that whatever issues arise with scalein the future, we are in a strong position to address them.

3.6.4. Scardf
Scala is being used increasingly within the DRI project.
The reasons for this are its scalability, compatibility withJava (Scala programs compile to JVM byte-code) andconcise syntax which results in less lines of code, better
readability and fewer opportunities for errors.
For this reason, scardf [28] is being considered for
future development of the Catalogue Service API within
DRI. The ScardfOnJena API appears to offer a slot in
replacement for the Java API currently being used. If the
project were being started over again now this may wellhave been the preferred language for the Catalogue
Service API, rather than Java.
3.6.5. Scale and performance
So far the DRI Catalogue has only been used for themodest amount of data currently held within DRI (theDurham Home Guard Records). Whilst no performanceissues have been encountered so far it is likely that thiswill become more of a concern as data accumulates.
Actual volumes are very difficult to predict and thenature of an archive is that it will always grow and nevershrink. For this reason extensibility has been a primaryconcern in all aspects of DRI development and thecatalogue is no different. Whilst we cannot predict future
performance we are encouraged by a number of factors:
Firstly the graph solution was chosen because of its
superior performance.

Page 97 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

4. The Future
So far what has been achieved with the DRI Catalogue isa fundamental remodelling of the catalogue using
Semantic Web technologies and a successful
implementation of a solution to the problem of closure.
With these new technologies in place further
enhancements of the catalogue can be considered whichwere not previously possible.

4.1. Named Entity Recognition
Using a semantic approach enables an Open World
Assumption. This means that it is “implicitly assumed
that a knowledge base may always be incomplete” [31].
It is always possible to add more information as we learn
it. As we extract more information from the content
going into the archive we can add it to our knowledgebase and this information can be transferred to Discovery
where it can be viewed and searched.
What could this really mean for people viewing therecord on the Web?
As a human being with a reasonable level of
education we can look at the following entry and make areasonable assumption that it refers to a soldier receiving
an award for conduct during the Second World War.

Page 98 of 162

It is easy to forget that the machine has no idea about
these concepts. As far as the computer is concerned thisis just a collection of string values. If we want thecomputer to help us search for information we need totell it about the concepts that we already understand. We
need to tell it that George John Potter is a Captain which
is a kind of officer, which is a kind of soldier, which is akind of person who serves in the army. We can tell thecomputer that he served in a “regiment” which is part ofan “army” and the regiment is called the “Royal Army
Medical Corps”. We can also tell it that we know this tobe a regiment within the “British Army”. If we identify
the Royal Army Medical Corps as a single entity we canthen say that other people also served in the Royal Army
Medical Corps and point at the same resource. With theaddition of these simple pieces of information suddenly
the computer can help us enormously in our search. The“Royal Army Medical Corps” is no longer just a piece oftext but a concept that the computer can relate to otherentities it also knows about. It can now tell us whatever it
knows about the Royal Army Medical Corps. For
example who else served in the Royal Army Medical
Corps? What theatres of combat or operations did it takepart in and when?
The follow is a machine readable representation of
Captain George John Potter's record previously
discussed. The nodes in the graph represent either
resources or literal values and the lines connecting thenodes are the properties.

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

4.2. Ontology-driven NLP
Let's say that we build up a dictionary (or ontology) of
these terms. As the documents are loaded into thearchive they could be scanned for these terms using a
Natural Language Processing tool such as GATE [32].
Let's take a real example from Discovery. The
following is the text of the recommendation for award
for the aforementioned George John Potter:“From 1 Aug 44 to 20 Oct 44 Bayeux, Rouen andAntwerp. During the period from 1 Aug to date thisofficer has carried the principal strain of establishing andre-establishing the hospital in three situations. His
unrelenting energy, skill and patience have been themainstay of the unit. His work as a quartermaster is themost outstanding I have met in my service. (A.R.ORAM)
Colonel Comdg. No.9 (Br) General Hospital”
I have underlined the terms that could reasonably beunderstood by the computer with the help of adictionary. We have place names, a date range, akeyword “hospital” a rank (Colonel) the name of a unit
(No.9 British General Hospital) and the name of aperson (A.R.Oram). The computer could reasonably be
expected to recognise A.R.Oram as he is himself thesubject of a record (he also received an award). Although
the computer would know him as Algar Roy Oram, it
would be computationally simple to add another namevalue with this common abbreviation. Likewise “Br”
could be recognised as an abbreviation for British.

As a result of this process the computer could perform
reification. In other words it could make statements that
may or may not be true but which are plausible
conclusions. For example, it could say “Colonel Algar
Roy Oram served with Captain George John Potter” or“Colonel Algar Roy Oram was based in Antwerp”. This
is inferred knowledge and may not be factual but it could
be marked as a theory in the knowledge-base, until sucha time as it can be shown to be (or not to be) a fact.

4.3. Semantic Search
Keyword searches tend to deliver a large volume ofirrelevant results because it is not usually possible tocommunicate the desired context to the search engine
mechanism. Semantic searching on the other hand allows
for selection of ontological terms when searching. For
example if you search for “George John Potter” in
Discovery you get 361 results whereas in fact, there is
only one result that actually refers to a person with thisname. Imagine if you were able to say that you werelooking for a match on person's name. To paraphrase
Bill Gates, context is king when doing Semantic Search.
This technique is known as query string extension.

Page 99 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

Semantic search offers other possibilities as well. If it
knows you are searching for a person named George
John Potter it would be possible for a semantic
knowledge base to look for terms that were closely
associated to matches. For example, George Potter'saward was given by a Colonel A.R. Oram. This
information could be held within the graph ofknowledge and therefore a semantic search could bringback results linking to Colonel Algar Roy Oram as well.
Through painstaking research it is possible to discover
this information now and this reveals that Algar Oram's
records describe in great detail the difficult anddangerous situation that he shared with George Potter in
Antwerp during the Second World War. In this way wecan discover new information about George Potter's
experiences during the war that are not directly expressed
but strongly implied. A semantic search could've
provided this information in seconds. This technique isreferred to as cross-referencing.
Because the knowledge base is a graph, it is possible
to go further into searches than traditional keyword
searching. For example in the case of George Potter you
could continue the search using the link to Algar Oram.
In this case you would find that he too was awarded a
medal and that this medal was awarded by a very senior
officer in the Royal Army Medical Corps who had
himself been decorated many times. Some of the medals
this individual received were for extraordinary acts ofbravery that saved many lives and the letters ofrecommendation make gripping reading. This may not
be what you were looking for originally but it provides
interesting context that you would not otherwise havefound. This is known as exploratory search.
Another possibility would be to allow machinereasoning within the catalogue. This would enable rules
to be applied. For example if the machine knew that
George Potter “served” with something called “No.9(British) General Hospital” in the “theatre of combat”
called “North West Europe (1944-1945)” it would bepossible to reason that certain other people “served with”
George Potter. This is new knowledge that is not
currently available in the catalogue. It is an example ofthe using the graph and the ontology to do reasoning.

Page 100 of 162

4.4. Linked Data (for the things the archive
doesn't have)Whilst TNA is a huge national source of information,
there is lots of pieces of official information that are
simply not there, but kept elsewhere. For example birth,marriage and death certificates from 1837 onwards areheld by the General Register Office or at local register
offices - prior to 1837 they are kept in parish registers;
military service records for soldiers and officers after
1920 are maintained by the Ministry of Defence; theLondon Gazette [33] is the official UK Government
Newspaper and has it's own knowledge-base of historical
articles. When it comes to places, an organisation such as
the Ordnance Survey would be a excellent source of
information. As long as an archive is kept in a silo, its
world view will remain incomplete.
Linked Data [34], the brainchild of World Wide
Web inventor Tim Berners-Lee, provides a way of unsiloing data and it is based on the standardised
technologies of the Semantic Web. By providing its
information as Linked Data, i.e. in RDF-based machinereadable formats, other organisations or individuals canconnect the dots between items of data. In the case ofGeorge Potter we know from his record that he set-up
field hospitals in Bayeux, Rouen and Antwerp. Whilst
the archive may not be an authority on these places,
someone could make a connection between a record heldin the archive and a resource that gives much more
information about these places. For example, DBPedia[35], which contains key information taken from
Wikipedia provides a great deal of useful and increasingly
reliable data. This could work both ways so not only
could external users make these connection but thearchive itself it could pull in information from otherplaces that provide Linked Data. For example at TNA,
UK location information could be pulled in from the
Ordnance Survey, allowing relevant maps and otherinformation to be shown within Discovery. At the sametime the archive could contribute knowledge back to
DBPedia, adding new entries and improving the qualityof existing entries on which it is an authority.

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

4.5. Crowd-sourced linking
TNA has a vast amount of information and it isunrealistic to think that a machine is going to be able toread everything and make all the right connections. It isgoing to make mistakes and it is going to miss things.
This is particularly the case with digitised documents, i.e.
scanned paper documents. Even with modern OCR
technology it is not possible to accurately read all these
items. Providing a way for researchers, archivists and thegeneral public to make connections would add enormous
value to the knowledge base. Allowing users to add tags
to a document is a common way to crowd-source
information but the terms tend to be very specific andindividualistic. Imagine if it was possible to selectdictionary terms from a semantic knowledge-base to
apply to these items, many new and previous overlooked
connections could be made.

I am grateful to a host of people whose knowledge andexpertise have made this work possible. In particular to
Bob DuCharme for his excellent book on SPARQL (nowworn out), Dr Harald Sack at the Hasso Plattner
Institute for his thorough course on Semantic Web
Technologies, the guys at Epimorphics who have given
us invaluable support and of course, above all, the folks
at The National Archives whose foward thinking has
enabled this project, in particular David Thomas, Tim
Gollins, Diana Newton, Peter Malewski and Adam
Retter.

Bibliography[1] The National Archives. Our new catalogue: the Discovery service.
http://www.nationalarchives.gov.uk/about/new-catalogue.htm[2] BBC. Domesday Reloaded http://www.bbc.co.uk/history/domesday/story. Copyright © 2013 BBC.
[3] The National Archives. Discovery http://discovery.nationalarchives.gov.uk.
[4] The National Archives. Freedom of Information Act 2000 http://www.legislation.gov.uk/ukpga/2000/36/contents.
[5] Free University of Berlin. D2RQ http://d2rq.org/.
[6] Apache Software Foundation. Apache Jena TDB http://jena.apache.org/documentation/tdb/.
Copyright © 2011-2013 Apache Software Foundation.
[7] Apache Software Foundation. Apache Jena Fuseki http://jena.apache.org/documentation/serving_data/.
Copyright © 2011-2013 Apache Software Foundation.
[8] W3C. SPARQL 1.1 Property Paths http://www.w3.org/TR/sparql11-property-paths/. Copyright © 2010 W3C.
[9] W3C. Terse RDF Triple Language http://www.w3.org/TR/turtle/. Copyright © 2013 W3C.
[10] GOV.UK. Choosing technology
https://www.gov.uk/service-manual/making-software/choosing-technology.html#level-playing-field.
[11] GOV.UK. Digital by Default Service Standard https://www.gov.uk/service-manual/digital-by-default.
[12] Tessella. Safety Deposit Box http://www.digital-preservation.com/solution/safety-deposit-box/.
Copyright © 2013 Tessella.
[13] Oracle. UUID http://docs.oracle.com/javase/6/docs/api/java/util/UUID.html.
Copyright © 1993, 2011 Oracle and/or its affiliates.
[14] W3C. SPARQL 1.1 Query Language http://www.w3.org/TR/sparql11-query/. Copyright © 2013 W3C.
[15] W3C. SPARQL 1.1 Update http://www.w3.org/TR/sparql11-update/. Copyright © 2013 W3C.
[16] The Apache Software Foundation. TDB Transactions.
http://jena.apache.org/documentation/tdb/tdb_transactions.html.
Copyright © 2011-2013 The Apache Software Foundation.
[17] Linked Data API http://code.google.com/p/linked-data-api/.
[18] Epimorphics. Elda Linked Data API Implementation http://code.google.com/p/elda/.
[19] Apache Software Foundation. Deliver Fuseki as a WAR file https://issues.apache.org/jira/browse/JENA-201.

Page 101 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive[20] W3C. Linked Data Cookbook
http://www.w3.org/2011/gld/wiki/Linked_Data_Cookbook#Step_3_Re-use_Vocabularies_Whenever_Possible.
[21] W3C. OWL Web Ontology Language Overview http://www.w3.org/TR/owl-features/.
Copyright © 2004 W3C.
[22] Dublin Core Metadata Initiative. DCMI Metadata Terms
http://dublincore.org/documents/dcmi-terms/. Copyright © 1995 - 2013 DCMI.
[23] W3C. RDF Vocabulary Description Language 1.0: RDF Schema http://www.w3.org/TR/rdf-schema/.
Copyright © 2004 W3C.
[24] W3C. XML Schema http://www.w3.org/XML/Schema. Copyright © 2000 - 2007 W3C.
[25] Dan Brickley and Libby Miller. FOAF Vocabulary Specification 0.98 http://xmlns.com/foaf/spec/.
Copyright © 2000 - 2010 Dan Brickley and Libby Miller.
[26] java.net. Jersey JAX-RS (JSR 311) Reference Implementation https://jersey.java.net/. Copyright © 2013.
[27] Agile Knowledge Engineering and Semantic Web (AKSW). Xturtle: an eclipse / Xtext2 based editor for RDF/
Turtle files http://aksw.org/Projects/Xturtle.html.
[28] scardf Scala RDF API http://code.google.com/p/scardf/.
[29] The National Archives. UK Government Web Archive http://www.nationalarchives.gov.uk/webarchive/.
[30] Ontotext. Ontotext OWLIM http://www.ontotext.com/owlim. Copyright © 2000-2013 Ontotext.
[31] Pascal Hitzler, Markus Krötzsch, and Sebastian Rudolph. Copyright © 2010 Taylor & Francis Group, LLC.
>978-1-4200-9050-5. Chapman & Hall/CRC. Foundations of Semantic Web Technologies.
[32] The University of Sheffield. GATE General Architecture for Text Engineering http://gate.ac.uk/.
Copyright © 1995-2011 The University of Sheffield.
[33] The London Gazette. The London Gazette http://www.london-gazette.co.uk/.
[34] Linked Data - Connect Distributed Data across the Web http://linkeddata.org/.
[35] DBPedia. DBPedia http://dbpedia.org/.

Page 102 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

A. The DRI VocabularyThis vocabulary has been laid out using the Turtle W3C format.
<http://nationalarchives.gov.uk/terms/dri>
rdf:type owl:Ontology ;owl:imports <http://purl.org/dc/elements/1.1/> .
:Batchrdf:type rdfs:Class ;rdfs:label "Batch"^^xsd:string ;rdfs:subClassOf rdfs:Class .
:Closure
rdf:type rdfs:Class ;rdfs:label "Closure"^^xsd:string ;rdfs:subClassOf rdfs:Class .
:ClosureTyperdf:type rdfs:Class ;rdfs:label "Closure type"^^xsd:string ;rdfs:subClassOf rdfs:Class .
:Collection
rdf:type rdfs:Class ;rdfs:label "Collection"^^xsd:string ;rdfs:subClassOf rdfs:Class .
:DeliverableUnit
rdf:type rdfs:Class ;rdfs:label "Deliverable unit"^^xsd:string ;rdfs:subClassOf :Item .
:Directory
rdf:type rdfs:Class ;rdfs:label "Directory"^^xsd:string ;rdfs:subClassOf :Resource .
:Export
rdf:type rdfs:Class ;rdfs:label "Export"^^xsd:string ;rdfs:subClassOf rdfs:Container .
:ExportStatus
rdf:type rdfs:Class ;rdfs:label "Export status"^^xsd:string ;rdfs:subClassOf rdfs:Class .
:File
rdf:type rdfs:Class ;rdfs:label "File"^^xsd:string ;rdfs:subClassOf :Resource .
:Item
rdf:type rdfs:Class ;rdfs:label "Item"^^xsd:string ;rdfs:subClassOf rdfs:Class .
:Manifestation
rdf:type rdfs:Class ;Page 103 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

rdfs:label "Manifestation"^^xsd:string ;rdfs:subClassOf :Item .
:PresentationManifestation
rdf:type rdfs:Class ;rdfs:label "Presentation manifestation"^^xsd:string ;rdfs:subClassOf :Manifestation .
:PreservationManifestation
rdf:type rdfs:Class ;rdfs:label "Preservation manifestation"^^xsd:string ;rdfs:subClassOf :Manifestation .
:RecordList
rdf:type rdfs:Class ;rdfs:label "Record list"^^xsd:string ;rdfs:subClassOf rdfs:Container .
:Resource
rdf:type rdfs:Class ;rdfs:label "Resource"^^xsd:string ;rdfs:subClassOf rdfs:Class .
:batchrdf:type rdf:Property ;rdfs:domain :Item ;rdfs:label "batch"^^xsd:string ;rdfs:range :Batch .
:closure
rdf:type rdf:Property ;rdfs:domain :DeliverableUnit ;rdfs:label "closure"^^xsd:string ;rdfs:range :Closure .
:closurePeriod
rdf:type rdf:Property ;rdfs:domain :Closure ;rdfs:label "closure period"^^xsd:string ;rdfs:range xsd:decimal .
:closureTyperdf:type rdf:Property ;rdfs:domain :Closure ;rdfs:label "closure type"^^xsd:string ;rdfs:range :ClosureType .
:collection
rdf:type rdf:Property ;rdfs:domain :Item ;rdfs:label "collection"^^xsd:string ;rdfs:range :Collection .
:completedDate
rdf:type rdf:Property ;rdfs:label "completed date"^^xsd:string ;rdfs:range xsd:date ;rdfs:subPropertyOf dcterms:date .
:descriptionStatus
rdf:type rdf:Property ;rdfs:domain :Closure ;Page 104 of 162

The National Archives Digital Records Infrastructure Catalogue: First Steps to Creating a Semantic Digital Archive

rdfs:label "description status"^^xsd:string ;rdfs:range xsd:decimal .
:directory
rdf:type rdf:Property ;rdfs:domain :Resource ;rdfs:label "directory"^^xsd:string ;rdfs:range xsd:string .
:documentStatus
rdf:type rdf:Property ;rdfs:label "document status"^^xsd:string ;:exportMember
rdf:type rdf:Property ;rdfs:label "export member"^^xsd:string ;rdfs:subPropertyOf rdfs:member .
:exportStatus
rdf:type rdf:Property ;rdfs:label "export status"^^xsd:string ;rdfs:range :ExportStatus .
:file
rdf:type rdf:Property ;rdfs:domain :Manifestation ;rdfs:label "file"^^xsd:string ;rdfs:range :File .
:parent
rdf:type rdf:Property ;rdfs:domain :Item ;rdfs:label "parent"^^xsd:string ;rdfs:range :Item .
:recordListMember
rdf:type rdf:Property ;rdfs:label "record list member"^^xsd:string ;rdfs:subPropertyOf rdfs:member .
:reviewDate
rdf:type rdf:Property ;rdfs:domain :Closure ;rdfs:label "review date"^^xsd:string ;rdfs:range xsd:dateTime ;rdfs:subPropertyOf dcterms:date .
:username
rdf:type rdf:Property ;rdfs:label "username"^^xsd:string ;rdfs:range xsd:string

Page 105 of 162

From trees to graphs: creating Linked Data
from XML
Catherine Dolbear
Oxford University Press<cathy.dolbear@oup.com>

Shaun McDonald
Oxford University Press<shaun.mcdonald@oup.com>

Abstract
This paper describes the use case at Oxford University Pressof migrating XML content to Linked Data, the businessdrivers we have identified so far and some of the issues thathave arisen around modelling RDF from an XML base. We
also discuss the advantages and limitations of schema.org
markup relative to our much richer XML metadata, anddescribe our experimental system architecture combining
stores for both XML documents and RDF triples.

1. Introduction: What We Need
Oxford University Press publishes a wide range ofacademic content in its online products, such as Oxford
Scholarship Online, Oxford Art Online, the Dictionaryof National Biography, Oxford Medicine Online,
Oxford Law Online, Oxford Scholarly Editions Online
and Oxford Islamic Studies Online to name just a few.
The content includes journals, chaptered books,
reference content, legal case studies and original literarytexts (ranging from Shakespeare to the Bible andtranslations of the Qur’an) across many different
disciplines.

Page 106 of 162

Since our products are largely subscription based, there
are varying levels of freely discoverable content on our
product sites, so our "Discoverability Programme" hasbeen working to release metadata and content to a widearray of users and to search engines. A major way of
doing this is the Oxford Index site, a publicly available
website described as our "Discoverability Gateway". Eachdocument (book chapter, journal article, original text or
entry in a reference work) in each of our online products
is given its own "index card" page on the Oxford Index.
We describe the Oxford Index to new users as a "digital
card catalogue", to use a library metaphor. This is stored"under the hood" as an XML file of all the document’s
metadata. We have already found that the exposure of
titles, authors, abstracts or short snippets of text from theonline document to the "open" web has increased thevolume of our content that’s indexed by Google andothers.

doi:10.14337/XMLLondon13.Dolbear01

From trees to graphs: creating Linked Data from XML

As well as Search Engine Optimisation (SEO), anothermajor business driver is the need to improve user
journeys within and between our various online
products. For example, we want to be able to suggest to auser looking at a reference entry on medical ethics fromthe Oxford Textbook of Medicine that they might also
be interested in a recent journal article on the same
subject; or to suggest to someone reading one of John
Donne’s poems on Oxford Scholarly Editions Onlinethat they might be interested in a monograph chapter
also about John Donne. While some of these links may
be generated dynamically (automatically generated
recommendations of the sort: "people who read what
you’re reading also looked at this other page") or driven
by an underlying search query based on metadata
elements, other links can be more definitively stated(such as citations, relationships between people,
authorship etc.) and may have already been manuallyedited. We call this latter category "static" links and havebeen storing these links as independent XML documents
in our metadata database which we call the "metadatahub". We have also grouped index cards about the same
topic or person together into what we call an"Overview", providing another way of navigating arounda topic. Index cards are linked to their overviews using
the "hasPrimaryTopic" link. As the number of links haveincreased, it has become more apparent that a graphmodel would be better suited to storing the links, and so
our thoughts have turned to RDF.

2. Metadata Hub and OxMetaML:
Where We're At
Developing a single XML schema for all this metadata
was no easy task, because the entire body of content asdata originates from many different sources, including
Mark Logic databases, FileMaker databases, SQL Server
stores and even spreadsheets, using a number of different
DTDs or none at all, and each having been developed toits particular product lifecycle by its particular productteam.
We analysed each data set identifying gaps andgathering product metadata where available. Initial
findings across larger sets helped identify a core set of
bibliographic fields to serve as a requisite benchmark.
Further findings uncovered where data was unavailable,
lacked standardization or sufficient workflows tofacilitate an additional "metadata" delivery stream.

Change requests were filtered into two categories: thosewith little to no adverse impact to the product’s business,and those requiring more significant changes. All nonimpactful changes were implemented – adding
identifiers, streamlining repositories, and authoring
scripts to automate upstream extraction. The analysis
results were updated, the core set of fields were extended,
and a draft schema model was created.
At the time, we had planned to implement theexisting Dublin Core model. There were a few factors
serving as major constraints. Oxford University Press
uses third party developers to build and maintain its
product websites. There are advantages to such a businessmodel, however, we realized we would no longer havecomplete control over data changes. Further, cost
efficiencies gained by code reuse when new sites arelaunched are only realized if changes to existing data
models are kept to a minimum. Finally, when newmodels are introduced, in such conditions, they tend toconflate with constructs solely intended to support site
functionality. Thus, while RDF/XML was the intent,
business needs for site development required a model
much more XML intensive. This coupled with theinherent challenge of filtering variant data from myriad
product silos into one, meant that our current model
uses an internal abstraction layer comprised of
OxMetaML - an XML schema suite of Dublin Core
nested within proprietary constructs, nested withinsimple RDF constructs, stored in a file store, theMetadata Hub.
The Metadata Hub (Hub) is our XML file store
loaded via a Pre-Ingestion Layer (PIL), where a collectionof automated, proprietary pipelines transform the various
source data into OxMetaML metadata records. One of
the advantages of OxMetaML is that it establishes asingle vocabulary for all links that had previously existed
in disparate silos. Each link is defined as an XMLserialized triple with a named predicate. Because many ofthe links are embedded in the record from which theylink, each link is encoded as an RDF resource with nospecific identifier. That is, as a blank node, which makes
further linking or querying difficult from an RDF
perspective. It is, however, efficient as XML, and for post
XML processing.
One of the post processing use cases calls for
dynamically creating static links. To do this, we index
the data with a full text search server, based on a Javasearch engine library extended with XML, among otherthings. It is a powerful search engine, allowing us topinpoint potential relationships between the subject of aquery (a single piece of content) and the entire corpus. It
is also normally quite fast. There are trade offs, however,when dealing with XML.

Page 107 of 162

From trees to graphs: creating Linked Data from XML

The search engine uses an inverted, terms based index,
where each term points to a list of documents that
contain it. For text documents, even for HTML where
markup is less strict, it is potent. The repetitive nature of
bibliographic metadata in highly structured XML,
however, inherently increases the time it takes tocomplete searches, especially when documents number inthe millions. An author name like Shakespeare could
appear in hundreds of thousands of documents.
Therefore, great care must go into the search engine
server's configuration.
One aspect controlled by configuration issegmenting, where the index is segmented into distinct
disk locations for greater efficiency. Segment sizes arecontrollable via the configuration file. From a file systemstandpoint, this is highly efficient. Basically, writing a 5
Gig file (or a collection of files amounting to it) to asingle disk space makes quick access nearly impossible.
With highly structured, repetitve XML, it becomesproblematic as it essentially requires multi-value fields.
"Multi-value field" is the term used for XML
elements that repeat per document. For instance, and not
insignificantly, keywords or index terms, which are very
relevant to a document's relational position within a
corpus, would be captured as multi-value fields. If theuse case requires that these fields be stored as well, thesearch will only return the last value indexed for eachone. In order to retrieve all relevant values from adocument, all values for a multi-value field must be
concatenated into a single, preferably separate, field, andthe engine must know to return that field. Further,
consideration must be given to which fields can bequeried, and which returned between the multi-value
field and its corresponding concatenated field.
The increase in query time is negligible bycomparison to configuration and load, which itself may
contribute to increased query time. Neither the schemaor configuration file, the heart of the index, can be
modified on the fly. There's no tweaking the index aseach configuration change requires reindexing all the
documents. In addition, while it is possible to use nearrealtime search (NRT), it can only be accomplished via
what are known as soft commits, where index changes(updates) are only made visible to search, as opposed tohard commits that ensure the changes have been saved
properly. Obviously, NRT is accomplished via a
combination of soft and hard commits. However, if youruse case requires that each query be followed up by
document retrieval in a matter of seconds, you are left
with the choice of either forgoing NRT, or performingsome impressive, architectural alchemy.

Page 108 of 162

3. An RDF Graph Model: Where
We're Going
Although our current metadata is stored in the XML
filestore serialised as RDF/XML, we still have some way
to go to represent the knowledge that it contains as an
RDF graph. This has come about for two reasons.
Firstly, because of the very different mindsets required
for XML, thinking in terms of documents and elements,
trees and sequential order; versus RDF where ourbuilding blocks are vertices and arcs. Where documents
are important, XML is a far better choice; whererelationships between things are important, RDF has theadvantage. One common tip is "XML for content, RDF
for metadata" and this is where we are now heading.
Since our metadata still includes abstract texts, which canrun to several paragraphs, we do not expect to completely
ditch the XML for RDF triples, but as more of theconcepts encoded in the XML metadata, such as authors,
become linked in their own right, and more importantly,
are given their own identifiers, potentially throughinitiatives such as the author identification scheme
ORCID , we expect to migrate more information to thegraph model.
The second reason we are still at the development
stage for our RDF model is because we really have twodifferent types of metadata, and are working towards
demarcating them more clearly. The first type is that
better known in the publishing world - bibliographicalinformation about the author, title, ISBN of the book orjournal to which the document belongs and other such
related metadata. In this case, since our primary output
for the bibliographical information is in HTML form on
the Oxford Index website, we have chosen to retain thedata as XML, which is easily parsed into HTML, andonly store the static links between documents, such as"references" links, as RDF.
The second type of metadata is information
concerning what the document is about, often called
contextual or semantic metadata. For example, wecurrently tag each document with the academic subject
to which it belongs, and some documents, the Oxford
Index Overviews, may also be tagged with information
about whether the Overview is about a Person, Event orother type of entity. There is significant scope for
expansion here, to provide more detail about the content
of the document. Semantic publishing workflows usually
do this by linking documents to an ontology, stored as
RDF or OWL-DL, which contains backgroundknowledge about the domain. For example, in thedomain of American Revolutionary history, we could
store the facts: John Adams and Samuel Adams wereFrom trees to graphs: creating Linked Data from XML

cousins, contemporaries of George Washington. John
Adams succeeded George Washington as President, andwas the father of President John Quincy Adams. Some ofthis information is available as Linked Data in DBpedia,
and we could augment it with additional information as
we expand our ontology. Therefore, if we identify
documents as being about John Adams, George
Washington and John Quincy Adams, say, based on
named entity recognition techniques we can then use theontology links to generate links between the documents
directly. Additionally, the ontology provides information
about the link type "father of", "successor of" etc. that
can help the user know how the two documents are
related to each other, thus further improving the user
journey.

We are storing the document-to-document links that areconsidered part of the bibliographic metadata as RDF
triples. However, we also need to record information
about the links themselves, for example, whether they
have been approved by a curator, the accuracy of the link
if it has been automatically generated, the date ofcreation etc. There are several ways of storing a fourth
piece of information in RDF. The recommended option
for Linked Data is to use named graphs, or "quads". This
assigns a name (URI) to each separate graph (or group of
triples), and hence allows additional information to be
attached to that graph URI. Although this is supported
in SPARQL, the RDF query language, unless we were to
put each triple in a separate graph, this approach would
not fulfil our needs, since we need to assign descriptive
information to each triple, not just each group of triples.
The alternative option is to assign a URI to each
triple and treat it as a resource in itself, so that further
statements can be made about the triple. This is known
as reification. For example the triple about Barack
Obama:<http://metadata.oup.com/10.1093/oi/authority.20110803100243337><http://metadata.oup.com/has_occupation>"President of the United States"

could be reified to:<http://metadata.oup.com/Statements/12345>
rdf:subject<http://metadata.oup.com/10.1093/oi/authority.20110803100243337>.
<http://metadata.oup.com/Statements/12345>
rdf:predicate<http://metadata.oup.com/has_occupation>.
<http://metadata.oup.com/Statements/12345>
rdf:object"President of the United States".

Then additional triples can be added about the Statement:<http://metadata.oup.com/Statements/12345>
oup:is_valid_from"20 January 2009".

This approach is usually not recommended for linked data because it increases the number of triples, and requires alarge number of query joins in order to return the RDF statements’ metadata. However, because we are not directly
publishing our document links, but simply storing them for insertion into XML which is later transformed into
HTML, reification is still an option for us.

Page 109 of 162

From trees to graphs: creating Linked Data from XML

We have adopted a modified form of this reification model, whereby each type of link is considered an RDFS class,and we create instances of that class which are objects of the oup:has_link predicate, and subjects of theoup:has_target predicate:<http://metadata.oup.com/10.1093/oi/authority.20110803100243337>
oup:has_link<http://metadata.oup.com/links/12345>.
<http://metadata.oup.com/links/12345>
rdf:type<http://metadata.oup.com/isPrimaryTopicOfLink>.
<http://metadata.oup.com/links/12345>
oup:has_target<http://metadata.oup.com/10.1093/acref/9780195167795.013.0922>.
<http://metadata.oup.com/links/12345>
oup:matchStatus"approved".

Much has been said about the drawbacks of the
RDF/XML syntax for serialising RDF. Unlike XML,RDF is fundamentally defined as a data model, not a
syntax. Although there are best practices for using
RDF/XML [Dodds2012], in our opinion, it is better touse one of the alternative serialisations of RDF, such as
Turtle [W3C2013] , to encode triples, and steer clear of
RDF/XML completely. Not only is RDF/XML verbose,and permissive of syntactic variation to describe the sameknowledge, it actually makes it harder on XML experts
to focus on modelling issues. The questions that need to
be answered in the design of an RDF Schema are what
concepts are we trying to encode, and what types of links
should we have between them? Issues of element order,
and whether to use rdf:about or rdf:Description are
orthogonal to knowledge modelling and don’t need to beaddressed if the Turtle syntax is used.
Initially, we are encoding the links between our XML
documents as triples, using a number of common
vocabularies such as Friend of a Friend/ and continuing
to use the Dublin Core terms we already had in theXML. We have also explored the use of bibo and PRISM
bibliographic vocabularies. We are testing whether tostore inverse triples explicitly, or to rely on inference or
SPARQL CONSTRUCT queries to generate the inverseat the output, as we deliver data to our front end
websites. Storage of links in both "directions" forexample, statements that a chapter "is part of" a book,and the book "has part" that chapter is obviously morecostly in terms of storage, but improves delivery
performance. Since we have to recombine the triples
back into the XML document for presentation on thefront end, generating the inverse triple using an inferenceengine is unlikely to be the most efficient method toacquire the information about the links, and requires theuse of OWL, rather than the less complex RDFS.

Page 110 of 162

4. Semantic PublishingExperiences: Where Others Are
A number of solutions to the kind of problems we’refacing have been discussed in the XML and RDF
communities: the various configurations of systemarchitecture "beast" combining an RDF triple store with
an XML database were discussed at XML Prague 2013
[Greer2013]. Greer’s "Consumer" model pulls in the
RDF triples via an XQuery application to combine themwith XML from a MarkLogic database, and anotheroption he mentions is to tag a node set with a URI
identifier, which corresponds to an RDF resource in a
triple store. This then explicitly links the XML
documents to the RDF triples, and is something we are
implementing through the use of DOIs (Digital Object
Identifiers) used both to identify the XML document
and, expressed as URIs, to identify the subject and objectof a triple expressing a relationship between two XML
documents.

From trees to graphs: creating Linked Data from XML

The BBC have combined the use of XML databases andRDF triple stores for their dynamic semantic publishingfor BBC Sport, the 2012 Olympics and now BBC News
sites [Rayfield2012]. Their journalists tag assets (such asvideo clips) according to an ontology, and this metadata
is captured in the OWLIM triple store, then combinedwith external sports statistics and content assets encodedin XML stored in a MarkLogic database. Content can be
aggregated using a combination of SPARQL for domain
querying and XQuery for asset selection. The content
transaction manager combines SPARQL and XQuery sothat queries for sports statistics related to a particularconcept, like "The league table for the English
Premiership" can be carried out across the triple store
and MarkLogic database. Another example of combining
XML with RDF is at Nature Publishing Group[Hammond2013], where they use XMP [Adobe], avocabulary using a subset of RDF that can be embedded
in XML documents, to bridge between their XML datastore and their triple store. While their XML content is
distributed across the organisation in a number ofdifferent silos, the triple store enables them to query the
integrated picture of all their data.

5. RDFa and schema.org
While semantic markup does not increase searchrankings directly, it has been shown to improve clickthrough rates significantly, as search results are more eyecatching and it's clearer to the user that the retrieved
document is a relevant answer to their query. For
example, when Best Buy added RDFa to their productpages, traffic to their site increased by 30%, and Yahoo!
has reported a 15% increase in click through rate forenriched links. We are still evaluating a number of
options for embedding structured metadata in our
discoverability pages. Although RDFa allows for richer
descriptions, and can provide our full metadata "under
the hood", the advantage of schema.org markup is that it
is fully supported by the major search engines. As an
example, we can add some simple markup to the
Overview on Barack Obama on the Oxford Index likeso:This does however require a mapping of our XML
elements such as "occupation" to schema.org vocabularyterms like "jobTitle", which can introduce semantic
mismatch. (Is "American Democratic statesman" really a
job title?). Other schema.org CreativeWork schemas,
such as Book and Article may map more closely on to
our XML, but overall, the drawback of schema.org isthat only very simple markup can be used, so it does notprovide a full alternative to an API on our metadata or
full linked data publication.

6. Conclusion
We are still in the early days of our journey from XML
to Linked Data, and a number of issues remain to beresolved. Firstly, we need to re-assess our business case to
identify the most effective output. Secondly, we need to
identify what proportion of our information should be
stored as triples, versus as XML: our strategy to date is tomigrate slowly, as more resources are assigned a URI andmore links created between those URIs, we can store the
new triples in the triple store. It is also a modelling issueof how much of the data is persistent - the more datachanges, the better it is to leave it as well-indexed XML
that can easily be searched, such that new links can be
dynamically created, rather than stored statically as RDF.
Thirdly, we need to decide whether to publish the triples
as RDFa embedded markup, or go the whole hog andpublish Linked Data, though again these may be twostages on a longer journey. And finally, we have yet toprove that a combination of an XML store fordocuments and triple store for links is really the bestarchitectural solution for our needs.

<div vocab="http://schema.org/" typeof="Person"
about="http://oxfordindex.oup.com/view/10.1093/oi/authority.20110803100243337"><span property="name">Barack Obama</span> <p/><span property="jobTitle">American Democratic statesman</span> <p/>born <span property="birthDate">4 August 1961</span> <p/></div>

Page 111 of 162

From trees to graphs: creating Linked Data from XML

Bibliography[Rayfield2012]

Jem Rayfield. Sport Refresh: Dynamic Semantic Publishing. 17 April 2012.
http://www.bbc.co.uk/blogs/bbcinternet/2012/04/sports_dynamic_semantic.html[Greer2013]Charles Greer. XML and RDF Architectural Beastiary. Proceedings of the XML Prague 2013
Conference. February 2013, 189-205.
http://archive.xmlprague.cz/2013/files/xmlprague-2013-proceedings.pdf[Hammond2013] Tony Hammond. Techniques used in RDF Data Publishing at Nature Publishing Group.
March 2013.
http://www.slideshare.net/tonyh/semweb-meetupmarch2013[Adobe]Adobe. Extensible Metadata Platform.
http://www.adobe.com/products/xmp/[W3C2013]
World Wide Web Consortium. Turtle Terse RDF Triple Language W3C Candidate
Recommendation. 19 February 2013.
http://www.w3.org/TR/turtle/[Dodds2012]
Leigh Dodds. Principled Use of RDF/XML. 12 June 2012.
http://blog.ldodds.com/2012/06/12/principled-use-of-rdfxml/

Page 112 of 162

xproc.xq - Architecture of an XProc processor
James Fuller
MarkLogic
Abstract

Our first example shows how XProc describes a pipeline
using XML markup.

XProc is a markup language that describes processing
pipelines which are composed of discrete steps that apply Example 1. XProc simple example
operations on sets of XML documents. This paper details out <p:pipeline xmlns:p="http://www.w3.org/ns/xproc"
the architecture, model and process flow of xproc.xq, an
version="1.0"
mylimit="10">XProc processor, implemented using XQuery 3.0.
Keywords: XProc, XQuery 3.0

1. Introduction
This article provides an in-depth overview of the primaryarchitectural components of xproc.xq, an XProc [1]processor which has been built using XQuery 3.0, on topof the MarkLogic database [2]. Where there is time I
highlight some of the more novel aspects of the systemand provide background on key design decisions.
The goals of developing xproc.xq (as with most of myopen source work) are of an entirely selfish nature;• Testbed XProc implementation for experimentation• Learn about XQuery and functional programming
development 'in the large'• Observe performance characteristics of XProc within
database context

2. XProc BackgroundThe ubiquity of XML creates the need for programmers
to be able to implement complex, scalable and extensible
processing work flows which work on sets of XML
documents using the broad and deep stack of XML
technologies available today. XProc [1], the XML
Pipeline language defined by the W3C, attempts toprovide developers with a tool that helps create complex
document work flows using a declarative description in
the form of pipelines.
A pipeline is a well worn abstraction in computing,yet loosely defined. For our purposes, we define pipelines
as a declarative model that prescribes a set of operations
which are to be applied to a set of XML documents.
Each operation has a consistent data interface allowing
for the flow of XML documents generating data for the
next operation to consume.

doi:10.14337/XMLLondon13.Fuller01<p:identity><p:input port="source" select="//p"/></p:identity><p:count limit="$mylimit"/></p:pipeline>

An XProc processor consumes the pipeline and applies
processing to XML document(s) supplied as input. The
example pipeline copies document(s) using thep:identity step then counts how many document(s)there are with the p:count step.
Having a declarative definition of a work flow,
separates the how to process from the what to process,leaving the XProc processor free to handle the 'how' as
implementation details and pipeline authors to describe
the 'what' to process with a catalog of operations. For
those who have never experienced XProc I will pass over
over many of the finer details of this example for now,
but I revisit in the XProc refresher section.
Unix pipelines are the oft quoted analogy when
explaining XProc. Unix pipelines work by allowing eachindividual shell command to consume and emit lines oftext data. Such a consistent data interface is similarly
found in XProc's ability to generate and consume XML,though its important to note that in Unix pipes, shellcommands work on each line of text versus an entire
document.
Much of the utility of Unix pipes comes from thefact that there are a lot of useful shell commands,
correspondingly XProc comes with a large set of built insteps as well as the facility to create your own steps. We
could extend the analogy by observing that shellcommands share a consistent selection language in theform of regular expressions where XProc leverages XPath.

Page 113 of 162

xproc.xq - Architecture of an XProc processor

The following year, in 2005, another W3C member
submission was proposed: "XML Pipeline Language(XPL) Version 1.0" (draft), submitted by Orbeon, Inc.,on 11 March and published on 11 April.2
It was identified as a goal to promote aninteroperable and standard approach to the processing of
XML documents and the working group started
meetings late 2005.
A set of use cases were developed and published in
20063 and work on the spec itself proceeded.
Several interim draft candidates were developed, with
2.1. Goals
the WG editor, Norman Walsh and member Vojtech
Toman developing in parallel reference implementations.
The XProc specification lists out twenty+ goals,
As is typical with any specification process, it took
embodying the guiding principles for development of the XProc much longer to achieve W3C Recommendation
language.
status, ratified in May 2010 [1].
I've taken the liberty to summarise into the following
list;
2.3. Brief refresher• The language must be expressed as declarative XML
and be rich enough to address practical An XProc pipeline document has a document root
element of p:pipeline or p:declare-step which contains
interoperability concerns but concise• The language must allow the inputs, outputs, and one or several steps. Steps are either implicitly orother parameters of a components to be specified with explicitly connected with documents flowing between
them. Each step type determines the kind of processing it
information passed between steps using XML• The language must define the basic minimal set of does on documents.
The example shown in the Introduction section,
mandatory input processing options and associatederror reporting options required to achieve illustrated how XProc describes a pipeline.
interoperability.
<p:pipeline xmlns:p="http://www.w3.org/ns/xproc"• Given a set of components and a set of documents,
version="1.0"
the language must allow the order of processing to be
mylimit="10">specified.
<p:identity><p:input port="source" select="//p"/>• Agnostic in terms of parallel, serial or streaming</p:identity>
processing<p:count limit="$mylimit"/>• The model should be extensible enough so that</p:pipeline>
applications can define new processes and make themAt first glance, you maybe able to deduce that this is a
a component in a pipeline.
• The model could allow iteration and conditional pipeline that has two components ('steps' is the XProc
processing which also allow selection of different term); an identity step which copies an XML document
and a count step that counts the number of documentscomponents as a function of run-time evaluation.
being passed to it from the p:identity step, but with the
2.2. History
twist that it p:count stops counting once it hits a certain
limit.
Pipelines in computing are an old concept, and in work
flow processing similarly ancient for markup languages.
As far back as 2004, a W3C Note set out
requirements for an XML processing model: "XML
Processing Model Requirements,"1 W3C Working
Group Note 05 April 2004.
If you work with XML today, its likely that you've
encountered or already built your own ad-hoc pipelines
using your favourite XML processor. XProc simply
formalises what could be viewed as the natural evolution
from Unix pipeline 'line by line' text processing to aricher, more complex work flow style of document
processing. This frees up the other XML technologies tofocus on what they are good at and for XProc to act as
the 'main control loop', orchestrating work flow
processes at a higher level of abstraction.

http://www.w3.org/TR/2004/NOTE-proc-model-req-20040405/
XML Pipeline Language (XPL) Version 1.0 (Draft) W3C Member Submission 11 April 2005 - http://www.w3.org/Submission/xpl/
3
XProc Use Cases - http://www.w3.org/TR/xproc-requirements/
1

2

Page 114 of 162

xproc.xq - Architecture of an XProc processor

None of this explains the select attribute xpath
expression on p:identity's p:input element. Its purpose
What defines the starting set of XML
is to instruct the processor to filter incoming documents
document(s) ?
on the input source port with an xpath expression, which
is very useful to select what you want to work on. EachHow do documents know how to flow frommatched <p/> element is turned into a document andstep to step ? What connects them ?
will be copied to the p:identity output result port.
XProc relies heavily on XPath as its built in selection
What does it mean that p:identity childmechanism and we will see it pops up everywhere.
p:input has a select attribute xpath value ofWith the magic of default readable ports, we can now'//p' ?
understand how each step's p:input uses a p:pipeelement to define where documents flow from, even if
How does the p:pipeline attribute @mylimit
we don't explicitly instruct them in our pipeline. The
define a reusable value that p:count uses
p:pipe uses a step's name attribute and port towithin its own limit attribute ?
unambiguously identify the binding and provides a rigid'flow' path at runtime, through which documents flow.
We can rewrite this pipeline to be more explicit, which
In summary, this pipeline takes a set of documents
answers some of our questions.
(from primary p:input) and sends the input into
p:identity (as defined with its p:input). The p:identity
step copies the p elements from them and passes them (as<p:declare-step name="mypipeline" version="1.0"
xmlns:p="http://www.w3.org/ns/xproc">a sequence of documents) to the p:count step. The
p:count step counts the number of items in the sequence<p:input port="source" primary="true"
up to a maximum of 10, outputting the count result tosequence="true"/>console (which is the job for primary p:output result<p:output port="result" primary="true"port).
sequence="true">
I've chosen this example to illustrate that XProc has<p:pipe step="mycount" port="result"/>some disconcerting 'warts', eg. the <p:empty/> within the</p:output>
p:with-option is anachronistic at best. You should now<p:option name="mylimit" select="10"/>have a sense of Process’s defaulting 'story'. Our<p:identity name="myidentity">
descriptionof a pipeline needs a sufficient level of detail<p:input port="source" select="//p">
for
an
XProc
processor to be able to puzzle out how to<p:pipe step="mypipeline" port="source"/>processthedocuments.
XProc provides implicit defaults</p:input></p:identity>
with its syntax to help make an already verbose language,<p:count name="mycount">
well less verbose. There is work needed at the W3C<p:input port="source">
XML Processing WG level to help make even more
<p:pipe step="myidentity" port="result"/>
syntaticalchanges to reduce verbosity.
</p:input>
What is not entirely clear with this example is;<p:with-option name="limit" select="$mylimit"><p:empty/></p:with-option></p:count></p:declare-step>Documents come in from the 'outside world' via theprimary p:input child defined under p:declare-step.
The p:pipeline was a kind of shorthand alias for the
more general p:declare-step step which incidently isalso used to define new steps.
The $mylimit value is an XProc option on thep:declare-step, the p:count has a defined option, whose
value is set using p:with-options.

Page 115 of 162

xproc.xq - Architecture of an XProc processor

Note
For those who are curious, I would highly recommend
installing what I consider the reference XProc
implementation, Norm Walsh's XML Calabash1 andrun against the example
calabash -isource=test.xml -isource=test2.xml test.xpl

This would yield the following result, outputted toconsole.
<c:result
xmlns:c="http://www.w3.org/ns/xproc-step">2</c:result>

Where the value of the c:result element reflects
depending on how many <p/> elements were passed
from the p:identity step, up to a limit of 10.
2.3.1. Process Model
In classic work flow systems it is common to implement
methods like Finite State Machine (FSM) [3] whichembed state information within each document.
In such systems, changes to state are the events which
control processing and the apparent 'flow' of documents.
Document flow based on state transitions implies that
there are no fixed paths between the processing 'steps'
enabling at runtime highly dynamic and complex workflows. In practice, event driven pipelines are also difficult
to implement and diagnosing issues with performance or
optimisations not very straightforward.
A saying attributed to several programming
luminaries declares that 'state' is the enemy of dynamic
computation and I think this applies to XProc. A moreamenable approach was needed to take advantage offunctional programming principles associated with its
declarative format.
In XProc, inputs flow into a pipeline from one stepto the next with results emitted at the end. The order ofthe steps is constrained by the input/output connections
between them rather then state contained within the
documents themselves. Document flow is a consequenceof the implicit and explicit binding of input source ports
to output result ports.

1

http://xmlcalabash.com/

Page 116 of 162

This allows implementations to be able to staticallyanalyze how to execute steps, be it in sequential orparallel fashion or to take advantage of somecharacteristic of the environment. With a minor loss of
freedom in specifying highly complex work flows(though its still possible with XProc) we gain a lot of
freedom in terms of process execution.
XProc also not have too bother maintaining the stateof each document which itself can be complicated andcostly in terms of performance. By eschewing with
document state altogether, XProc avoids the issues
associated with recovering state when it gets lost or
corrupted.
Using a 'stateless' process flow model means that
XProc itself minimises constraining parallel or streaming
processing scenarios. Streaming is a particularly difficult
scenario to enable, for example, if there is significant
reordering between the input document trees and output
document trees, its been observed [4] that one should
avoid even attempting to stream, conversely even mature
XML processor like Saxon [5] has only partial streamingin place for its XML Schema validator, xpath processor
and XQuery/XSLT processor [6].
Lastly, be aware that there are plenty of pipelines one
could author that can cause side effects which invalidate
streaming or parallel processing. This caveat is more to
do with the technologies underlying any specific step
processing (p:eval, p:xslt, etc) versus XProc.
2.3.2. Steps
Steps can have options and parameters, some steps arecalled 'compound' steps and embody multiple nested
pipelines.
New steps can be defined, using p:declare-step, that areused in exactly the same manner as the built-in steps.
Note that whenever you have create a pipeline, thatpipeline itself can also be reused as a step in otherpipelines.
Custom steps can be bundled up into library (p:library)
and reused in other pipelines by importing using ap:import element.

xproc.xq - Architecture of an XProc processor

Compound and Multi container Steps

A compound step contains a subpipeline. Multicontainer steps contain two or more subpipeline(s).
<p:declare-step>

Declares an XProc pipeline. This step can define a
new reusable XProc step for use by other pipelines.
When used within another pipeline it acts as if it
was an atomic step, regardless if it contains more
subpipelines.
<p:pipeline>

Is an alias of p:declare-step with default implied
inputs and outputs, itself reusable as a step inother pipelines. When invoked within anotherpipeline it acts as if it was an atomic step,
regardless if it contains more subpipelines.
<p:choose>

Multi container step which selects one of anumber of alternative pipelines based on test
criteria<p:for-each>

Iterates over a sequence of documents with a
specific subpipeline<p:group>Groups a sequence of steps together as a
subpipeline<p:try>Multi container step that provides a try
subpipeline which if fails is caught with an error
exception handling subpipeline<p:viewport>

Iterates a pipeline over inner selections of eachdocument in a set of documents.
Atomic Steps<p:count>

Count the number of documents in source input
sequence.
<p:delete>

Delete items matched by pattern from the source
input.
<p:directory-list>

Enumerate a directory's listing into result output.
<p:error>

Generate an error that throws at runtime.
<p:escape-markup>

Escape XML markup from source input.
<p:filter>

Filter documents with dynamically created selectexpressions<p:http-request>

Interact with resources identified byInternationalized Resource Identifiers (IRIs) overHTTP.
<p:identity>

Make an exact copy of an input source to theresult output.
<p:insert>

Insert an XML selection into the source input.
<p:label-elements>

Create a label (ex. @XML:id) for each matchedelement, and store the value of the label within an
attribute.
<p:load>

Load an XML resource from an IRI providing it as
result output.
<p:make-absolute-uris>

Make the value of an element or attribute in thesource input an absolute IRI value in the result
output.
<p:namespace-rename>

Rename the namespace declarations.
These steps are the basic building blocks of XProc <p:pack>pipelines with each carrying out a single XML operation.
Merge two document sequences.
Atomic steps fully encapsulate the processing they apply. <p:parameters>
Most atomic steps accept input and emit output. All
Make available a set of parameters as a c:paramatomic steps will never themselves contain subpipeline(s).
set XML document in the result output.
Required steps: These steps are provided by a <p:rename>conformant XProc processor.
Rename elements, attributes, or processing
instruction.
<p:add-attribute>

Add a single attribute to a set of matchingelements.
<p:add-xml-base>

Explicitly add or correct XML:base attributes on
elements.
<p:compare>Compare two documents for equivalence.

<p:replace>

Replace matching elements.
<p:set-attributes>

Set attributes on matching elements.
<p:sink>

Accept source input and generate no result output.
<p:split-sequence>

Divide a single sequence into two.

Page 117 of 162

xproc.xq - Architecture of an XProc processor<p:store>

Store a serialized version of its source input to a
URI.

Additional steps: The XML Processing WG from time
to time publishes W3C notes on additional steps thatexist in the XProc step namespace.

<p:string-replace>

Perform string replacement on the source input.
<p:unescape-markup>

Unescape the source input.
<p:unwrap>

Replace matched elements with their children.
<p:wrap>

Wrap matching nodes in the source documentwith a new parent element.
<p:wrap-sequence>

Produce a new sequence of documents.
<p:xinclude>

Apply XInclude processing to the input source.
<p:xslt><p:template><p:in-scope-names>

A developer may also define their own steps (using
p:declare-step) which when combined with p:libraryprovides a powerful reuse componitisation.

Community defined extensions
There are also many extensions being defined by thecommunity which are being defined which some
XProc processors may support.

XSLT evaluation on style sheet input source.

2.3.3. Known Implementations
Optional Steps: These steps are optionally provided by
The following is transcribed from a list of XProc
an XProc processor.
processors being tested at tests.xproc.org1:<p:exec>

Apply an external command to the input source.
<p:hash>Generate a cryptographic hash (message digest,
digital fingerprint) and inserts in document.
<p:uuid>

Generate a Universally Unique Identifier (UUID).
<p:validate-with-relax-ng>

Validate the input XML with RelaxNG schema.
<p:validate-with-schematron>

Validate the input XML with Schematron schema.
<p:validate-with-xml-schema>

Validate the input XML with XML schema.
<p:www-form-urldecode>

Decode the x-www-form-urlencoded string into a
set of XProc parameters.
<p:www-form-urlencode>

Encode a set of XProc parameter values as an xwww-form-urlencoded string.
<p:xquery>

XQuery evaluation on xquery input source.
<p:xsl-formatter>

Render an XSL version 1.1 document (as in XSLFO).

1

XProc Known Implementations - http://xproc.org/implementations/Page 118 of 162

Calabash
Norman Walsh is building an open-source
implementation in Java. Calabash is built on topof the Saxon APIs and uses XPath 2.0 as its
expression language.
Calumet
EMC’s Java-based XProc processor. The processor
features an extensible architecture and is easy toembed in other Java applications. Free fordeveloper use.
QuiXProc Open
Innovimax's GPL, Java implementation based on
XML Calabash adding Streaming and Parallel
Processing. The is also a Commercial product at
http://quixproc.com.
TubularTubular is a Java implementation based on
immutable objects, in order to facilitate theaddition of parallelism support, thus reducing theneed for locking mechanisms.
XProcerity
XProcerity is a Java implementation focused
primarily on high performance in multi-threaded
environments, such as high-traffic enterprise web
applications.
xprocxq
An earlier implementation of Jim Fuller's xprocxq
is an experimental bootstrap implementation of
W3C XProc Draft Specification, written in
XQuery, for the eXist XML Database [7].

xproc.xq - Architecture of an XProc processor

2.3.4. XProc vnext
The following is a sampling of non trivial deploymentsof XProc in use today, Anecdotal evidence seems to point
towards the fact that for the right scenario, XProc can bequite a power tool.
http://mesonet.info/ - real-time citizen world wide
weather station
http://code.google.com/p/daisy-pipeline/wiki/
XProcOverview- The DAISY Pipeline is an opensource, cross-platform framework for document-related
pipelined transformation. It supports the migration ofdigital content to various formats efficiently andeconomically, facilitating both production anddistribution of DAISY Digital Talking Books
http://balisage.net/Proceedings/vol8/html/
Williams01/BalisageVol8-Williams01.html
validating RESTful services
https://github.com/gimsieke/epubcheck-xproc epub checker implemented using XProc
Unfortunately, there is just as strong feedback thatindicates that there are many 'rough edges' in XProc v1.0
which is a barrier to wider adoption in both the XML
and broader development communities.
• too verbose or the need to be overly explicit• some constructs unwieldy (parameters)• deficient (only use string values in options andvariables)• hard to work with other non-XML data
• we require mechanism for sanctioning step definitions
without full blown W3C specification
Adoption, while slow, has had a steady uptake over theensuing two and half years since becoming a W3C
Recommendation. The W3C XML Processing WG is
now preparing for work on version 2.0 of thespecification, by first creating a draft set of requirements1
with a short set of goals that attempt to address
deficiencies;
1. Improving ease of use (syntactic improvements)
2. Improving ease of use (ex. increasing the scope forworking with non XML content)
3. Addressing known shortcomings in the language
4. Improve relationship with streaming and parallel
processing
The requirements document also traces how well XProc
v1.0 satisfied previous use case requirements. This 'scorecard' helps focus working on scenarios that were not at
all or partially addressed.

The next version of XProc is very much a case of 'fix
what is broken' and be judicious with adding only thebare minimum required.
The following list is a sample of concrete actions
being considered;• Fix Parameters - Change parameters to be more like
options which imply adopting the XSLT 3.0
extensions to the data model and functions andoperators to support maps
• Non XML document processing - Provide native
processing of non XML processing with a constrained
scope (possibly using a resource manager)• Drop XPath 1.0 support - Remove any *must* typerequirements for supporting XPath 1.0.
• Allow options and variable to contain arbitrary
fragments - Relax the constraint that variables andoptions can ony be defined as a string or
xs:untypedAtomic.
• Fix "non-step wrapper"- Remove the concept of'Non-step wrappers' by making p:when/p:otherwise
in p:choose and p:group/p:catch in p:try compoundsteps and get rid of the notion "non-step wrapper".
• Syntax changes - For example, allow Attribute Value
Template (AVT).
This is an ongoing discussion, so please feel free to offer
suggestions and join the debate (Details at XML
Processing WG home page2)

3. The xproc.xq project
The xproc.xq project is an open source project hosted atgithub [8] and offers use under the liberal Apache v2.0
license3. xproc.xq is an implementation of an XProc
processor using XQuery v3.0 with vendor specific plugins.
Currently, only support for MarkLogic [2] exists but plans
are in place to support other XQuery processors that support
XQuery 3.0 (eXist, Saxon, XQilla).
In late 2008, the author created a prototype XProc
processor using XSLT v2.0 [9] under eXist XML
Database [7]. This proof of concept led to the start ofdevelopment of xprocxq, with an initial implementation
created as an extension to the eXist XML database server.
The goals were set out as follows;• creation of an XProc processor with XQuery• avoid using XSLT, mainly as the author had engaged
in development of pipeline like processing in XSLTand wanted to avoid reliance upon it• use eXist support for first class functions to underpin
execution control

XProc vnext language requirements - http://www.w3.org/XML/XProc/docs/langreq-v2.html
XML Processing Working Group - http://www.w3.org/XML/Processing/3
Apache v2.0 License - http://www.apache.org/licenses/LICENSE-2.0.html

12Page 119 of 162

xproc.xq - Architecture of an XProc processor• acknowledge limitations of XQuery by implementing
a usable subset of XProc (at the time both XQuery
3.0 and XProc v1.0 were draft standards)• leverage performance of an XProc processor
embedded in a database context
eXist [7] has excellent extensibility characteristics making
it is easy to develop extensions, but it became quite achallenge to get a usable subset of XProc conformance.
Most of the issues were related to XQuery v1.0
suitability as a language for implementing such a
processor versus anything particularly difficult with eXist
itself.
XQuery v1.0 lack of first class function support
meant a heavy reliance on eXist specific functions, moretroubling was that many fundamentals of xprocxq
process flow were controlled by util:eval() (eXist
extension for dynamically executing constructed
XQuery). Additionally, due to a reluctance to employ
XSLT there were several issues in the implementation of
XProc steps, which turns out to be a mistake as XProc
makes heavy use of XSLT style match [9] expressions.

4. xproc.xq architecture
4.1. Design
The diagram provides a 'fly over' view of xproc.xq
application architecture, in the context of its process
flow. Using XQuery allows us to focus on the applicationarchitecture aspects, which is why there are no lower
level components on display (xml parser, xpath engine,
xml schema valiator ...).

Tip
Incidently, it is now possible to use XQuery 3.0 to
fully emulate XSLT matching expressions as
exemplified by John Snelson's transform.xq [10]In 2010, when XProc became a recommendation, planschanged to completely refactor xprocxq to work with
Saxon [5], which started to support many of theemerging ideas in XQuery 3.0. This had a positive effecton the internal processing and with robust first class
function support was able to remove most evals but still aheavy reliance on vendor specific extensions throughout
the codebase.
The final iteration of the XProc processor, known as
xproc.xq, started in 2012 and is built on top of
MarkLogic [2] XQuery v3.0 support. This refactor
focused on isolating as much of the codebase as possible The XProc processor advances through three stages when
into pure XQuery v3.0 and push any vendor specific processing an XML pipeline.
code into a pluggable module. This enables future • static analysis- consume and parse the XProc
pipeline, generating a runnable representation of said
versions of xproc.xq to support other XQuery vendors.
pipeline• dynamic evaluation- engine that dynamicallyevaluates the runnable pipeline representation• serialization- output interim results for further
processing or final results

Page 120 of 162

xproc.xq - Architecture of an XProc processor

Before we precede a word of caution. I am using Example 2. unordered XProc Example
nomenclature which is more appropriately applied to<p:declare-step version='1.0' name="main">development within compiled languages. Building an<p:input port="source"/>
XProc processor using a dynamic, interpreted language<p:output port="result">
like XQuery often brings the terminology in use into
<p:pipe step="i1" port="result"/>
question, mainly because engineering tradeoffs are being</p:output>
considered which in compiled languages would feel like
<p:identity name="i1">'cutting corners'. This comes with the territory of<p:input port="source">
pushing XQuery beyond its intended limits and with
<p:pipe step="i3" port="result"/>that said we now drill down into more detail of each</p:input>
phase.
</p:identity>

4.2. Static Analysis Phase
The static analysis phase consumes the original pipeline,
parsing it and generates a highly decorated version of thispipeline.
This decorated pipeline version can be considered theinternal model used throughout all subsequent
processing and provides a 'single point of truth' in termsof lookup, namespace declarations, step bindings,
variable & option definitions. The static phaze also takes
care of adding a unique internal default name to each
step, as well as reordering steps according to the flow as
defined by connections (bindings) between steps.
When a pipeline is created, the syntax represents anauthors intent but itself does not contain enoughinformation to be 'runnable' by an XProc processor. As
we've seen in previous sections, XProc has a suite of
implied processing behaviors which need to be 'teased'
out, we also need to take care of other concerns, likeensuring the order of steps follow how their implicit andexplicit bindings have been created. To illustrate what I
mean, nothing stops a developer writing an obfuscatedpipeline where the order of step processing is unclear.
The following is an example (taken directly from theW3C XProc test suite1) which illustrates the kind ofproblems that the static analysis phase needs to accountfor.

<p:identity><p:input port="source"><p:pipe step="main" port="source"/></p:input></p:identity><p:identity name="i3"/></p:declare-step>In this example, the p:identity step named 'i1' input
port means its not the first step in the XProc work flow.
Its actually the second step defined which takes in thepipeline's input that is the first to be processed. And
perhaps what will surprise most readers is that the firststep is actually the last to process, based on its stepbinding with p:identity 'i3'. Deriving the final processorder is a function of making all step binding
relationships explicit.
The next example shows a pipeline where the author
took care to layout the steps in their true process order.
Example 3. ordered XProc Example<p:declare-step version='1.0' name="main"><p:input port="source"/><p:output port="result"><p:pipe step="i1" port="result"/></p:output><p:identity><p:input port="source"><p:pipe step="main" port="source"/></p:input></p:identity><p:identity name="i3"/><p:identity name="i1"><p:input port="source"><p:pipe step="i3" port="result"/></p:input></p:identity></p:declare-step>

1

XProc Test Suite - http://tests.xproc.org/

Page 121 of 162

xproc.xq - Architecture of an XProc processor

Its a feature of XProc that the author does not have to Example 5. xproc.xq entry point
explicitly control process order by ordering XML
xquery version "3.0";
elements. Being able to insert a step or make changes tostep bindings without having to go trace through an import module namespace xprocxq =entire pipeline checking ordering makes life easier for"http://xproc.net/xprocxq" at "/xquery/xproc.xq";
developers.
declare namespace p="http://www.w3.org/ns/xproc";

Side Effects in pipelines
Connections between steps define order in XProc but
this is not a promise or guarantee of the actual processorder. For pipelines that rely upon side effects,
unexpected results may occur (non-deterministic
processing, like downloading a file from the internet or
saving a file to the file system). The XProc
specification discusses this in the XProc specification1
Now lets take a look at the abstract syntax tree that getsgenerated by the static analysis phase, using a variation of
our original xproc example.
Example 4. XProc example<p:pipeline xmlns:p="http://www.w3.org/ns/xproc"
version="1.0"><p:identity><p:input port="source"/></p:identity><p:count limit="10"/></p:pipeline>

let $pipeline :=<p:pipeline xmlns:p="http://www.w3.org/ns/xproc"
version="1.0"><p:identity/><p:count limit="10"/></p:pipeline>
let $stdin := (<doc>
<p>mary had a little lamb</p></doc>,<doc><p>a</p><p>b</p></doc>)
returnxprocxq:xq($pipeline,$stdin)

The simplest way to run this is to cut and paste into
MarkLogic query console(http://localhost:8000/qconsole) and choose thecontent source that xproc.xq has been set up with.

If you have not previously installed xproc.xq to run on
your MarkLogic instance, now is the time to review
xproc.xq installation to install and deploy xproc.xq.
The default method of running xproc.xq is to import the
xprocxq XQuery library module and invoke with thesimple xprocxq:xq() entry point, as shown in thefollowing code listing.

1

XProc specification H. Sequential Steps, parallelism, and side-effects - http://www.w3.org/TR/xproc/#parallelism

Page 122 of 162

xproc.xq - Architecture of an XProc processor

The parameters the function accepts are defined as;• $pipeline - the XProc pipeline being processed• $stdin - A sequence of XML Document(s) to be
Example 6. result of XProc processing
placed on the primary input source port of thepipeline<c:result xmlns:c="http://www.w3.org/ns/xproc-step">• $bindings - n/a2• $options - Sequence containing options that will</c:result>override pipeline option values
But what we really want is to be able to analyze the• $outputs - n/a
decorated tree version that is generated 'under the covers'• $dflag - When set to 1 outputs decorated tree
during the static analysis phase. This can be achieved
representation of XProc pipeline and ll input/output
using an overloaded version of the xprocxq:xq()
port values
function, demonstrated below;• $tflag - When set to 1 outputs timing information
By supplying a value of 1 for the $dflag, we are
Example 7. getting debug output from xproc.xq
instructing xproc.xq to emit the decorated pipeline tree.
xquery version "3.0";
We also get returned all values of any input or output
port which for the time being we will ignore.
import module namespace xprocxq =Cut and paste the above into query console"http://xproc.net/xprocxq" at "/xquery/xproc.xq";(http://localhost:8000/qconsole) and run.
Running this will return a c:result element containing
the count of unique documents

declare namespace p="http://www.w3.org/ns/xproc";
let $pipeline :=<p:pipeline xmlns:p="http://www.w3.org/ns/xproc"
version="1.0"><p:identity><p:input port="source"/></p:identity><p:count limit="10"/></p:pipeline>
let $stdin:= (<doc><p>mary had a little lamb</p></doc>,<doc><p>a</p><p>b</p></doc>)
let $bindings := ()let $options := ()let $outputs := ()let $dflag:= 1
let $tflag:= 0
returnxprocxq:xq($pipeline, $stdin, $bindings,$options, $outputs, $dflag, $tflag)

Page 123 of 162

xproc.xq - Architecture of an XProc processor

Example 8. debug output<xproc:debug episode="11600574566574829649" xmlns:xproc="http://xproc.net/xproc"><xproc:pipeline><p:declare-step version="1.0" mylimit="10" xproc:type="comp-step"
xproc:default-name="!1" xmlns:p="http://www.w3.org/ns/xproc"><ext:pre xproc:default-name="!1.0" xproc:step="true"
xproc:func="ext:pre#4" xmlns:ext="http://xproc.net/xproc/ext"><p:input port="source" select="/" xproc:type="comp" primary="true"><p:pipe port="result" xproc:type="comp" step="!1" xproc:step-name="!1"/></p:input><p:output xproc:type="comp" port="result" primary="true" select="/"/></ext:pre><p:identity xproc:step="true" xproc:type="std-step"
xproc:func="std:identity#4" xproc:default-name="!1.1"><p:input port="source" select="//p" xproc:type="comp" primary="true"><p:pipe port="result" xproc:type="comp" step="!1.0" xproc:step-name="!1.0"/></p:input><p:output xproc:type="comp" port="result" sequence="true" primary="true" select="/"/></p:identity><p:count limit="10" xproc:step="true" xproc:type="std-step"
xproc:func="std:count#4" xproc:default-name="!1.2"><p:input port="source" select="/" xproc:type="comp" primary="true"><p:pipe port="result" xproc:type="comp" step="!1.1" xproc:step-name="!1.1"/></p:input><p:output xproc:type="comp" port="result" primary="true" select="/"/><p:with-option xproc:type="comp" name="limit" select="10"/></p:count><ext:post xproc:step="true" xproc:func="ext:post#4"
xproc:default-name="!1!" xmlns:ext="http://xproc.net/xproc/ext"><p:input port="source" primary="true" select="/" xproc:type="comp"><p:pipe port="result" xproc:type="comp" step="!1.2" xproc:step-name="!1.2"/></p:input><p:output primary="true" port="result" xproc:type="comp" select="/"/></ext:post></p:declare-step></xproc:pipeline><xproc:outputs>..... snipped for brevity .....
</xproc:outputs></xproc:debug>

We are specifically interested in the xproc:pipeline
Table 1.
element which contains the abstract syntax tree
(decorated pipeline).
Level #
Level #.#
The code highlighted in bold shows how each step now !1 p:declarehas a unique internal @xproc:default-name attribute. The stepnaming convention for these default names is outlined!1.0 ext:pre step - internal step that is responsible
within the XProc specification [1] itself and provides afor bringing input into the pipeline via p:pipe on
unique id to each step element. Nesting as well as a way
p:input reference to !!
to determine its nesting level, which is needed to deal
!1.1 p:identity step - standard atomic step thatwith nested subpipelines. The input and output porttakes its input from the result of !1.0
p:pipe elements now point to their sources using thesedefault-names.
!1.2 p:count step - standard atomic step thattakes its input from the result port of !1.1
The following table outlines how @xproc:default-nameare used;!1! ext:post step - internal step that takes its input

from the result port of !1.2 and responsible for
placing the result to the outputter.

Page 124 of 162

xproc.xq - Architecture of an XProc processor

The ext:pre and ext:post steps are not extension steps
to be specified by pipeline authors but added during
static analysis to facilitate piping source and result ports
to the parent of the pipeline. All these steps do is copy
their inputs to their outputs (similar to p:identity inthat respect) to facilitate bringing in data from outside
the pipeline itself.
For our single branch pipeline example, this means thatthe !1 p:declare-step p:input is passing the standard
input (set during xprocxq:xq() invoke) to ext:prep:input. In the same manner, the !1! ext:post p:output
is being piped to !1 p:declare-step p:output ending
upon being emitted as the return result from the
xprocxq:xq() function invoke.
These extension steps are also responsible for making
sure XML documents flow correctly between steps andsubpipelines.
Moving on from extension steps, lets now remark upon
the xproc:type attribute which exist on every element.
This attribute identifies an element's component type,within the XProc vocabulary; explicitly marking out
elements makes dynamic evaluation simpler. The
following list outlines all the available types;• comp-step: indicates a compound step
• *-step: indicates a (standard | optional | extension)
atomic step• comp: indicates an ancillary component (all otherelements that are not steps themselves e.g.. p:input,
p:output, p:pipe, p:option, etc...)Finally, the @xproc:step and @xproc:func attributes areused to map the step element with the internal functionthat does the processing for the step, which we explain in
more detail in the next section.

But what about pipelines that contain compound or
multi container steps? One of the strengths of XProc is
its ability to construct complex multi branching workflows, but how can we model these branching work flows
in a form thats easy for the evaluator to process?
The next diagram shows a complex pipelines,
containing a p:choose that switches to one subpipeline or
the other, based on some condition. At the top level, weview each step atomically, that is we have only two steps
p:choose and p:store.

During runtime, p:choose invokes a new instance of theevaluator of the chosen subpipeline. The subpipelines aremodeled as separate ordered sequences, with only one ofthem ever actually selected to process during runtime.

4.3. Dynamic Evaluation Phase

We do not spawn or clone instances of the evaluator,
execution still occurs on a single code path. The nextOnce a decorated pipeline has been built, its the job of
diagram shows how nested functional composition
the evaluation stage to execute the pipeline and manage
provides the mechanism for achieving this.
the inputs and outputs sloughing off each step of theprocess.
From the view of the evaluator, every type of pipeline
is constructed to be an ordered sequence of steps. The
diagram shows this for our example pipeline.

Page 125 of 162

xproc.xq - Architecture of an XProc processor

Reading the diagram from top to bottom and left toright illustrates programmatic execution flow.
Once we get into dynamic evaluation, we see that themain engine is the novel application of an XQuery 3.0
left-fold() function. The decorated pipeline steps are
represented by a an ordered sequence of steps identified
by their @xproc:default-name. This solves the problem of
how to reduce complex process sequences into a single
process sequence, with itself reducing to a single final
output ('turtles all the way down').
Example 9. xproc.xq evaluation stage 'engine'left-fold($xproc:eval-step-func,$starting-input, (!1, !1.0, !1.1, !1.2, !1!))

Page 126 of 162

If any of the steps in the process sequence represent
another process sequence (e.g. a subpipeline) our
ReduceReduce algorithm naturally 'reduces' up the value
as if the original step was atomic.
The parameters of the left-fold() function are
described below;• $xproc:eval-step-func - step evaluator function
defined and passed in at runtime. This function is
responsible for running each steps function.
• $starting-input - every evaluation process sequence
has a starting input• sequence of @xproc:default-name-. Names are uniqueto a pipeline and while we could have passed in the
step's functions themselves it seemed to make more
sense to pass around the step's ID.

xproc.xq - Architecture of an XProc processor

You may recall that decorated pipelines defines an@xproc:func attribute and that is looked up for execution
by $xproc:eval-step-func function (which itself isdefined as xproc:evalstep#4 by default). Defining at
runtime, a function that runs each step function, proves
to be a powerful and flexible idiom that opens up a range
of interesting possibilities for redefining and enhancing
step processing without amending xproc.xq codebase.
(we have some fun doing just this later in the paper).
The other important responsibility of the dynamic
evaluation stage is to ensure that with each step processedthat its inputs and outputs are placed within the resource
manager, making them available in a consistent and easymanner for other steps to use. The resource manager isvendor specific and in the case of xproc.xq running on
MarkLogic [2], we take advantage of the database tostore this data. With other vendors we may have to push
to disk or keep process results within in memory data
structures (such as map:map extensions in many of the
XQuery processors).
Today, the resource manager just serves up andmanages internal IRI's which are constructed from theXProc episode system property1 and @xproc:defaultname but the plan is to use the resource manager tooverride URI to provide xml catalog2 like functionality.

In the case of xproc.xq it was important to provide an
entry point 'documented' module (xproc.xq) which is
the XQuery Library module developers would import
into their own XQuery projects. Having a rigid interface
encapsulates the implementation, just a fancy way ofsaying we can make changes to xproc-impl.xqy withoutchanging the entrypoint that other modules may invoke.

4.4. Serialisation
xproc.xq implements a naive serialisation strategy, as all
it does today is looks up from the resource manager the
output result port for the step named !1!, which in the
XProc naming convention is always the top level (andlast) result port.
The only other responsibility of the serialisation stage
is to output debug information (previously shown)which contains the decorated pipeline and a dump of all
port values.
As work progresses on xproc.xq, the serialisation layer
will need to become more sophisticated, especially in thearea of performance as we want to be able to persist
documents to the underlying database efficiently.

Isolating all vendor specific code within the util.xqy andresource.xqy modules assists in making it easier toprovide support for other XQuery vendors going forward
into the future.

4.6. A word about testing in xproc.xq

Tests for xproc.xq are contained within the src/tests
directory of the distribution and provide coverage for4.5. Code Layout
individual steps, as well as end to end testing from the
xprocxq:xq() entrypoint.
XQuery being a relatively youthful language with Tests are identified using a %test:case annotation withlightweight reuse mechanisms, means it is important to most tests employing the assert:equal() function to test
put some thought into how to design a project's code for equivalence between two inputs. The following codelayout.
listing shows how a typical test looks like;1
2

XProc system properties - http://www.w3.org/TR/xproc/#f.system-property
XML catalog - http://en.wikipedia.org/wiki/XML_Catalog

Page 127 of 162

xproc.xq - Architecture of an XProc processor

Example 10. test:testIdentity

5.1. XQuery 3.0 to the rescue

declare %test:case function test:testIdentity() {let $actual := std:identity(<test/>, (), (), ())
returnassert:equal($actual, document{<test></test>})};Introducing XQuery 3.0 into the xproc.xq project
represented a significant and positive turning point in its
development. In previous incarnations, using XQuery
v1.0, xproc.xq had serious issues as there were subtle andhard to debug issues arising due to the usage of evalfunctions, which were employed to achieve flexible
execution of pipelines and steps.
Problems with performance were appearing,
especially where we had nested evaluation. Each invokeof the eval function raised the overall computing
resources 'cost' through the need to clone an entire
execution environment.
As Michael Kay observed in his 2009 Balisage 'You
Pull, I’ll Push: on the Polarity of Pipelines' paper [12],XQuery can be used to create 'pull' pipelines which takeadvantage of using function calls as the primary
composition mechanism, in the case of xproc.xq weachieve this using fn:left-fold() to encapsulate nested
function calls. The cited work also demonstrates that pull
pipelines are not particularly good at broadcasting tomultiple execution streams of execution, but are good at
merging multiple inputs. As all branching mechanisms in
XProc naturally resolve to only one execution path at any
point in runtime, using a 'pull' style pipeline seems torepresent a good match.
Here are just a few reasons why XQuery 3.0 is• Using a Reducer, such as left-fold(), in
combination with dynamic function calls underpin
the heart of xproc.xq dynamic evaluation engine. Itmeans we have no problems with performance or anyof the aforementioned issues with spawned
environments for evaluation.
• XQuery 3.0 annotations feature is employed to
identify in the codebase step functions. As we canquery which functions have this annotation at
runtime it vastly simplifies xproc.xq extensibility
mechanism making it straightforward to author new
steps in pure XQuery.
• The choice of the 'flow' work flow model is a perfectmatch for a functional programming language which
has functions as first class citizens. All step inputs andoutputs are written once and never mutatedthereafter. Changing state 'in-place' is destructive andcan represent a loss of fidelity, as xproc.xq has acomplete trace of every steps input and outputs it isvery easy to diagnose and test.

The test suite runner used by xproc.xq is based on the
Rob Whitby's excellent XRay [11] which has both a web
and command line GUI.

XRay also defines the annotations %test:setup and%test:teardown for setting up the test environment.
Overall, XRay's set of features, robustness and speed
make it the perfect choice for XQuery testing in
MarkLogic [2] and can highly recommend its usage.
One gap in automation revolves around running thecurrent W3C XProc test suite which are not yet covered
under the XRay system and uses a custom test runnerbuilt in XQuery. Time permitting these tests will alsofind their way under XRay automation.
Another known deficiency is mock testing within
XProc. which could be addressed with implementationof a Resource Manager.

5. Some Design Decisions
There have been many design decisions over the past few
years of xproc.xq development, but some have been more
impactful then others.

Page 128 of 162

xproc.xq - Architecture of an XProc processor

5.2. Steps with XSLT & XQuery

5.3. Nascent Resource Manager

As explained in the xproc.xq history section, I had
embarked on large pipelining projects in early 2000'swith XSLT and even had gone so far as to implement aprimitive version of xproc.xq using XSLT v2.0.
As I had originally targeted xproc.xq to run in thecontext of an XML database, it seemed a reasonablechoice to use XQuery as the implementation language (itbeing the stored proc language for most XML databases).
The collory to this decision was to banish XSLT v2.0
which turned out to be the wrong approach. My original
concerns had revolved around using XSLT v2.0 within
the dynamic evaluation phase, but I in so deciding I also
opted out of using XSLT v2.0 anywhere in xproc.xq
development. XSLT's polymorphism and dynamic
dispatch makes static analysis hard enough if you arecreating an XSLT processor and even more difficult if
the static analysis is being performed at a higher
application level.
XProc dependency on XSLT match patterns
combined with the fact that many of the steps lentthemselves to implementation using XSLT v2.0, I had
inadvertently created a more difficult development path.
Another aggravating factor was the gap between maturityof XSLT 2.0 and XQuery v1.0, which usually resulted insome hodge podge of XQuery v1.0 with vendor
extensions.
Finally, during the transition from using Saxon [5] toMarklogic [2] XQuery processor, I changed my decisionand decided to use XSLT v2.0 in the implementation ofsteps.
This was enormously useful as it had an immediate
effect of simplifying many of the buggy step functions
and enabling the development of some functions which
had proved complicated to even start developing.

xproc.xq implements a simple resource manager thatprovides durable storage of every input and output that is
generated throughout the dynamic evaluation phase.
Today, this resource manager is used for internal lookupof port values and is useful when debugging pipelines.
As it exists today in the codebase, xproc.xq is wellplaced to expose resource manager functionality directly
to XProc developers, in the form of switches, options or
extension steps. An enhanced resource manager could
provide;• pluggable storage api - could mean that storage
backends could be easily swapped with no change to
the XProc code. Absolute URI's could be overriden or
new scheme's could be contemplated (ex.
resource://).
• mock testing - being able to 'override' URI's that referto real resources would make testing easier andprovide greater coverage
There has been much discussion on the W3C XML
Processing Working Group about the idea of formalising
the concept of a resource manager within XProc. I am
not personally convinced that the resource manager
needs to be a sanctioned feature of XProc, so far its felt
like an implementation detail, though this is one of thosefeatures where experience in the field should be gathered
before rushing to specification.

6. Having some fun
Now that we've dived into the innards of xproc.xq and
explained some of the more significant design decisions,its time to corroborate the impact of these decisions bydirectly experiencing xproc.xq.

6.1. Run step function in XQuery
Why let xproc.xq have all the fun ? We can run
individual steps from XQuery, as long as we know what
to pass to the step function.
Each step function has the same functional signature;• primary input - input sequence of XML document(s)• secondary input sequence of XML document(s)• options - you will need to use specific xproc:options
syntax for specifying• variables - n/a
The example shows how we can apply the p:addattribute step operation to an XML document;

Page 129 of 162

xproc.xq - Architecture of an XProc processor

Example 11. invoking a step in XQuery
xquery version "3.0";
import module namespace std ="http://xproc.net/xproc/std"at "/xquery/steps/std.xqy";
declare namespace p = "http://www.w3.org/ns/xproc";declare namespace xproc = "http://xproc.net/xproc";
std:add-attribute(<test/>,(),<xproc:options><p:with-option name="match"select="*"/><p:with-option name="attribute-name"select="id"/><p:with-option name="attribute-value"
select="'test'"/></xproc:options>,())

6.2. Extending xproc.xq with pure xquery
steps
Creating an extension step using pure XQuery is
straightforward in xproc.xq. You just add your new step
function into the src/steps/ext.xqy, ensuring to mark itwith the %xproc:step annotation
Example 12. new ext:mynewstep function
declare %xproc:step function ext:mynewstep($primary, $secondary,$options, $variables){<my-new-step>my new step</my-new-step>};

All that is required is to add this new step's definition
into the internal XProc library which defines extensions,at src/etc/pipeline-extensions.xml.

Page 130 of 162

xproc.xq - Architecture of an XProc processor

Example 13. ext step library<p:library xmlns:p="http://www.w3.org/ns/xproc" xmlns:ext="http://xproc.net/xproc/ext"
xmlns:xproc="http://xproc.net/xproc" name="xprocxq-extension-library"><p:declare-step type="ext:pre" xproc:step="true" xproc:bindings="all" xproc:support="true"
xproc:func="ext:pre#4"><p:input port="source" primary="true" sequence="true" select="/"/><p:output port="result" primary="true" sequence="true" select="/"/></p:declare-step><p:declare-step type="ext:post" xproc:step="true" xproc:func="ext:post#4" xproc:support="true"><p:input port="source" primary="true" sequence="true" select="/"/><p:output port="result" primary="true" sequence="true" select="/"/></p:declare-step><p:declare-step type="ext:xproc" xproc:step="true" xproc:func="ext:xproc#4" xproc:support="true"><p:input port="source" primary="true" select="/"/><p:input port="pipeline" primary="false" select="/"/><p:input port="bindings" primary="false" select="/"/><p:output port="result" primary="true"/><p:option name="dflag" select="0"/><p:option name="tflag" select="0"/></p:declare-step><p:declare-step type="ext:xsltforms" xproc:step="true" xproc:func="ext:xsltforms" xproc:support="true"><p:input port="source" sequence="true" primary="true" select="/"/><p:output port="result" primary="true" select="/"/><p:option name="xsltformsURI"/><p:option name="debug"/></p:declare-step><p:declare-step type="ext:mynewstep" xproc:step="true" xproc:func="ext:mynewstep" xproc:support="true"><p:input port="source" sequence="true" primary="true" select="/"/><p:output port="result" primary="true" select="/"/></p:declare-step></p:library>

The library markup contains some extra attributes,
which assist the dynamic evaluation stage validate a step's
signature.
You will never need to use a p:import statement to use
the extension steps as they library is loaded
automatically.
The standard and optional steps are implemented in asimilar manner, but these libraries should only containsteps defined in the XProc specification.

Page 131 of 162

xproc.xq - Architecture of an XProc processor

6.3. BYOSR (bring your own step runner)

Example 15. anonymous function as step runner

The following code listing shows the overloaded versionof the xprocxq:xq() function where you can pass it afunction that evaluates each step function (the default forthis is $xproc:eval-step-func). During dynamic
evaluation, this step runner is responsible for executing
each step's function.

xquery version "3.0";

Example 14. step runnerxquery version "3.0";
import module namespace xprocxq ="http://xproc.net/xprocxq"
at "/xquery/xproc.xq";
import module namespace xproc ="http://xproc.net/xproc"
at "/xquery/core/xproc-impl.xqy";
import module namespace u ="http://xproc.net/xproc/util"
at "/xquery/core/util.xqy";
declare namespace p="http://www.w3.org/ns/xproc";
let $pipeline :=<p:declare-step version='1.0'><p:input port="source" sequence="true"/><p:output port="result"/><p:count/></p:declare-step>
let $stdin := (<document><doc xmlns=""/></document>,<document><doc xmlns=""/></document>,<document><doc xmlns=""/></document>)
let $dflag:= 0
return xprocxq:xq($pipeline,$stdin, (), (), (),$dflag, 0,$xproc:eval-step-func)

Looking up $xproc:eval-step-func() insrc/core/xproc-impl.xqy we see it has 4 parameters inimport module namespace xprocxq ="http://xproc.net/xprocxq"
at "/xquery/xproc.xq";
import module namespace xproc ="http://xproc.net/xproc"
at "/xquery/core/xproc-impl.xqy";
import module namespace u ="http://xproc.net/xproc/util"
at "/xquery/core/util.xqy";
declare namespace p="http://www.w3.org/ns/xproc";
let $pipeline :=<p:declare-step version='1.0'><p:input port="source" sequence="true"/><p:output port="result"/><p:count/></p:declare-step>
let $stdin := (<document><doc xmlns=""/></document>,<document><doc xmlns=""/></document>,<document><doc xmlns=""/></document>)
let $dflag:= 0
let $tflag:= 0
let $bindings := ()let $options := ()let $outputs := ()returnxprocxq:xq($pipeline, $stdin, (), (), (),$dflag, 0,function($step, $namespaces, $input, $ast) {$xproc:eval-step-func($step, $namespaces, $input, $ast)})

Everything works as it does before, all we've done isits signature, so we could easily rewrite as an anonymous provide an anonymous function wrapper around our
dynamic invocation.
function, as shown in bold in the amended code listing• $step - contains the xproc:default-name for the step
• $namespaces - list of declared namespaces• $input - usually contains primary input port value fora step• $ast - the decorated pipeline is passed in and used asa kind of 'lookup' table

Page 132 of 162

xproc.xq - Architecture of an XProc processor

What if we wanted to do some additional processing With xproc.xq release, it now transitions from my own
with each step or preprocess any of the parameters ? We personal 'workbench' to something I hope others find as
can use this anonymous function idiom to trivially insert useful as I've found it instructive in developing.
new runtime behaviors without having to amend a lineof the core xproc.xq code itself.
A. Getting and installing xproc.xq
Example 16. enhancing step runner with anonymous
functionfunction($step, $namespaces, $input, $ast) {let $log := u:log("processing step: " ||$step || " at " ||fn:current-dateTime())
return$xproc:eval-step-func($step, $namespaces, $input, $ast)}

The only dependency for xproc.xq is an installed andavailable MarkLogic Server version 6.03 or later,
Download xproc.xq fromhttps://github.com/xquery/xproc.xq and follow the upto-date installation instructions contained in the README.

The example uses u:log() function which is a wrapper tothe vendors own log function.
When we run the pipeline now and observe MarkLogic's
ErrorLog.txt file, we see a trace of each step's name andthe timestamp when processed.
Alternately, we could have opted to implement a stepthat does this logging, but this would be presumably a
step that does nothing with the input and worst does not
output anything (or just copies input to output like
p:identity step). Just like functions with no input or
return no output, steps that do nothing with the flowing
XML document(s) indicate that some 'impure'processing with side effects is going on. This idiom
allows you to mitigate the impact 'out of band'processing.

7. Summary
XProc provides the foundation which to create a facade
over the bewildering array of XML technologies inexistence today. In this sense, XProc can be the the layer
which exposes a component architecture designed forprovisioning reusable software.
The short term future for xproc.xq is;• continue enhancing quality and adherence to the
XProc specification• document performance within the MarkLogic
database and enhance serialisation/persistence to thedata layer• With help from the community, I would like to see
xproc.xq running on other vendor XQuery 3.0
processors.
• replace parsing with solution based on Gunther
Rademacher's REx Parser Generator1

1

http://www.bottlecaps.de/rex/

Page 133 of 162

xproc.xq - Architecture of an XProc processor

Bibliography[1] XProc: An XML Pipeline Language, W3C Recommendation 11 May 2010
http://www.w3.org/TR/xproc/[2] MarkLogic
http://www.marklogic.com[3] Finite-state machine. (2013, September 10). In Wikipedia, The Free Encyclopedia. Retrieved 10:11,
September 12, 2013, from http://en.wikipedia.org/w/index.php?title=Finite-state_machine&oldid=572362215[4] Zergaoui, Mohamed. “Memory management in streaming: Buffering, lookahead, or none. Which to choose?”
Presented at International Symposium on Processing XML Efficiently: Overcoming Limits on Space, Time, orBandwidth, Montréal, Canada, August 10, 2009. In Proceedings of the International Symposium on
Processing XML Efficiently: Overcoming Limits on Space, Time, or Bandwidth. Balisage Series on Markup
Technologies, vol. 4 (2009). doi:10.4242/BalisageVol4.Zergaoui02.
[5] Michael Kay's XSLT & XQuery Processor
http://www.saxonica.com[6] Kay, Michael. “A Streaming XSLT Processor.” Presented at Balisage: The Markup Conference 2010,
Montréal, Canada, August 3 - 6, 2010. In Proceedings of Balisage: The Markup Conference 2010. Balisage
Series on Markup Technologies, vol. 5 (2010). doi:10.4242/BalisageVol5.Kay01.
[7] eXist XML Database
http://www.exist-db.org[8] The xproc.xq project hosted at GitHub
https://github.com/xquery/xproc.xq[9] XSL Transformations (XSLT) Version 2.0. Michael Kay, editor. W3C Recommendation. 23 January 2007.
http://www.w3.org/TR/xslt20/[10] Transform.xq - A Transformation Library for XQuery 3.0. John Snelson. XML Prague 2012.
http://archive.xmlprague.cz/2012/files/xmlprague-2012-proceedings.pdf[11] Rob Whitby's XRay
https://github.com/robwhitby/xray[12] Kay, Michael. “You Pull, I’ll Push: on the Polarity of Pipelines.” Presented at Balisage: The Markup
Conference 2009, Montréal, Canada, August 11 - 14, 2009. In Proceedings of Balisage: The Markup
Conference 2009. Balisage Series on Markup Technologies, vol. 3 (2009). doi:10.4242/BalisageVol3.Kay01.

Page 134 of 162

Lazy processing of XML in XSLT for big data
Abel Braaksma
Abrasoft<abel@abrasoft.net>

Abstract

2. Introduction

In recent years we've come to see more and more reports onprocessing big XML data with XSLT, mainly targeted atstreaming XML.This has several disadvantages, mainly
because streaming is often implemented as forward-only
processing, which limits the expressive power of XSLT and
XPath.
In this paper I present an alternative which processes
XML in a lazy manner by not fully loading the whole XML
document in memory and by timely dismissing XML
fragments.
We will find that this solves many document-centric
XML use-cases for large datasets, while leaving the fullpower of XSLT at your fingertips. In addition, the small
memory footprint of this method makes it ideal for scenarios
such as mobile devices where memory is limited.

Since the dawn of XML there have been many attempts
to load XML efficiently. In this paper we are onlyinterested in those methods that enable XML processingof large datasets for scenarios where available main
memory is not enough to contain the XML in memory
at once. These methods typically fall in one of thefollowing categories:• Streaming;• Lazy loading;• Succinct data structures;• Compression.
Streaming is arguably the best known method forprocessing large XML documents. Its main features arehigh speed of parsing and constant low memory
footprint. With most streaming approaches, each node isvisited once in a depth-first left-to-right traversal. Thismeans that its biggest limitation that it cannot
accomodate for free-ranging traversal through the tree,
for instance, a normal DOM tree would be out of thequestion. Several types of streaming XML exist, from
SAX in Java [SAX] and XmlReader in .NET [XRDR] to
the highly versatile but complex STX engine [STX]. In
the XSLT domain, streaming is available as an extension
to Saxon [SSTRM] and since XSLT 3.0 it is alsoavailable for compliant processors that implement thisoptional feature, like Saxon [Saxon] and the upcoming
Exselt [Exselt] processors.
Lazy loading is a method where nodes that are notrequired are not loaded from the storage medium,
henceforth resulting in a smaller memory footprint and,depending on the scenario, faster processing and loading
times. Typically, this method works best from an XML
database, but can also applied from disk. One such
example is Oracle's XDB [XDB], but they call it Lazy
Manifestation. I'm not aware of XSLT processors
supporting this directly, but we will see that bycombining streaming and classical approaches, it ispossible to mimic this behavior and to get similar
benefits.

Keywords: XML, XSLT, XPath, big-data,
lazy-processing, lazy-loading

1. Disclaimer
This paper is based on the publicly available versions ofXPath 3.0, XSLT 3.0 and XDM 3.0 as of January 10,2013 [XSLWD][XPCR][XDM3]. Since neither of these
specifications is final, it is possible that references anddetails change before the final specification has received
Recommendation status. Some items discussed here havenot (yet) made it into the public Working Draft.

doi:10.14337/XMLLondon13.Braaksma01

Page 135 of 162

Lazy processing of XML in XSLT for big data

Succinct data structures [Jacobsen] are a way ofminimizing storage required by the data structures that
store references to the objects in a DOM tree. Some ofthese methods are briefly described below. This is a wayof compressing the data structure without having todecompress them when traversing the data tree. Thismethod was presented at XML Prague 2013 on a poster
by [Joannou] and the VTD parser [VTD] is a practicalavailable example.
Compression can be applied in many forms. A
compressed tree requires uncompressing prior toprocessing it, which makes it less suitable for XML
processing with XSLT. However, partial compression, or
on-the-fly compression, for instance of text nodes only,
may have a significant benefit to make the overallmemory footprint smaller.
Combinations of these methods exist, again theposter by Joannou showed that on-the-fly compressionof text-nodes, possibly combined with lazy-loading, canyield a memory footprint for the whole XDM or DOM
to be less than 100%, in certain cases even as low as 50%of the on-disk size, without compromising onfunctionality.
This paper focuses on lazy loading of XML as an
approach to optimizing XML processing with XSLT. We
will find that current standards of XSLT, namely XSLT
1.0 and XSLT 2.0, do not provide any means for lazyloading, but with the upcoming feature of streaming in
XSLT 3.0, combined with the traditional XDM
approach of processing, we have powerful tools at handto simulate lazy loading without the need for a specificXML parser that does so. However, we will also see thatit depends on implementations to make use of theavailable information in the stylesheet.

3. The challengeWith streaming, it is possible to process virtually any sizeof input XML with XSLT. However, streaming XSLT isseverely limited. While the XSL Working Group hasgone to great lengths to make streaming as useful as
possible, it is by its very nature forward-only, whichmakes it very hard to process a large document where therequirement is to use free-ranging expressions. In otherwords, when you need to look back and forward throughthe document from any given point in the process.
References and index building are examples of theserequirements.
The approach presented here will make it possible toprocess XML documents that do not fit in memory atonce, but would fit in memory if only the nodes that areneeded for the processing were loaded. This approach I
call lazy processing of XML.

Page 136 of 162

4. The method
With classical XSLT 1.0 and 2.0, the processor will build
an XDM tree of the whole XML source document. With
streaming, the XML source document is not maintained
in memory but instead is loaded on the fly with only thecurrent node, without its siblings or children, is typically
kept in memory. Streaming helps primarily in those cases
where the XML document does not fit in memory as awhole, or when you are reading from a source thatcannot give the whole document at once, for instancewhen processing a Twitter or news feed.
We consider streaming a whole document without
the need of processing a given node, skipping that node.
Skipping a node gives the XSLT processor a hint that it
does not need to load that node in memory, simply
because it is not needed. This is only partially true, ofcourse, as the position of subsequent nodes relies on theskipped node, and thus certain properties must always beloaded, but the node itself and its contents and children,
are not required. In its simplest form, the following
example can skip the whole document except for the root
element:<xsl:mode streamable="yes" /><xsl:template match="/root"><xsl:value-of select="@creation-date" /></xsl:template>This example only reads out an attribute of the root nodeand doesn't do any further processing. We will use thisexample in the timings to find out the difference
between a processor skipping nodes and a processor
actually processing individual nodes.
A counter-example that requires all nodes to beprocessed is the following:<xsl:strip-space elements="*" /><xsl:mode streamable="yes" />This example merely processes all nodes through thedefault template rules and will output it as text, skipping
white-space-only text nodes.
For measuring the difference in processing speed
between those two methods, skipping all, and processing
all, I added a small template to remove all text nodes
from the output stream, to make sure writing the output
stream does not tamper with the timings. This found aremarkable difference with streaming processors, where
the first example was processed in 4.5 seconds and thesecond in 6.1 seconds. Similar differences in timings
were seen with different sizes of the input.
This observation is the basis of selectively processing
nodes, which we'll examine in the next sections.

Lazy processing of XML in XSLT for big data

5. Filtering nodes
The essence of lazy processing XML is to not load any
nodes that are not used. In its most simple form,
consider that you want to filter out certain elements froman XML document. If you are not interested in theseelements, you do not need to load them.
Suppose you are only interested in the chapters of abook, but not in the contents, you could've previously
written a stylesheet as follows:<xsl:template match="chapter"><chapter><xsl:value-of select="@title" /></chapter></xsl:template><xsl:template match="paragraph" />Assuming the structure of your input is as follows:<chapter title="some title" /><paragraph> some paragraph</paragraph><chapter title="some title" /><paragraph> some paragraph</paragraph><paragraph> some paragraph</paragraph><chapter title="some title" /><paragraph> some paragraph</paragraph>

This will effectively only output the titles. But what you
are essentially doing here is telling the processor toprocess the paragraph nodes, but then to do nothing.
This has become easier in XSLT 3.0, where we can
explicitly tell the processor to ignore unmatched nodes
when they are encountered:<xsl:mode on-no-match="deep-skip" />This declaration means that all nodes that are notspecifically matched will be skipped, including all its
children. This is mainly a simplified way of writing what
was already possible, but it also tells the processor it canquickly skip over unmatched nodes, and ignore all its
children.
However, in the classical approach, this helps us little
with lazy loading, because as long as we still have to load
the whole document in the XDM, the skipped nodes arestill parsed. For large documents, this can be quite aperformance hit. In cases where the document still fitted
into memory, there was no noticable performance gainof using deep-skip.
In trivial scenario's, a processor could ignore those
nodes while loading the XML into the XDM, but if aprocessor were indeed to do so, it will have to take extra
care of the positions of nodes, because positions don'tchange, even when a node is skipped.

An alternative approach, where the programmer could
explicitly tell the processor what nodes to skip, was aptly
called xsl:filter, i.e:
<xsl:filter operation="ignore" select="paragraph"/>However, this approach violates the node identities andthe node traversal. Parsing the same document with orwithout a filter, would yield different node identies,
because of the missing nodes. It could've been a shortcut
for a micro-pipeline, but that technique is already wellknown and programmers have more expressive power
capturing the result of a transformation in a variable andre-processing it.
Some processors, namely Saxon, have an extension
function that allows a certain kind of filtering. In thecase of Saxon it is called Burst Mode Streaming [Burst]
and it allows you to create a single select statement that
only selects those nodes from an input stream that you
require. However, the returned nodes are parentlessnodes and it is not possible to traverse between the
disconnected nodes.

6. Lazy loading of XML
The limitations of skipping nodes as a means for
programmer-induced lazy loading do not fit its purpose.
The idea of lazy loading is not to skip nodes, but to beable to process a node when we need it, and to only load
it when we need it.
However, the nature of XML is that it does not
maintain any information on node length, so unless thesource of the XML is a system that does have thisinformation, like a database, we will still require,
whatever approach we'll take, to load each node at least
once, if not only just to know where the next node will
start.
There are few systems around that apply a form oflazy loading in the traditional sense. Taking the previous
paragraph, it may come as no surprise that all of thesesystems are in fact databases. In a way, they cheat,because any XML database must have at least read the
XML already. What they do is, when you create a queryof the data, that they only provide you with the actual
data of that query when you actually process thatparticular node, of element, which typically maps to acertain row in a table. Oracle takes this approach in xDB
for instance.

Page 137 of 162

Lazy processing of XML in XSLT for big data

Given the above, what can we consider lazy loading of 8. Using streaming to achieve lazy
XML if we apply the term to other systems than
databases? When XML is loaded into a DOM or an loading
XDM, a lot of time is spent on constructing the data
model. Skipping constructing the data model, or not In the previous filtering example, we were trying toloading the node information in memory, is lazy loading simply output all the chapter titles. Let's rewrite thatexample using streaming:
XML.

7. Streaming processing of XML
using XSLT

<xsl:mode on-no-match="deep-skip"
streamable="yes" /><xsl:template match="chapter"><chapter><xsl:value-of select="@title" /></chapter></xsl:template>At the core of the new features of XSLT 3.0 lies one
simple declaration, xsl:mode, which sets the features aparticular mode must operate under. The feature ofinterest here is turning streaming off or on for any Now that was easy, wasn't it? We simply mark theparticular mode:
default unnamed mode as streamable and leave the rest
as it was. This works, because the only template rule in<!-- turn streaming on for the default,
this stylesheet has one downward expression only
unnamed mode -->(actually, the attribute axis is not considered a downward<xsl:mode streamable="yes" />expression, it is possible to access multiple attributes in a<!-- turn streaming on for mode withsingle sequence constructor).
the name "streaming" -->What has changed is the mode of operation. The<xsl:mode name="streaming" streamable="yes" />processor will not load the document into an XDM in
When streaming is turned on, all template rules for this the tradition way. Instead, it will now load it streamed,
mode must be guaranteed streamable. Whether a that is, it will not keep a memory of the tree of nodes
template rule is guaranteed streamable of not is beyond that are being processed.
In this example, we have applied lazy loading to thethe scope of this paper, but bottom line is that eachtemplate rule must have at most one downward whole document. One could argue, that the only node
expression and no upward epxressions or otherwise free- that was necessarily loaded completely, is the title
attribute node, because it was needed to create the text
ranging expressions.
According to the XSLT 3.0 specification, streaming node with xsl:value-of. It was not needed to load thewhole chapter element, nor was it needed to load themeans the following [Streaming]:
paragraph elements, let alone the children of theseelements. Of course, the XML parser is required to
The term streaming refers to a manner ofprocess these elements and all their attributes to at least
processing in which documents (such as
the minimal extend required to determine the wellsource and result documents) are notformedness and, in the case of schema-awareness, therepresented by a complete tree of nodes
validity. But it was not required, in fact it wasn't evenoccupying memory proportional toallowed, to create an XDM tree of any of these nodes.
document size, but instead are processed
Let us see how this performs in our original extreme"on the fly" as a sequence of events, similar
example:in concept to the stream of events notified
by an XML parser to represent markup in<xsl:mode streamable="yes"lexical XML.
on-no-match="deep-skip" />In other words, the memory will remain constant.

Page 138 of 162<xsl:template match="/root"><xsl:value-of select="@creation-date" /><!-- continue processing to see
on-no-match in practice --><xsl:xsl-apply-templates /></xsl:template>Lazy processing of XML in XSLT for big data

The difference with the original example is that we now
force the processor to process all nodes. However, we
also tell the processor to deep-skip any nodes that are notmatched. The result of timing this is similar to theoriginal example in The Method section above, it is only
marginally slower. This shows us that this approach gives
us a strong handle on how to lazily load elements only
when we need them.
Now that we know how to not load the nodes, howwould we go about loading the nodes once we actuallyneed them? How can we break out the streaming modeand load one element, and all its children, or perhaps amore fine-grained selection, into memory?
XSLT 3.0 comes with several aides to achieve this.
The main approaches are the following:• fn:snapshot• fn:copy-of• micro-pipelining
One might think that simply switching modes would beenough, but switching modes is not allowed. Once you
are in a guaranteed streamable mode, you cannot simply
break out by applying a non-streaming mode.

8.1. fn:snapshot and fn:copy

There's currently an easy way out of this. The trick is tochange the context in the for-each loop to be the context
of the result of the snapshot. For instance, you canrewrite the above example as follows:<xsl:mode streamable="yes" /><xsl:mode streamable="no" name="non-streaming" /><xsl:template match="employees"><xsl:for-eachselect="employee[position() = 3]/copy-of()"><xsl:apply-templates select="."mode="non-streaming" /></xsl:for-each></xsl:template><xsl:template match="employee"
mode="non-streaming">....
</xsl:template>

Note that we moved the if-statement inside the XPathexpression. If we did not do so, the benefits of selectively
copying only the needed elements would be void.

8.2. Micro-pipelining
Micro-pipelines have been possible since XSLT 2.0. In
its simplest form, they take a (modified) copy of data
into a variable and process it again. In the case of
streaming, the possibilities of micro-pipelining get a new
dimension and make it possible to effectively create veryfine-grained lazy loading XSLT transformations.
An example of micro-pipelining:

You can take a snapshot of a node and its children at any
time, which copies all the nodes and makes themavailable for normal processing. This is the easiest way offully loading a particular node that you are interested in.
In the following example, this is demonstrated by<xsl:variable name="filtered">loading only the third employee and all its children:<xsl:mode streamable="yes" /><xsl:mode streamable="no" name="non-streaming" /><xsl:template match="employees"><xsl:for-each select="employee"><xsl:if test="position() = 3"><xsl:apply-templates select="fn:snapshot(.)"
mode="non-streaming" /></xsl:if></xsl:for-each></xsl:template><xsl:template match="employee"
mode="non-streaming">....
</xsl:template>

Note the switching of modes. In the current state of thestandard, this is not allowed by the streamability rulesthat are in the current internal draft of the spec. The
previous, and current public Working Draft does allow
this type of processing though. It is not sure whether thiswill become possible or not in the final specification.

<xsl:stream href="feed.xml"><xsl:for-each select="news/news-item[@date > '2013-06-10']"><xsl:copy-of select="." /></xsl:for-each></xsl:stream></xsl:variable><xsl:apply-templates select="$filtered/news-item" />This example lazily loads all news items, and then creates
an XDM of only the items after June 10, 2013.

9. More advanced lazy loading
A prime example where lazy loading can be applied iswhen the requirements clearly state that we do not needto look at all elements in the source document, and when
the selection of these skippable elements form asignificant part of it. For instance, consider that you arecreating a document containing only the abstracts andtitles of all papers, and all papers are inside one
document. If the XML is similar to DocBook, you canmake use of the deep-skip method examples like
provided above.

Page 139 of 162

Lazy processing of XML in XSLT for big data

If we expand the requirements a little further and only
want to show the abstracts of papers that have been
written by a certain author, simply using deep-skip will
not be possible, because we cannot create a pattern thattests the contents of text nodes (this is considered a freeranging expression, because the whole text node must be
loaded to know whether it matches a certain string). In
this scenario, the copy-of function comes to our help, as
it allows a free-ranging expression and the expression canbe used by the processor to return only a limited set ofnodes. For instance:<xsl:copy-of
select="article/info/copy-of()[author/personname = 'John']/(abstract | title)" />The predicate, as above, without the fn:copy-of function,
would be consider free-ranging because it requires
processing the personname elements and its children. By
adding the fn:copy-of function, this works, as thefunction creates a copy of the current node which canthen be processed in the normal, non-streaming way.

10. Always lazy loading XML with
XSLT

11. Limits of lazy loading XML
While there is no limit to the size of the input XML
when using streaming, lazy loading has its limits. In fact,
you have to be really careful. In particular, you need toconsider the following rules:• Do not use snapshots when your stream should run
indefinitely, unless you've confirmed that yourprocessor discards the used memory correctly;• The size of the snapshots multiplied by the maximum
amount of taken snapshots should not exceedavailable memory;• Minimize the size of the snapshots by micropipelining them and removing all unnecesary datafrom the stream• Do not use snapshots in inner loops
While the first rule is obvious (indefinitely running input
stream will yield indefinite required size for snapshots),
the second one can be hard to measure. Depending on
the processor and the chosen XML parser, as a rule ofthumb, take the size on disk and multiply that by 2.5 to
4.0. Multiply the result with the expected amount ofsnapshots to be taken and you should have a fair guess ofthe memory needed.
If your processor has an extension function to discard
the memory used, or to flush the output stream to disk,
your reach is virtually unlimited.

It would be nice if we had a way to process any XSLT
using streaming and lazy loading automatically.
However, the complexity of the possible scenarios makes
12. Lazy processing performance
it impossible for an automated process to determine
statically how the best lazy loading approach should
To determine the performance of lazy loading, I've
work. Even more, this is the main reason why you
created several scenario's with a large input file, which I
cannot simply turn every stylesheet into a streaming
processed in one of four ways:
stylesheet.
For instance, if your requirement is to visit the • Deep: use deep-skip, only read an attribute of the root
element of the source document, this is the minimum
preceding-sibling axis often, or, worse, you need to use
possible transformation.
the preceding axis in a predicate inside a pattern, it is
Deep02: deep-skip, processing 2% of the elements of•unlikely that streaming or lazy loading will help.
thesource document, a typical use-case where only a
Similarly, if you need to find a string in any node, there's
small
part of the original document needs to be
no way lazy loading will help, because you need to visit
processed.
every node anyway.
When dealing with large documents, one needs to • Shallow02: shallow-skip, processing 2% of theelements of the source document, same as previous,
limit oneself. Using the lazy loading technique, you canbut forces the processor to go over each node to find aachieve the best of both worlds. Instead of using freematch.
ranging expressions on the whole tree, you need toconsider filtering the input tree to a smaller subset, or if • Text: the same transform as above, but this time thenon-matching nodes are matched and a one-character
that is not possible, use streaming and lazy loading tostring is output to prevent the processor from
specifically load the parts only when you need them.
optimizing the match away.

Page 140 of 162

Lazy processing of XML in XSLT for big data

All transformations are equal, except for the last one,
where an addition match="node()" is added. The other
differ only in the setting of the on-no-match attribute of
the current mode. The first one however uses a deliberate
non-selecting apply-templates and only matches against
the root node. This we use to have a baseline to compare
the measurements to: it is the simplest transformation
possible, and it measures how fast the processor dismisses
the input document once it knows that the selectstatement selects nothing.
As input for the transformation I used an XML
corpus of all Shakespeare's plays. The combined plays
where copied in one XML file, which surmounted toabout 10MB and then copied again until the sizes
100MB, 500MB and 5GB were reached. This file isrelatively document-centric, which is a requirement forgood tests, because using a vanilla XML file gives theprocessor a lot of chances to read-ahead, which does notgive very meaningful results.
The processor used for all tests was Saxon 9.5.0.2 forJava and it was run on a 2x 3.3GHz Xeon 48GB Dell
Precision Workstation with a Windows 7 64 bit
operating system. To eliminate timer errors, each test was
run 5 times, the lowest and highest values were removed
and the other three arithmetically averaged. All results
are shown in seconds.
Let us look at some graphs. The first graph shows
processing a 100MB and a 500MB input file using
traditional in-memory non-streaming processing. The
deep02 and shallow02 bars are noticeably of the same
size. This means that the processor doesn't optimize thescenario of deep-skip, or, perhaps more likely, because
the document is already loaded in memory anyway, there
is no benefit for deep-skip versus shallow-skip. The testof the 5GB document is not included, because it didn't
load in the available memory. The largest measured
memory size was for the shallow02 test, the processor
took about 4.1GB.

This gives us a good starting point to compare the inmemory processing to the streaming processing. I ran
exactly the same tests, except that the modes were
switched to streaming. Three things became immediately
apparent:• Processing with streaming takes longer, on average
about 20-25%• Memory remains constant on about 200MB (thoughstill quite high, I didn't try to force the Java VM toforce less available memory)• On the longer runs, memory slowly dropped during
the process to about 140MB on the longest run. I
have not been able to find a satisfying explanation forthis behavior.
That streaming takes longer may come as a surprise.
However, streaming, and more particularly streaming
using the XSLT 3.0 standard, is fairly new. In fact, theprevious version of Saxon, 9.4 didn't yes support thestreaming feature (it did support streaming extensions
though). Many features of streaming, in particular theones related to guaranteed streamability analysis, are still
heavily under development. As Michael Kay mentions
himself in [Kay], features first, performance later. In
theory, streaming can be much faster than in-memory
processing, especially when the stylesheet is designedwith streaming in mind.
The main reason for looking at the streaming
timings, is to find the difference of lazy processing or
loading XML nodes. What happens when we skip nodes?
What happens when we only load a subset of the nodes?
The deep02 and shallow02 only touch a subset of thenodes and skip the rest:

Page 141 of 162

Lazy processing of XML in XSLT for big data

The graph shows a very clear performance gain when
deep-skipping nodes in a streaming process. This
stylesheet mimics the behavior that you find when you
deliberately use fn:copy-of or fn:snapshot, but I wasn't
able to use these functions correctly, the implementation
still treats too many scenarios as not streamable. Hence I
stuck to using the more straightforward approach ofskipping nodes. We can conclude three things here:• Processing all nodes versus processing one node takes
double or more time (note that we only measure the"touching" of nodes by matching them, we don't
actually do anything complex with the nodes, which
would clearly make this distinction even bigger)• There's an expected and noticeable performancedifference between using deep-skip and shallow-skip.
Even in a scenario where only 2% of the input
document is actually matched, deep vs shallow
skipping shows a big difference. It is more challenging
to write a stylesheet for deep-skipping, but it is worth
the effort when processing large documents.
• Lazy loading has a clear advantage over processing all
nodes. Thinking carefully about the design of an
XSLT stylesheet becomes more and more important,because a naive implementation may easily load too
many nodes, which costs the processor too much timeto process. This difference is much more apparent
than in in-memory processing.
Let us look at one more graph to see what happens when
we try to load a really large document of 5GB:

Page 142 of 162

The document processed here was exactly 10x the size ofthe 500MB document. The timings are almost exactly
10x as long as well. In streaming environments, thislineair performance behavior is very expected, because of
the way streaming works. With in-memory processing,many performance comparisons that have been done
over the years, have shown a non-lineair performancegraph [Zavoral].

13. Further improvements using
succinct data models
Beyond the XSLT 3.0 standard, more improvements are
possible. These improvements must be done on the
XML data model. Several researches and implementation
exist currently, that apply the succinct data model
principles to XML loading.
During the International Symposium on Information
Processing 2009 in Huangshan, China, Yunsong Zhang,
Lei Zhao and Jiwen Yang presented a method for reusing
the parsed XML tree to optimize repetitive processing[Zhang]. They called this R-NEMXML and the essential
idea they presented was to encode the node information
in a 64 bit integer and store the actual data of the treeelsewhere. The integer contains type information and an
offset to the actual node in another storage medium.
There paper was a follow up on NEMXML, presented inJune the same year, which showed a non-extractive
method of processing XML.

Lazy processing of XML in XSLT for big data

In XML Prague in February 2013, Stelios Joannou,
Andreas Poyias and Rajeev Raman of the University ofLeicester presented a poster about their SiXML (Succinct
Indexing XML) parser [Joannou], which took the
NEMXML idea one level further to provide spaceefficient data structures for in-memory parsing of large
XML. They did so by storing the data structures in apointerless data structure as parenthesized strings, known
as succinct data structures [SiXML]. SixDOM and
SiXML was pioneered in 2009 by O'Neil Delpratt in histhesis at the University of Leicester [Delpratt]. Earlier
work similar to this, but using a different approach canbe seen in the VTD XML parser as mentioned in the
introduction, which came forth from the 2003 concept
presented by XimpleWare. VTD stands for Virtual
Token Descriptor [VTD] and stores the location of dataof nodes using an offset, as opposed to using objects torepresent the data, which has a relative overhead that
bloats the DOM model.
For our processor [Exselt] we are currently
investigating the SiXML model as an updateable model(it is currently read-only) and we try to expand that using
lazy loading of the data content of nodes, dependent on
the structure of the XSLT, as described in the previous
sections in this paper.

14. Conclusion
We have seen that XSLT 3.0 provides several ways tolazily load elements, or to at least lazily process them.
While certain environments provide lazy loading out of
the box, especially in certain database systems, when it
comes down to processing input from a memory stream,a local or a remote file, lazy loading was not yet a feature.
With the addition of streaming to XSLT 3.0 came a few
other enhancements to the language that facilitate
handling large datasets, that make it possible without
using extensions to load a node, with all its children no
demand or to skip nodes by using filtering based on theavailable matching templates and the settings in thecurrent mode.
By testing these assumptions using a real-world
scenario, we've shown that lazy loading in streaming
processing has significant benefits. While the processors
still need to optimize this kind of processing, it is very
encouraging that in a simple comparison of different
approaches, we see already gains of 50% and more.
If these approaches can be combined with an XML
parser that can serve content of nodes lazily, the resulting
processing speed and required memory may go evenfurther down.

Page 143 of 162

Lazy processing of XML in XSLT for big data

Bibliography[Burst]Burst Mode Streaming.
http://saxonica.com/documentation9.4-demo/html/sourcedocs/streaming/burst-mode-streaming.html.
[Delpratt] Space efficient in-memory representation of XML documents. PhD. thesis at University of Leicester https://lra.le.ac.uk/handle/2381/4805. O'Neil Davion Delpratt. 2009.
[Exselt]
Exselt, a concurrent streaming XSLT 3.0 processor for .NET. http://exselt.net. Abel Braaksma.
[Jacobsen] Succinct static data structures. Ph.D. thesis, Pitssburgh, PA, USA. G.J. Jacobsen. 1988.
[Joannou] In-Memory Representations of XML Documents with Low Memory Footprint. Poster XML Prague 2013,
part of SiXML. Stelios Joannou, Andrieas Poyias, and Rayeev Raman. 2013.
[Kay]
Streaming the identity, confusing running times. http://saxon-xslt-and-xquery-processor.
13853.n7.nabble.com/Streaming-the-identity-confusing-running-times-td11839.html.
[SAX]
Simple API for XML. http://www.saxproject.org/.
[Saxon]Saxon by Saxonica. http://saxonica.com. Michael Kay.
[SiXML] Succinct indexable XML. http://www.cs.le.ac.uk/SiXML/.
[SSTRM] Streaming in Saxon using saxon:stream.
http://saxonica.com/documentation9.4-demo/html/sourcedocs/streaming/.
[Streaming] Streaming definition of XSLT 3.0.
http://www.w3.org/TR/xslt-30/#streaming-concepts.
[STX]
Streaming Transformations for XML.
http://stx.sourceforge.net/.
[VTD]
VTD-XML: Virtual Token Descriptor. http://vtd-xml.sourceforge.net/.
[XDB]
Lazy Manifestation in Oracle's XDB.
http://docs.oracle.com/cd/B19306_01/appdev.102/b14259/xdb10pls.htm.
[XDM3] XQuery and XPath Data Model 3.0, W3C Candidate Recommendation 08 January 2013.
http://www.w3.org/TR/2013/CR-xpath-datamodel-30-20130108/.
Norman Walsh, Anders Berglund, and John Snelson.
[XP3]
XML Path Language (XPath) 3.0, Latest Version.
http://www.w3.org/TR/xpath-30/.
Jonathan Robie, Don Chamberlin, Michael Dyck, and John Snelson.
[XPCR]XML Path Language (XPath) 3.0, W3C Candidate Recommendation 08 January 2013.
http://www.w3.org/TR/2013/CR-xpath-30-20130108/.
Jonathan Robie, Don Chamberlin, Michael Dyck, and John Snelson.
[XRDR]
XmlReader .NET BCL class.
http://msdn.microsoft.com/en-us/library/system.xml.xmlreader.aspx.
[XSLT3] XSL Transformations (XSLT) Version 3.0, Latest Version.
http://www.w3.org/TR/xslt-30/. Michael Kay.
[XSLWD] XSL Transformations (XSLT) Version 3.0, W3C Working Draft 1 February 2013.
http://www.w3.org/TR/2013/WD-xslt-30-20130201/. Michael Kay.
[Zavoral] Perfomance of XSLT Processors on Large Data Sets. ICADIWT '09. Second International Conference on
theApplications of Digital Information and Web Technologies
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5273945. Filip Zavoral. Jana Dvorakova.
[Zhang]
R-NEMXML: A Reusable Solution for NEM- XML Parser. International Symposium on Information
Processing 2009 in Huangshan, China
http://www.academypublisher.com/proc/isip09/papers/isip09p155.pdf. Yunsong Zhang. Lei Zhao.
Jiwen Yang.

Page 144 of 162

Using Distributed Version Control Systems
Enabling enterprise scale, XML based information development
Dr. Adrian R. Warman
IBM United Kingdom Limited<Adrian.Warman@uk.ibm.com>

1. Disclaimer
Any views or opinions expressed in this paper are thoseof the author, and do not necessarily represent official
positions, strategies or opinions of International Business
Machines (IBM) Corporation.
No guarantees are offered as to the timeliness,accuracy or validity of information presented.

2. Introduction
Enterprise scale technical writing, or “information
development”, typically requires many people, working
on many aspects of content, across many time zones. At
this level, traditional “word processor” tooling simply
cannot cope, particularly where more than one person
must work on the same or closely related content. By
contrast, XML-based information development, where
content is divided into discrete information units, solves
many of the problems. But not all of them.
In particular, where several people must work on thesame files concurrently, or where the timescale for therequired work might conflict with other delivery
obligations, technical writers can find themselves insituations where updates might be lost or broken.
These are classic problems for software developers,
too, and it turns out that some software development
techniques can usefully be applied to the problem ofmanaging XML-based documentation at an enterprisescale. Specifically, Distributed Version Control System(DVCS) tooling can be applied to everyday
documentation tasks. In this paper, we explain how a
DVCS might be used by a product documentation teamto help deliver diverse documentation sets, created using
XML, for large scale parallel delivery.

3. Definitions
It is helpful to clarify exactly what is meant by specificterms and concepts mentioned within this paper. We
prefer straightforward, pragmatic definitions to moreformal, rigorous definitions.

doi:10.14337/XMLLondon13.Warman01• Information Development refers to the combinationof resources and processes used to create content thatis readily usable by the intended audience. So, if atechnical writer uses a word processor to create somenotes about how to install a software package, that is
information development. If a software developer
creates some extended comments within application
source code, and those files are run through an
automatic documentation generation tool (such as
javadoc) to create HTML reference guides, that too is
information development. If a marketing team createssome visually impressive video files that cannot be
used by someone with a visual impairment, that is not
information development.
• An Information Set is a logically-distinct collectionof content that has been created or assembled toaddress a specific concept, task or purpose. A manual
for a product running on a specific platform is an
information set. A chapter on installing a product is
an information set. A collection of documentation
source files containing all the materials necessary to
produce an information set is an information stream.
• DITA is the Darwinian Information Typing
Architecture, and is a form of XML markup, where
well-formed and valid XML files are used to holdhuman-readable documentation source. This enables
enterprise scale work, such as reuse of content, while
applying strict controls to help ensure consistency.
• A Distributed Version Control System, or DVCS, isa peer-to-peer file management tool. It helps trackand manage changes to files, with no single repository
being the definitive storage point. The distributed
nature means that the ability to merge, and manage
merges, of source files is essential. Each DVCS user
has an entire copy of the source repository, making it
possible to carry out substantial work ‘offline’.
Traditionally aimed at software development, theapplication of DVCS to information development
provides some interesting and significant benefits, assuggested by this paper.

Page 145 of 162

Using Distributed Version Control Systems

4. A simple example of workflow
Workflow in enterprise scale information development isremarkably similar to software development. Assume thata new feature is being implemented within a product.
This feature will be the subject of specification anddesign documents. These describe details such as
implementation constraints, performance targets, test
cases, and so on. Similarly, if the feature must be
documented because it is usable by customers, there will
be usage considerations such as the target audience, thelevel of detail required, whether to include examples andto what level of detail, what output format to use, and soon. Other essential considerations include translation
and accessibility compliance, but these are not discussed
further in this paper.

As a simple example, let us assume that an establishedproduct, called ‘ProductEx’, is about to be released on an
additional hardware platform. This requires substantialchanges to the product documentation, such as platformspecific installation and troubleshooting details. At thesame time, a new feature called ‘FastMemEx’ is being
introduced and will be available on the established andnew platforms. The feature modifies the behavior of thesoftware product to make it run faster, but a trade-off isthat the software requires more memory to operate. It is
possible that the performance testing carried out during
development might conclude that FastMemEx is notready to be included with the next release of ProductEx,
and would be deferred to a later date. Waiting until the
FastMemEx“go or no-go” decision is made would notleave enough time for the documentation to be written,
therefore it must be possible to remove the FastMemEx
documentation quickly if necessary.
Even this minimal example makes it clear that there
are often substantial information development tasks in a
project. It represents a typical scenario for information
development. In addition to any other development ormaintenance work, the example identifies two distinct
new information streams, each of which has implications
for the other stream. Ideally, both these new streams will
be delivered, but it is possible that the productcomponent associated with one stream might bedeferred, and therefore the updates for that component
must be held back.
We can represent this scenario diagrammatically, as
shown in Figure 1, “Flow of streams and merging”.

Figure 1. Flow of streams and merging

The problem for technical writers is how to manage each of these challenges.

Page 146 of 162

Using Distributed Version Control Systems

5. Basic XML content creation
using DITA

The dita content in exploring-the-dita-ot.dita isactual ‘product’ documentation. Even without knowing
any DITA, you should be able to recognize content suchas the topic title, a simple paragraph, and a citation
Part of the solution is to use an appropriate reference to another publication.
documentation source format. As mentioned above,
These files can be created and modified using any
DITA is a well-formed XML based content markup text editor. Ideally, you would use a tool that is XMLlanguage. It is easy to find more details about DITA, but aware, so that the proper checks are in place to ensure
for the purposes of this paper, the following two example that the files are well-formed and valid. There are also
files adapted from samples provided with the DITA some more advanced tools that go some way towards
Open Toolkit implementation are helpful.
WYSIWYG presentation, although in practice these canbe more frustrating than helpful because many enterprisequickstart.ditamap
scale documentation projects make use of attributetagging to conditionalize the build according to a specific<?xml version="1.0"
encoding="UTF-8"?>
product, platform, audience, and so on.
<!DOCTYPE map PUBLIC "-//OASIS//DTD DITA Map//EN"
The key point about these examples is that they show
"map.dtd">‘simple’ text files as the basis of extremely large,<map xml:lang="en-us">
enterprise scale, documentation. The use of XML as the<title>Getting started</title>‘hosting’ structure helps ensure consistency and<topicref
href="quickstartguide/exploring-the-dita-ot.dita" reliability in creating output deliverables.
collection-type="sequence"><topicref
href="readme/installing-full-easy.dita"/><topicref
href="quickstartguide/rundemo.dita"/><topicref
href="quickstartguide/runmore.dita"/></topicref></map>
exploring-the-dita-ot.dita<?xml version="1.0"
encoding="UTF-8"?><!DOCTYPE task PUBLIC "-//OASIS//DTD DITA Task//EN""task.dtd"><task id="exploring-the-dita-ot"><title>Getting Started with the DITA Open Toolkit</title><shortdesc>The<ph><cite>Getting Started Guide</cite></ph> isdesigned to provide a guided exploration of the
DITA Open Toolkit. It is geared for an audiencethat has little or no knowledge of build scripts
or DITA-OT parameters. It walks the novice user
through installing the full-easy-install versionof the toolkit and running a prompted build.
</shortdesc></task>The ditamap in quickstart.ditamap provides
navigational and structural information. It determines
which ‘topic’ files appear in the final publication, and inwhat order.

6. DVCS principles
As we saw in the earlier diagram, when writing enterprisescale documentation, you are very likely to encounter
situations where two or more people are working on thesame documentation set. Often, they will work on
different files, but from time-to-time they need to update
the same file at the same time. Anecdotally, but not
surprisingly, many of the changes tend to be within files
that are in close ‘proximity’ to each other in the overall
structure, rather than being distributed ‘randomly’
throughout the content.
This need to have several people working on acommon set of files is not unique to technical writers.
Software developers encounter exactly the samechallenges. They have tackled the problem by extending
the concept of a Version Control System (VCS) into aDistributed Version Control System (DVCS).
In its simplest form, a VCS stores historical recordsof files in a repository. Each time a file is modified, a new
copy of the updated files is stored or “checked in” to therepository. The main objectives are to make it easy to
backup all the work, and to be able to “roll back” toearlier versions of the files.
A VCS traditionally has a single, central server tohost the repository. This makes it easy to backup thecontent at regular intervals. However, it means thatpeople working on the files must first copy them to theirlocal machine (“check out”), make any changes, thenupload the changed files (“check in”) back to the central
server.

Page 147 of 162

Using Distributed Version Control Systems

The VCS model does have limitations. In particular:
1. You must have effectively continuous network
connectivity to check out or check in to the server.
2. It is complicated to deal with different threads ofdevelopment on your local machine.
3. There is a risk that your changes to a file will beoverwritten by changes made to the same file byanother developer, who checks in their changes after
you have checked in yours.
One solution to these problems is to use a DVCS. A key
characteristic of a DVCS is that everyone has a complete
copy of the entire repository. Surprisingly, this can be
more cost-effective in terms of storage space than a
traditional VCS, for reasons explained in Appendix A.
Any changes made to files locally must be available toeveryone else who has a copy of the repository. Thismeans that a DVCS is intentionally designed to share file
updates between the repositories, and to accommodatechanges made to the same file by different developers.
The sharing works in a way that:• Is as quick and efficient as possible.
• Makes the merging of separate changes to the same
file automatic.
• Ensures that any problem merging the changes results
in a “fail gracefully” report.
The last point is especially important. If something doesgo wrong during the merge, none of the changes are lost,and you are able to work on the file directly to resolve
the problems.
In Figure 2, “Merging of documentation streams”,
you can see different threads of development work froma real documentation project using a DVCS called git.
At frequent intervals, the changes in a given thread canbe seen merging back into other threads.
Figure 2. Merging of documentation streams

Page 148 of 162

7. Using a DVCS for multi-branchcontent
Note
In this paper, we do not have the space to describe
DVCS operation in detail. For more information
about typical DVCS tools and tasks, see either git or
Mercurial.
In a DVCS such as git, each of the work streams iscreated as a 'branch', where the current collection of files
is duplicated in a separate development strand. Thisbranch can be edited independently of the original files.
At any time, you can save the current state of the files ina branch, and then switch to any other branch. There isno specific limit to the number of branches you might
create; in practice, creating too many branches would
make it difficult to keep track of them yourself.
Branching means that each task can be split off into
its own stream of activity. You can safely work on all thefiles within the branch, knowing that any changes you
make can be re-integrated into the main branch later.
In git, the main branch is called master. To startworking on the task of documenting a new feature, you
might create a new branch as follows:
git branch MyNewFeature

However, nothing obvious happens until you 'switch to',or “check out” the new branch:
git checkout MyNewFeature

You have now taken copies of all the files in your projectfor use in the new branch. You can proceed to create andupdate the various files associated with the new feature.

Using Distributed Version Control Systems

It is important to realize that documentation files might
be correct or incorrect according to three precise
Notemeasures:
A DVCS does not normally duplicate files when a new 1. Whether the file is well-formed.
branch is created. A duplicate file is created only when 2. Whether the file is valid.
the original file is modified. This makes branch 3. Whether the file is semantically meaningful.
creation and switching quick and easy.
A DVCS cannot assist you with the third measure, notleast because a (comparatively) simple tool like a DVCS
Every so often, you save your work by checking in the cannot interpret the vagaries of human communication.
changes. In git, this is called making a “commit”. A However, the way in which a DVCS can support the
typical commit action might look something like this:first and second measures suggests a possible future
development that might one enable support for semantic
assessment.
... various file editing tasks
git add <list of new or modified files>
Most DVCS systems can be extended
git commit -m "Corrected the syntax explanation for the new feature." and modified in many ways;... continue editing files.
customization was an important principle in developing
git. For example, when you check in your updates using
Similarly, you will want to bring in the other changes
that colleagues have made to documentation files in their the git commit command, some extra tasks can be runusing an extension called a “hook”. A very usefulbranches. Assuming they have checked in their changes
application of this is to run some tests on the files as they
to the master branch, you can bring those changes intoare prepared for check in. In the following code segment,
your files by using the command:the command-line utility xmllint is applied to each
git merge master
DITA file just before it is checked in. The script ensures
All being well, the external changes are merged safely that only DITA files are tested in this way.
into your files. Eventually, however, a problem is likely for FILE in `exec git diff-index --cached \to occur, where you and your colleagues have been trying--name-status $against | egrep '^(A|M)' | awk \to change the same section of the same file. When this'{print $2;}'` ; dohappens, the DVCS might not be able to resolve the if ( echo $FILE | egrep -q '.+\.(dita|ditamap)$' )discrepancy. Instead, it reports that the merge has failed, # if the filename ends in .dita or .ditamap
and tells you details of the affected file or files. For the thenxmllint --valid --noout $FILE > /dev/null 2>&1
merge to complete, you must edit the broken files and fix
RESULT=$?
the problem. For any broken file, git makes all the# echo "Checking $FILE, return code: $RESULT"
necessary information available to you, including:if [ $RESULT -ne 0 ]then• The file contents as they were before you or your
EXIT_STATUS=1
colleague made any changes.
echo "Invalid DITA markup: $FILE"• The changes you made to the file.
fi• The changes your colleague made to the file.
# echo "EXIT_STATUS now $EXIT_STATUS"
Using these three details, you should have everything you # else
need to solve the merging problem. At worst, you know # echo "Ignoring $FILE"
who else has been editing the file, and so could talk to fi
them about the specific changes and how you might The xmllint command checks that a given file is bothcome to an agreement on what the final content should well-formed and valid according to the XML catalog
be.
defined on the system. In this case, the DITA DTDs
After going through several cycles of editing files andhave been included in the XML catalog, which means
merging in changes from colleagues, you are ready tothat every time you check in your DITA files, they are
merge your changes back into the master branch. To do
automatically tested to be well-formed and valid. A
this, you switch back to the master branch, then merge
potential enhancement would be to modify the git hookin the changes from your working branch:so that semantic analysis might also be performed.
git checkout mastergit merge MyNewFeature

Page 149 of 162

Using Distributed Version Control Systems

8. Using a DVCS to manage
multiple deliveries
A common reason for using DITA and similar markup
tools for enterprise scale documentation tasks is that they
support conditional or filtered building of content. Thisis an exceptionally useful capability, and is based on theuse of XSL-style pattern matching to include or exclude
content, according to precise conditions.
A good example is where a content file is created that
describes a task in general terms, but also includes
platform- or version-specific details. One option for the
final document delivery would be to build a single
documentation set that includes all of the platform- orversion-specific information. Within this large collection,
each instance of (say) platform dependent content might
be clearly identified using a small icon or logo. However,given that markup must be included anyway to identify
content specifics, a second option is to have entirely
separate builds: one for each of the specific variants. The
result is a much larger number of documents, but each
one is specifically tailored for a precise audience.
What we are describing here is another way ofviewing the documentation streams. Previously, we haveidentified a main documentation stream (master), withseparate branches established for clearly defined updatetasks. These branches are then merged or reintegrated
back into master as required.
But the branching model of a DVCS allows us toadopt a different model, where content that is common
to all documentation is present in one branch, andcontent that is unique to a given platform or productversion can be isolated off in a separate branch. This
concept can be seen diagrammatically in Figure 3, “Useof DVCS for multiple deliveries”.

Page 150 of 162

It is important to note that this diagram is not illustratinga deployment model; indeed some DVCS specialists
actively discourage the view that a DVCS should be used
for deployment purposes, where the existence of branch
represents the live content on a production system.
Rather, the diagram shows that content may beassembled to produce some release-ready material, inmuch the same way that ingredients are assembled prior
to working through a recipe to create a meal.
Using a DVCS for multiple deliveries offers several
advantages in comparison to the traditional single-stream
content model, over and above those already described.
For example, re-use of content is much each easier toachieve, promoting consistency and simplicity. The
assembly-on-demand approach to multiple deliveries has
some similarities to a Just-In-Time delivery model; there
is no need to 'retire' a documentation stream if it is nolonger required - instead you simply omit it from thefinal build process.
At the same time, it must be admitted that using a
DVCS in this way is pushing the boundaries of what was
originally intended. In a small scale, real projectexperiment, where content for multiple deliveries wasassembled prior to deployment on a hosting system, themechanism worked well. However, the difference incomparison to the earlier description of using DVCS isthat there is no requirement to store the resulting merged
content. Therefore, if there had been problems during
the merge, it is possible that the effort required to resolve
the problems might have exceeded the benefit ofseparating out the deliverable streams. It will beinteresting to see whether larger scale tests of DVCSbased multi-delivery remain viable.
Figure 3. Use of DVCS for multiple deliveries

Using Distributed Version Control Systems

9. Summary

By contrast, a DVCS user has a complete set of all files in
the repository. It is a quick and easy task to switch to any
other snapshot or indeed branch within the repository,
for the simple reason that all the files are available locally.
In most cases, many of the files are unchanged fromsnapshot to snapshot. Further, software or
documentation source files are typically in a text format.
Both these aspects mean that DVCS software can apply
compression techniques extremely effectively. Here aresome numbers from an actual project:

In this paper, we have reviewed the basic principles ofenterprise-level documentation tasks. We have outline
some of the challenges faced when producing
documentation that comprises different strands of work,
such differing content or timescales. We have shown
how some of these challenges reflect similar problems
encountered when managing large scale softwaredevelopment concept. We have introduced a softwareoriented solution: Distributed Version Control Systems. Table A.1. Storage requirements compared: VCS andUsing these building blocks, we have explained how DVCS
DVCS technology might be used to help enable
AspectVCS
DVCS
enterprise scale, XML-based information development.
Server copy of entire repository
800+ MB 331 MB

A. Storage requirements for a local
repository

Local copy of repository andsnapshot

1.6 GB

1 GB

A common example of a VCS is subversion. This tool
provides a central server that hosts a repository for all the B. Useful resources
working files of a project. At any time, a developer mightuse a client program (svn) to check out a 'snapshot' of Choosing a Distributed Version Control System
the project files from a precise point in time. To switch
Choosing an XML Schema: DocBook or DITA?
to an older or more recent snapshot, the developer uses
Distributed Revision Control
the svn command again to request the correct files from
DITA Open Toolkit
the server. This means that a subversion user only everhas a single set of files, and must communicate with theserver to switch to a new set. If there is no network
access, no switching is possible.

Page 151 of 162

A complete schema definition language for the
Text Encoding Initiative
Lou Burnard
Lou Burnard Consulting<lou.burnard@retired.ox.ac.uk>

Sebastian Rahtz
IT Services, University of Oxford<sebastian.rahtz@it.ox.ac.uk>

Abstract

1. Introduction

For many years the Text Encoding Initiative (TEI) has useda specialised high-level XML vocabulary known as ODD in
the “literate programming” paradigm to define its
influential Guidelines, from which schemas or DTDs inother schema languages are derived. This paper describes a
minor but significant modification to the TEI ODDlanguage and explores some of its implications. In thecurrent ODD language, the detailed content model of anelement is expressed in RELAX NG, embedded inside TEI
markup. We define a set of additional elements which
permit the ODD language to cut its ties with existing
schema languages, making it an integrated and independent
whole rather than an uneasy hybrid restricted in its features
to the intersection of the three current schema languages.
This may pave the way for future developments in the
management of structured text beyond the XML paradigm.
We describe the additional features, and discuss the
problems of both implementing them, and of migrating
existing TEI definitions.

The Text Encoding Initiative (TEI) began in the late1980s as a conscious attempt to model existing and future
markup systems. The original TEI editors, Lou Burnard
and Michael Sperberg-McQueen, had spent much of
their careers trying to find satisfactory ways of expressing
the rugosities of typical humanities datasets using thedatabase modelling techniques common in the IT
industry at that time. They naturally turned to the same
techniques to help draw up a formal model of textual
features, and their representations in different markup
schemes. The following figure, taken from an early paperon the topic, typifies the approach: distinguishingsharply between the features perceived in a text, theirrepresentation by the application of tags, and the namesthat might be used for those tags.

Page 152 of 162

doi:10.14337/XMLLondon13.Rahtz01

A complete schema definition language for the Text Encoding Initiative

Figure 1. Abstract model from TEI EDW05, 1989

This exercise in modelling started to become more thantheoretical quite early on in the life of the TEI, notably
during 1991, when the TEI's initial workgroups started
to send in their proposals for textual features which theyfelt really had to be distinguished in any sensible
encoding project. It rapidly became apparent thatsomething better than a Hypercard stack or relational
database would be needed to keep track of the tags they
were busy inventing, and the meanings associated withthem. In particular, something able to combine text andformal specifications in a single SGML document wasneeded. Fortunately Donald Knuth had been here before
us, with his concept of “literate programming”.1.

1

2

In the autumn of 1991, Michael Sperberg-McQueen and
Lou Burnard started seriously thinking about ways of
implementing the idea of a single DTD which could
support both the documentation of an encoding scheme
and its expression as a formal language. Our thoughts
were necessarily constrained to some extent by the
SGML technology at our disposal, but we made aconsidered effort to abstract away from that in the true
spirit of literate programming as Knuth eloquently
defines it elsewhere: “Instead of imagining that our maintask is to instruct a computer what to do, let us
concentrate rather on explaining to human beings what
we want a computer to do.” 2 The documentation foreach element in the proposed system thus needed toprovide informal English language expressions about its
intended function, its name and why it was so called, theother elements it was associated with in the SGML
structure, usage examples and cross-references to places
where it was discussed along with formal SGML
declarations for it and its attribute list. Relevant portionsof these tag documents could then be extracted into therunning text, and the whole could be reprocessed toprovide reference documentation as well as to generate
document type declarations for the use of an SGML
parser. The following figure shows a typical example ofsuch a tag document“Literate programming is a methodology that combines a programming language with a documentation language,
thereby making programs more robust, more portable, more easily maintained, and arguably more fun to write thanprograms that are written only in a high-level language” (http://www-cs-faculty.stanford.edu/~uno/lp.html)
Donald Knuth, Literate Programming (1984)

Page 153 of 162

A complete schema definition language for the Text Encoding Initiative

Figure 2. Tagdoc for <resp> element in P2 (colorized version)

Note here how the SGML declarations are embedded as floating CDATA marked sections, effectively isolating themfrom the rest of the document, and thus making it impossible to process them in any way other than by simple
inclusion. Such refinements as, for example, checking that every element referenced in a content model has its own
specification are hard or impossible. There is also ample scope for error when the structural relationships amongst
elements are redundantly expressed both in DTD syntax, and in human readable form using the <parents> and<children> elements. Nevertheless, this system 1

1

The first full specification for an ODD system is to be found in TEI working paper ED W29, available from the TEI archive at http://
www.tei-c.org/Vault/ED/edw29.tar. It defines a set of extensions to the existing “tiny.dtd” (an early version of a simple TEI-compliant
authoring schema, not unlike TEI Lite), which adds new elements for documenting SGML fragments, elements and entities. It also
specifies the processing model which the markup was intended to support. An ODD processor was required to• extract SGML DTD fragments• generate reference documentation (REF) form
• generate running prose (P2X)
A processor to carry out the reverse operation (that is, generate template ODD specifications from existing DTD fragments) is also
described. Although intended for the use of TEI Workgroups, in practice ODD processors built to this model were used only by the
TEI editors.

Page 154 of 162

A complete schema definition language for the Text Encoding Initiative

At the fifth major revision of the guidelines (P5, released in 2007 after 6 years of development), the TEI switched tousing RELAX NG as the primary means of declared its content models, both within the element specifications which
had replaced the old tag documents as input, and as output from the schema generation process. As a separate
processing step, XML DTDs are also generated from this same source, while W3C schema is generated from the
RELAX NG outputs using James Clark's trang processor. Another major change at TEI P5 was the introduction andextensive use of model classes as means of implementing greater flexibility than had been achievable by using SGML
parameter entities. Both of these changes are reflected in the part of the TEI P5 specification for the <respStmt>
element shown in the following figure:
Figure 3. Parts of <respStmt> element in P5 (XML)Page 155 of 162

A complete schema definition language for the Text Encoding Initiative

Where TEI P2 had used embedded DTD language toexpress content models, TEI P4 had expressed themusing string fragments still recognisably derived from
SGML DTD language. In TEI P5, we moved toembedding RELAX NG in its own namespace, thus
placing that schema language in a privileged position,and inviting the question expressed internally as the
Durand Conundrum.1 No contrainst is placed on editors
as to the features of RELAX NG they can use, so it iseasy to make something which cannot be converted toW3C Schema by trang (eg interleave), or which is notcovered by the conversion to DTD, or which duplicateswork being done elsewhere. The last of these isparticularly worrisome, as attributes are managed
separately in the ODD language language, but the
RELAX NG content model fragment may add attribute(or child element) declarations.

2. What's not ODD?
In the current source of TEI P5, there is extensive use of
several different XML vocabularies:• Examples in TEI P5 are presented as if they belonged
to some other "TEI Example Namespace"; thishowever is merely an ingenious processing trick tofacilitate their validation;• Element content models are expressed using a subsetof RELAX NG, as discussed in the previous section;• Datatypes are expressed in a variety of ways, mapping
either to built-in W3C datatypes (as defined in the
W3C Schema Language) or to RELAX NG
constructs;• Some additional semantic constraints (for example,
co-dependence of attributes and element content) are
expressed using ISO Schematron rules.
• Specialist vocabularies such as XInclude, MathML
and SVG are used where appropriate.
Everything else in a TEI-conformant ODD specification
uses only constructs from the TEI namespace. In thispaper, we describe a further extension of the ODD
language to replace at least some of the cases listed above.

1

2.1. Element content models
ODD is intended to support the intersection of what is
possible using three different schema languages. In
practice, this reduces our modelling requirements quite
significantly. Support for DTD schema language inparticular imposes many limitations on what would
otherwise be possible, while the many additional facilities
provided by W3C Schema and RELAX NG for content
validation are hardly used at all (though some equivalentfacilities are now provided by the <constraintSpec>element). A few years ago, the demise of DTDs was
confidently expected; in 2013 however the patient
remains in rude health, and it seems likely that support
for DTDs will continue to be an ongoing requirement.
We therefore assume that whatever mechanism we use to
specify content models will need to have the following
characteristics:• the model permits alternation, repetition, andsequencing of individual elements, element classes, or
sub-models (groups of elements)• only one kind of mixed content model — the classic(#PCDATA | foo | bar)* — is permitted• the SGML ampersand connector — (a & b) as ashortcut for ((a,b) | (b,a)) is not permitted• a parser or validator is not required to do look aheadand consequently the model must be deterministic;that is, when applying the model to a document
instance, there must be only one possible matchinglabel in the model for each point in the document
We think these requirements can easily be met by thefollowing small incremental changes to the ODD
language:

The Durand Conundrum is a jokey name for a serious question first raised by David Durand when the current TEI ODD XML
format was being finalised at a meeting of the TEI Technical Council held in Gent in May 2004. David pointed out that the TEI's
mixed model was a compromise solution: like other XML vocabularies, the TEI was perfectly hospitable to other namespaces, so we
could equally well embed our TEI additions within a natively RELAX NG document. A similar suggestion is made in Eric Van der
Vlist's RELAX NG (O'Reilly, 2011), which proposes a hybrid language called "Examplotron" in which the documentation is expressedusing the XHTML vocabulary, the document grammar is expressed using RELAX NG, and additional constraints are expressed using
Schematron. See further http://examplotron.org

Page 156 of 162

A complete schema definition language for the Text Encoding Initiative

Specification
At present, references to content modelcomponents use the generic <rng:ref> element. As
a consequence, naming conventions have been
invented to distinguish, for example references toan element or attribute class (name starts with"model." or "att.") from references to a predefinedmacro (name starts with "macro.") or fromreferences to an element (name starts withsomething other than "model." or "macro.").
Although these name changes are purely a matterof convenience, we suggest that it would be better
to use the existing TEI ODD elements<elementRef>,<classRef>and <macroRef>elements.
For example,<rng:ref name="model.pLike"/>becomes<classRef key="model.pLike"/>

Repeatability
In RELAX NG, this is indicated by special
purpose grouping elements <rng:oneOrMore> and<rng:zeroOrMore>. We propose to replace these by
the use of attributes @minOccurs and @maxOccurs,
which are currently defined locally on the<datatype> element. Making these also available
on <elementRef>, <classRef> and <macroRef>elements, gives more delicate and consistentcontrol over what is possible within thecomponents of a content model.
Sequence and alternation
Sequencing and alternation are currently indicatedby elements defined in the RELAX NG namespace(<rng:choose>, <rng:group>, etc.) We replace theseby similar but more constrained TEI equivalents<sequence> which operates like <rng:group> toindicate that its children form a sequence within acontent model, and <alternate> which operates
like <rng:choose> to supply a number ofalternatives.

In this example ((a, (b|c)*, d+), e?) we have asequence containing a single element, followed by arepeated alternation, a repeated element, and an optional
element. This would be expressed as follows:<sequence><sequence><elementRef key="a"/><alternate minOccurs="0" maxOccurs="unbounded"><elementRef key="b"/><elementRef key="c"/></alternate><elementRef key="d" maxOccurs="unbounded"/></sequence><elementRef key="e" minOccurs="0"/></sequence>

Repetition can be applied at any level.
In ((a, (b*|c*))+, for example, we have a repeatedsequence. This would be expressed as follows:<sequence maxOccurs="unbounded"><elementRef key="a"/><alternate><elementRef key="b" minOccurs="0"
maxOccurs="unbounded"/><elementRef key="c" minOccurs="0"
maxOccurs="unbounded"/></alternate></sequence>

A mixed content model such as (#PCDATA | a |model.b)* might be expressed as follows:<alternate minOccurs="0" maxOccurs="unbounded"
mixed="true"><elementRef key="a"/><classRef key="model.b"/></alternate>

References to model classes within content models pose aparticular problem of underspecification in the current
ODD system. In the simple case, a reference to a model
class may be understood as meaning any one member ofthe class, as assumed above. Hence, supposing that themembers of class model.ab are <a> and <b>, a content
model<classRef key="model.ab" maxOccurs="unbounded"/>For handling character data, we follow the W3C Schema is exactly equivalent toapproach and define an attribute @mixed for each<alternate maxOccurs="unbounded">container element.The two simple cases of empty<elementRef key="a"/>content, and of pure text content, will be covered by an<elementRef key="b"/>empty <content> element.
</alternate>
We now provide some simple examples, showing how
some imaginary content models expressed using RELAX
NG compact syntax might be re-expressed with theseelements.

Page 157 of 162

A complete schema definition language for the Text Encoding Initiative

However, sometimes we may wish to expand model
references in a different way. We may wish to say that areference to the class model.ab is not a reference to any of
its members, but to a sequence of all of its members, orto a sequence in which any of its members may appear,and so forth. This requirement is handled in the current
ODD system by over-generating all the possibilities,
again using a set of naming convention to distinguishamongst them. We propose instead to control thisbehaviour by means of a new @expand attribute on<classRef> (modelled on an existing @generate on<classSpec>), but with the advantage of being usable atthe instance level.
For example,<classRef key="model.ab" expand="sequence"/>is interpreted as a,b while<classRef key="model.ab"
expand="sequenceOptional"/>is interpreted as a?,b?,<classRef key="model.ab"
expand="sequenceRepeatable"/>is interpreted as a+,b+ and<classRef key="model.ab"
expand="sequenceOptionalRepeatable"/>is interpreted as a*,b* Note that the ability to specify
repetition at the individual class level gives a further levelof control not currently possible. For example, a model
containing no more than two consecutive sequences ofall members of the class model.ab could be expressed
quite straightforwardly:<classRef key="model.ab" maxOccurs="2"
expand="sequence"/>2.2. Datatyping and other forms of validation
Validation of an element's content model is but one ofmany different layers of validation that a TEI user may
wish to express in their ODD specification. The current
system also provides mechanisms to constrain thepossible values of attributes by means of datatyping andalso, increasingly, by explicit constraints expressed using
languages such as ISO Schematron. It seems reasonable
to ask how many of these additional layers may beincorporated into our proposed new vocabulary.

Page 158 of 162

The vast majority of TEI attributes currently define their
possible values by reference to a datatype macro which isdefined within the ODD system, where it is mapped
either to a native RELAX NG datatype or to an
expression in RELAX NG syntax. This indirection allows
the schema builder to add a small amount of extra
semantics to an underlying "bare" datatype. For example
data.duration.iso,data.outputMeasurement,
data.pattern, data.point, data.version, and data.word
all map to the same datatype (token as far as a RELAX
NG schema is concerned; CDATA for an XML DTD). As
their names suggest, however, each of these TEI
datatypes has a subtly different intended application,
which an ODD processor may use in deciding how topresent the corresponding information, even though themapping to a formal schema language is identical in eachcase.
Given the existence of this TEI abstraction layer, it
seems unnecessary to propose further change to the way
attribute values are constrained in the ODD system. At
the time of writing, there are still a few attributes whose
values are expressed directly in RELAX NG syntax, butthat is a corrigible error in the Guidelines source code.
The most commonly used datatype macro isdata.enumerated, which maps to another frequently used
datatype data.name, and thence to the underlying
RELAX NG datatype for an XML Name. The difference
between an enumeration and a name is, of course, that a(possibly closed) list of possible values can be provided
for the former but not for the latter. In the ODD system,
for every datatype declared as data.enumerated, a sibling<valList> element should be provided to enumerate anddocument all or some of the possible values for this
attribute. This ability to constrain and document
attribute values is of particular interest because it permits
TEI schema-specifiers to define project-specificrestrictions and semantics considerably beyond thoseavailable to all schema languages.

A complete schema definition language for the Text Encoding Initiative

A further layer of constraint specification is provided by
the <constraintSpec> element which may be used toexpress any kind of semantic constraint, using any
suitable language. In the current TEI specifications, theISO-defined Schematron language is deployed to replaceas many as possible of the informally expressed rules forgood practice which have always lurked in the Guidelines
prose. This facility allows us to specify, for example, theco-occurrence constraint mentioned in the previous
paragraph (that the specification for an attribute with adeclared datatype of data.enumerated should also containa <valList>). It also allows an ODD to make more
explicit rules such as “a relatedItem element must have
either a @target attribute or a child element” or “theelement indicated by the @spanTo attribute must follow
the element carrying it in document sequence”, which
are hard to express in most schema languages.
For our present purposes, it is important to note that the
TEI <constraintSpec> element was designed to support
any available constraints language. Although the current
generation of ODD processors assume the use of ISO
Schematron, there is no reason why future versions
should not switch to using different such languages as
they become available without affecting the rest of the
ODD processing workflow or the ODD language itself.
As such, we see no need to modify our proposals to takethis level of validation into account.

3. Discussion
The ideas presented here were first sketched out in thesummer of 2012, and greeted positively at the ODDWorkshop held following the DH 2012 conference in
Hamburg. An earlier version of this paper was presented
at the TEI Conference in Texas in November 2012. In
this section we briefly summarize some of the comments
received.

At first blush, our proposals seem to flout the TEI
philosophy of not re-inventing the wheel. The TEI does
not and should not take on itself the task of inventing a
new XML vocabulary for such matters as mathematics ormusical notation where perfectly acceptable and wellestablished proposals are already in place. However the
TEI has arguably already gone down the road of definingsome aspects its own schema language, (for example, by
providing constructs for representing element andattribute classes, and for associating attribute lists andvalue lists with element declarations) and this proposal
simply continues along the same path. It should also be
noted that there are three competing standards for
schema language in the marketplace (DTD, RELAX
NG, W3C Schema) each with its own advantages. By
making ODD independent of all three, we make it easierto profit from the particular benefits of each, as well as
providing the ability to document intentions notnecessarily expressible using any of them.
Resolving the Durand conundrum in this way, rather
than taking the alternative approach of embedding TEI
documentation elements in the RELAX NG namespace,is clearly a compatible expansion of the current scheme
rather than an incompatible change of direction which
would not break existing systems or documents.
As a concrete example, consider the occasionallyexpressed desire to constrain an element's content to be asequence of single specified elements appearing in any
order, that is, to define a content model such as (a,b,c,d)but with the added proviso that the child elements may
appear in any order. In SGML, the ampersand operator
allowed something like this; in RELAX NG the<interleave> element may be used to provide it, butthere is no equivalent feature in W3C Schema or DTD
languages, and we have not therefore proposed it in ourlist of requirements above.

Page 159 of 162

A complete schema definition language for the Text Encoding Initiative

Suppose however that the Technical Council of the TEI
decided this facility was of such importance to the TEI
community that it should be representable in TEI ODD.
It would be easy enough to add a new grouping element
such as <interleave> (or add an attribute@preserveOrder taking values TRUE or FALSE to ourexisting proposed <sequence> element) to represent it.
Generating a RELAX NG schema from such an ODD
would be simple; for the other two schema languages one
could envisage a range of possible outcomes:• an ODD processor might simply reject the construct
as infeasible;• an ODD processor might over-generate; that is, it willproduce code which validates everything that is valid
according to the ODD, but also other constructs thatare not so valid;• an ODD processor might over-generate in that way,
but in addition produce schematron code to remove“false positives”.
For example, consider the following hypothetical ODD<interleave><elementRef key="a"/><elementRef key="b" maxOccurs="2"/><elementRef key="c"/></interleave>

In XML DTD or W3C schema languages (which lackthe <rng:interleave> feature), an ODD processor canrepresent these constraints by generating a content modelsuch as(a|b|c)+and at the same time generating additional Schematron
constraints to require the presence of no more than one<a> or <c> and up to two <b>s. An extra twist, in thiscase, is that if there are more than two <b> elements, theymust follow each other.
As a second example, consider the need forcontextual variation in a content model. For example, a<name> or <persName> appearing inside a “data-centric”
situation, such as a <listPerson> element, is unlikely tocontain elements such as <del> or <corr> which are
however entirely appropriate (and very useful) whenidentifying names within a textual transcription. In alinguistic corpus, it is very likely that the child elements
permitted for <p> elements within the corpus texts will
be quite different from those within the corpus header— the latter are rather unlikely to include any part ofspeech tagging for example.

1

At present only ISO schematron rules allow us to definesuch contextual rules, although something analogous tothem is provided by the XSD notion of base types. It is
not hard however to imagine a further extension to the
ODD language, permitting (say) an XPath-valued@context attribute on any <elementRef>, <macroRef>, or<classRef> restricting its applicability. Thus, the content
model for <p> might say something like<elementRef
key="s"context="ancestor::text"
maxOccurs="unbounded"
minOccurs="1"/><macroRef key="macro.limitedContent"
context="ancestor::teiHeader"/>

to indicate that a <p> within a <text> element must
contain one or more 
 elements only, whereas one
within a TEI Header must use the existing macrodefinition limitedContent.
However, before embarking on such speculative
exercises, it is clear that further experimentation isneeded, to see how easily existing content models may be
re-expressed, and what the consequences of such a
conversion would be for existing processing tools.

4. Implementation
In order to prove the language additions proposed above
can adequately replace the current embedded RELAX
NG, we have completed four pieces of work:• Formal definition of the new elements in the TEIODD language; this presents no problems.
• Implementation, in the ODD processor, of theconversion from the new language elements to thetarget schema languages (DTD, RELAX NG, W3C
Schema), each according to its abilities• Conversion of the existing content models in RELAX
NG to the new format• Testing of the resulting generated schemas to ensurethat they are at least as permissive as the old ones.
The last of these tasks is covered by the already extensive
testing undertaken after each change to the TEI source; asuite of test schemas and input files is checked,1 and it isexpected that these will already catch most errors.
The second task is not problematic. The TEI ODDprocessing engine is written in XSLT and already works
through a complex process to arrive at a generated DTD
or RELAX NG schema (XSD is generated by trang); thework of adding in the extra processing is simply an
extension of existing code.

The TEI Guidelines themselves are also, of course, a very large TEI document which is extensively checked against itself.

Page 160 of 162

A complete schema definition language for the Text Encoding Initiative

This leaves the problem of converting the existing 573
models in the TEI Guidelines. These fall into fourgroups:• Reference to a macro, predefined patterns of which
the TEI provides 8 (which is a small enough number
to be translated by hand if necessary)• Simple reference to a members of a class of elements• A content model of plain text, <empty>, or a data
pattern• Complex hand-crafted model with branches ofchoices and nesting
An analysis shows that the final group covers 269 of thecontent models:In practice, the majority of these are amenable torelatively simple automated conversion.

5. Next steps
The extended schema language for the TEI as describedhere is implemented in a preliminary release (May 2013),
and will be completed in time for the Members Meetingof the TEI Consortium in October 2013. It is expected
to be made an optional part of the TEI language by theend of 2013, even if the TEI Guidelines themselves do
not convert to using it internally.

Table 1. Types of content models in TEI
Type

Numberclass members

28

empty, data and text

62

macro

194

simple name

20

other

269

Page 161 of 162

Charles Foster
XML London 2013
Conference Proceedings
Published by
XML London
103 High Street
Evesham
WR11 4DN
UK
This document was created by tranforming original DocBook XML sources
into an XHTML document which was subsequently rendered into a PDF byAntenna House Formatter.
1st edition
London 2013
ISBN 978-0-9926471-0-0