dedoc/paragraph_dataset huggingface dataset:
&md; "&mdash;
Figure 6: Simple Dragoman style sheet
If the encountered tag is a start tag, the first of the two actions
is activated, and if it is an end tag, the second is activated. Either
or both startText and endText may be empty strings. If the one-
character action is a quotation mark, the tag is replaced by the
startText or endText, respectively, subject to print suppression; using
a minus sign in place of the quotation mark dictates that printing
of text be suppressed, a plus sign dictates that printing of text be
restarted, an asterisk dictates that the replacement text be printed
regardless of print suppression, and an ampersand dictates that the
tag content be interpreted as if it were an entity reference. Except
for the last of these actions, there is no reliance on tags in the text
being properly matched. Tags not matching any indicator in the
table are printed as text (subject to print suppression) by replacing
the left angle bracket by &lt; so that it is interpreted as character
data rather than as a tag.
Attributes are handled similarly, except that attributes are identi-
fied by the indicator tagID.attrname; the actions (text replacement,
print suppression, etc.) occur before and after the attribute value,
respectively; and attributes not matching any indicator have their
values suppressed by default. Entity references are also handled
similarly to tags, except that there is only one action specified per
indicator, and it must begin with a quotation mark to indicate text
replacement.
A style sheet has the following structure:
• It begins with a plus or minus, signalling whether print is
initially on or suppressed.
• After a line with two percent signs, it lists tags or attributes
in arbitrary order, with a pair of actions specified for each.
• After another line with two percent signs, it lists entities in
arbitrary order, with a single action specified for each
For the simple style sheet shown in Figure 6, printing is initially
on, the content between match tags are displayed in red, d:tract
start and end tags are simply ignored (the tags are replaced by
empty strings), a line break is inserted after printing the value
of the attribute val on a d:tract tag, the printing of any content
occurring between greek start and end tags (inclusive) is suppressed,
and the entity reference &md; is passed on to the HTML renderer
as an entity reference, namely &mdash;. All other tags and entity
references are passed to the renderer as character data and any
other attributes on the d:tract and greek tags are ignored.
For the Alberti Magni e-corpus, style sheets were created to
display matched text in an easily readable form, either before or
after the light editing of the works. In addition, a style sheet was
created to display text with all tags uninterpreted so that the exact
structure is visible. Printing the same paragraph with each of these
(a) unedited
(b) edited
(c) show tags
Figure 7: Display of text using three style sheets
style sheets is shown in Figure 7. Two additional style sheets were
created for displaying large portions of the documents. The first
of these suppresses the printing of most text fields but preserves
section headings and titles, with suitable inclusion of markup for
unordered lists to display a text’s structure, and displays a red as-
terisk for every match (so that each asterisk appears underneath
the heading for the section within which the match occurs), as
illustrated by the output fragment in Figure 8; in response to click-
ing on an asterisk, the system displays the text page on which the
corresponding match occurs. The second additional style sheet pre-
serves the single- and two-column formatting of the source texts,
as shown in Figure 9.
Like Lector, on which it is based,Dragoman uses a “best-effort”
approach to interpreting markup. Unlike CSS [15], it does not rely
on the tagged text conforming to well-formed XML in order to
display text. Instead, any errors in markup are ignored, relying
instead on the user’s interpretation of the rendered display to dis-
cover errors in the input text or in the style sheet. This forgiving
nature has repeatedly proven especially useful during periods in
a project’s life cycle in which the code for structure recognition
(Section 3) is under development or undergoing change.
Figure 8: Search for “licit”: Text suppression and hyperlink
generation with a style sheet
Figure 9: Using a style sheet that preserves columns
6 RETROSPECTIVE AND FURTHERWORK
With the ubiquity of search engines and web browsers, it may seem
that all retrieval applications can be easily addressed with off-the-
shelf tools. However, scholars in the humanities have demanding
needs for text search and display. Some of these have been ad-
dressed by the software developed for the Alberti Magni e-corpus
8 project. In particular, after preparing the scholar’s texts in a suit-
able XML-tagged form, a system built on top of sgrep for search
and Dragoman for display can address many of those 9 needs.
Alternative XML-aware search engines (such as BaseX [20], eX-
ist [21], Wumpus [3], or XQEngine [16]) could equally well have
been used in this project, simplifying some solutions but requir-
ing more effort to address other concerns. For example, using an
XQuery engine with full-text search capability [12], would allow
phrase searches across tags, thus eliminating the shortcoming in
using sgrep when confronted by phrase instances that cross page
and column breaks and across shifts to and from italic and bold text.
However, because pages and columns form a hierarchical structure
that is not nested with respect to logical text structures from books
and chapters down to paragraphs and sentences, it is necessary to
use milestone tags; thus searching within pages or columns or dis-
playing the contents of a single page or column is less convenient
with XQuery than it is with 10 sgrep. No matter which engine is
chosen, code must be implemented to address the normalization of
Latin orthography, and for word-based (as opposed to region-based)
systems, Latin-specific stemming rules must be implemented.
In retrospect, would it have been preferable to use XQuery in
place of sgrep to serve as the query language for building a search
engine for the Alberti Magni e-corpus? Because XQuery is defined
by a suite of W3C recommendations [33], adopting that language
provides the option of choosing among several well-maintained
software implementations and the likelihood that such software
will continue to be available for the foreseeable future. XQuery is
a far more powerful language than sgrep, which is restricted to
identifying and extracting regions from a document in the order that
they appear. The computational power of sgrep is therefore more
similar to XPath Version 1.0 [4] (but with the addition of intersection
and difference of sets and not being restricted to XML) than it is to
XQuery. However, the needs of the scholars wishing to access the
Alberti Magni e-corpus are for a simple query interface that finds
regions of text from a collection of documents and presents matches
to queries in the order that they appear in those documents, and
this computational requirement does not require the expressive
power of XQuery. Were a similar project to re-start today, serious
consideration should be given to building the engine with software
that supports XPath Version 3.0 or XQuery, but using a simple
region algebra such as the one supported by sgrep (or Wumpus, if
the corpus is large enough to require an index) should also receive
due consideration. The amount of implementation effort to connect
the user interface to the search software is likely to be quite similar
for both approaches.
Converting matched text from XML to an HTML display form
can also be implemented using XML technology based on W3C rec-
ommendations, namely CSS [15]. As is true for search, the needs for
8The software is freely available at https://github.com/fwtompa/e-corpusSE.
9One scholar recently wrote: “I am using the engine a lot these days for my research,
and it is extraordinarily useful to me.” Another posted the comment: “The Alberti Magni
E-Corpus ... [has] some very nice search features. (Which is important, because Google
Books is sometimes really crazy about its word recognition in modern languages, much
less in Latin.)” A web site states: “l’opera di Alberto Magno scaricabile in formato pdf
e, soprattutto, liberamente ricercabile on line ... si tratta di uno strumento che avrà
ancora per molti una sua utilità.”
10See https://wiki.tei-c.org/index.php/Milestone-chunk.xquery for a solution in
XQuery, implemented as util:get-fragment-between in eXist.
the Alberti Magni e-corpus do not require the computational power
of general transformations. Instead, Dragoman uses only a simple
tag interpretation algorithm that is easily explained. The tasks of
writing several style sheets and controlling which style sheet to
invoke is required, regardless of the implementation technology.
The most difficult part of starting a project with a new corpus
is to convert the text into XML that reflects its logical structure,
an extremely challenging task when physical layout must be inter-
preted [8], but also quite challenging when the input is plain text
with embedded font 11 information. In most of the text, each feature
to be tagged can be recognized fairly easily, but unexpected difficul-
ties arise when the features overlap in unanticipated ways. These
problems do not result from the choice of software, but rather they
are inherent in working with human-generated data: regardless of
the approach taken to recognize structure, every form of feature
interleaving that appears in the corpus must be accommodated by
the conversion system. In several projects, we have successfully
written converters that process the input sequentially in multiple
passes. An alternative approach worthy of investigation is to drive
the recognition process by a grammar that reflects the structures
to be recognized. Unfortunately, developing such grammars for
documents that have not yet been transformed into XML may be
an equally challenging task due to the presence of many unantic-
ipated exceptions that are commonly present in documents that
were written over decades using just pen and paper [34]. Based on
his experiences during several projects, Cummings has similarly
observed:
"The process of using an incremental ad hoc interme-
diate format is quite common in such conversions. In
many cases it is difficult to know very much about the
structure of the document until much of the conver-
sion has been done, hence an incremental approach
allows a greater ease of recovery from mistaken con-
version steps. However, it highlights a basic problem
in such attempts to manipulate legacy files: that until
they are in some format for which tools exist, it is
difficult to examine or validate their structure." [6]
The specific tag set chosen for the Alberti Magni e-corpus can be
improved. It has several idiosyncratic aspects, such as the chosen
division names (Figure 2) and the use of <frag> and <cont> tags to
represent words split across page and column breaks. If the scholars
responsible for managing the e-corpus wish to share the tagged
documents with other scholars, they will be well-served to first
transform the tagging to be TEI-compliant [29]. Because the corpus
has already been structured as well-formed XML, this should be
far simpler than applying tags to a new document, and standard
XML transformation tools, such as XSLT [17], can surely be used
without difficulty. Thereafter, it may be worthwhile to develop a
DTD [24] or XSD [22, 25] schema that describes the constraints on
document structures found in the collection.
When the Alberti Magni e-corpus project was started, it was
intended that the software should be easily applied to any scholar’s
11The XML tags that are output by today’s OCR readers often represent physical
properties of the text only, such as columns and typefaces. However, more sophisticated
document analysis can also be applied to capture logical structure [13].
corpus withminimal customization. Both sgrep andDragoman pro-
vide simple mechanisms for adapting the search and display to new
corpora. Therefore, the system could easily be used by scholars who
work with other document collections. However, the functionality
that is provided by the code base that ties these two components
together with the query interface is specific to the needs of the
Albertus Magnus scholars: the various modes of search, the re-
quirement to display snippets with location information, and the
ability to browse documents at most one page at a time may not
fulfill all the needs of other scholars. For example, the software pro-
vides no ability to rank results, no ability to pose a query in which
some terms use normalized spellings and others do not, no ability
to compare two documents directly (e.g., to find similar passages,
to analyze text differences, or to examine primary and secondary
sources together), no ability to link a passage in a document to its
critical apparatus (except by proximity), no ability to attach new
annotations to a document, and so forth. A more widely acceptable
scholars’ search engine still requires much work.
User requirements for the software were developed in consulta-
tion with University of Waterloo Professors Delbert Russell (for
Campsey) and Bruno Tremblay (for Albertus). The implementation
benefitted from assistance with earlier versions from two (then
undergraduate) students: Jingchi (Evan) Chen (January 2004-April
2004 for the Camsey project) and Shadab Ul Rashid (May 2008-
December 2008 for the Alberti Magni e-corpus project) and from
repeated discussions with Andrew Kane, who also provided valu-
able feedback to improve this paper. Facilities were provided by
the University of Waterloo, and financial support for the students
was provided by NSERC, the Natural Sciences and Engineering
Research Council of Canada, and by CFI, the Canada Foundation
for Innovation.
[1] David M. Berry. 2011. The computational turn: Thinking about the digital hu-
manities. Culture Machine 12 (2011), 1–22.
[2] Chris Biemann, Gregory R Crane, Christiane D Fellbaum, and Alexander Mehler.
2014. Computational Humanities-Bridging the gap between Computer Science
and Digital Humanities (Dagstuhl Seminar 14301). Dagstuhl Reports 4, 7 (2014),
80–111.
[3] Stefan Büttcher. 2018. Wumpus File System Search. http://www.wumpus-
search.org/. (accessed April 2018).
[4] James Clark and Steven DeRose. 1999. XML Path Language (XPath) Version 1.0.
W3C Recommendation. W3C. http://www.w3.org/TR/1999/REC-xpath-19991116.
[5] Mariano P Consens and Tova Milo. 1998. Algebras for querying text regions:
Expressive power and optimization. J. Comput. System Sci. 57, 3 (1998), 272–288.
[6] James Cummings. 2006. Liturgy, drama, and the archive: Three conversions from
legacy formats to TEI XML. Digital Medievalist 2 (2006).
[7] James Cummings. 2008. The Text Encoding Initiative and the Study of Literature.
In A Companion to Digital Literary Studies, Susan Schreibman and Ray Siemens
(Eds.). Blackwell, Chapter 25, 451–476.
[8] Andreas Dengel and Faisal Shafait. 2014. Analysis of the Logical Layout of
Documents. In Handbook of Document Image Processing and Recognition, David
Doermann and Karl Tombre (Eds.). Springer, Chapter 6, 177–222.