dedoc/paragraph_dataset huggingface dataset:
Figure 1: Nesting structure of Albertus Magnus’s Super IV
libros Sententiarum; all undefined non-terminals represent
running text, typically divided into paragraphs and sen-
tences and interleaved by page and column breaks
search the Oxford English Dictionary (OED) and display the re-
sults [30].
This paper summarizes the approach taken to develop an engine
for searching collections of texts, based on the needs of the Alberti
Magni e-corpus project. Section 2 describes the conventions used to
mark up the text as part of the proof-reading and correction effort
that follows initial optical character recognition. Next, Section 3
describes the challenges encountered when converting the marked-
up text to XML in preparation for search and display. Section 4
describes the functionality of the search engine, and Section 5
describes the text formatter that displays document fragments by
replacing XML tags with HTML codes and suppressing XML fields,
as desired. Finally, Section 6 summarizes some lessons learned and
outlines areas for further research and development.
2 INITIAL TEXT MARKUP
A scholar in the humanities, having chosen the texts to analyze,
often begins by mechanically scanning myriads of printed doc-
ument pages, applying optical character recognition (OCR), and
laboriously correcting errors (by comparing the output against
the original images, perhaps aided by a language-specific spelling
checker). This digitization process results in a stream of text, with
some typography and layout preserved.
Successful text search and display requires that structural units
and features within a document are recognized by the search engine.
This typically includes the recognition of layout structure (pages
and columns), logical structure (chapters, sections, paragraphs, and
sentences), and embedded features (headings, italic text, Hebrew
text, and illustrations). All documents in the editions scanned for
the Alberti Magni e-corpus are laid out on pages, but only some
place the text into columns. All have sentences and paragraphs,
and many are hierarchically arranged in units designated as LIBER
(book), TRACTATUS (tract), and CAPUT (chapter).
During the proof-reading stage, some initial markup to capture
the structural units and features can also be added to the text. OCR
software captures not only the sequence of letters that comprise
a text, but also its spacing and typography, and thus some of this
markup may be derivable from a parallel analysis of the physical
layout of the scanned pages [23]. Beyond what the OCR system
provides, some structural markup is simple to insert manually.
Unfortunately, there is a cost to adding markup by hand: the more
markup that is required, the greater the additional cognitive burden
that is placed on the proof reader and the greater the time to insert
markup by hand [18]. For the Alberti Magni e-corpus, the following
markup conventions were designed to balance completeness and
simplicity:
• enclose all page and column numbers within slashes (e.g., /24/
for the start of page 24, /135B/ for the start of the second column
on page 135);
• insert $err#corr$ to indicate that the erroneous text in the original
manuscript err has been corrected to corr ;
• surround transliterations of Greek letters with [GREEK: and ];
and similarly surround transliterations of Hebrew letters with
[HEBREW: and ];
• indicate the presence of graphics by the notation [ILLUSTRATION];
• identify source text upon which Albertus Magnus is commenting
by inserting [@author:] at the start and [@] at the end, where
author is the name of the source text’s author;
• surround tables with |- at the start of each table’s first line and
-| at the end of each table’s last line, and insert | between table
columns on each line;
• where non-standard section headings are used, supplement them
with synthetic headings in a standardized form; for example, to
indicate that the heading Cuius primus tractatus est de homine
indicates the start of the first “tractatus”, which is entitled “De
homine”, the scholar inserts the mark up:
+TRACTATUS I%DE HOMINE+Cuius primus tractatus est de homine
The resulting documents have sufficient structure explicitly and
unambiguously marked that it can be converted into well-formed
XML that includes tags marking all textual elements of interest to
the researcher. However, the ability to transform a document col-
lection into XML in theory does not imply that it is straightforward
to perform the transduction in practice.
3 RECOGNIZING TEXT STRUCTURE
Before new texts are added to the e-corpus, they need to be con-
verted to XML, often built upon the TEI tag set developed by the
Text Encoding Initiative [7, 29]. The Alberti Magni e-corpus re-
quires tags that mark pages, columns, titles, structural sections,
paragraphs, sentences, hierarchical locations that serve as targets
for citations into the text, and various other features present in the
corpus. Rather than building individual taggers for each document,
a single conversion pipeline was designed to be applied to all of the
works in the e-corpus; in this way, as conversion errors are found,
the tagging pipeline can be corrected and reapplied to all documents
to ensure that similar, but undetected, errors in documents already
processed are also corrected.
For the Alberti magni e-corpus, the input text is stored in rich
text format (RTF), using predetermined markup conventions as
described in the previous section. The texts are first converted to
HTML using 4 rtf2html. Thereafter the text passes through a series
of perl scripts, the first of which removes superfluous white space
and HTML encoding (but preserving italics and bold tags). Next,
simple atomic elements are tagged: Greek, Hebrew, and illustration
indicators are replaced by tags; table delimiters are replaced by
corresponding HTML table, row, and cell tags; page and column
markers are replaced by empty “milestone” tags with attributes
encoding the page and column numbers; and editorial corrections
marked as $err#corr$ are changed to <corr w="err">corr</corr>.
4https://sourceforge.net/projects/rtf2html/
Fashioning a Search Engine to Support Humanities Research DocEng ’18, August 28–31, 2018, Halifax, NS, Canada
These changes appear to be straightforward, and it is fairly sim-
ple to implement a converter that handles the vast majority of
the text correctly. However, complications arise when the various
markup conventions overlap in the text or occur in unanticipated
contexts. The remainder of this section illustrates some of the com-
plications encountered when recognizing the structure found in
the Alberti Magni e-corpus.
Page layout conventions are such that page and column bound-
aries are usually placed where there is white space in a text. How-
ever, they can also occur in the middle of words. In order to recog-
nize a complete word that crosses a page or column boundary and
still record exactly where the word is split, an element is introduced
tomark the starting fragment of a brokenword, with an attribute en-
coding the whole word, and a second element is introduced to mark
the second fragment of the word. Thus, for example, the tags in
<frag w="sicut">sic</frag><col n="16b"/><cont>ut</cont> record
where the word sicut breaks between the bottom of the first col-
umn on page 16 and the top of the next. To complicate things
further, a page or column break can also occur in the middle of a
correction: $con/488A/trariuum#con/488A/trarium$; thus conver-
sion code must be carefully written to accommodate this variant.
Furthermore, if a page or column break is surrounded by italic text,
the slashes around the page or column number may well also be in
italics: these four variants are indistinguishable by a proof-reader
of the text in printed form. Again, conversion code must be able to
handle all these variants.
Inserting tags that reflect the collected works’ structures requires
significantly more attention. The starts of sections are often clearly
indicated by key words or phrases at the start of input lines (espe-
cially if supplemented by synthetic section headers as described
in the previous section), and these markers are often followed by
section numbers, but there is sufficient variety to make this task
non-trivial. There are 23 markers signifying the start of various
types of numbered sections (see Figure 2), and a variety of number-
ing conventions are used. For example, some structures use Roman
numerals (e.g., CAPUT IV.) and other use Arabic numerals (e.g.,
Sermo 39); the work Sermones parisienses is divided into numbered
sections without preceding key words (e.g., I, II), and Determinatio
Magistri Alberti de novo spiritu is similarly divided, but using Arabic
numerals (e.g., 1, 2), further complicated by the presence of a sec-
tion numbered 1a between the section numbered 1 and the section
numbered 2; and some sections containing Biblical commentary are
numbered by ranges of the verses referenced (e.g., Versus 43-44).
In addition, recognizing the starts of structures is further compli-
cated by the possible occurrence of corrections for the structure
name, the section number, or both (“$ACTICULUS#ARTICULUS$ XV.”
vs. “ARTICULUS $XI.#IX.$” vs. “$ARTICULUS#ARTICULUS 5 III.$”).
Having recognized a section heading, any text following it with-
out an intervening blank line is tagged as a title for that section.
Text following a blank line is then interpreted as the body of the
section, consisting of paragraphs with nested sentences. Because
paragraphs are separated by blank lines in the input, marking para-
graph boundaries is straightforward. A naïve approach to marking
sentence boundaries was adopted, based simply on punctuation
5All examples in this section are present in the input texts and were discovered while
developing the conversion pipeline.
division opening nesting
d:liber LIBER
d:tract TRACTATUS >d:caput
d:caput CAPUT
d:caput Capitulum
d:caput CAPITULUM
d:caput CAP.
d:caput Caput
d:in-caput IN CAPUT >d:in-prooem
d:in-caput In Caput
d:dist DISTINCTIO
d:Lombard [Pierre Lombard:] -d:art
d:art ARTICULUS
d:pars PARS
d:qi QUAESTIO INCIDENS leaf
d:quaes QUAESTIO
d:mem MEMBRUM
d:part PARTICULA
d:subpart SUBPARTICULA
d:quaesit QUAESITUM leaf
d:ver VERSUS leaf
d:sermo Sermo leaf
d:signum § leaf
d:num leaf
Figure 2: Indicators for starts of numbered sections
followed by a capitalized word, but more sophisticated methods
could be applied if it were found to be necessary [19].
The end of a section is implicitly indicated by the start of some
other section that cannot be nested within it. For example, consider
the works with sections marked LIBER and nested sections marked
TRACTATUS, which in turn have nested sections marked CAPUT: each
LIBER section ends when the next section marked LIBER is encoun-
tered; similarly, a section marked TRACTATUS ends when a section
marked either LIBER or TRACTATUS begins. However, the structure is
not necessarily this regular: LIBER I in the work Topica starts with
two CAPUT sections before the first TRACTATUS with its nested CAPUT
sections begins; several works have TRACTATUS and CAPUT sections,
without first being divided into LIBER sections; all the LIBER sec-
tions in the work Politica are divided into CAPUT sections without
any divisions marked TRACTATUS. Many other works use radically
different structures and substructures, as indicated in Section 1
above, making it important to characterize the constraints on the
nesting of structures. Figure 2 lists all the structural divisions in the
e-corpus that correspond to (usually) numbered sections, together
with the possible text sequences that occur before the section num-
ber appears. The third column indicates constraints on the nesting
of those divisions, as follows:
leaf no nested substructures
>struc disallowed in struc
-struc no nested substructures and disallowed in struc
There are 23 additional rules for those substructures (such as pro-
logues, sections marked EXPOSITIO TEXTUS, etc.) that are never num-
bered. Using these tables, the ends of structures are recognized
much like a shift-reduce parser processes context free languages [9].
Peri hermeneias (ed. Borgnet, 1890), Lib.II, tract.1, cap.13, p.92a
Topica (ed. Borgnet, 1890), Lib.I, cap.2, p.45a
De animalibus (ed. Stadler, 1916-1920), Lib.XXIII, cap.2, p.504
De fato (ed. Mandonnet, 1927), art.1, p.399
Super Iob (ed. Weiß, 1904), cap.13, v.28, p.179
Super Ieremiam (frag.) (ed. Meersseman, 1932), in cap.1, v.5, p.7
Super Threnos (ed. Borgnet, 1893), praefatio, p.246a
Super prophetas minores (ed. Borgnet, 1892), Amos, prol., p.181
De IV coaequaevis (ed. Borgnet, 1895), tract.4, q.61, art.2,
part.1, quaesit.2, p.203b
Super IV libros Sententiarum (ed. Borgnet, 1893-4), Lib.IV,
dist.27, A-B, divisio et expositio textus, p.250b
Summa theologiae (ed. Borgnet, 1894-1895), Pars II, tract.1, q.4,
m.2, art.5, part.3, q.inc.1, p.103b
Determinatio Magistri Alberti de novo spiritu (ed. de Guibert,
1931), 1-a, p.117
Epistula de ungelt (ed. Rieder, 1901), p.47
Figure 3: Examples of labels to identify match locations
The last major challenge to address when recognizing structure
is to place markers that can help to identify the locations of matches
found by the search engine. For each work, the specifications of
how to present each match are determined by the needs of the
scholars, as illustrated in Figure 3 with examples from the Alberti
Magni e-corpus.
Because each work is stored and searched in a separate file, the
search engine can extract the name of a work, including the editor
and year, from a “metadata” table associating that name with the
file name. The remainder of the location information for a match
needs to be determined primarily from the nesting of structures
surrounding the match point in the text.
Unfortunately, there are two pieces of the location information
that are associated with non-nested structures: (1) both page and
column numbers are stored as milestones rather than via nesting,
and (2) commentary on Lombard’s text in Super IV libros Senten-
tiarum refers to the previous one or more sections of that text that
do not have their own separate commentary. Thus, in Figure 3,