dedoc/paragraph_dataset huggingface dataset:
the location within Super IV libros Sententiarum refers to the sec-
ond column on page 250, within the section marked Divisio et
Expositio Textus (divEtExp in Figure 1) that follows sections A
and B of Lombard’s text, found within the th 27 distinction within
Liber IV. With these rules, and a table mapping XML structured
division names to the abbreviations (e.g., d:liber to “Lib.”, d:mem to
“m.”), an attribute value capturing the complete location information
(except the name of the work) can be inserted into each section
start tag and into each page and column milestone in the corpus.
The detailed location information for a search match is then set
to the last location attribute value that precedes the match point,
prepending it with the information found in the metadata table.
4 CORPUS SEARCHING
The original OED search engine was driven by queries expressed
using the Pat region algebra [10, 28]. At 570MB, the OED e-corpus
was very large for its time, requiring its search engine to use a
“suffix array” index to achieve reasonable performance [11]. The
format chosen for text display was specified by style sheets using
Lector [26], and the resulting display commands were executed
on the X Window System (X11). More recently, the OED search
software was revised to run as a web application, replacing Pat by
the Wumpus [3] search engine, which also uses a region algebra
to specify searches. At the same time, Lector was replaced by
Dragoman a simple, but similar, XML-to-HTML converter driven
by style sheets (as described in Section 5).
The Alberti Magni e-corpus is an extensive text collection, but
its size remains modest for search engines running on today’s
hardware:
60 works 18,974 pages 9.07 million words 130MB
As a result, a simple main memory search system is sufficiently pow-
erful to provide suitable functionality without the inconvenience
of indexing. The sgrep “structured grep” utility [14] is designed to
search XML-like text without relying on an index, and it is thus
highly suitable as a backbone for a search engine in this context.
The combined search and display code is invoked from a web
page that uses a front-end handler to read parameters for the search,
calls a back-end search engine (here, sgrep), formats the results
(using Dragoman), and presents the results on a subsequent web
page that provides a form for specifying the next query. The code
depends on several configuration files that are referenced from a
single “directory file” read at compile time by the front-end han-
dler. This directory file specifies the location of the metadata table
(which stores the names and locations of files to be searched), the
commands required to invoke the search engine, the location of the
style sheets for controlling how to display search results, the loca-
tion of the list of searchable units (e.g., paragraphs, pages, sections),
and the location of a file containing messages to be displayed by
the front end. Because the metadata table, list of style sheets, list of
searchable units, and list of messages are all referenced indirectly,
their contents can be changed as desired without recompilation.
Given a query, the sgrep system (like Pat and Wumpus) returns
matching regions rather than searching for documents that contain
similar words, where a region can be any contiguous sequence of
one or more characters within the document being searched. The
system finds all text regions that match a query exactly, returning
either the number of such regions or the extents themselves (as
pairs of offsets for the start and end of each region or as strings of
characters found between those offset pairs). The sgrep query lan-
guage is a “region algebra” [5] that treats string queries as searches
for regions whose contents exactly match the query string; includes
operators for testing for region inclusion, adjacency, and proxim-
ity; and includes operators for combining lists of regions through
conjunction, disjunction, and negation.
The reliance onmatching regions leads to some surprising results
for users familiar with the word-based retrieval engines commonly
used to search the web. For example, a search for the string et
in Albertus’s Mineralia matches 4788 two-character regions that
contain the string et, only 3026 of which are instances of the word
et. A search for “ et ” (the word surrounded by blanks) identifies
2906 instances of the word, but other instances are abutted by
punctuation marks or XML tags. If the goal is to find all instances of
the word “et,” the solution is to provide an option to find all regions
that correspond to maximal contiguous sequences of letters and
returning those that coincide with the regions matching the string
et after case-folding. As a second surprising example, a search
for regions matching in ipsum returns 4 matches of the phrase in
Albertus’sMineralia, but it misses the instance on page 10 where the
two words are separated by a column break. A somewhat acceptable
solution to searching for the phrase consisting of these two words
is to find all sentences that contain the string in followed a blank
followed (eventually) by the string ipsumwithin that same sentence;
unfortunately this returns 39 spurious matches that include other
intervening words as well. As a consequence, the results returned
when searching for phrases must be carefully examined.
Importantly, sgrep also supports the formation of regions that
extend from a region in one list until the nearest region in another
list. For example,
define(UNIT_STAG,( "<d:" .. ">" ))
first creates a region list L1 of all regions (of length 3) matching
the string “<d:” and a second list L2 of all single-character regions
that contain “>”, then forms regions from the start of each region
x ∈ L1 to the end of each region y ∈ L2 where there does not exist
z ∈ L1 ∪ L2 such that z , x , z , y, and z falls within the region
extending from the start of x to the end of y. That is, the list of
regions defined by UNIT_STAG above includes each region defined
by the closest pairs of “<d:” and “>”, inclusive of both ends. Instead
of “..”, the notation “._” includes the start regions but excludes the
end regions, “_.” excludes the start regions and includes the end
regions, and “__” excludes both the start and end regions from the
returned regions list (i.e., the returned regions extend from the ends
of the first regions to the starts of the second paired regions).
To use sgrep, it is necessary to describe the encoding of struc-
tures to be searched and returned (e.g., pages, paragraphs, titles).
Paragraphs are straightforward, since they are encoded as XML
elements having the generic identifier p:
define(PARA,(ELEMENTS("p")))
Titles are slightly more complicated: some titles are synthesized
for the e-corpus, and so both they and their original print forms are
tagged differently than section titles that are maintained without
alterations, and the names of works in the corpus also use distinctive
tags. As a result, for this corpus, titles are encoded as XML elements
with any of four generic identifiers:
define(TITLE,( (ELEMENTS("title") or
ELEMENTS("synTitle") or ELEMENTS("printTitle") or
ELEMENTS("origTitle")) ))
Finally, because pages and columns are indicated by milestones
only, their extents must be described as extending from one mile-
stone (inclusive) to the next (exclusive):
define(PG,(ELEMENTS("page") ))
define(PAGE,( (start ._ PG) or (PG ._ PG) or (PG .. end) ))
define(COL,(ELEMENTS("col") ))
define(COLEND,( COL or PG ))
define(COLUMN,( (start ._ COL) or
(COL ._ COLEND) or (COL .. end) ))
Similarly to identifying pages and columns based on such extents,
the location text to be displayed for a match (as in Figure 3) can be
constructed from the one found in the extent that is defined from
the preceding location attribute, regardless of tag, (inclusive) to the
location attribute that follows (exclusive).
In all, 26 sgrep rules are defined to support boolean search within
any of six structures (paragraphs, titles, sentences, sections, pages,
or columns) present in the Alberti Magni e-corpus and to support
the searches needed to highlight matches and display the locations
of the matched results.
With these rules defined, to find all paragraphs that contain the
string “elicit”, the search engine first identifies all paragraph regions
P that contain regions (of length 6) exactly matching the search
string. It next finds all regions M that exactly match the search
string within regions in P . It then finds all regions L that start at
one location attribute and end with another and contain the start of
the regions exactlymatching the search string (i.e., the “e” of ”elicit”).
The search engine outputs the number of matching regions in P and
the number of matching regions inM . After that, for each region in
P , it constructs the location identifier for the first location attribute
in L that falls within that region and prints the location followed by
the contents of the matching paragraph from P , surrounding each
word containing a match inM with <match> and </match> to provide
highlighting. Figure 4 shows the two paragraphs in the document
“De sex principiis” that contain the three instances of words that
include the string “elicit”, each identified by the locations of those
matches. Similar processing is applied when searching for phrases
Figure 4: Search for paragraphs containing “elicit”
Table 1: Equivalences for orthographic normalization
ae ci d j k m oe ph v y
e ti t i c n e f u i
(i.e., strings that include blanks, such as "quod equus et 6 homo"),
ordered words (e.g. "prima*secundum"), or boolean combinations of
these (e.g., "dicit Aristoteles" + (bestia | animal | brut)).
Variations in Latin spelling, especially when the sources are var-
ied, add a complication to search. For example, the letters j and i
are often used interchangeably, as are the letter pairs ci and ti; the
letter h is sometimes omitted and apostrophes are used inconsis-
tently. To simplify search, therefore, the search engine includes an
option for searching with normalized spelling, where the following
rules are used for orthographic normalization: remove diacritics,
ignore capitalization, ignore h and apostrophe (’), use the equiva-
lences in Table 1, and ignore repeated letters. Thus, for example,
philosophiae is normalized as filosofie, Tyrrhenum is normalized
as tirenun, and prooemio is normalized as prenio.
A complication arises from region-based search that is not present
in word-based search. With exact match, a search for the string
philosophia matches all instances of the prefix of philosophiae.
However, philosophia is normalized as filosofia, so a search for
its normalized form does not match the normalized form of in-
stances of philosophiae. As a result, users may be surprised to
learn that search under normalization can produce fewer results
than search for the same string without normalization. To adjust
for this anomaly, a normalized search for any string ending in a
is similarly interpreted as searching also for that string ending in
ae; a normalized search for any string ending in o is interpreted as
searching also for that string ending in oe; a normalized search for
any string ending in c is interpreted as searching also for that string
ending in ci; and a normalized search for any string ending in p
is interpreted as searching also for that string ending in ph. This
approach guarantees that no matches are missed, although some
false positives might be returned, such as instances of the string
ille when searching for strings matching illa (normalized).
Search engines based on inverted indexes (postings lists) typ-
ically support normalized search by incorporating orthographic
normalization as part of text tokenization prior to indexing. How-
ever, because sgrep does not use an index, an alternative method
is needed to search for normalized text. The solution is to build
a “shadow” text X for each document D in which each word wi
is replaced by its normalized form ni , but the offset of ni in X re-
mains unchanged from the offset ofwi in D. Subsequently a search
for normalized query terms in X returns a list of matching offsets.
These offsets are then used to locate the corresponding instances of
the unnormalized words in D. Figure 5 shows a partial result from
a search with normalized spelling.
The indexing cost of this approach is the time to normalize all
documents and an amount of space equal to the sum of the sizes
of the original documents. For a corpus supporting research in
the humanities, even one as large as the Alberti Magni e-corpus,
6As stated earlier, the results of searching for phrases will miss instances interrupted
by tags.
Figure 5: Search for “alicujus” with normalized spelling
this is quite manageable: it takes only a couple of 7 seconds to
create the 130MB normalized version of the Alberti Magni e-corpus.
Because of the need to merge lists, the engine’s response time for
searching all sixty documents increases with normalization from
6 seconds (unnormalized) to 9 seconds (normalized) for a rarely
occurring string (such as convertibilia), from 10 to 12 seconds for
a frequently occurring string (such as illa), and from 13 seconds
to 19 seconds for an extremely common string (such as a).
5 DISPLAYING MATCHED TEXT
After a search, the matching portions of a document need to be dis-
played, as illustrated in Figures 4 and 5. To this end, the Dragoman
text display subsystem processes text sequentially, replacing simple
XML opening and closing tags, attributes, and entity references by
the text specified in a user-selected style sheet and chosen by the
designer to reflect the structure of the text through typography.
The result is then passed to an HTML renderer (i.e., a web browser)
for display.
The style sheet dictates how to interpret tags, attributes, and
entity references as they are encountered in a left-to-right traversal
of the text to be displayed, transforming the string into HTML. An
important characteristic of the formatter is that not only can tags,
attributes, and entity references be replaced by formatting or other
symbols, but they can also be used to suppress arbitrarily large
portions of text. Optionally, a matched opening and closing tag
pair or a tag’s attribute can be interpreted to indicate that the text
between the tags or the attribute’s value be treated as if it were
an entity reference (effectively interpreting the opening tag as the
start of an entity reference and the closing tag as its end).
Each line of a style sheet has three tab-separated fields, and as
each tag is encountered in the text to be displayed, Dragoman
searches for a line in the style sheet matching the tag’s tagID in the
first field (the indicator):
tagID action startText action endText
7All reported performance statistics are based on single-threaded execution under
Ubuntu 16.04 on a shared 2 CPU - Intel E5-2697 v3 Xeon machine with 256GB Ram.
match "<font color="red"> "</font>
d:tract " "
d:tract.val " "<br/>
greek - +