SlowGuess/ACL_2020_OCR huggingface dataset:
Markus Gartner, Kerstin Jung
Institute for Natural Language Processing
{markus.gaertner,kerstin.jung}@ims.uni-stuttgart.de
Corpus query systems exist to address the multifarious information needs of any person interested in the content of annotated corpora. In this role they play an important part in making those resources usable for a wider audience. Over the past decades, several such query systems and languages have emerged, varying greatly in their expressiveness and technical details. This paper offers a broad overview of the history of corpora and corpus query tools. It focuseses strongly on the query side and hints at exciting directions for future development.
Annotated corpora have always been the backbone for many fields in NLP and other disciplines related to linguistics. Whether serving as an invaluable source of empirical evidence for foundational research or doubling as gold-standard training input for fueling the furnaces of our machine learning factories, their importance cannot be overemphasized. But especially for the empirically motivated user base, corpora are only ever as good as the means available to explore them. And the primary means of exploring linguistically annotated corpora have always been (dedicated) corpus query tools and corpus query languages in their manifold shapes.
In this paper we intend to give a thorough chronology of the major interplay between corpus progression and query tool evolution, with a strong focus on the latter. We start with an overview on relevant aspects of corpora and how they changed over the past  $\sim 30$  years in Section 2. Section 3 elaborates on the observable phases in query tool development. In Section 4 we discuss alternative corpus query approaches based on general purpose data(base) management solutions and provide pointers to related work in Section 5. Section 6 summarizes some of our observations and with
Section 7 we finally hint at our vision for future directions in corpus query system development.
2 Once Upon a Corpus - Trends in Corpus Evolution
Though corpus linguistics dates back further, major online catalogs such as those from LDC<sup>1</sup> and ELRA<sup>2</sup> list corpora starting from the early 1990s. In the following decades corpus trends have varied along several dimensions, both technical and content-related. This section discusses such features and gives examples for their evolution. Since this overview is an introduction to digital corpus query systems, we mainly focus on written and annotated corpora.
With a focus on written corpora, character encoding is a decisive factor when estimating the publication date. Starting from plain ASCII (Everts,  $2000^{3}$ , Graff and Cieri, 2003) and language/script specific encodings, such as ISO/IEC 8859 (Armstrong-Warwick et al., 1994; Federico et al., 2000), nowadays many corpora come with a (mostly) language independent UTF-8 encoding (Ion et al. (2012); Prasad et al. (2019) and compare Schäfer (2015) with Schäfer and Bildhauer (2012)), which is also able to capture symbols relevant for transcription and annotation.
Similar to character encoding, the preferences regarding the representation format for corpus content changed over time. Many corpora established in the 1990s come in an SGML format (Liberman, 1989; Amaryllis, 2001; Graff, 1995). In the next decade, XML-based corpora followed (Chiao et al. (2006) and compare Hajic et al. (2001) and Pajas
$^{1}$ Linguistic Data Consortium, https://catalog.ldc.upenn.edu/
$^{2}$ European Language Resources Association, http://catalogue.elra.info/
$^{3}$ Earlier version published 1997 by ELRA: ISLRN 628-817-117-400-1
6307
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6307-6321 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
and Štepánek (2005)) and since corpora were also made accessible over the web, relational database management systems (RDBMSs) became a valuable backend for corpus storage (Davies, 2005). Today we face a multitude of formats ranging from sophisticated and specialized XML encodings to simple tabular formats and often a corpus comes with more than one representation (Petran et al., 2016; Bick, 2018). Especially since the first CoNLL shared tasks<sup>4</sup>, their tabular format to encode sequence-based annotations and relations has been majorly developed (Nivre et al., 2016).
Regarding included languages, multilingual and (partly) parallel corpora appear early (Liberman, 1989; Armstrong-Warwick et al., 1994; Graff and Finch, 1994), however, there was a rise of parallel corpora in the first decade of the current century. Prominent examples are Europarl (Koehn, 2005), the CESTA Evaluation Package (Hamon et al., 2006) and the Prague Czech-English Dependency Treebank 1.0 (Cmejrek et al., 2005). On the other hand, with the rise of web corpora, language detection became more important to only crawl (or keep) web data for a specific language.
Corpus size is a less discriminative factor than one might think, since many early corpora came as collections of sub-corpora. Armstrong-Warwick et al. (1994) already contains 90 million words and LDC's Gigaword initiative started in 2003 (Graff and Cieri, 2003), while many small corpora for specific topics or containing manual annotations are constantly being created. Nevertheless, with recent web corpora, e.g. ENCOW16 $^5$  and iWEB $^6$ , several billion tokens pose new challenges for the design of both storage and search facilities.
While for spoken corpora domain selection is often tailored to the research question at hand (cf. Talkbank (MacWhinney et al., 2004)), for written corpora (and especially annotated ones) there is a bias towards news and official documents, which was superseded by multi-domain web corpora starting in the late 2000s (e.g. the WaCKy initiative (Baroni et al., 2009) and COW) and, in the follow up, the increasing number of corpora of computer-mediated communication and social media<sup>7</sup>. Like
with the language setting, for web-corpora the challenge is no longer to include more languages or domains, but to identify and/or restrict them to a sensible subset. Collections of historical language data have also been available for some time, e.g. the Corpus of Middle English Prose and Verse<sup>8</sup> and with the rise of the Digital Humanities many further corpora are created and/or enhanced with linguistic annotations, such as the Drama Corpora Project<sup>9</sup>, where some corpora have been enhanced with lemma information.
Most corpora come with annotations, the earlier ones mainly with flat and word-based annotations, mostly including part-of-speech, such as the ECIELSNET Italian & German tagged sub-corpus<sup>10</sup>. Regarding the structural aspect, stand-off syntactic annotations became more feasible with emerging treebanks, while over time the focus changed from phrase-based (Brants et al., 2004) to dependency tree structures (Hajic et al., 2001). The current decade has also seen an increase in the richness of annotation layers of morphological, syntactical and semantical description, including highly concurrent annotations belonging to the same description layer, e.g. Ide et al. (2010) or Schweitzer et al. (2018).
3 A Brief History of Querying
We observed three major phases or generations in the history of corpus query systems, which are roughly aligned to the last three decades. The following is meant as a comprehensive but not exhaustive chronology of corpus query systems and approaches. Space does not permit we provide in-depth descriptions for every system mentioned but instead refer to Section 5 for pointers to existing work that discusses and compares certain (families of) query systems in detail.
3.1 First Generation - Humble Beginnings
The history of corpus querying systems has been for the most part tightly connected to the gradual expansion of the targeted corpus resources. As such the initial wave of corpus query tools during the 1990s was mostly geared towards text corpora:
The COSMAS<sup>11</sup> lineage remains until today<sup>12</sup>
$^{4}$ https://www.conll.org/previous-tasks  
 $^{5}$ COrpora from the Web (COW), English sub-corpus, https://corporafromtheweb.org/  
 $^{6}$ https://www.english-corpora.org/  
 $^{7}$ Annual conference on computer-mediated communication and social media corpora started in 2013 https:// sites.google.com/site/cmccorpora/
$^{8}$ https://quod.lib.umich.edu/c/cme/  
 $^{9}$ https://dracor.org/  
 $^{10}$ ISLRN 869-857-775-378-7  
 $^{11}$ Corpus Search, Management and Analysis System, http://www.ids-mannheim.de/cosmas2/  
 $^{12}$ The initial version COSMAS I has been in continuous service from 1992 till 2003 and COSMAS II ever since 2002
6308
the public query front-end for the large corpus collection hosted at the IDS (Bodmer, 2005), offering keyword in context (KWIC) visualization in a browser-frontend and various query constraints.
In contrast the Linguistic DataBase program (LDB) (Halteren and Heuvel, 1990) features a very expressive tree-based query syntax and also ships with a tree editor. In addition it provides an ingenious event-based approach for extracting information from a corpus during search.
The Corpus Workbench (CWB) architecture (Christ, 1994) with the Corpus Query Processor (CQP) as its core component is maybe the most widely used corpus query system as of today, serving as the backend for many corpus exploration websites. Having been under continuous maintenance to keep up with the demands of the new century (Evert and Hardie, 2011), it provides a solid set of simple yet expressive search features, such as regular expressions over tokens and token content, flexible structural boundaries, support for parallel corpora or the ability to enrich a corpus during ingest with external data that can then be used for querying, e.g. WordNet (Miller, 1995) categories.
Emu (Cassidy and Harrington, 1996) was designed for speech corpora with multiple levels of segmentation. Primarily a hierarchical speech data management system, it also supports label- and position-based queries for collections of tokens.
Similarly the MATE Workbench (Mengel, 1999; Mengel et al., 1999; Heid and Mengel, 1999; Isard et al., 2000) also targets combinations of text and speech data in the form of XML annotation files. It provides full boolean operations over hierarchical and time-based constraints in a logic-style query language, but no direct support for quantifiers.
3.2 Second Generation - The Rush for Rapid Feature Expansion
At the dawn of the  $21^{\mathrm{st}}$  century the second and larger wave of query systems emerged. Initially focused heavily on treebanks annotated for phrase-based syntax, a later trend shifted more towards supporting dependency syntax annotations, with an overall theme of increasing expressiveness with new approaches to query syntax and constraints.
TIGERSearch (König and Lezius, 2000; Lezius, 2002) was among the first with its logic-based query language to target phrase-based treebanks conforming to the TIGER model (Brants et al., 2004). It inspired many of the later query ap
proaches, but was quickly surpassed wrt expressiveness due to limited negation or quantification<sup>13</sup>.
The ICE Corpus Utility Program (ICECUP)[14] introduced a completely new direction of development. Wallis and Nelson (2000) emphasized the complexity required to transform a two-dimensional tree description into a linear sequence of textual expression and made an argument for a graphical query approach. Their fuzzy tree fragments act as visual (under-)specification of the targeted phrase-based tree structures and are then matched against instances in a corpus. The appeals of this approach are diverse: It enables example-based searching by allowing the user to start from an existing instance in the corpus, transform it into a query and then relax the constraints on that query to generalize it[15]. Not having to learn a formal query language and annotation schemes first, also lowers the barrier to entry for successful querying.
As a dedicated treebank query tool TGrep2 (Rohde, 2001) offers a rich query syntax for phrase-based treebanks. Notable features are conjunction, disjunction and negation for relations, over 30 predefined basic link types and the ability for users to simplify complex queries by using macros.
Usually corpus query tools depend on the target data already being annotated. Gsearch (Corley et al., 2001) however lets the user query unstructured text data by parsing it on the fly with a chart parser. Gsearch queries contain phrase-based constraints with limited boolean operators and the results are emitted in SGML.
VIQTORYA $^{16}$  (Steiner and Kallmeyer, 2002) is another tool to query phrase-based treebanks. Its query syntax is very similar to TIGERSearch $^{17}$  and queries are translated for the RDBMS backend.
Outside the domain of monolingual corpora ParaConc (Barlow, 2002) combines typical concordancer functionality such as surface search and
13The developers decided to forgo universal quantification due to computational cost and tractability (TIGERSearch Help, section 10.3) but also proposed an extension of the language with universal quantification and the implication operator. Marek et al. (2008) mention a solution based on set operations over multiple queries. This "allows to express queries which need a universal quantifier if expressed in a single query". Unfortunately the referenced term paper is not available online.
$^{14}$ Designed for ICE-GB, the British component of the International Corpus of English (Nelson et al., 2002).
15Described by Wallis and Nelson (2000) as the 'get me something like that' query method.
16 Visual Query Tool for Syntactically Annotated Corpora 17 Consisting of the same quantifier-free subset of first-order logic, but different precedence definition of internal nodes (cf. Steiner and Kallmeyer (2002) and Clematide (2015)).
6309
KWIC result view with regex and tag search and applies it to parallel corpora as targets.
The CorpusSearch (Taylor, 2003; Randall, 2008) command line tool for phrase-based syntax expects tree search configurations provided via query files with a boolean query language over a variety of tree predicates and regular expressions. Limitations on disjunction and negation and lack of quantification[18] make it slightly less expressive.
With full first-order logic the Finite Structure Query (FSQ) tool by Kepser (2003) offers access to the complete TIGER model, including arbitrary secondary edges and support for regular expressions in a graphical user interface (GUI). It is however limited to rather small corpora due to poor scalability of the query evaluation process.[19]
To access multi-modal and highly cross-annotated data in the NITE Object Model Library (Carletta et al., 2003), Evert and Voormann (2002) specified the NITE Query Language (NiteQL) based on MATE. Information from various segmentation levels can be extracted and combined in a logic-style language, including limited quantification. To honor the nature of multi-modal data they also propose a level of "fuzziness" for time operators with a configurable fuzziness interval.
Based on the MdF (Monads-dot-Features) Database and its query language QL by Doedens (1994), Emdros (Petersen, 2004) implements a text database for annotated texts. Its query syntax uses bracket nesting to express hierarchical relations and it surpasses TIGERSearch in several aspects of expressiveness, e.g. existential negation<sup>20</sup>.
While previously mentioned query systems were either freely available or bound to the licensing model of associated corpus resources (e.g. ICE-CUP), the popular Sketch Engine (Kilgarriff et al., 2004) commercialized<sup>21</sup> corpus management and exploration in a web-based platform (Kilgarriff et al., 2014). Extending the CQP, its own query language CQL offers efficient access to corpora available on the platform (Jakubíček et al., 2010).
Around the same time ANNIS was published
(Dipper and Götze, 2005) and started a successful ecosystem with the corpus metamodel SALT, the converter framework PEPPER and ANNIS itself as search module with its query language AQL. AQL is a very expressive query language on top of the graph-based model of SALT and an extension of the TIGERSearch syntax. Notable improvements over TIGERSearch are the access to concurrent annotations for the same layers, a rich set of segment relations to choose from and the generalization of directed relations in a query to be applicable for any type of edge in the corpus graph (e.g. syntax, coreference or alignments in parallel corpora). Queries in ANNIS can be constructed textually or graphically in a browser environment. It has been under continuous development for about 15 years now (Zeldes et al., 2009; Krause and Zeldes, 2014), resulting in the richest collection of result visualizations available in any corpus query system.
The Linguist's Search Engine (LSE) (Resnik and Elkiss, 2005) applies the query-by-example concept in a browser-based setting: A user provides a natural language example containing the desired phenomenon and receives a parse tree usable for querying. Relaxation or removal of constraints from this tree then yields increasingly generalized instances from built-in or custom collections[22].
The emergence of XPath $^{23}$  as a way of querying the tree-structure of various XML-based corpora offered new directions for corpus query languages. Bird et al. (2006) introduced LPath as an extension of XPath to overcome its limitations regarding the lack of expressible horizontal relations, a feature crucial for querying linguistic data. A later extension turned it into a first-order complete variant named LPath $^{+}$  (Lai and Bird, 2005).
Faulstich et al. (2006) also used an extension of XPath called DDDQuery to query complex annotation graphs of historical texts $^{24}$ . While using a RDBMS as backend, they do not directly translate queries into SQL. Instead user queries are first transformed into a first-order logic intermediate representation which in turn is translated into SQL.
The Prague Dependency Treebank (PDT) (Hajic et al., 2001; Hajic, 2006) is a richly annotated corpus. Its unique characteristic is a tectogram
22The "Getting Started Guide" (http://hdl.handle.net/1903/1324) for LSE mentions TGrep2 as the search component. In Resnik and Elkiss (2005) this information is missing and the screenshots do not show textual TGrep queries anymore, so the actual query evaluation backend is unknown.
$^{23}$ https://www.w3.org/TR/xpath
$^{24}$ http://www.deutschdiachrondigital.de/
18The way negation on arguments to search-function calls is handled allows to express certain quantified relations though.
19The author of FSQ discusses those limitations in (Kepser, 2004) and proposes a solution based on monadic second-order logic which was later implemented in MonaSearch.
20See Petersen (2005) for a brief comparison of the two systems including benchmarks on example queries.
21An open-source part under the label NoSketch Engine with the Manatee backend for indexing and search is also available at https://nlp.fi.muni.cz/trac/noske.
6310
matical layer which also includes annotations for coreference, deep word order, topic and focus. To provide users with adequate tools for access to this complexity, NetGraph (Ondruška et al., 2002; Mírovský, 2006) allows creation of tree queries for various layers both textually and graphically.[25]
Stockholm TreeAligner (Lundborg et al., 2007; Marek et al., 2008) continues the trend of extending the TIGERSearch language and applies it to parallel corpora. Its main improvement is the (re)introduction and implementation of universal quantification to overcome this central weakness.
Classic query tools for text corpora such as CQP lack the ability to efficiently deal $^{26}$  with common features of annotations for morphologically rich languages, such as positional tagsets or non-disambiguated annotation instances. POLIQARP $^{27}$  (Przepiórkowski et al., 2004; Janus and Przepiórkowski, 2007) is an indexer and query tool loosely based on the CQP approach with a client-server architecture and a variety of available client implementations. Initially targeted towards rich word-level annotations, such as in the IPI PAN Corpus (Przepiórkowski, 2004), it was later extended to also cover syntactic-semantic treebanks.
What's wrong with my NLP? by (Riedel, 2008) is primarily meant as a visualization tool with the ability to highlight differences between two concurrent dependency annotations (e.g. a gold standard and automatic predictions) with search options based on surface forms, tags and as a neat feature also including aforementioned diffs.
Maryns and Kepser (2009a) extended the expressiveness of FSQ to monadic second-order logic in MonaSearch. It features a GUI for viewing text-only "flat" results and defining queries of enormous expressiveness. However, due to the limitations of the underlying MONA framework (requiring binary tree structures), the system can only target collections of proper trees.
PML-TQ $^{28}$  (Pajas and Štepánek, 2009; Štepánek and Pajas, 2010) is effectively the successor of NetGraph, being designed to handle
the rich multi-level annotations in the PDT. Its graphical client $^{29}$  is directly integrated into the tree editor TrEd (Pajas, 2009) to support graphical query construction. Queries in PML-TQ are expressed as a mandatory selection part in bracket-syntax and an optional list of instructions to generate result reports. The latter of those two parts was groundbreaking in that it allows for an unprecedented freedom in selectively extracting information from any successful match during a search and creating various aggregations or statistics from it. Besides excellent result handling its query language is also quite powerful, including quantification and negation of sub-queries.
3.3 Third Generation - New Challenges
During the last decade the speed at which new query tools have been developed or published slowed down considerably. At the same time continued growth in size of corpus resources rendered some of the earlier approaches inapplicable (cf. (Kepser, 2004) for a discussion on the limitations of FSQ), calling for innovative alternatives. The three most common themes of this era were (i) scalability and adaptability of search backends to keep up with the explosive growth of corpora, (ii) reducing the barrier to entry for a wide(r) range of potential users and (iii) working towards unification or standardization of query languages.
$\mathbf{GrETEL}^{30}$  (Augustinus et al., 2012) is another implementation of the example-based search concept for the LASSY corpus (van Noord et al., 2013). Users provide sentences or example fragments and mark the areas of interest. Examples are then parsed, the subtrees for the specified part(s) of the input extracted and subsequently translated into XPath queries to run against the corpus in XML format. Further query options include the ability to specify whether or not pos, lemma or surface form of tokens in the subtree should be considered for the query. Since the user is effectively shielded from the tree representation and formal query formulation, GrETEL requires neither knowledge of an actual query language nor about the annotation scheme or underlying theories of the corpus.
Fangorn (Ghodke and Bird, 2012) addresses the challenge of querying treebanks too large to be loaded into memory, a scenario prohibitive for
29The modular architecture supports multiple scenarios, such as a client-server setup with an RDBMS backend or an integrated index-less query evaluator in Perl for local data.
36Greedy Extraction of Trees for Empirical Linguistics
[25] Besides NetGraph the tree visualizer and editor software TrEd (Pajas, 2009) also can be used to search in PDT and other tree structures via user macros defined in Perl. It does however not offer a query language for non-programmers.
26This does not imply their expressiveness being insufficient for this task, but rather that such queries can become quite bloated and their construction cumbersome for users.
$^{27}$ POLyinterpretation Indexing Query And Retrieval Processor
28 Prague Markup Language - Tree Query
6311