SlowGuess/ACL_2020_OCR huggingface dataset:
query tools with custom evaluation engines. They use Apache LUCENE $^{31}$  in a client-server setup to manage large numbers of phrase structure trees. Its query language follows the LPath scheme but lacks regular expressions support on label content.
Unlike the majority of other systems in recent years, we developed ICARUS $^{32}$  (Gartner et al., 2013) as a standalone desktop application for visualization and example-based search $^{33}$  with a custom query evaluation system and no indexing or dependency on another database technology. Initially designed for querying dependency treebanks it underwent multiple extensions to make it compatible with annotations for coreference (Gartner et al., 2014) and prosody $^{34}$  (Gartner et al., 2015) and also to incorporate automatic error mining as a means of exploration (Thiele et al., 2014). Its bracket-style query language is similar to PML-TQ but lacks quantifiers and a dedicated section for result preparation instructions. While queries can be defined both textually or graphically, the preferred way is to use the graphical query editor that also provides contextual help for getting started easily.
CLARIN Federated Content Search $^{35}$  (CLARIN-FCS) is a successful example of unifying query access to multiple distributed corpus resources hosted by different parties and with diverse native query frontends. Its query language FCS-QL is heavily based on POLIQARP but also only meant to cover a small intersection of the expressiveness of common corpus query tools.
On the level of standardization CQLF $^{36}$  (Banski et al., 2016) provides an initiative that aims at providing means for comparability and interoperability of corpus query languages. In its first phase $^{37}$  CQLF-1 defines classes and features for the description of query languages for single-stream data.
A unified serialization format for CQLF-1 is available with KoralQuery (Bingel and Diewald, 2015), a JSON-LD based and theory-neutral cor
pus query protocol. It serves as the internal query representation $^{38}$  of  $\mathbf{KorAP}^{39}$  (Banski et al., 2014; Diewald et al., 2016), the designated successor of COSMAS II. While CLARIN-FCS multiplexes a query defined in a common (limited wrt expressiveness) query language to multiple query processors, KorAP lets the user choose up-front among several query languages $^{40}$  that all can be processed by the system in a microservices architecture $^{41}$ .
Similar to Fangorn, SETS $^{42}$  (Luotolahti et al., 2015) is geared towards very large treebanks, this time targeting dependency syntax with a query language inspired by TRegex $^{43}$ . It is browser-based with a RDBMS backend and uses an elaborate query evaluation process: SETS generates and compiles optimized code for matching tokens for each query and only retrieves the minimal token sets from the database needed for evaluating a query.
Multilingwis $^{44}$  (Clematide et al., 2016) provides exploration in multiparallel corpora (Graen et al., 2016). Focused on result presentation and reducing the required expert knowledge, it simplifies the process of finding translation variants.
Other notable events in this time period include the modernization of CQP "for the new millennium" (Evert and Hardie, 2011) and the introduction of graphANNIS (Krause et al., 2016), a graph database backend for ANNIS3 as an alternative to the former RDBMS-based relANNIS.
4 Technological Alternatives
Many of the systems we presented in Section 3 use various forms of database technology as their storage or evaluation backend. Typically every such database or information management system already ships with its dedicated query language, such as SQL for RDBMSs, SPARQL for the RDF format, XPath and XQuery for XML documents, CYPHER for Neo4j and other graph-based databases or Apache LUCENE with its own query dialect for accessing the text database.
31https://lucene.apache.org/
$^{32}$ Interactive Platform for Corpus Analysis and Research, University of Stuttgart
An integrated interface for plugging in dependency parsers allows users to generate parses for example sentences that can then be converted into queries and relaxed iteratively.
34With various similarity measures usable for expressing query constraints based on the PaIntE model by Mohler (2001)
35https://www.clarin.eu/content/ content-search
$^{36}$ Corpus Query Lingua Franca. Part of ISO TC37 SC4 Working Group 6 (ISO 24623-1:2018).
$^{37}$ CQLF is an ongoing long-term effort, with CQLF-2 currently being worked on at the stage of a committee draft.
38The high level of abstraction it implements and the ver-. bosity required to express simple queries combined with JSON syntax results in limited human readability.
$^{39}$ Korpusanalyseplattform der nachsten Generation ("Corpus analysis platform of the next generation")
40At the time of writing it supports the following query languages: Poliqarp, FCS-QL, AQL, CQP 1.2, COSMAS II
$^{41}$ KorAP builds on a variety of (storage) technologies, including several RDBMS variants, LUCENE and also the graph database Neo4j (http://neo4j.com/).
42 Scalable and Efficient Tree Search
43A "Tree regular expression" language in TGrep2 style
44 Multilingual Word Information System
6312
This does of course prompt the question on the necessity of developing dedicated corpus query languages when more often than not the actual query evaluation is just offloaded to an existing database technology. Already Jarke and Vassiliou (1985) mentioned a plethora of (technical) factors to be considered when deciding on a (database) query language. Mueller (2010) on the other hand takes the perspective of scholarly users, providing arguments especially targeting the aspects of usability from a humanistic point of view, describing the handling of search results as "Achilles heel of corpus query tools". Having previously examined those factors in (Gartner and Kuhn, 2018), we also agree on the continuing necessity of dedicated corpus query systems and query languages to bridge the gap between formal/technical expressiveness and the usability factors decisive for corpus users. Especially future directions as the ones we propose in Section 7 demand architectures that are more complex than the mere translations of data and queries.
There have however also been approaches or use case analyses to completely store and query linguistic corpora with OWL (Burchardt et al., 2008), XQuery (Cassidy, 2002) or a via RDBMS (e.g. content of the DIRNDL corpus (Eckart et al., 2012) in its entirety has for a long time only been available through direct SQL queries), but historically speaking those cases generally represent a minority.
A lot of work has been invested already into laying the theoretical foundations for various aspects of and approaches to corpus querying, as well as into evaluating and comparing existing query systems. We distinguish between three types of contributions, namely (i) requirement analyses, (ii) evaluations of individual query languages or approaches and (iii) actual performance comparisons between multiple systems (feature-based or benchmarks).
Several contributions listing requirements for corpus query systems have been previously mentioned in Section 4. In addition, Mirovský (2008) provides a list of required language features for querying PDT and Lai and Bird (2004) do so for treebanks in general, specifically related to navigation, closures over relations and going "beyond ordered trees" in order to query more complex structures. This list of functional requirements is later extended on in Lai and Bird (2010) with features such as temporal organization and non-navigational
requirements. While not exclusive to corpus query systems, technical aspects related to feasibility (e.g. scalability or computational complexity) or long-term maintainability (e.g. interoperability and extensibility) are also frequently emphasized by Lai and Bird (2004), Kepser (2003) and others. Besides the usability-focused scholarly position of Mueller (2010) around aspects of answer time, maintenance cost and the management of search results, we previously discussed additional non-technical requirements related to the general readability or post-processing capabilities of a query language and its learnability in Gartner and Kuhn (2018), the latter being a crucial factor for achieving wide-spread use in humanistic fields.
Formal evaluations of query languages are somewhat rare, e.g. (Lai and Bird, 2010) for LPath and  $\mathrm{LPath}^+$ , (Kepser, 2004) for MonaSearch or in part (Kepser, 2003) for FSQ. Instead the vast majority of evaluations use example queries of varying complexity to compare different query languages or systems. Notable early work on query complexity was done by Lai and Bird (2004), comparing several query languages $^{45}$  based on a set of linguistic information needs of increasing complexity. The example queries they provide have proven to be a good baseline for comparing the capabilities of query languages and subsequently found their way into many later tool evaluations, such as (Petersen, 2006a) for Emdros or in Clematide (2015) when highlighting features of particular query languages. Yet another evaluation approach was used by Frick et al. (2012) when they applied the classes defined in CQLF-1 as evaluation criteria in the comparison of COSMAS II, POLIQARP and AQL.
Clematide (2015) provides a very thorough reflection and categorization of the various families of corpus query languages: text corpus, treebank, path-based $^{46}$  and logic-based. A point he makes that resonates well with other surveys is the importance of striking the right balance between usability and technical aspects in any practical situation.
In some cases actual performance benchmarks have been published, such as testing Emdros with different RDBMS backends (Petersen, 2006b),
45TRep2, TIGERSearch, Emu, CorpusSearch, NiteQL, LPath
46We argue for a more differentiated view on path-based query languages: While Clematide (2015) considers PML-TQ to be part of this family, we propose to move it together with ICARUS into a tree-based category of query languages, as their use of bracketed tree-expressions to describe structural relations represent a slightly different approach.
6313
comparisons between TIGERSearch and Emdros in Petersen (2005), MonaSearch and TIGERSearch in (Maryns and Kepser, 2009b) and Luotolahti et al. (2015) benchmarking SETS against ICARUS. However, due to the rapid change in technologies and the architectural differences between query systems, it tends to be very difficult to provide accurate and meaningful performance comparisons and readers are advised to carefully examine whether the reported use cases are applicable to their own.
6 Key Observations & Shortcomings
In this section we intend to condense some of our observations after analyzing a large number of query systems. We focus on the following two aspects suitable for pointing out challenges (stemming from past shortcomings) and motivating directions in development of future corpus query systems, protocols or architectures.
6.1 Shifting Design Goals
The different generations of corpus query systems listed in Section 3 are the results of design processes with generally very distinct goals. The first generation in Section 3.1 can be seen as the initial step to have some means of querying beyond the search functions of grep or any text editor.
Subsequently, the second time period described in Section 3.2 represents a general exploration phase: Approaches in almost every direction were implemented, either as proof of concept for new query features or to address very specific linguistic theories or phenomena. Many of those implementations however were not scalable to the degree demanded by the rapid growth $^{47}$  of corpora.
As such the general trend in Section 3.3 was to overcome those limitations and provide scalable systems with also increased usability. At the same time the overall expressiveness of query languages provided took a step backwards. Especially concepts like closures over relations, (universal) quantification or existential negation often got rationalized in favor of performance in younger systems. Our vision of a hybrid architecture sketched in Section 7 is intended to overcome those limitations by utilizing and combining the different strengths of systems involved (such as the robust performance of indexing systems and the expressiveness and flexibility of custom query evaluation engines).
6.2 Fragmentation & Limited Reusability
With the enormous amounts of resources that have been invested into creating this zoo of corpus query languages and systems, it is surprising how little reuse and unification has occurred over the years. We attribute this trend to a variety of frequently recurring factors, particularly the following:
- Due to the lack of standards regarding the categorization of expressiveness of query languages it has always been extremely difficult to determine whether an existing system could meat all the requirements a new project, user scenario or corpus resource posed, leading to redundancy.[48]
- The technological heterogeneity<sup>49</sup> involved also represented a major issue that only slowly is being overcome by the emergence of standards for corpus storage and interchange formats or the shift to more modular architectures such as microservices or plugin-engines, making it much easier to adapt a system to new requirements.<sup>50</sup>
- Especially early query systems often emerged as an interface for a very particular corpus, a specific format or to support the phenomena a certain project was interested in. As such, the limited resources typically available for short-term funded projects rarely allowed for extending previous monolithically designed work. Newly implemented (and often isolated) solutions focusing on a narrow selection of very specific query features or annotations were a common result.
7 The Final Frontier – An Outlook
With several dozens of systems contributing their individual variations, the pool of available corpus query tools and languages has become quite large. Navigating this ocean in order to find the right tool for the job and then learn to use it can already be as much effort as manually investigating the data at hand. Fortunately the CQLF standardization initiative aims at providing developers with the means of locating their tools on a map of query features, so that prospective users may find them without an odyssey. While this effort is still in an early stage, we are looking forward to having catalogs available
48 An aspect that CQLF is now addressing, removing the need of essentially reverse engineering a tool or studying its source code, as time constraints together with the lack of standardization often went along with poor documentation.
49Ranging from platform/language lock-ins to format/storage dependencies, often in a monolithic composition.
50Such as new query features, formats, storage/database solutions, standalone apps or various client-server architectures.
47Growth continually occurred both in size (number of primary units) and complexity (number of annotation layers).
in the not too distant future, allowing us to browse for query languages based on our individual information needs. However, many questions regarding the future of corpus querying still remain, two of which we consider of particular importance and will discuss in the following sections.
7.1 One Language to Query Them All?
Today we have a cluttered buffet of corpus query languages to pick from depending on our information needs. Interestingly they all share the pros and cons of being designed as formal languages with the goal of taciturnity, meaning that for the untrained eye they usually represent just a weird salad of letters and special characters.[51] This is particularly noteworthy, as all modern corpus query tools feature a rich GUI and could easily employ a more verbose query language while at the same time shield users from the time overhead when creating queries by clever auto-completion or recommendation functions.
Likewise, today's corpus queries are not self-contained to the level of for instance SQL queries, which are composed of dedicated parts for scope selection, actual constraints and result preparation. Usually only the constraint part is present in corpus query languages, with only a few exceptions  $^{52}$ , leaving additional configurations (result size limit, search direction, case sensitivity) exclusively to external components, such as the GUI, hampering the reproducibility of search results severely.
A fully self-contained and human-readable query protocol that can embed any existing query language and augment it with (boilerplate) statements to bind the query content to actual corpora and annotation layers, provide information about the query dialect and its version and store configuration and result preparation instructions, would go a long way towards unification and potential interoperability of corpus query systems.
7.2 Towards a Hybrid Architecture?
The typical architecture of corpus query systems today is a monolithic one and contains from bottom to top (i) a backend storage or custom data model,
(ii) a custom query evaluator or query interface to said backend and (iii) a query parser or translator to process the raw user query. Choices in technology or algorithms for (i) through (iii) definitively dictate the basic nature and structure of the information that can be queried. They usually make it very difficult, if not impossible, to implement changes or extensions retrospectively or from the outside. A strong dependency on indexing to access large corpora also presupposes a priori knowledge of what information is meant to be searchable, frequently confining corpus query tools to the role of being mere finding aids within a research process.
We would like to see them become true enablers instead, allowing queries to go far beyond of what a corpus has to offer with its bare annotations alone and for example include the following extensions to create more informed search solutions:
- Use knowledge bases and similar external resources to allow more generalized queries, e.g. "find verbal constructions containing a preposition in combination with some sort of furniture".
- Add (semantic) similarity measures (e.g. word embeddings) and other approaches for increased fuzziness to improve example-based search.
- Offer true scripting support for users to extent or customize the ability provided by a system. While this might affect performance in unpredictable and detrimental ways, raw (distributed) computing power and clever use of pre-filtering can offset the impacts on performance.
Naturally all of these proposed features (and especially the last one) require a drastically different and quite heterogeneous architecture. Taking the microservices approach of KorAP as an example, it is easy to imagine a hierarchically organized architecture of query translation and evaluation services working together (by partially answering queries, filtering the results or otherwise post-process them) to provide the optimal combination of freedom in expressiveness and performance guarantees. Space does not permit we provide a detailed description of such a hybrid approach. Instead we refer to (Gartner, to appear) for an overview of our ongoing efforts to design and implement a hybrid corpus query architecture and associated query protocol. Twenty years ago this might have seemed utterly unrealistic, but advances in information management systems and distributed computing certainly put this vision within technical reach.
$^{51}$ Kaufmann and Bernstein (2010) investigated the usability of natural language queries for interfaces to the semantic web with positive results. It would be interesting to see similar studies on corpus query interfaces.
52cf. PML-TQ for exemplary post-processing instructions, allowing to treat results as tabular data and to perform various transformation and aggregation operations on it, including textual reports.
Amaryllis. 2001. Amaryllis corpus - evaluation package. ELRA. ISLRN: 786-395-313-491-8.
Susan Armstrong-Warwick, Henry S. Thompson, David McKelvie, and Dominique Petitpierre. 1994. Data in your language: The ECI Multilingual Corpus 1. In Proceedings of the International Workshop on Sharable Natural Language Resources, Nara, Japan.
Liesbeth Augustinus, Vincent Vandeghiste, and Frank Van Eynde. 2012. Example-based treebank querying. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 3161-3167, Istanbul, Turkey. European Language Resources Association (ELRA). ACL Anthology Identifier: L12-1442.
Piotr Bański, Joachim Bingel, Nils Diewald, Elena Frick, Michael Hanl, Marc Kupietz, Piotr Pežik, Carsten Schnober, and Andreas Witt. 2014. KorAP: the new corpus analysis platform at IDS mannheim. Human language technology challenges for computer science and linguistics. 6th language & technology conference December 7-9, 2013, Poznan, Poland, pages 586 - 587, Poznan. Uniwersytet im. Adama Mickiewicza w Poznaniu.
Piotr Bański, Elena Frick, and Andreas Witt. 2016. Corpus Query Lingua Franca (CQLF). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA).
Michael Barlow. 2002. ParaConc: Concordance software for multilingual parallel corpora. In Proceedings of the Third International Conference on Language Resources and Evaluation. Workshop on Language Resources in Translation Work and Research, Las Palmas, Canary Islands - Spain, pages 20-24.
Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209-226.
Eckhard Bick. 2018. Arbobanko - A treebank for Esperanto. In Proceedings of CICLing 2018 - 19th International Conference on Computational Linguistics, Germany. Springer.
Joachim Bingel and Nils Diewald. 2015. *KoralQuery - A General Corpus Query Protocol*, volume 111, pages 1-5. Linköping University Electronic Press.
Steven Bird, Yi Chen, Susan B Davidson, Haejoong Lee, and Yifeng Zheng. 2006. Designing and evaluating an XPath dialect for linguistic queries. In 22nd International Conference on Data Engineering (ICDE'06), pages 52-52. IEEE.