# Using FLOSS for Storing, Processing and Linking Corpus Data

Damir Mukhamedshin, Olga Nevzorova, Alexander Kirillovich

► To cite this version:

Damir Mukhamedshin, Olga Nevzorova, Alexander Kirillovich. Using FLOSS for Storing, Processing and Linking Corpus Data. 16th IFIP International Conference on Open Source Systems (OSS), May 2020, Innopolis, Russia. pp.177-182, 10.1007/978-3-030-47240-5_17. hal-03647263

HAL Id: hal-03647263

https://inria.hal.science/hal-03647263

Submitted on 20 Apr 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Using FLOSS for Storing, Processing and Linking
Corpus Data

Damir Mukhamedshin¹ [0000-0003-0078-9198], Olga Nevzorova¹ [0000-0001-8116-9446] and Alexander Kirillovich¹,² [0000-0001-9680-449X]

¹ Kazan Federal University, Kazan, Russia

² KazD JSCC RAS, Kazan, Russia

{damirmuh, onevzoro, alik.kirillovich}@gmail.com

**Abstract.** Corpus data is widely used to solve different linguistic, educational and applied problems. The Tatar corpus management system (http://corpus.antat.ru) is specifically developed for Turkic languages. The functionality of our corpus management system includes a search of lexical units, morphological and lexical search, a search of syntactic units, a search of N-grams and others. The search is performed using open source tools (database management system MariaDB, Redis data store). This article describes the process of choosing FLOSS for the main components of our system and also processing a search query and building a linked open dataset based on corpus data.

**Keywords:** corpus linguistics, corpus manager, linked open data

# 1 Introduction

In this paper, we discuss the development of the corpus management system for the Tatar National Corpus “Tugan Tel” [1] (Fig. 1). The main components of the corpus data management system are the following: Database, Search engine and User interface.

Development of a corpus data management system cannot begin without a clear un-
derstanding of which tools will be used to store and process corpus data. Therefore, one
of the important factors when choosing and using tools for data storing and processing
is for us their open source code, which is fundamental to ensure transparency and flex-
ibility of the tool.

When choosing tools for processing and storing corpus data, we were faced the task
of finding a set of FLOSS to ensure high speed of performing search queries (no more
than 0.1 seconds for direct search queries, no more than 1 second for reverse search
queries), wide search capabilities (at least direct and reverse search, search in parts of
word forms and lemmas, mixed search, phrase search), and the possibility of further
growth of system performance and functionality.
<figure><img src="image_2.png" /><figcaption>Fig. 1. The corpus manager GUI</figcaption></figure>

The general architecture of the corpus management system is represented at Fig. 2. This model includes three main components: a web interface, a search engine, and a database. Tatar texts with morphological markup of each token are loaded into the system database. Interaction between a corpus management system and users is performed through user queries.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Abstract structure of the corpus data management system</figcaption></figure>

The search results contain the set of sentences which include the tokens from user queries. We also make the corpus dataset built according to the standards of the LLOD project.
## 2 Using FLOSS in Corpus Management System

To choose FLOSS for storing and processing of corpus data, we formed a list of the most important selection criteria:

1. Performance. The solution should produce an average of at least 10 search operations per second.

2. Functionality. The solution should provide a possibility of direct and reverse search by wordforms and lemmas, search by parts of wordforms and lemmas, and phrase search.

3. Compatibility with other software. FLOSS for data storage should work with FLOSS for data processing and vice versa.

4. Completeness of documentation and presence of a large community. Everyone should be able to continue its development, focusing on the documentation and the accumulated experience of the community.

5. Development prospects. The solution should develop and maintain modern technologies.

We analyzed information in public sources [4] and chose FLOSS according to criteria 2-5. The resulting FLOSS set is presented in Table 1.

<table><caption>**Table 1.** Chosen FLOSS for storing and processing corpus data.</caption><thead><tr><td>Name</td><td>Type and main features</td></tr></thead><tbody><tr><td>memcached<sup>1</sup></td><td>Storage system of keys and values in memory</td></tr><tr><td>memcacheDB<sup>2</sup></td><td>Distributed key-value storage system. It supports transactions and replication</td></tr><tr><td>Redis<sup>3</sup></td><td>Non-relational distributed data storage system. It allows storing strings and arrays and making selections from them</td></tr><tr><td>FoundationDB<sup>4</sup></td><td>NoSQL database with a shared nothing architecture. It provides an ordered key-value store with transactions</td></tr><tr><td>Sphinx<sup>5</sup></td><td>Search system consisting of an indexer and a full-text search module</td></tr><tr><td>Elasticsearch<sup>6</sup></td><td>Distributed RESTful search engine. It allows making full-text search for structured data and performing complex search queries</td></tr><tr><td>MySQL<sup>7</sup></td><td>Relational DBMS</td></tr></tbody></table>

<footnote n="1">¹ https://github.com/memcached/memcached</footnote>

<footnote n="2">² https://github.com/LMDB/memcachedb</footnote>

<footnote n="3">³ https://github.com/antirez/redis</footnote>

<footnote n="4">⁴ https://github.com/apple/foundationdb</footnote>

<footnote n="5">⁵ https://github.com/sphinxsearch/sphinx</footnote>

<footnote n="6">⁶ https://github.com/elastic/elasticsearch</footnote>

<footnote n="7">⁷ https://github.com/mysql/mysql-server</footnote>
To verify compliance with the first criterion, we conducted a series of experiments on writing and searching based on the generated data [3]. The best results were shown by the MySQL + Redis suite, which was chosen by us to store data in the corpus management system

To build the system based on the selected components, additional elements must be included. So, to bind the PHP interpreter and the Redis data warehouse, we use the PhpRedis extension. In order for the PHP interpreter and MySQL DBMS to work in tandem, we use the php-mysql package, namely the mysqlnd driver, which allows working with the DBMS using its own low-level protocol.

Thus, the scripts executed by the PHP interpreter allow operations with data both from the Redis data storage and from the MySQL DBMS. Performing operations in the order necessary to solve the tasks assigned to the corpus data management system, PHP scripts are the link between indexes and cached data stored in Redis storage, index tables and text data stored in MySQL DBMS.

Let us consider how the processing of a simple direct search query *alma* *tat* / *apple* *en* is executed. The process execution of such query is shown in Fig. 3.

First of all, the script that performs this task searches the identifier of the word-form *alma* in the Redis database, by making a request using the Redis extension. Already at this point, the script may report an error in the query, if the identifier of the desired wordform is not found in the corpus data.

In the second step, the script makes a request to the MySQL database using mysqlnd. In this case, the script needs to get a list of sentences containing the desired wordform. This data is stored in the index table of corpus data. In response, the script receives a list of context (sentence) identifiers, according to which in the third step the required number of sentence texts is requested. The received data is displayed to an user in the form of an HTML document or a JSON structure.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Process execution of search query *alma*</figcaption></figure>
## 3 Building a Linked Open Dataset Based on Corpus Data

The corpus has been published on the Linguistic Linked Open Data cloud [4]. The dataset is represented in terms of NIF [5], OntoLex-Lemon [6], LexInfo, OLIA [7] and MMoOn [8] ontologies.

<figure><img src="image_5.png" /><figcaption>Fig. 4. LLOD publishing platform</figcaption></figure>

LLOD publishing platform consists in the following open-source components (Fig. 4). The RDF is stored in the OpenLink Virtuoso triplestore. The dataset is available via deferrable URI's and SPARQL endpoint. Deferrable URI's are accessible throws the LodView RDF browser, based on the Apache Tomcat webserver. The query interface is powered by YASQE [8].

## 4 Conclusion

The solutions presented in this article are applied in the developed corpus management system. Using FLOSS significantly reduced the development time and ensured the transparency of processes, flexibility, and possibility of in-depth analysis, as well as opportunities for further development.

The corpus data management system is used to work with Tatar texts. The total volume of the collection of Tatar texts is about 200 million wordforms. The average execution time of the direct search query does not exceed 0.05 seconds in 98% of cases, and the reverse search is performed by the system within 0.1 second in 82% of cases, which exceeded the expected system performance. In many ways, such performance is provided by FLOSS, used for data storage and processing. Also, thanks to FLOSS, search capabilities were expanded in the system. The search using category lists, the complex search using logical expressions, the search for named entities and other functions were added to the advanced version of the system.

**Acknowledgements.** The work was funded by Russian Science Foundation according to the research project no. 19-71-10056.
References

1. Suleymanov, D., Nevzorova, O., Gatiatullin, A., Gilmullin, R., Khakimov, B.: National corpus of the Tatar language “Tugan Tel”: grammatical annotation and implementation. C. Vargas-Sierra (ed). Selected Papers from the 5th International Conference on Corpus Linguistics (CILC2013) [Special issue]. Procedia-Social and Behavioral Sciences, (95), 68–74 (2013). doi:10.1016/j.sbspro.2013.10.623.

2. Katkar, M., Kutchhii, S., Kutchhii, A.: Performance Analysis for NoSQL and SQL. International Journal of Innovative and Emerging Research in Engineering, 2(3), 12–17. (2015).

3. Mukhamedshin, D., Suleymanov, D., Nevzorova, O.: Choosing the Right Storage Solution for the Corpus Management System (Analytical Overview and Experiments). In: Bouhlel M. and Rovetta S. (eds). Proceedings of the 8th International Conference on Sciences of Electronics, Technologies of Information and Telecommunications (SETIT’18), Vol. 1. Smart Innovation, Systems and Technologies, vol. 146, pp. 105–114. Springer, Cham (2019). doi:10.1007/978-3-030-21005-2_10.

4. Cimiano, P., Chiarcos, C., McCrae, J.P., and Gracia, J.: Linguistic Linked Open Data Cloud. In: Cimiano, P., et al. (eds.) Linguistic Linked Data: Representation, Generation and Applications, pp. 29–41. Springer (2020). https://doi.org/10.1007/978-3-030-30225-2_3.

5. Hellmann, S., Lehmann, J., Auer, S., and Brümmer, M.: Integrating NLP using Linked Data. In: Alani, H., et al (eds.) Proceedings of the 12th International Semantic Web Conference (ISWC 2013), Part II. Lecture Notes in Computer Science, vol. 8219, pp. 98–113. Springer, Berlin, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41338-4_7.

6. McCrae, J.P., Bosque-Gil, J., Gracia, J., Buitelaar, P., and Cimiano, P.: The OntoLex-Lemon Model: Development and Applications. In: Kosem I., et al. (eds.) Proceedings of the 5th biennial conference on Electronic Lexicography (eLex 2017), pp. 587–597. Lexical Computing CZ (2017).

7. Chiarcos, C.: OLIA – Ontologies of Linguistic Annotation. Semantic Web 6(4), 379–386 (2015). https://doi.org/10.3233/SW-140167.

8. Klimek, B., Arndt, N., Krause, S., Arndt, T.: Creating Linked Data Morphological Language Resources with MMoOn - The Hebrew Morpheme Inventory. In: Calzolari N., et al. (eds.) Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC’16), pp. 892–899. ELRA, 2016.

9. Rietveld, L., Hoekstra, R.: The YASGUI Family of SPARQL Clients. Semantic Web, 8(3), 373-383 (2017). https://doi.org/10.3233/SW-150197.