LAUDATIO-Repository

Thomas Krause, Anke Lüdeling, Carolin Odebrecht, Laurent Romary, Peter Schirmbacher, Dennis Zielke

► To cite this version:

Thomas Krause, Anke Lüdeling, Carolin Odebrecht, Laurent Romary, Peter Schirmbacher, et al.  
LAUDATIO-Repository: Accessing a heterogeneous field of linguistic corpora with the help of an  
open access repository. Digital Humanities 2014, Jul 2014, Lausanne, Switzerland. hal-01574399

HAL Id: hal-01574399

https://inria.hal.science/hal-01574399

Submitted on 14 Aug 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire **HAL**, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Accessing a heterogeneous field of linguistic corpora with the help of the open access repository **LAUDATIO**

<figure><img src="image_1.png" /></figure>

Thomas Krause*, Anke Lüdeling*, Laurent Romary+, Peter Schirmbacher*, Carolin Odebrecht*, Dennis Zielke*
Humboldt-Universität zu Berlin*, Inria France+
www.laudatio-repository.org

Digital Humanities, Lausanne – Switzerland '14

DFG

Deutsche
Forschungsgemeinschaft

<figure><img src="image_2.png" /></figure>

Inria
INVENTEURS DU MONDE NUMÉRIQUE

How can we access and (re-)use
heterogeneous research data with the
help of an open access repository?

<figure><img src="image_3.png" /></figure>

## 1. LAUDATIO-Repository²¹

- display structured metadata, search for and download corpora without registration

- upload new annotations to existing corpora and upload new corpora

- based on the Fedora Repository Software⁴

- customized web interface

- faceted search with Elastic Search¹⁷

- automatic registration of PIDs for every version of a corpus¹⁸

- open access to corpora via CC-Licenses¹⁶

- connected to the search and visualization system ANNIS⁸ for corpus analysis

- metadata schema with TEI ODD³

- conversion between different formats with SaltNPepper⁷

## 2. Research data

Examples for corpus formats for editing and accessing data

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

Historical German corpora already included in the LAUDATIO-Repository

- Deutsche Diachrone Baumbank⁹

- Fürstinnenkorrespondenzkorpus¹⁰

- German Manchester Corpus¹¹

- Kasseler Junktionskorpus¹²

- Märchenkorpus¹³

- Referenzkorpus Althochdeutsch¹⁴

- Register in German Science¹⁵

- plenty of different formats and content standards are used

## 3. Metadata model for corpora

- structured metadata for corpora, documents and annotations metadata for textual corpora⁵,⁶

- used for searching and displaying corpora

- describes the corpus, the textual primary sources and the annotation

- specification with TEI XML + ODD³,⁴

- ODD customization and documentation and validation with RELAX NG XML syntax schema²⁰

- rich and structured metadata for heterogeneous data

<figure><img src="image_9.png" /></figure>

## 4. Specific corpus architecture – exemplified with KAJUK

- historical corpus of German ‘Nähesprache’¹ of 17th – 19th e.g. private letters

- syntactically annotated, e.g. cohesion and clauses

- token and span annotation, pointing annotation

### KAJUK Format

```xml
<lb n="28,3009">
  <KOR type="subj">Es</KOR>
  <praed><V ID="Fin"><VV>hatten</VV></V></praed>
  <FOK>auch</FOK>
  <subj>die Bätzgesdörfer</subj>
  eine Härt Schaff in unserm Dorf</lb>
  <line n="12"/>
  <lb n="28,29,3009">
    <J IR="kop"><KON>und</KON></J>
    <subj type="E" dir="V">die Bätzgesdörfer</subj>
    <praed><V ID="Fin"><VV>erhilten</VV></V></praed>
    sie mit Gottes Hulf
  </FOK>auch</FOK>
  < subj real="Pron">sie</subj>
  <praed><V ID="Fin"><VV>gaben</VV></V></praed>
  dem Hauptman darvon,</lb><!-hier line-->
  <line n="13"/>
```

- annotated in an idiosyncratic SGML style

- without tokenization

- the very same category such as 'subj' can be either annotated as an element or a value of an attribute

- mixture of unary and binary tags

- comments are annotation

- attribute values of 'lb' elements refer to each other

- documentation of both: the formats and their content in LAUDATIO

- LAUDATIO is open for every format and any content

- applicable for structured and unstructured formats

### KAJUK represented in relANNIS

<figure><img src="image_10.png" /></figure>

- tokenization of the corpus with a mapping of categories in token and span annotations

- renaming and structured grouping of attributes, element names and their values

- attribute values of 'lb' elements are realized as pointing relations

## 5. One metadata format for heterogeneous corpora

- import of the TEI XML metadata in the LAUDATIO-Repository

- configuration of the interface with the help of a JSON configuration file¹⁹

- description of the annotation guidelines (e.g. what is meant by 'subj')

- description of the type and preparation of the annotation in each format

- manual or automatic annotation, which annotation tool was used

- details (e.g. 'subj' is either element or value of attributes 'EB' or 'type')

```xml
<namespace name="subj" rend="Syntactical"
  xml:id="subj" corresp="Wortgruppenkategorie">
  <tagUsage gi="subj">Subjekt.</tagUsage>
</namespace>
```

- structured metadata for all parts of a corpus

```xml
<appInfo n="1" style="Manual">
  <application ident="XML" style="XML" version="1.0"
    type="XMLAnnotation" subtype="NA">
    <label>XML Editor</label>
    <p>XML Element 'subj'. Wert des XML Attribut 'type' und 'EB'.</p>
  </application>
</appInfo>
```

<figure><img src="image_11.png" /></figure>

LAUDATIO provides a open access platform for sharing historical corpora. Information about research on the existing corpora and their preparation helps you
- to either extend or build new corpora,
- to do your own research,
- to evaluate and re-use already existing, research and replicate results.

- Open Access and (Re-)use via structured access over all types of corpora

## 6. References

[1] Ágel, V., Hennig, M. (2007) DFG-Projekt "Explizite und elliptische Junktion in der Syntax des Neuhochdeutschen" Forschungsnotiz. In Zeitschrift für Germanistische Linguistik 35, 185-189. [2] Burnard, L., Bauman, S. (Ed.) (2008) TEI P5: Guidelines for Electronic Text Encoding and Interchange. Oxford. [3] Burnard, L., Rahtz, S. (2004) RelaxNG with Son of ODD. Extreme Markup Languages. [4] Lagoze, C., Payette, S., Shin, E., Wilper, C. (2006). Fedora: an architecture for complex objects and their relationships. International Journal on Digital Libraries, 6(2). 124-138. [5] Odebrecht, C. (2014) Modeling Linguistic Research Data for a Repository for Historical Corpora. Digital Humanities 2014 Conference. 8.7.-12.7.2014, Lausanne. [6] Odebrecht, C., Krause, Th. (2013) Metadata in an Infrastructure for Historical Corpora. SFB 732 Incremental Specification in Context. Kolloquium. 20.6.2013, Stuttgart. [7] Zipser, F., Romary, L. (2010) A model oriented approach to the mapping of annotation formats using standards. In Workshop on Language Resource and Language Technology Standards, LREC 2010. La Valette, Malta. [8] Zeldes, A., Ritz, J., Lüdeling, A., Chiarcos, C. (2009) ANNIS: A search tool for multi-layer annotated corpora. In Proceedings of Corpus Linguistics 2009. Liverpool. [9] DDB http://hdl.handle.net/11022/0000-0000-2107-3 [10] Fürstinnenkorrespondenzkorpus http://hdl.handle.net/11022/0000-0000-20A6-0 [11] GerManc http://hdl.handle.net/11022/0000-0000-24E3-7 [12] KAJUK http://hdl.handle.net/11022/0000-0000-2102-8 [13] Märchenkorpus http://hdl.handle.net/11022/0000-0000-1F5B-9 [14] DDD-AHD http://hdl.handle.net/11022/0000-0000-1F9C-F [15] RIDGES http://hdl.handle.net/11022/0000-0000-2106-4 [16] Creative Commons-Licenses https://creativecommons.org/licenses/ [17] ElasticSearch http://www.elasticsearch.org/ [18] Handle-PIDs, Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen http://epic.gwdg.de/wiki/index.php?title=EPIC:API:v2:contribution [19] JSON http://json.org/ [20] RELAX NG http://relaxng.org/ [21] LAUDATIO stands for Long-term Access and Usage of Deeply Annotated Information; Technical documentation LAUDATIO-Repository http://rtd.cms.hu-berlin.de/docs/laudatio-repository-documentation/
LAUDATIO-Repository

Accessing a heterogeneous field of linguistic corpora with the help of an open access repository

Thomas Krause, Anke Lüdeling, Carolin Odebrecht, Laurent Romary, Peter Schirmbacher,

Dennis Zielke

Humboldt-Universität zu Berlin

An open access to digital historical research data for historical linguistics enables a fruitful exchange of research sources and research methods. To achieve this goal the LAUDATIO-Repository provides a long-term open access to historical corpus linguistic data. By developing the LAUDATIO-Repository we also want to explore how to build repositories that are useful for a set of well defined communities but are also flexible enough to be used and extended to serve other communities not considered beforehand. Considering the user community's needs requires a clear understanding of the community's user scenarios and research.

Digitalization and annotation of historical texts for linguistic purposes is a methodically and technologically challenging task that consumes a lot of time and resources. Linguistic analysis uses various annotations and methods such as multiple segmentations (Krause et al. 2012), normalizations (Bollmann, Petran & Dipper 2011), syntax trees (Nivre 2008), dependency annotations (Nivre et al. 2007) and (semi-) automatic consistency checks (Dickenson & Meurers 2003). Having the chance to either look up these methods and their results, the corpus itself, or to re-use existing corpora for further research may facilitate and enrich the research methods and possibilities of the linguistic community. Via an extensive documentation of the whole preparation phase including annotation, tools and the resulting data allows a uniform and structured access to such a heterogeneous field of research data and thereby a first establishment of best practice standards.

There are many tools and formats that are used to annotate and process historical corpora, e.g. for token based annotations and parses (see e.g. Schmidt 2009, Brugman & Russel 2004). That's why the repository is open to all formats. The repository is based on Fedora 3.6 (Lagoze et al. 2006) and a self-developed web interface that uses its API. Each format of a corpus is stored as whole in a separate binary data stream. The collection of data streams at a certain point in time form the corpus object. This mechanism is flexible enough to cover the variety of existing annotation tools such as EXMARaLDA, @annotate or ELAN and it can also cover new formats (even future ones not invented yet).

We also developed an unified meta-model for (historical) text corpora which is powerful enough to cover a heterogeneous field of corpus linguistic data and can be modified according to future developments in the research field<fnref n="1" />. For each corpus in the repository metadata in this meta-model must exist and on each import the metadata is automatically validated against the scheme. To enable a flexible modeling we chose a customization of TEI xml with the help of an ODD specification. The meta-model indirectly refers to basic concepts for using corpora not only

<footnote n="1">¹ The current documentation of the metadata can be accessed under http://korpling.german.hu-berlin.de/schemata/laudatio/doc/S6/corpus/</footnote>
in the repository: A corpus is defined by the sum of documents, the actual (historical) texts. A document is the sum of all applied annotations. The mapping of this meta-model and the TEI header structure takes the user scenarios designed for the LAUDATIO-Repository into consideration: Every corpus need a structured and uniform documentation which comprises information about the corpus creation, the texts used for the corpus and every kind of annotation which is applied. Crucial for the reasons given above, the whole preparation process is covered as well.

In order to assist re-usage of existing corpus resources and minimize duplicate work researchers are able to find, download and import corpora on their own. The detailed metadata does not only support understanding the corpus, it is also used for searching for corpora in a structured manner and allows searching for corpora that are already suited for their research question or are a good base for further annotations. Thus, users are able to search for certain annotation methods, tools or annotation values. We use the metadata to provide a faceted search (see Tunkelang 2009 for an extensive discussion) using ElasticSearch<fnref n="2" /> as a technical backend.

Via a flexible technical basis system and an extensive corpus documentation which is enable a structured view and search, an access to a heterogeneous field of historical corpora is provided by the LAUDATIO-Repository which will be presented in the demo session.

## References

Bollmann, Marcel, Petran, Florian, Dipper, Stefanie (2011) Rule-Based Normalization of Historical Texts. *Proceedings of Language Technologies for Digital Humanities and Cultural Heritage Workshop Hissar, Bulgaria, 16 September 2011*. pp. 34–42.

Brugman, Hennie, Russel, Albert (2004). Annotating Multimedia/ Multi-modal resources with ELAN.In: *Proceedings of LREC 2004, Fourth International Conference on Language Resources and Evaluation*.

Dickinson, Markus, Meurers, Detmar (2003) Detecting Errors in Part-of-Speech Annotation. *Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics (EACL-03)*. Budapest, Hungary.

Krause, Thomas, Lüdeling, Anke, Odebrecht, Carolin, Zeldes, Amir (2012) Multiple Tokenization in a Diachronic Corpus. *Exploring Ancient Languages through Corpora Conference (EALC), 14.-16.Juni 2012*.

Lagoze, Carl, et al. "Fedora: an architecture for complex objects and their relationships." *International Journal on Digital Libraries* 6.2 (2006): 124-138.

Nivre, Joakim (2008) Treebanks. *Corpus Linguistics: An International Handbook*. Vol. 1. Berlin: Mouton De Gruyter, 2008. pp. 225-41.

<footnote n="2">² ElasticSearch website, http://www.elasticsearch.org, accessed 29 Oct. 2013</footnote>
Nivre, Joakim, Hall, Johan, Nilsson, Jens, Chanev, Atanas, Eryigit, Gülsen, Kübler, Sandra, Marinov, Svetoslav, Marsi, Erwin (2007) MaltParser: A language-independent system for data-driven dependency parsing. *Natural Language Engineering* 13/02. pp. 95-135.

Schmidt, Thomas (2009) Creating and Working with Spoken Language Corpora in EXMARaLDA. In: Lyding, V. (ed.) *LULCL II: Lesser Used Languages & Computer Linguistics II*. pp. 151-164.

Tunkelang, Daniel (2009) Faceted search. *Synthesis Lectures on Information Concepts, Retrieval, and Services* 1.1. pp. 1-80.