# Encoding language diversity in TEI: a description of regional and non-standard languages in multilingual diachronic corpora

Juliette Janes, Rachel Bawden, Thibault Clérice, Rasul Dent, Lucence Ing, Oriane Nédey, Benoît Sagot

► To cite this version:

Juliette Janes, Rachel Bawden, Thibault Clérice, Rasul Dent, Lucence Ing, et al.. Encoding language diversity in TEI: a description of regional and non-standard languages in multilingual diachronic corpora. TEI Conference 2025, TEI Consortium, Sep 2025, Cracovie, Poland. hal-05249807

HAL Id: hal-05249807

https://hal.science/hal-05249807v1

Submitted on 11 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Encoding language diversity in TEI: a description of regional and non-standard languages in multilingual diachronic corpora

Juliette Janès<fnref n="1" />, Rachel Bawden<fnref n="1" />, Thibault Clérice<fnref n="1" />, Rasul Dent<fnref n="1" />, Lucence Ing<fnref n="1" />, Oriane Nédey<fnref n="1" />, Benoît Sagot<fnref n="1" />

¹ Inria, Paris, France

## Keywords

TEI ODD, fine-grained language description, non-standard languages, multilingual corpora

## Abstract

Currently, French minority and regional languages lack digital resources, especially for cultural preservation and equitable representation. With the emergence of new tools for speakers, it is crucial to build stable corpora for under-resourced languages. In France, initiatives like DIVITAL<fnref n="1" /> and the earlier RESTAURE project<fnref n="2" /> have developed resources for specific languages, but much remains to be done.

To address this gap, the COLaF<fnref n="3" /> project aims to develop corpora and Natural Language Processing (NLP) tools for the languages of France, spanning different languages and dialects, time periods, geographic regions and social contexts. Even within one language, variation happens depending on temporal, geographical and sociolinguistic factors (Labov, 1989), especially for non-standard languages. While NLP increasingly addresses linguistic diversity (Broeder, 2006) and TEI elements exist for this purpose, to our knowledge, no previous project has tackled the issue of a standardized structure for linguistic variation and its associated parameters in TEI corpora.

<footnote n="1">¹ https://divital.gitpages.huma-num.fr/fr/</footnote>

<footnote n="2">² https://restaure.unistra.fr/</footnote>

<footnote n="3">³ COLaf ('Corpus et Outils pour les Langues de France) 'Corpus and Tools for the Languages of France'), is an ongoing project funded by Inria to contribute to the development of free corpora and tools for French and other languages of France. For further information see: https://colaf.huma-num.fr/en/</footnote>
We propose a TEI schema to enable the fine-grained description of language variation, with a
particular focus on non-standard languages.<fnref n="4" />,<fnref n="5" /> We use a combination of <language>, <particDesc>
and <settingDesc> elements to document the manifestation of a language as it is spoken in a specific
place, at a specific time and within a particular context. As BCP47 does not include all the 'regional
languages' we would like to work on, we use Glottolog identifiers (Nordhoff, 2011), for a wider range
of languages. We will later propose various language classifications to browse the corpus to address
the multiple representations of languages and dialects. We specify both elements to include detailed
metadata on the language, individuals associated with the document, and collection location, using
controlled vocabularies where possible<fnref n="6" /> (Figures 1, 2 and 3). The metadata is linked to the text via
attributes @who, @corresp, @hand, and @xml:lang (Figure 4).

<footnote n="4">⁴ The ODD schema and document is available here: https://github.com/DEFI-COLaF/metadata</footnote>

<footnote n="5">⁵ The ODD has been already used for various multi-speaker corpora such as the Molye
(https://github.com/DEFI-COLaF/Molye), Concours Picard
(https://github.com/DEFI-COLaF/Datasets_text/tree/main/Picard_Concours) and Forum Occitania
corpora.</footnote>

<footnote n="6">⁶ For example, Glottolog identifiers for the language id, Geonames identifiers (https://www.geonames.org/) for the place id, etc.</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1. Example of a content for a `<language>` element</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 2. Example of a content for a `<person>` element</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 3. Example of a content for a `<place>` element</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4. Example of a linked content</figcaption></figure>
Bibliography

Broeder, Daan, et Peter Wittenburg. « The IMDI metadata framework, its current application and future direction ». *Int. J. Metadata Semant. Ontologies* 1, n° 2 (1 octobre 2006): 119-32. https://doi.org/10.1504/IJMSO.2006.011008.

Labov, William. *Sociolinguistique*. Traduit par Alain Kihm. 1 vol. Le sens commun. Paris: Minuit, 1989.

Nordhoff, Sebastian, et Harald Hammarström. « Glottolog/Langdoc: Defining Dialects, Languages and Language Families as Collections of Resources ». In *Proceedings of the First International Workshop on Linked Science* 2011, 783:7. Bonn, Germany: CEUR, WS, 2011. https://ceur-ws.org/Vol-783/paper7.pdf.

TEI Consortium, eds. "2.4.2. Language Usage" TEI P5: Guidelines for Electronic Text Encoding and Interchange. [2025-01-24]. https://tei-c.org/release/doc/tei-p5-doc/fr/html/HD.html#HD41 (2025-04-07)

TEI Consortium, eds. "16.2 Contextual Description" TEI P5: Guidelines for Electronic Text Encoding and Interchange. [2025-01-24]. https://tei-c.org/release/doc/tei-p5-doc/en/html/CC.html#CCAH (2025-04-07)
## About the authors

**Juliette Janès** is a R&D engineer in the ALMANaCH team at Inria, Paris. Her current work focuses on corpus production and the development of NLP tools for the COLaF project on languages of France.

**Rachel Bawden** is a Researcher at Inria in the ALMANaCH lab in Paris. She is a holder of a "springboard" chair in the PRAIRIE institute, and her research focus is machine translation and low-resource scenarios.

**Thibault Clérice** is a Researcher at Inria in the ALMANaCH lab in Paris. His research focuses on digital and computational humanities for classical and medieval periods.

**Rasul Dent** is a doctoral student at Inria in the ALMANaCH lab in Paris. His work focuses on language identification for historical and contemporary languages of France, and especially French-based Creoles.

**Lucence Ing** is a research engineer in the ALMANaCH team at Inria Paris, specializing in computational linguistics for medieval French. She holds a PhD on lexical obsolescence in medieval French and is the project manager of the COLaF Project.

**Oriane Nédey** is currently a PhD candidate in Computer Science at Inria Paris (team ALMANaCH, Défi COLaF). Her thesis project focuses on machine translation for low-resource dialectal variants, especially dialect continua, with Occitan as principal use case.

**Benoît Sagot** is a Senior Researcher at Inria, where he heads the ALMANaCH lab in Paris. He holds a chair at the PRAIRIE institute and was the 2023–2024 holder of the Collège de France chair of computer science. His research focuses on natural language processing and computational linguistics.