<figure><img src="image_1.png" /></figure>

HAL
open science

International standards for the identification and the
description of languages and their varieties

Laurent Romary

► To cite this version:

Laurent Romary. International standards for the identification and the description of languages and their varieties. Harmonizing language data, De Gruyter, pp.35-60, 2025, 9783119148023. 10.1515/9783112208212-003 . hal-05415482

HAL Id: hal-05415482

https://hal.science/hal-05415482v1

Submitted on 14 Dec 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# International standards for the identification and the description of languages and their varieties

Laurent Romary, Inria

**Abstract:** The present chapter explores the need for and methods of standardizing language identification and description, especially in digital and linguistic contexts. With thousands of languages and countless varieties spoken globally, the task of accurately identifying, coding, and representing them is complex but crucial for interoperability, information management, and linguistic research. The article presents two key ISO standards — ISO 639 for language codes and ISO 21636 for describing language varieties — and discusses how these standards can help professionals across domains manage linguistic diversity. It also proposes enhancements to the TEI (Text Encoding Initiative) framework to better accommodate the nuanced identification and documentation of languages and their varieties. Ultimately, the article advocates for more coherent, flexible, and comprehensive digital language resources, emphasizing the importance of community consensus, data modeling, and training in standardization practices.<fnref n="1" />

**Keywords:** standardization, language description, language code, language varieties, TEI guidelines

## 1. Standardizing languages, why and how?

### 1.1 Languages – what is there to be standardized?

The Ethnologue register<fnref n="2" /> lists around 7,000 languages, while the Glottolog<fnref n="3" /> database (Hammarström et al., 2024) identifies over 8,000 languoids (families, languages, dialects, see (Forkel and Hammarström, 2022)). In reality, there is far greater variability in how a language is spoken (Reppen et al., 2002) depending on geographical context, the circumstances of communication, and also on social or educational parameters<fnref n="4" />. When we look at the level of individuals, we find that each one of us embodies a whole spectrum of linguistic forms, shaped by our origins, our learning history, and the groups with which we have interacted. We learn to master different registers, we activate different vocabularies according to the situation, and all of this evolves over time.

This variety of linguistic forms that we can observe at both the level of an individual — and even more so among a group of individuals — can render the task of standardized language identification almost impossible in its entirety. Yet, we intuitively know how to recognize a language we are familiar with and we can communicate fluidly within groups sharing a given language. In the context of the widespread digital exchanges, we must be capable of exchanging oral or textual information within computer systems that rely on the accurate identification of the languages that are being used.

In this context, standardization offers professionals in different fields — language specialists, archive and library curators or IT professionals exchanging language data — means to identify

<footnote n="1">¹The author acknowledges the use of Large Language Models (LLMs) as writing aids in phrasing this chapter, based on the authors' notes, ideas, and concepts. The author retains full responsibility for the content.</footnote>

<footnote n="2">² https://www.ethnologue.com</footnote>

<footnote n="3">³ https://glottolog.org</footnote>

<footnote n="4">⁴ See for instance the work of Eugenio Coseriu (1970) on linguistic variation.</footnote>
languages so that the information arrives where it should arrive, and is processed in the right way.

Behind this generic idea of why standardization is important in the domain of languages, there is a deeper question of what is to be standardized. Should we try to catalogue languages so that they fit into a box associated with a code? Or should we standardize the method to identify languages so that professionals themselves have a word to say about, the level of granularity at which a language should be considered? The present chapter intends to offer a compromise between these two perspectives.

## 1.2 A whole range of possible applications

Language encoding is essential for digital applications as it enables systems to identify, process, and present content in the correct language, ensuring seamless communication across linguistic diversity and platforms. In library settings, properly encoded language metadata allow for accurate cataloguing, searchability, and retrieval of documents within a multilingual archive. On the web, it ensures that pages are correctly displayed and indexed, enhancing accessibility and user experience. Language encoding also supports spell-checking by enabling software to apply the right linguistic rules, improving accuracy. In smartphones, it facilitates multilingual interfaces, predictive text, and voice recognition. Crucially, standardized language encoding promotes interoperability, allowing different systems and applications to exchange and interpret linguistic data consistently, supporting global communication and information management.

A strong framework for describing and identifying languages and their varieties is of course essential to academic purposes dealing in particular with language resources. This is the case for any corpus gathering activity. Exemplary applications range from dealing with specific heritage objects such as epigraphic research (Bodard, 2010) to large-scale ambitions to harvest online content for training machine learning systems, t (Ortiz et al., 2019)<fnref n="5" />. And of course, language identification is key to all dictionary projects where framing the intended language variety has a strong impact on the actual lexical coverage.

The ultimate application of language description and encoding is clearly that of field linguistics where a scholar has to determine the granularity at which he/she wants to observe linguistic data to define the dimensions (in the sense of ISO 21636, as we shall see below) to consider and the language families he/she wants to connect the observations to. A strong standardisation framework is essential to be able to compare observations across researchers and linguistic spheres. For instance, it is important when previous evidence such as recordings, lexicons or grammatical descriptions in order to compare observations over time. As demonstrated by the exemplary work carried out in the Wibarab<fnref n="6" /> project, each observation should be characterized according to the actual variety it represents.

As we can see from the examples above, there might be a tension between constraints on the linguistic precision and the technical effectiveness of language identification. Still, it is not imaginable to provide a different technical solution for each application, with the risk that proprietary solutions lack a proper maintenance of schemas and associated tools. What we shall see in this chapter is that ISO has probably defined, over the course of more than 50 years of activity on the subject, a rather reasonable path with both a stable framework for the identification of languages and a flexible environment for the description of varieties.

<footnote n="5">⁵ See also https://oscar-project.org/</footnote>

<footnote n="6">⁶ “What Is Bedouin-type ARABic?”, https://wibarab.acdh.oew.ac.at</footnote>
1.3 The standardisation landscape

This chapter will focus on the two main standardisation efforts within the International
Organisation for Standardisation (ISO): ISO 639, a standard defining a code for identifying
languages , and ISO 21636, a standard for describing language varieties. Additionally, we will
refer to other standards in the course of this chapter. It is thus useful to elicit some basic
elements for those readers who are not necessarily accustomed to standardisation.

* Standards are documents that have no legal power but create a social contract between two parties having to exchange information, in our case, the exchange of digital information;

* Standards are being developed in different bodies, such as ISO, but also the W3C (World Wide Web Consortium) or the TEI (Text Encoding Initiative);

* Standards are usually the result of a consensus within a community. They rely on a design process involving experts working jointly on a document or, maintain an authority list of objects.

We need to keep in mind that a real-life application is always based on a whole range of
standards, issued by potentially different standardisation bodies
Each contributes to the
definition of specific aspects in the representation of a digital object or elicits the way standards
can actually be composed. Fig. 1 shows the various standardisation layers that allow the
encoding of the language in which a text is expressed within the following element according to
the TEI guidelines.

We can illustrate this with the following snippet:

<quote xml:lang="fr-FR" xmlns="http://www.tei-c.org/ns/1.0">Demain,
dès l'aube, à l'heure où blanchit la campagne...</quote>

Typically, language identification requires the provision of a localization code, a country code taken from ISO 3166, or a script from ISO 15924, when the language may be written in two different ways (e.g. for Serbian). The IETF standard BCP<fnref n="7" /> 47, combined with the IANA language subtag registry<fnref n="8" />, elicits how such standards may be combined. In turn, the W3C recommendation for XML specifies how to incorporate a language identification tags within an XML element, ie. using the @xml:lang<fnref n="9" /> attribute. Finally, the TEI guidelines, based on the XML recommendation, define a vocabulary of elements together with their possible combinations and the condition of integration of the @xml:lang attribute to them.

<footnote n="7">⁷ Best Current Practice, one of the ways standardisation documents are named in IETF.</footnote>

<footnote n="8">⁸ https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry</footnote>

<footnote n="9">⁹ We use the notation @XXX to indicate the name of XML attributes in the flow of our text and <YYY> for XML elements.</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: The multilayered standardisation landscape</figcaption></figure>

## 1.4 Identifying language as an object to be standardised

The central question when contemplating the standardisation of language codes is to define what a language actually is. There is a large literature on the subject and it is not the aim of the current chapter to dive into the details of the linguistic debates. Still, we want to provide here a glimpse into the complexity of the issue before we provide more details of ISO 639.

From a scholarly point of view, we can only speak of a language if we have observables. Observables can be primary sources such as written traces or recordings, or secondary sources, typically observations by field linguists, that a certain language exists and has certain linguistic properties. At this stage, we can already perceive two elements that contribute to the difficulty of identifying a language: a) the wide variety of evidence that we may have, from fragmented written traces to a whole body of observation and literature; b) such traces can never be about the language as a whole but are a compilation of elementary productions being part of the same "language". What we address here is indeed a central notion: whatever we do in the domain of linguistic documentation or description can only be conceived bottom up from the individual forms used by a given speaker, called his or her idiolect. On this basis, a language can be defined by a degree of commonness between idiolects, i.e. how much the way two speakers express themselves allows them to be understandable to one another. This is what we usually call mutual intelligibility.

A direct consequence is the difficulty to identify languages without actually describing them at least along the dimensions that are relevant for grouping or differentiating idiolects or groups thereof. The current chapter is thus an attempt to present the standardisation activities within ISO together with proposals as a contribution to the complex issues of linguistic description.

# 2. Language coding with ISO 639

## 2.1 ISO 639 principles

The main objective of the ISO 639 standard<fnref n="10" /> is to provide a simple means to guarantee semantic interoperability between digital systems exchanging linguistic information. Therefore, it provides identifiers that uniquely refer to languages. ISO 639 defines a *language code*, made of

<footnote n="10">¹⁰ ISO 639:2023 Code for individual languages and language groups</footnote>
elementary *language code elements* for each language covered by the standard, together with identifiers and names for each of the code elements.

In line with the vision provided in the introduction, the notion of language in ISO 639 is understood as the largest mutually intelligible set of idiolects<fnref n="11" />. The scope of ISO 639 is explicitly that of natural languages, as opposed for instance to computer languages.. It covers spoken, written and signed languages, including extinct or historical languages. Under certain conditions, e.g. a living user community, it comprises constructed languages, such as Esperanto. In addition to individual languages, the standard covers macro-languages. For ISO 639 legacy reasons, this covers previously identified “languages” which actually are a superset of several distinct languages, for example Arabic or Chinese. It also applies to language groups when several languages share several linguistic or geographical properties, e.g. Caucasian languages..

In this framework, ISO 639 language code elements are associated with one to three identifiers:

* a mandatory three-letter identifier (give examples here);

* an optional second three-letter identifier, for some languages, corresponding to a, so-called "bibliographic use" or "terminological use";

* a two-letter identifier for a reduced number of "major, mostly national individual languages", also known as set 1 of ISO 639 (183 languages as of March 2025) - give examples here

For instance, the language with the reference name "French" has three identifiers, *fr*, *fra* (terminological use) and *fre* (bibliographical use).

In addition to these identifiers, ISO 639 provides them with language names. For each language code element, there is a single reference name. To achieve this, a differentiating element may be added to the name; for instance, "Leti (Cameroon)" vs. "Leti (Indonesia)". Additional language names may be provided in English or French as part of the standard<fnref n="12" />. They correspond to names used in the literature or to forms used for indexing in archives or libraries. For example, the language with identifier *lrt* is associated with the reference name "Larantuka Malay" and an additional "inverted" name "Malay, Larantuka".

The evolution of the ISO 639 language code is ensured by the ISO 639 maintenance agency which has been set up in 2023 and receives all online change requests addressed to it. Such change requests may both tackle the evolution of the language code at large, with proposals to create, merge, split or deprecate existing codes, as well as proposals of changes in naming existing language code elements. Typically, change requests related to the reference name of a code element are quite common in the work of the ISO 639 maintenance agency. They are often related to the observation that the reference name can be seen as derogatory for the corresponding speaker community or that another name has become a far more common designation for the language.

The maintenance of the ISO 639 language code is an activity that covers both linguistic but also political aspects since it can impact on the essence of certain communities or states. This is made all the more difficult because the existing language code is quite comprehensive and relies on a long-standing history (see below). The review mechanism for ISO 639 is essentially based on change requests issued by the community about which the maintenance agency should make

<footnote n="11">¹¹ The definition of individual language precisely says “largest set of idiolects, used by different speakers, which are all interconnected through high mutual intelligibility, or through a chain of high mutual intelligibility, or which are socio-politically considered as a unit equivalent to such a largest set” (cf. https://www.iso.org/obp/ui/fr/#iso:std:iso:639:ed-2:v1:en:term:3.1.3)</footnote>

<footnote n="12">¹² ISO 639 anticipates that its language code may be associated with other names by ISO member bodies.</footnote>
an informed decision, possibly based on asking for further experts' opinions. Still, in order to ensure some stability of the overall ISO 639 framework, there is little margin for creativity and the text of the ISO 639 standards clearly scopes conditions for such changes, in particular in terms of cross-intelligibility. Among the other constraints abiding to the evolution of the ISO 639 language code, we should also mention here that an identifier can never be re-used twice, even if it is part of a language code element that has been deprecated.

Another important observation at this stage is that the ISO 639 language code is the result of eight decades of standardisation activities and it is only recently that a single coherent framework for its maintenance has been set in place. As a result, some of the language code elements would not have necessarily been designed in the same way with the current rules, but are kept in place for legacy and thus backward compatibility reasons. This is in particular one of the reasons why macro-languages exist in the ISO 639 framework. Still the new maintenance agency has put procedures in place whereby it documents — on a private github server — all decisions and action it takes, thus offering the possibility at each stage to fully justify its choices and also to create a corpus of case law for the years to come.

## 2.2 Looking back at history to understand the current stage of ISO 639

Even if the present chapter is intended to give the clues for a proper understanding and usage of the ISO 639 language code as it is now, we cannot avoid giving some historical elements as to how the standard was set up and evolved over time. This is all the more important that it impacts on both the structure of the ISO 639 language code and the current organisation of the maintenance agency.

The work on ISO 639 started as early as the 1960s with the publication in 1967 of a first recommendation (ISO/R 639) which had a scope beyond the sole encoding of languages (also covering countries and other identifiers), but which paved the way for the definition of an elaborate multi-part standard over the years.

The first part, published initially in 1988 as ISO 639 and in 2002 as ISO 639 part 1, addressed the encoding of languages using two-letter identifiers and thus could only address "major, mostly national individual languages". Still, it was somehow timely available for accompanying the development of web technologies such as HTML in particular<fnref n="13" />.

The limits of the 2-letter basis of ISO 639-1 for encoding a wide variety of languages was soon addressed by extending the ISO 639 framework with three letter identifiers and addressing, within ISO 639-2 (published in 1998), the encoding of a first larger group of languages. For this work, the Library of Congress took the lead in the definition and thereafter the maintenance of the standard and impelled a strong library orientation in both the coverage and the choice of identifiers, so that it would actually match their own indexing practices. This explains for instance why a score of languages still has two three-letter identifiers.

The ISO 639-3 standard was published in 2007 to provide a more "comprehensive coverage of languages" by systematizing the allocation of codes to a portfolio that would not be limited to the mainstream languages as used in libraries or digital applications at that time. The work around this standard was strongly influenced by the existing Ethnologue registry maintained by SIL International, which as a consequence became the registration authority associated with the standard.

<footnote n="13">¹³ The first version of HTML was announced in 1991 (cf. Berners-Lee and Fischetti, 2000), but the @lang attribute only appeared in HTML 4 (1999 — https://www.w3.org/TR/html4/), and defined according to RFC 1766 (1995 — https://www.ietf.org/rfc/rfc1766.txt).</footnote>
When ISO 639:2023 was published, it replaced the 5 existing ISO 639 parts so that, as described above a single language code was created encompassing the whole range of languages, language groups and macro-languages that were covered by them. Still, it was decided not to be completely disrupting, both from the point of view of the language maintenance organisation as of the language code structure:

* The three existing registration authorities associated to the former ISO 639 parts were integrated into the structure of the ISO 639 maintenance agencies as *language coding* agencies, so that their expertise would be kept in the work of the new agency;

* The scope of the four ISO 639 parts (1,2, 3 and 5) were redesigned into the new language code as representing so called sets...

When looking at the organisation of the ISO 639 language code from a distance, we can see that the sets do not play a fundamental role. They are more a vestigial organ from the past than something that may impact on specific applications. All in all, language code elements are homogeneous units, whatever the number of identifiers that are associated with them and their belonging to some of these sets. We can anticipate that the notion of set may vanish in the years to come.

## 2.3 Which digital commons for ISO 639? A TEI-based proposal

At this stage, it is important to raise the issue of providing a comprehensive digital resource associated with the ISO 639 language code so that it could cover the whole spectrum of potential applications. As it stands currently, the availability of ISO 639 lies within a quite fragmented landscape. Because of the historical context of the creation of ISO 639, the three language coding agencies have their own websites where they provide access to the language code with more or less emphasis on the sets they are responsible for. Beyond these official sources, ISO 639 is available at various places, in particular Wikipedia, across various pages, and of course as part of the IANA Language Subtag Registry which maintains the reference implementation of BCP 47. Besides, those various sources are based on heterogeneous data models for representing ISO 639 content, in as much as we can call data models flat tables on a web page. Ideally, we should design both a precise data model and a resource that serves as a basis for all downstream consumers.

One issue with providing such a resource to all kinds of possible users is that it should be comprehensive enough not only for basic language identification mechanisms, like checking that two resources are similarly coded, but should allow extra consistency checks when needed. Typically, one should be able to identify that a legacy linguistic resource is coded using an identifier that is no longer in use. This implies that the ISO 639 resource should not only make active language code elements available, but also retired ones, such as Languedocien (Inc), which has been merged with Occitan (oci). This last example also shows that it may be useful to even provide the full change history associated with the language code at large.

At an early stage in our attempt to design a data model for describing languages and language varieties that would cover the various objectives and constraints that we have just delineated, it was clear that a central application domain could be linguistic resources and possibly digital humanities. This is why it occurred to us that a possible implementation of such a model should be compatible as much as possible with the general framework offered by the TEI guidelines, which are now the most widely adopted standards in these fields. Indeed, the TEI guidelines already contain a `<language>`<fnref n="14" /> element, which, although its current scope is reduced to the tiny

<footnote n="14">¹⁴ https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-language.html</footnote>
use case of registering the languages used within a TEI document, could be an interesting basis
for a more elaborate descriptive component.

In the rest of this paper, we will thus place ourselves in the hypothesis of deeply revising this
element to fulfil our purpose, while, in our view, not so much abusing the existing background of
the TEI guidelines. We will try to show that this proposal could cover the basic identification
needs related to ISO 639, but also more in-depth documentation associated with language
varieties, as we shall see in the next section. Finally, we would like to see this new definition of
the <language> element as a general tool that could be associated with a whole range of
linguistic observations.

To deal with the basic needs of ISO 639, we propose to use the existing TEI attributes on
<language> in the following manner:

* The @type attribute can indicate the type of language the description is about, individual language, macro-language or language group<fnref n="15" />. It is complemented by a @subtype attribute to indicate whether the language is considered as living, extinct, constructed, historical or special (for reserved ISO 639 codes);

* The @status attribute, which can take the values active or retired to mark the status of the language code element in the ISO 639 code and, as mentioned above, allow one to trace legacy code elements which may occur in long lasting document repositories for instance;

* The @ident attribute, in accordance with its current meaning in the TEI guidelines, would be kept to provide the identifier under which the language will be referenced in the rest of the encompassing TEI document. In general, it should be the identifier that BCP 47 would actually favor, e.g. two-letter identifiers for languages in set 1.

Identifiers are naturally provided with the <ident><fnref n="16" /> element associated with the following attributes:

* The @type attribute is used in strong alignment with the TEI guidelines practices to indicate the nature of the object being identified, and thus uniquely set to "languageIdentifier";

* The @subtype attribute is used to determine which of the ISO 639 is provided and can take one of the following values: "ISO639-2" (for two letter identifiers), ISO639-3L (for three letter identifiers for bibliographic use), ISO639-3T (for three letter identifiers for terminological use);

The names associated to the language code element can be easily represented using the TEI
element <name><fnref n="17" /> in combination with the following attributes:

* The @type attribute is used in strong alignment with the TEI guidelines practices to indicate the nature of the object being named, and thus uniquely set to "languageName";

* The @subtype attribute may be used in addition to @type when the components of the name are inverted (e.g. "Creole French, Réunion"), and thus uniquely set to "invertedName";

<footnote n="15">¹⁵ As we shall see in the next section, these values will have to be complemented to deal with language varieties as well.</footnote>

<footnote n="16">¹⁶ https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-ident.html</footnote>

<footnote n="17">¹⁷ https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-name.html</footnote>
* @xml:lang is provided to indicate, in conformance to the XML recommendation, the language in which the name is provided. Its content should be conformant to BCP 47. By construction, it is always English (en) for language reference names.

As an example, we can see below the two forms for the creole French of La Réunion (rcf) with first the reference name and the additional name with an inverted (bibliographic like) name.

```xml
<name type="languageName" role="languageReferenceName"
      xml:lang="en">Réunion Creole French</name>
<name type="languageName" subtype="invertedName" xml:lang="en">Creole French, Réunion</name>
```

We can note that with this model, we could easily add names in other languages when, for instance, a national standardization body would like to take up the standard in its national catalogue.

With such a data model associated with the transformation of the `<language>` element, we see that we have all the building blocks to integrate the core components of ISO 639 features. For instance, the entry associated to French (fr) could be represented as follows:

```xml
<language status="active"
           ident="fr"
           type="individual"
           subtype="living">
    <ident
        type="languageIdentifier"
        subtype="ISO639-3B">fre</ident>
    <ident
        type="languageIdentifier"
        subtype="ISO639-3T">fra</ident>
    <ident
        type="languageIdentifier"
        subtype="ISO639-2">fr</ident>
    <name
        type="languageName"
        role="languageReferenceName" xml:lang="en">French</name>
    <name type="languageName" xml:lang="fr">Français</name>
</language>
```

In addition, we can record the evolutions undergone by a given language code element by adding the `<event>`<fnref n="18" /> element to the possible descriptive children of `<language>`. Its @type attribute categorizes the event and can take the values “change”, “merge”, “duplicate”, or “split”. Depending on the event type, a child element `<desc>`<fnref n="19" /> provides a more precise description of the event either by pointing to the language element it has been replaced with or, in the case of a split, the list of languages that has resulted from its deprecation.

```xml
<language status="retired" xml:id="auv" ident="auv">
    <name type="languageName" role="languageReferenceName"
          xml:lang="en">Auvergnat</name>
    <event type="merge" when="2007-03-14">
        <desc>
            <ref target="#oci">oci</ref>
        </desc>
    </event>
</language>
```

<footnote n="18">¹⁸ https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-event.html</footnote>

<footnote n="19">¹⁹ https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-desc.html</footnote>
3. Beyond languages - language varieties - the 21636 series

3.1 The principles behind ISO 21636

The objectives of the work carried out around the ISO 21636 series within ISO committee TC 37/SC 2 (see chapter 17 for general information on ISO committees) are clearly complementary to the work achieved with ISO 639 and its maintenance agency. In a way, it has taken the reverse direction of what was initially intended with the now withdrawn part 6²⁰ of ISO 639, which intended to provide 4 letter identifiers for language variants, with the general agreement that it would be vain to try to catalogue in a comprehensive manner language variants or, as specified in ISO 21636, language varieties.

The ISO 21636 series actually aims at providing a descriptive method within which a wide range of types of language varieties can be described in coherence with the underlying setting offered by ISO 639. As a matter of fact, it identifies a language variety as a subset of an individual language which not only fulfils some structural criterion in terms of linguistic properties, but also some external criteria related to the conditions related to where the variety is used, by whom and under which socio-cultural circumstances. As such it follows the same principles that we saw for ISO 639, inspired by Lieb (1999), that a language or a variety thereof corresponds to a set of idiolects that share certain characteristics, with at its centre, the notion of intelligibility.

The description of a language variety can be made along what ISO 21636 calls dimensions which, either in isolation or by combining several of them, may allow one to elicit how the variety can be singled out within a given individual language. The possible dimensions provided by ISO 21636 are the following ones:

* The *space dimension* is one of the most prominent dimensions to identify a language variety. It lies at the basis of the definition of dialects, seen as the way a language is used, (usually spoken) in a certain area and is often central to field linguistic studies;

* The *time dimension* can be used to characterise the period over which a given variety was to be observed, either for historical purposes or simply to characterise a specific trend for instance;

* The *social group dimension* corresponds to social circumstances where a certain variety is used. It can be related to certain types of gatherings, social strata or professional exchanges;

* The *situation dimension*, which qualifies the social setting within a linguistic interaction takes place. Although ISO 21636-1 focuses on the use of this dimension for describing registers, we extend it here to cover the description of any kind of social context, whether it impacts on the used register or not;

* The *person dimension*, which characterizes a language spoken by a single person, can be seen as a special case of the social group dimension, and although a personal variety is not necessarily homogeneous<fnref n="21" />, there are many cases where this dimension can be

<footnote n="20">²⁰ Withdrawn: ISO 639-6:2009 Codes for the representation of names of languages Part 6: Alpha-4 code for comprehensive coverage of language variants.</footnote>

<footnote n="21">²¹ It may vary according to the situation, the interlocutors etc., all in all along many of the dimensions contemplated in ISO 21636.</footnote>
useful in concrete applications. On can think of identifying an author of a series of texts
(or online contributions), collecting a dictionary of a famous author (The dictionary of
Goethe<fnref n="22" />) and of course to document each observation made during a field linguistics
campaign;

* The *medium dimension* allows one to characterize a variety according to the actual way it has been communicated, i.e. whether it is just spoken, written or by default multimodal. The standard anticipates other modalities such as signed, whistled, drummed or resulting from specific augmenting devices in cases of impairment;

* Although it is positioned as a specific dimension in ISO 21636, we could see the *proficiency dimension* simply as a property attached to either the social group or the person dimension;

* Finally, ISO 21636 introduces a *communicative functioning* dimension which allows one to express how much the variety is specific to a person or group of persons with enhanced, normal or constrained communicative capacities. This may allow one to qualify specific impairments, diseases or above the average eloquent capacities.

It should be noted that many of these dimensions may rely on other standards that may exist to
characterise them, such as the provision of spatial or temporal information, but also specific
terminologies related to socio-cultural contexts for instance.

As we have seen above and we will further explore when further formalizing language varieties
in the context of the TEI, there may even be dimensions that can be grouped as part of the
description of a speaker or a speaker community, rather than to be seen as an autonomous
property of a language variety.

## 3.2 Providing even more descriptive capabilities to the TEI `<language>` element

The definition of a language variety provided by the ISO 21636 series together with the
provision of a comprehensive set of dimensions along which such varieties can be characterised
is a very good basis to get one step further into the TEI based language modelling work we have
started in the previous section. It seems we can now deeply revamp the `<language>` element so
that we design a real "linguographic" element aimed at offering a variety of language descriptive
capabilities. This notion is incidentally close to the concept of language profile introduced in the
Wibarab project (see Engler and Schopper, 2025).

We thus extend here the proposal made earlier by considering that the `<language>` element
could bear all dimensions provided by ISO 21636 and be used as a component within a TEI
document header to identify and describe the varieties used in its content or, when used within
the body of a document, to list language references made within a text, as would be the case for
text texts related to linguistic theory or grammar.

The core usage of the `<language>` element would then be very similar to its current purpose in
the TEI guidelines which declares the occurrence of a language together with its identifier
expressed according to the BCP 47 framework.

In the rest of this section, we suggest integrating the implementation of the 21636 dimensions
as further child elements to `<language>`, with a systematic reuse of existing TEI elements. As we
shall see and depending upon the intended application, some of the dimensions may be either
reorganised into specific structures or even positioned elsewhere in the TEI header when
necessary.

<footnote n="22">²² See http://gwb.uni-trier.de/de/</footnote>
First, we need to extend the definition of the attributes attached to the `<language>` element to deal with varieties:

* By adding the value `variety` to `@type`;

* By adding additional values on `@status` to qualify the condition under which the variety has been put together, e.g., `observed`, `inferred`.

The space dimension can be easily implemented by using the `<settingDesc>`<fnref n="23" /> element as a child of `<language>`. It does not have to be typed unless the locations have an impact for the definition of varieties (e.g. urban vs. countryside). The locations concerned can be described by means of elements such as `<place>`<fnref n="24" /> or `<listPlace>`<fnref n="25" />. The TEI further provides various sub elements to them to refine their description, such as `<placeName>`<fnref n="26" />, `<location>` (for geographical details or addresses), etc.

```xml
<settingDesc>
  <place type=""/>
  <placeName/>
  <location type="address|geopolitical"/>
  <idno/>
  <note/>
</place>
</settingDesc>
```

By means of the `<idno>`<fnref n="27" /> element, the description of the place associated with a language variety can also be limited to a reference to an online standard such as ISO 3166 codes or the M49 Standard Country or Area Codes for Statistical Use<fnref n="28" /> with the possible reference to the respective identifiers as illustrated in the following example:

```xml
<language ident="fr-FR">
  <name type="languageVariety">French as used in France</name>
  <place type="country">
    <placeName xml:lang="fr">France</placeName>
    <idno type="UN-M49">250</idno>
    <idno type="ISO-alpha3">FRA</idno>
  </place>
</language>
```

The implementation of the time dimension can be easily made by means of the `<date>` element, which, together with the large palette of dating attributes (`@when`, `@notBefore`, `@notAfter`, `@from`, `@to`), but also by naming a given period by using `<name>` as a child, allows one to represent a whole variety of temporal spans.

```xml
<language>
  <date notBefore="1900" notAfter="1940">
    <name type="period">early 20th century</name>
  </date>
</language>
```

<footnote n="23">23 https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-settingDesc.html</footnote>

<footnote n="24">24 https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-place.html</footnote>

<footnote n="25">25 https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-listPlace.html</footnote>

<footnote n="26">26 https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-placeName.html</footnote>

<footnote n="27">27 https://tei-c.org/release/doc/tei-p5-doc/en/html/ref-idno.html</footnote>

<footnote n="28">28 https://unstats.un.org/unsd/methodology/m49/</footnote>
Here again, there is a strong standardization background for the expression of temporal
information with the two parts of ISO 8601. Since the TEI guidelines potentially offer a whole
range of attributes associated to the <date> element, the main issue will be to provide precise
guidelines concerning a subset of these and how to use them to make sure that different
varieties could be easily compared along this temporal dimension.

The situation dimension, in the extended sense signalled above, can be made by means of the
TEI <domain><fnref n="29" /> element.

Note: This element needs to be revamped after a variety of possible implementation in such
projects as Wibarab are considered.

Since the social group dimension corresponds to the characterisation of a given speaker
community one or several characteristics, there is again a straightforward implementation in
the TEI framework using here the <personGrp> element.

<personGrp>
  <age/>
  <gender/>
  <faith/>
  <education/>
  <nationality/>
  <residence/>
  <occupation/>
  <socecStatus/>
</personGrp>

For instance, a simple description of business English could be as follows:

<language>
  <name type="languageVariety">business English</name>
  <personGrp>
    <occupation>business</occupation>
  </personGrp>
</language>

And here, a more complex example of a person group encoding inspired from the SHAWI
project<fnref n="30" /> (cf. Mörth and Procházka, 2017):

<personGrp>
  <age>Different ages</age>
  <gender>Male and female</gender>
  <faith>Muslim (Sunni)</faith>
  <education>No specific level of education. Any formal education received in Turkish, no formal education in Arabic</education>
  <nationality type="citizenship">Turkish</nationality>
  <nationality type="ethnicity">Arab</nationality>
  <residence>No specific residence</residence>
  <occupation>No specific occupation</occupation>
</personGrp>

<footnote n="29">²⁹ https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-domain.html</footnote>

<footnote n="30">³⁰ https://www.oew.ac.at/acdh/research/linguistics/research/corpus-linguistics/shawi-the-shawi-type-arabic-dialects</footnote>
<socecStatus>The speakers of this variety were formerly sheep and goat rearing nomads.</socecStatus>
<note/>
<langKnowledge>
<langKnown tag="tr"/>
<p>All of the informants speak Turkish and Shawi Arabic, although they received no formal education in Arabic and therefore have no knowledge of Modern Standard Arabic. Data of other language knowledge was not collected.</p>
</langKnowledge>
</personGrp>

Just as we have <personGrp> to implement the social group dimension, the TEI <person> element is the ideal candidate to implement the person dimension. Depending on the context, this element may contain further elements to either name the speaker or writer of the language (like in the example of the Goethe-Wörterbuch mentioned above), or just provide characteristics useful for the specification of the linguistic sample that is being recorded, as would be the case in field linguistics or when studying for instance the specificities of non-native speakers of a language (cf. Falko corpus). These kinds of examples are typical reasons why we suggest implementing the proficiency dimension as a component of the person dimension. This is all the more natural in the TEI framework as both elements ( <person> and <personGrp>) actually contain the <langKnowledge> element that is perfectly fit to the purpose.

<langKnowledge>
<langKnown tag=""/>
</langKnowledge>

By doing so, we can describe the linguistic profile of a person, whose idiolect we would want to qualify:

<person sex="m" role="speaker collaborator">
<-- other details omitted -->
<langKnowledge>
<langKnown tag="mix">Mixtepec-Mixtec</langKnown>
<langKnown tag="en">English</langKnown>
<langKnown tag="es">Spanish</langKnown>
</langKnowledge>
</person>

Here below, we further illustrate the person with the following example where we characterize
a young non-native female speaker:

<language ident="en-x-nnyw">
<person>
<gender value="W">woman</gender>
<age>between 18 and 22</age>
<langKnowledge>
<langKnown tag="en" level="nonNative"/>
</langKnowledge>
</person>
</language>

This is typically the kind of information that you would attach to corpora dedicated to language
acquisition such as the Falko corpus<sup>31</sup> (Reznicek, 2012), where they try to attach to each entry

<footnote n="31"><sup>31</sup> https://www.linguistik.hu-berlin.de/de/institut/professuren/korpuslinguistik/forschung/falko</footnote>
the necessary metadata related to the education history, the first language (L1) together with the age and gender.

The characterization of the medium dimension for a language variety can be implemented by means of the <channel> element where we would suggest making the @mode attribute mandatory and constrained with a semi-open list of explicit values.

<channel mode="multimodal">face-to-face conversation</channel>

Contrary to what is currently suggested within the TEI guidelines, we would suggest to avoid using mixed values for the @mode attribute (e.g. sw for spoken to be written, when dictating for instance), and use the @type attribute to clearly differentiate between the primary mode (how the language content is being produced) and the secondary mode (the intended use). For instance, the English language used in drama could be described as:

<language ident="en-x-drama">
  <channel type="primary" mode="written"/>
  <channel type="secondary" mode="spoken"/>
</language>

The implementation of the communication functioning dimension can also be covered by adapting the existing <interaction> element in the TEI guidelines. The simplest way would be to provide appropriate values for @type such as “regular”, “enhanced” and “constrained” and provide an additional description as the content of the element. For instance, if we were to characterize the variety associated to drunken speech, we could have:

<interaction type="constrained">drunken speech</interaction>

With such a model for language description purposes we could imagine that language resource projects could systematize the precise description of the languages they deal with and further share such descriptions so that a real catalogue of linguographic objects could be made available for any type of TEI application as a common resource.

In some cases, though, some of the descriptive elements may not be directly associated to the languages present in a given language resource, but rather associated to the description of the language resource itself. If we take for instance the description of the medium using the <channel> element presented above, a given project may either consider that it characterises a specific variant within a heterogeneous corpus of, say spoken, written etc. content, or, on the contrary, that it characterizes the way the corpus has been gathered by using the <textDesc> element within the <profileDesc> of the corpus.

As an example of how this model could be used to share more precise information about the languages used within a given language resource, we can foresee that, as suggested just above, a <language> element such as the one we are aiming at could be associated to each BCP 47 tag in a project, in particular when variant or private use subtags are present, where the meaning of the actual subcomponents may not be straightforward just from looking at the coding.

As an application of this and being aware that it would require some in-depth work to provide a complete coverage, we show below how a simple entry in the IANA language subtag registry<fnref n="32" /> (left) could fit into the proposed TEI encoding (right):

Type: language

Subtag: fr

<language ident="fr"

source="https://www.iana.org/assignments/"

<footnote n="32">³² https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry</footnote>
Description: French
Added: 2005-10-16
Suppress-Script: Latn

language-subtag-registry/language-subtag-registry">
  <name type="IANA-Description">French</name>
  <date type="whenAdded" when="2005-10-16"/>
  <ident type="Suppress-Script">Latn</ident>
</language>

In a mid-term perspective, this would present the advantage of proving a way to compare and align descriptions across standardisation initiatives.

## 4. Putting things in practice

The claim we make in this chapter is that any project, initiative or environment which has to signal the use of languages within a given corpus of linguistic content should go beyond the sole provision of tags that would not completely elicit the intention behind them. We have tried to show how the TEI guidelines, in complement to the background and features of such standards as ISO 639 and ISO 21636 could actually fulfil this need and even ensure a continuum with authority lists such as the IANA language tag registry and very specific projects.

In the context of a concrete corpus project, the proposed framework should be seen as a tool by implementers to help them precisely identify their needs and in particular make clear the level of granularity at which languages have to be defined for the purpose of the project and try to stay coherent in the precision with which languages are identified and documented.

If we consider the first class of use cases mentioned in the introduction, i.e. the basic interoperability between generic computer systems (web pages, computer interfaces etc.), we can observe that they can only work when the sender and the receiver of the information rely on the same level of granularity in terms of language identification. This implies that designers are more or less forced to ground their developments on the most common languages as covered by set 1 of ISO 639, possibly extending to set 2 with locale information for large linguistic regions (e.g. fr-CA), typically to facilitate the work of spell checkers.

In the case of large-scale corpus projects like the OSCAR project that we mentioned in the introduction, the situation is quite similar since the information available from scrapped online material is shallow. The corpus is actually sorted per language on the basis of a language classifier trained with corresponding language samples. New languages can only be identified if there is a training corpus at hand, meaning that, for instance, identifying a certain variety will require that it has enough available online presence, but also that it can be tagged with enough precision, so that it can be distinguished from other samples.

A project such as Wibarab could be seen as situated on the opposite side of the language granularity spectrum since it aims at identifying the language varieties of Bedouin communities down to each observation, more less boiling down to idiolects. Still the actual characterisation of these varieties closely depends on the linguists that will associate both intrinsic (e.g. phonetic or lexical) or extrinsic features (locational, situational, composition of the observed group) as relevant for the underlying research agenda. In a way, as is the case when studying the introduction of a new language in the ISO 639 language code, the experts will have to define
precise criteria as to why two observations belong to the same variety or should be distinguished.

Finally, we could even think of using the framework as a way to provide metadata for elementary observations or recording as available on online platforms such as wiktionary. If we take for instance the entry for the esperanto word *scribis* in the French wiktionary<fnref n="33" />, we can see it contains an audio recording qualified for both the location and the language level of the speaker. A proposal encoding in this respect (within an *<entry>* element for the corresponding word) could be:

<cit type="recording">
<media
    url="https://fr.wiktionary.org/wiki/Fichier:LL-Q143_(epo)-Wierzbowski-skribis.wav"
    type="audio/wav"/>
<language ident="eo-Wierzbowski">
<date type="recording" when="2023-04-13"/>
<person>
  <persName>Wierzbowski</persName>
  <langKnowledge>
    <langKnown tag="eo" level="good"/>
  </langKnowledge>
</person>
<settingDesc>
  <place>
    <settlement>Białystok</settlement>
    <country>Pologne</country>
  </place>
</settingDesc>
</language>
</cit>

It should be noted that one should not necessarily rely on the technical framework used for a given application to control the descriptive granularity that has to be employed. For instance, let us take a project that uses, and rightly so, the TEI guidelines for representing its language resources. As we have seen, the TEI relies on the generic @xml:lang attribute, whose content is based on IETF BCP 47, which allows a whole range of possibilities from the provision of a simple 2 letter code for usual languages down to precisely localized varieties. As a result, the encoder must have a clear strategy of which type of language information he/she will be using, for which type of object in his or her TEI representation. Just to make this point more concrete, in the case of the encoding of a lexical resource, there may be simple language codes for the working language (the one used for documenting the content) and more precise codes for the object languages (the ones being described in the lexical resource), taking into account the high level of precision that the lexicographer may need in identifying a specific variety.

## 5. Conclusion

In this chapter, we have tried to explore methods for standardizing language identification and description, especially in digital and linguistic contexts. With thousands of languages and countless varieties spoken globally, the task of accurately identifying, coding, and representing them is complex but crucial for interoperability, information management, and linguistic research. The article presents two key ISO standards — ISO 639 for language encoding and ISO

<footnote n="33">³³ https://fr.wiktionary.org/wiki/skribis</footnote>
21636 for describing language varieties — and discusses how these standards help professionals across domains manage linguistic diversity. It also proposes enhancements to the TEI (Text Encoding Initiative) framework to better accommodate the nuanced identification and documentation of languages and their varieties.

We hope this chapter has contributed to a better understanding of the tenets of the ISO standards related to language identification and language variety description and offered some perspectives for future designers of digital components, whether in applicative or research projects. Part of the methodology we have tried to put forward is that there must be a conceptual continuum from idiolects to large coverage language families, which can be coupled with a coherent data model applicable to any kind of granularity level in between. To this purpose, we have suggested a possible implementation using a revamped version of the TEI <language> element, but the same model could potentially be instantiated in other frameworks. All in all, the questions that are to be answered before the beginning of any kind of such projects is knowing what you want to describe for a given project, what approximations are allowed, what constraints are to be applied, and at the end of the day what you really want to differentiate as a language or language variety.

Finally, we want to insist that the proper use of existing standards is strongly based upon the wide dissemination of standardisation information not only within developers' communities, but also practitioners of language descriptions and content providers in the language industry at large. This requires a solid training strategy both in universities but also from international infrastructures such as CLARIN<fnref n="34" /> and DARIAH<fnref n="35" /> so that a large compendium of online or in- presential training resources are offered on a regular basis. Even if language identification and description standards seem to be very technical objects from a certain distance, they are so essential in providing interoperable digital systems that they should be taken very seriously within a wide range of academic and industrial contexts.

## Acknowledgement

My warmest thanks to all the colleagues and friends who have provided input or feedback on this text, in particular: Susanne Alt, Gaby Bodard, Veronika Engler, Charly Moerth, Daniel Schopper, Toma Tasovac and of course the editors of the present volume.

## References

Berners-Lee, T. (1999) *Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web by Its Inventor* (Harper San Francisco) <https://dl.acm.org/doi/abs/10.5555/580812>.

Bodard, G. (2010). EpiDoc: Epigraphic documents in XML for publication and interchange. *Latin On Stone: epigraphic research and electronic archives*, 101-18.

Coseriu, E. (1970) *Einführung in die strukturelle Betrachtung des Wortschatzes* (Narr).

Engler, V. and D. Schopper (2025) "The WIBARAB Database. Data model and analytic functions", in: *Wiener Zeitschrift für die Kunde des Morgenlandes*. 115, 77-102. <https://doi.org/10.25365/phaidra.684_04>

<footnote n="34">³⁴ https://www.clarin.eu/</footnote>

<footnote n="35">³⁵ https://www.dariah.eu/</footnote>
Forkel R. and H. Hammarström (2022) ‘Glottocodes: Identifiers Linking Families, Languages and Dialects to Comprehensive Reference Information’, *Semantic Web*, 13.6, pp. 917–24, doi:10.3233/SW-212843.

Hammarström, H. (2024) and others, *Glottolog* 5.1, doi:10.5281/zenodo.14006617

Lieb, H.-H. (1993) *Linguistics Variables: Towards a unified theory of linguistic variation*. Philadelphia, Amsterdam: John Benjamins Publishing Company.

Mörth K. and S. Procházka (2013) ‘The Vienna Corpus of Arabic Varieties: Building a Digital Research Environment for Arabic Dialects’, in Lisan Al-Arab: Studies in Contemporary Arabic Dialects. Proceedings of the 10th International Conference of AIDA. Qatar University (LIT Verlag, 2017), pp. 209–18.

Ortiz Suárez, P. J., B. Sagot, and L. Romary (2019) ‘Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures’, in *7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7)* (Leibniz-Institut für Deutsche Sprache).

Ortiz Suárez, P.J. L. Romary, B. Sagot (2020) A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages. *ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics*, Seattle / Virtual, United States. (10.18653/v1/2020.acl-main.156). (hal-02863875v2)

Reppen, R., D. Biber, and S. Fitzmaurice (2002) *Using Corpora to Explore Linguistic Variation, Studies in Corpus Linguistics* (John Benjamins Publishing Company) <http://digital.casalini.it/9789027296160>

Reznicek, M., Lüdeling, A., Krummes, C., Schwantuschke, F., Walter, M., Schmidt, K., Hirschmann, H., Andreas, T. (2012): *Das Falko-Handbuch. Korpusaufbau und Annotationen Version 2.01*

## Cited standards

BCP: 47 *Tags for Identifying Languages*, IETF, September 2009

ISO/R 639:1967 *Symbols for languages, countries and authorities*

ISO 639-3:2007 *Codes for the representation of names of languages – Part 3: Alpha-3 code for comprehensive coverage of languages*

ISO 639:2023 *Code for individual languages and language groups*

ISO 3166-1:2020 *Codes pour la représentation des noms de pays et de leurs subdivisions – Partie 1: Codes de pays*

ISO 8601-1:2019 *Date and time Representations for information interchange – Part 1: Basic rules*

ISO 8601-2:2019 *Date and time Representations for information interchange – Part 2: Extensions*

ISO 15924:2022 *Information and documentation — Codes for the representation of names of scripts*

UN M49, *Standard Country or Area Codes for Statistical Use (Series M, No. 49)*, United Nations, https://unstats.un.org/unsd/methodology/m49/
<figure><img src="image_4.png" /></figure>