# LISN @ SIGMORPHON 2023 Shared Task on
Interlinear Glossing

Shu Okabe, François Yvon

► To cite this version:

Shu Okabe, François Yvon. LISN @ SIGMORPHON 2023 Shared Task on Interlinear Glossing. The 20th SIGMORPHON workshop on Computational Morphology, Phonology, and Phonetics, Association for computational linguistics, Jul 2023, Toronto, Canada. 10.18653/v1/2023.sigmorphon-1.21 . hal-04186388

HAL Id: hal-04186388

https://hal.science/hal-04186388v1

Submitted on 23 Aug 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
LISN @ SIGMORPHON 2023 Shared Task on Interlinear Glossing

Shu Okabe and François Yvon

Université Paris-Saclay & CNRS

LISN, rue du Belvédère

91405 Orsay, France

{shu.okabe,francois.yvon}@limsi.fr

Abstract

This paper describes LISN's submission to the second track (open track) of the shared task on Interlinear Glossing for SIGMORPHON 2023. Our systems are based on Lost, a variation of linear Conditional Random Fields initially developed as a probabilistic translation model and then adapted to the glossing task. This model allows us to handle one of the main challenges posed by glossing, i.e. the fact that the list of potential labels for lexical morphemes is not fixed in advance and needs to be extended dynamically when labelling units are not seen in training. In such situations, we show how to make use of candidate lexical glosses found in the translation and discuss how such extension affects the training and inference procedures. The resulting automatic glossing systems prove to yield very competitive results, especially in low-resource settings.

# 1 Introduction

LISN participated in the ‘open track’ of the shared task on interlinear glossing of SIGMORPHON 2023 (Ginn et al., 2023) with two submissions. Figure 1 presents the format of the sentences for this shared task. In this track, the source sentence **T** is overtly segmented into morphemes (**M**), which yields an explicit one-to-one correspondence between each source morpheme and the corresponding gloss (**G**), thanks to the Leipzig Glossing Rules convention (Bickel et al., 2008). A translation **L** in a more-resourced language (English or Spanish) is also provided, except for Nyangbo. An obvious formalisation of the task that we mostly adopt, is thus to view glossing as a sequence labelling task performed at the morpheme level.

As can be seen in Figure 1, there are roughly two categories of glosses: *grammatical glosses* indicating the grammatical function of the morpheme (e.g., *GEN1*) and *lexical glosses* expressing a meaning (e.g., *son*).<fnref n="1" /> While the grammatical glosses

<table><caption>Figure 1: A sample entry in Tsez: source sentence (T), and its morpheme-segmented version (M), glossed line (G), and target translation (L)</caption><tr><td>T</td><td>Nesis</td><td>l<sup>1</sup>ono</td><td>uži</td><td>zown.</td></tr><tr><td>M</td><td>nesi-s</td><td>l<sup>1</sup>ono</td><td>uži</td><td>zow-n</td></tr><tr><td>G</td><td>he.OBL-GEN1</td><td>three</td><td>son</td><td>be.NPRS-PST.UNW</td></tr><tr><td>L</td><td colspan="4">He had three sons.</td></tr></table>

of a language constitute a finite set of labels, the variety of lexical glosses is unknown, which is one of the main challenges of the task, especially in small training data conditions.

To accommodate such cases, we assume that lexical glosses can be directly inferred from the translation tier. Recent works on automatic gloss generation, such as (McMillan-Major, 2020; Zhao et al., 2020), also rely on a similar assumption and leverage the available translations. In our model, we will thus consider that the set of possible labels for the morphemes in any given sentence consists of the union of (a) all the grammatical glosses, (b) lemmas occurring in the target translation, (c) frequently-associated labels from the training data. By using a variant of Conditional Random Fields (CRFs) (Lafferty et al., 2001), which enables such local restriction of the set of possible labels, our glossing model can be viewed as an extension of previous sequence labelling systems based on CRFs such as (Moeller and Hulden, 2018; McMillan-Major, 2020; Barriga Martínez et al., 2021). In our approach, using translations as labels during training raises the issue of aligning the translation and the source sentence, which we handle with the neural word alignment model of Jalili Sabet et al. (2020). As alignments are computed at the morpheme level, this technique does not apply for the ‘closed track’, where the source segmentation is not part of the training annotations.

Our participation is motivated by two factors:

<footnote>lexical glosses in our submission.</footnote>

<footnote n="1">¹We consider ‘compound’ glosses such as ‘he.OBL’ as</footnote>
to evaluate the model performance across varying training data sizes (from a few dozen to thousands of sentences) and to challenge its ability to handle a variety of high-resource languages in the target translation. Section 2 describes our system, while Section 3 presents our experimental settings. Section 4 reports the complete set of results obtained with our models.

<figure><img src="image_2.png" /><figcaption>Figure 2: Illustration of our approach to label the example source sentence **M** of Figure 1. **G** represents the set of all grammatical glosses in the training data, **T** the set of words occurring in the translation **L**, **D** the set of lexical labels from the training dictionary, and **R** the reference lexical labels seen in training. During training, automatic alignments between **M** and **T** are used.</figcaption></figure>

## 2 System description

Our glossing system uses two main technological components: we (a) rely on an automatic alignment model between the lexical glosses and the target translation during training, which also allows us to exploit additional information regarding target words, such as their Part-of-Speech (PoS) tag or their position; (b) use an extended version of CRFs which allows us to locally restrict the set of possible labels to carry out the glossing task. Figure 2 summarises the main ideas behind our approach.

### 2.1 Aligning lexical glosses with target words

To align the lexical glosses with the target translation, we use the multilingual aligner SimAlign (Jalili Sabet et al., 2020), which relies on the cosine similarity of the source and target unit embeddings. Three heuristics are available to extract the alignments from a similarity matrix; we use the Match method in our submission, since it gave the best results in preliminary experiments. This method considers the alignment task as a maximal matching problem in the bipartite weighted graph containing all possible alignment links between lexical glosses and target words. This heuristic notably

ensures that all lexical glosses are aligned with a target word.<fnref n="2" />

<figure><img src="image_3.png" /><figcaption>Figure 3: Example of SimAlign alignment between lexical glosses and an English translation (Tsez sentence).</figcaption></figure>

Figure 3 displays an example of alignment computed with the Match method. We can note that most alignments are trivial because both units are either identical (e.g. ‘one’) or have the same lemma (e.g. **son**/sons). The remaining links are also of great interest in our case. For the alignment pair (khan/’kings’), although the gloss itself is not in the translation, they are synonyms and share valuable properties such as their PoS tag. Besides, the alignment of **be** with ‘for’ is obviously wrong and only exists because of the constraint of aligning every lexical gloss. Nevertheless, frequent lemmas such as **be** occur in multiple sentences, and their possible labels are observed in the training reference annotations.

### 2.2 Label and label features

Our approach views glossing as a sequence labelling task, meaning that the basic output label for each morpheme is the gloss itself. Our implementation of the CRF model (see below) also enables us to simultaneously predict *label features*, which are arbitrary linguistic properties that can be derived from the label. In our experiments, we chose to incorporate such additional information, which will yield more general, hence more robust, feature functions. In all systems, we thus predict three properties of the label: (a) the actual gloss *g*, (b) a binary category *b* about its nature (GRAM for grammatical glosses, or LEX for lexical glosses), and (c) its projected PoS tag *p* that we collect from the aligned target word.<fnref n="3" />

## 2.3 Probabilistic sequence labelling model

Our system reuses Lost (Lavergne et al., 2011), a probabilistic model initially devised for statistical machine translation. With Lost, it is possible to label arbitrary segments of a source sentence with

<footnote n="2">²Unless there are more lexical glosses than words in the translation.</footnote>

<footnote n="3">³As grammatical morphemes have no aligned target words, we use the generic label GRAM for all grammatical glosses.</footnote>
‘phrases’ from a large bilingual dictionary and to effectively search for the best possible labelling given a set of trained feature weights. Compared to the original translation task, using Lost for automatic glossing brings several simplifications. In particular, there is no need to consider multiple segmentations of the source as the segmentation in morphemes is observed, nor to consider multiple source reorderings, as the translation is also always observed. We thus only focus below on the features of Lost that are relevant for the glossing task.

Lost uses a discriminative model based on the theory of Conditional Random Fields (Lafferty et al., 2001). In a standard CRF, for a sequence *x* of *T* observations, the probability of the corresponding label sequence *y* ∈ 𝒴^T is computed as:

$$p_{\theta}(\mathbf{y}|\mathbf{x}) = \frac{1}{Z_{\theta}(\mathbf{x})} \exp \left\{ \sum_{k=1}^{K} \theta_k G_k(\mathbf{x}, \mathbf{y}) \right\}, \quad (1)$$

where $G_k$ are the feature functions with associated weights $\theta_k$ and $Z_{\theta}(\mathbf{x})$ is the partition function summing over all possible label sequences. In practice, the features usually test local properties (unigram or bigram). Training is performed by maximising the penalised conditional log-likelihood on a set of fully labelled instances.

Implementing this model for machine translation or for our glossing task is challenging. This is because the set of all possible labels is significantly larger than for most sequence labelling tasks, which means that the computational cost of computing $Z_{\theta}(\mathbf{x})$ can get prohibitive, even for sequences of moderate sizes. The implementation we use, Lost (Lavergne et al., 2011), enables us to specify a local (i.e. for a sentence-specific) set of labels, which defines a restricted *search space* both in training and inference: this means that the normaliser in (1) will only consider a restricted number of possible labellings. Using this implementation, the forward-backward computations performed during training remain tractable, even when the number of possible labels gets extremely large. This feature of Lost is also useful here, as we can restrict the set of possible *lexical* glosses by defining a specific search space for each sentence, as we explain below.

## 2.4 Defining the search space

During training, we define the search space associated with the source *x* made of *T* morphemes as comprising all sequences of *T* labels from either:

the set of known grammatical glosses ($G$), the lemmas of the words in the translation ($T$), the most frequent lexical glosses associated with the source morphemes in the training set (this can be viewed as a dictionary $D$), and the gold glosses ($R$) for reference reachability (Liang et al., 2006). This ‘simple’ label set comprises two parts: one ($G$) is common to all sentences, while the remaining labels are defined on a per-sentence basis. In formal terms, the search space is thus ($G \cup T \cup D \cup R)^T$. As explained in Section 2.2, we also consider label features, where the basic labels are augmented with various additional information.

Training the CRF model also requires supervision information, provided here by the reference glosses, from which we readily derive the reference sequence of labels in the search space (an example of reference output labels can be seen on the right-hand part of Figure 4).

During inference, since we have no access to the reference labels, the test search space only comprises the union of the grammatical glosses, the lemmas from the translation, and the labels from the dictionary ($G \cup T \cup D$).<fnref n="4" /> Table 1 displays an example output label from each label set for the S1 setting.

<table><caption>Table 1: Example of output labels extracted from each label set (S1 setting), using the example of Figure 3. The reference label set $R$ is only used during training.</caption><thead><tr><th>set</th><th>g</th><th>b</th><th>p</th></tr></thead><tbody><tr><td>G</td><td>GEN1</td><td>GRAM</td><td>GRAM</td></tr><tr><td>T</td><td>king</td><td>LEX</td><td>NOUN</td></tr><tr><td>D</td><td>khan</td><td>LEX</td><td>NOUN</td></tr><tr><td>R</td><td>khan</td><td>LEX</td><td>NOUN</td></tr></tbody></table>

## 2.5 Feature set

Our two submissions, S1 and S2, use the same model and share most features computed on the source morpheme input. However, the latter extends the former system with additional features.

The input to Lost is the source morpheme *s*, from which we also deduce the following features: its position *p* within the word coded as a numerical value (from 0 to *n*) for complex words, or as ‘F’ for free morphemes, its length *l* in characters,

<footnote n="4">⁴When a lemma is both in the translation and dictionary or repeated in the translation, we still create distinct paths in the search space, as these can be associated with different features (e.g. their PoS and position). The search algorithm will then pick the most likely option.</footnote>
<table><caption>Figure 4: Example of input, outputs, and associated features to Lost for the Tsez reference sentence of Figure 1.</caption><thead><tr><th rowspan="2">i</th><th rowspan="2">input<br>source<br>morph. <i>m</i></th><th rowspan="2">position<br>(in word) <i>t</i></th><th colspan="4">S1 features</th><th colspan="2">S2 features</th><th rowspan="2">reference<br>gloss <i>g</i></th><th rowspan="2">outputs<br>GRAM or<br>LEX <i>b</i></th><th rowspan="2">PoS<br>tag <i>p</i></th><th colspan="3">S2 features</th></tr><tr><th>length<br><i>l</i></th><th>first 3<br>letters <i>d</i></th><th>last 3<br>letters <i>e</i></th><th>copy<br>src <i>cs</i></th><th>position<br>src <i>ps</i></th><th>copy<br> trg <i>ct</i></th><th>position<br> trg <i>pt</i></th></tr></thead><tbody><tr><td>0</td><td>nesi</td><td>0</td><td>4</td><td>nes</td><td>esi</td><td>0</td><td>1/4</td><td>he.OBL</td><td>LEX</td><td>PRON</td><td>0</td><td>1/4</td></tr><tr><td>1</td><td>s</td><td>1</td><td>1</td><td>s</td><td>s</td><td>0</td><td>1/4</td><td>GEN1</td><td>GRAM</td><td>GRAM</td><td>-1</td><td>-2</td></tr><tr><td>2</td><td>1<sup>5</sup>ono</td><td>F</td><td>5</td><td>1<sup>5</sup>o</td><td>ono</td><td>0</td><td>2/4</td><td>three</td><td>LEX</td><td>NUM</td><td>0</td><td>3/4</td></tr><tr><td>3</td><td>uži</td><td>F</td><td>3</td><td>uži</td><td>uži</td><td>0</td><td>2/4</td><td>son</td><td>LEX</td><td>NOUN</td><td>0</td><td>4/4</td></tr><tr><td>4</td><td>zow</td><td>0</td><td>3</td><td>zow</td><td>zow</td><td>0</td><td>3/4</td><td>be.NPRS</td><td>LEX</td><td>VERB</td><td>0</td><td>2/4</td></tr><tr><td>5</td><td>n</td><td>1</td><td>1</td><td>n</td><td>n</td><td>0</td><td>4/4</td><td>PST.UNW</td><td>GRAM</td><td>GRAM</td><td>-1</td><td>-2</td></tr></tbody></table>

and its first and last three letters (d and e respectively). Figure 4 displays an example of input and the associated features.

With all these inputs, we compute unigram and bigram feature functions, detailed in Table 2. On top of the basic unigram and bigram features involving the gloss (top of the table), we also consider the binary category *b* and PoS tag *p* to compute more general feature functions (middle of the table). The idea is to capture associations between specific grammatical labels occurring after a given PoS tag (e.g. (VERB, PST.UNW) with the bi-pos-gloss feature).

In the S2 system, we add two more features: first, a binary variable (uni-copy-trg-src), which is True only for lexical glosses that occur letter-for-letter in the source sentence, to account notably for copied words (e.g. proper nouns). Second, we add a categorical feature (uni-pos-src-trg) encoding information about the relative position of the current morpheme with each target word in the translation, to lower the probability of high-distortion source-target associations. This categorical encoding is computed by chunking each sequence into four parts and reporting the chunk numbers: for instance, the value '(1/4, 3/4)' is used when matching a morpheme in the first quarter of the source sentence with a target word in the third quarter of the target sentence. For any unaligned target word, we use -1 as the corresponding position; for grammatical glosses, we assign the value -2 for the corresponding target word.

# 3 Experimental conditions

## 3.1 Languages

Our (partial) official submission for S1 considers the following five (out of seven) languages: Tsez (ddo), Gitksan (git), Lezgi (lez), Natugu (ntu; surprise language), and Uspanteko (usp; target translation in Spanish). For our second submission (S2),

we could only consider three languages (Tsez, Gitksan, and Lezgi). Since our system relies on the translation to get the lexical glosses, we could not run our models on Nyangbo (nyb), although the corpus has a similar size to other languages we studied. For all submissions, we rely solely on the provided training datasets; no external resource was used.

We have run S2 on Tsez and Uspanteko subsequently and will also report these results below.

## 3.2 Pre-processing

The PoS tags and lemmas are obtained with spaCy,<fnref n="5" /> using the en_core_web_sm and es_core_news_sm pipelines for English and Spanish translations respectively.

All lemmas from the translation are lowercased except when the associated PoS tag is a proper noun ('PROPN').

## 3.3 SimAlign settings

Since the glosses and the translation are in the same language, we use the embeddings from the English BERT (bert-base-uncased) (Devlin et al., 2019) when the target language is English and mBERT ('bert-base-multilingual-uncased') when it is in Spanish (for Uspanteko). We can note here that our model is compatible with multiple target languages, SimAlign being an off-the-shelf multilingual (neural) aligner.

Our preliminary experiments showed that the embeddings from the 0-th layer yielded the best alignments, especially compared to the 8-th layer, which seems to work best in most alignment tasks. A plausible explanation is that contextualised embeddings are unnecessary here because lexical glosses do not constitute a standard English sentence (for instance, they do not contain stop words, and their word order reflects the source language word order).

<footnote n="5">⁵https://spacy.io/.</footnote>
<table><caption>Table 2: Unigram and bigram features for our submissions: S1 features about the main gloss label on top, S1 features involving the two other general outputs, and S2 additional features at the bottom.</caption><thead><tr><th>Feature</th><th>Test</th><th>Example (cf. Figure 4 i = 5)</th></tr></thead><tbody><tr><td>uni-gloss</td><td>1(g<sub>i</sub> = g)</td><td>PST.UNW</td></tr><tr><td>bi-gloss</td><td>1(g<sub>i</sub> = g) ∧ 1(g<sub>i-1</sub> = g')</td><td>(be.NPRS, PST.UNW)</td></tr><tr><td>uni-gloss-morph</td><td>1(g<sub>i</sub> = g) ∧ 1(m<sub>i</sub> = m)</td><td>(PST.UNW, n)</td></tr><tr><td>uni-gloss-position</td><td>1(g<sub>i</sub> = g) ∧ 1(t<sub>i</sub> = t)</td><td>(PST.UNW, 1)</td></tr><tr><td>uni-gloss-length</td><td>1(g<sub>i</sub> = g) ∧ 1(l<sub>i</sub> = l)</td><td>(PST.UNW, 1)</td></tr><tr><td>bi-gloss-morph</td><td>1(g<sub>i</sub> = g) ∧ 1(g<sub>i-1</sub> = g') ∧ 1(m<sub>i</sub> = m)</td><td>(be.NPRS, PST.UNW, n)</td></tr><tr><td>uni-gloss-start</td><td>1(g<sub>i</sub> = g) ∧ 1(d<sub>i</sub> = d)</td><td>(PST.UNW, n)</td></tr><tr><td>uni-gloss-end</td><td>1(g<sub>i</sub> = g) ∧ 1(e<sub>i</sub> = e)</td><td>(PST.UNW, n)</td></tr><tr><td>uni/bi-bin</td><td>1(b<sub>i</sub> = b) (∧1(b<sub>i-1</sub> = b'))</td><td>GRAM ((LEX, GRAM))</td></tr><tr><td>uni/bi-pos</td><td>1(p<sub>i</sub> = p) (∧1(p<sub>i-1</sub> = p'))</td><td>GRAM ((VERB, GRAM))</td></tr><tr><td>uni-bin-morph/position/length</td><td>1(b<sub>i</sub> = b) ∧ 1(m<sub>i</sub> = m)/1(t<sub>i</sub> = t)/1(l<sub>i</sub> = l)</td><td>(GRAM, n) / (GRAM, 1) / (GRAM, 1)</td></tr><tr><td>uni-bin-start/end</td><td>1(b<sub>i</sub> = b) ∧ 1(d<sub>i</sub> = d)/1(e<sub>i</sub> = e)</td><td>(GRAM, n) / (GRAM, n)</td></tr><tr><td>bi-position-bin</td><td>1(t<sub>i</sub> = t) ∧ 1(t<sub>i-1</sub> = t') ∧ 1(b<sub>i</sub> = b)</td><td>(0, 1, GRAM)</td></tr><tr><td>bi-bin-gloss</td><td>1(g<sub>i</sub> = g) ∧ 1(b<sub>i-1</sub> = b')</td><td>(LEX, PST.UNW)</td></tr><tr><td>bi-gloss-bin</td><td>1(b<sub>i</sub> = b) ∧ 1(g<sub>i-1</sub> = g')</td><td>(be.NPRS, GRAM)</td></tr><tr><td>uni-pos-morph</td><td>1(p<sub>i</sub> = p) ∧ 1(m<sub>i</sub> = m)</td><td>(GRAM, n)</td></tr><tr><td>bi-pos-gloss</td><td>1(g<sub>i</sub> = g) ∧ 1(p<sub>i-1</sub> = p')</td><td>(VERB, PST.UNW)</td></tr><tr><td>bi-gloss-pos</td><td>1(p<sub>i</sub> = p) ∧ 1(g<sub>i-1</sub> = g')</td><td>(be.NPRS, GRAM)</td></tr><tr><td>uni-pos-start/end</td><td>1(p<sub>i</sub> = p) ∧ 1(d<sub>i</sub> = d)/1(e<sub>i</sub> = e)</td><td>(GRAM, n) / (GRAM, n)</td></tr><tr><td>uni-copy-trg</td><td>1(ct<sub>i</sub> = ct)</td><td>-1</td></tr><tr><td>uni-copy-trg-src</td><td>1(ct<sub>i</sub> = ct) ∧ 1(cs<sub>i</sub> = cs)</td><td>(-1, 0)</td></tr><tr><td>uni-posi-ts</td><td>1(pt<sub>i</sub> = pt) ∧ 1(ps<sub>i</sub> = ps)</td><td>(-2, 4/4)</td></tr><tr><td>uni-gloss-morph-pts</td><td>1(g<sub>i</sub> = g) ∧ 1(pt<sub>i</sub> = pt) ∧ 1(m<sub>i</sub> = m) ∧ 1(ps<sub>i</sub> = ps)</td><td>(PST.UNW, -2, n, 4/4)</td></tr></tbody></table>

## 3.4 Parameter settings

We always use Lost with the default setting, using only the $l_1$ regularisation penalty $\rho_1 = 0.5$ and keeping the $l_2$ penalty term to $\rho_2 = 0$. This setting gave the best results on average in our preliminary experiments.

## 3.5 Metrics

We use the same evaluation metrics as in the Shared Task: morpheme accuracy, word accuracy, BLEU, and differentiated precision, recall, and F1-score for grammatical (gram) and lexical (stem) glosses.

# 4 Results

Table 3 reports the results for the organiser's baseline<fnref n="6" /> and our systems on the development dataset, while Table 4 gives the corresponding test numbers. We only present the word- and morpheme-level (overall) accuracy, which are the two official metrics of the Shared Task results.<fnref n="7" /> We also report the

results of S2 for Tsez and Uspanteko, which were not available at the time of submission.

<table><caption>Table 3: Accuracy (overall) at the word (top) and morpheme (bottom) levels for the baseline and our two systems on the development dataset. Star-marked values correspond to runs that were not available at the time of submission.</caption><thead><tr><th>model</th><th>ddo</th><th>git</th><th>lez</th><th>ntu</th><th>usp</th></tr></thead><tbody><tr><td>baseline</td><td>74.2</td><td>25.0</td><td>32.6</td><td>-</td><td>75.9</td></tr><tr><td>S1</td><td>83.6</td><td>40.2</td><td>84.4</td><td>88.2</td><td>76.5</td></tr><tr><td>S2</td><td>84.5*</td><td>43.8</td><td>85.1</td><td>88.5</td><td>77.3</td></tr><tr><td>baseline</td><td>85.0</td><td>30.0</td><td>50.1</td><td>-</td><td>81.3</td></tr><tr><td>S1</td><td>91.0</td><td>55.5</td><td>87.3</td><td>92.1</td><td>82.7</td></tr><tr><td>S2</td><td>91.5*</td><td>58.8</td><td>88.2</td><td>92.4</td><td>83.4*</td></tr></tbody></table>

Our systems are consistently better than the baseline, with larger gaps when few training sentences are available (cf. Gitksan or Lezgi). Our second system slightly improves the accuracy on the development set; a similar trend can also be observed on the test set.

Compared to other submitted systems, we reached the best word accuracy for Gitksan and

<footnote n="6">⁶https://github.com/sigmorphon/2023glossingST/tree/main/baseline.</footnote>

<footnote n="7">⁷https://github.com/sigmorphon/2023glossingST/blob/main/results.md.</footnote>
<table><caption>Table 4: Accuracy (overall) at the word (top) and morpheme (bottom) levels for the baseline and our two systems on the *test* dataset. Star-marked values correspond to runs that were not available at the time of submission.</caption><thead><tr><th>model</th><th>ddo</th><th>git</th><th>lez</th><th>ntu</th><th>usp</th></tr></thead><tbody><tr><td>baseline</td><td>75.7</td><td>16.4</td><td>34.5</td><td>41.1</td><td>76.6</td></tr><tr><td>S1</td><td>84.9</td><td>28.4</td><td>83.4</td><td>88.8</td><td>76.3</td></tr><tr><td>S2</td><td>85.5*</td><td>31.5</td><td>83.0</td><td>89.3</td><td>76.7*</td></tr><tr><td>baseline</td><td>85.3</td><td>25.3</td><td>51.8</td><td>49.0</td><td>82.5</td></tr><tr><td>S1</td><td>91.4</td><td>50.8</td><td>87.2</td><td>92.6</td><td>82.4</td></tr><tr><td>S2</td><td>91.8*</td><td>51.1</td><td>87.0</td><td>92.8</td><td>82.7*</td></tr></tbody></table>

Natugu and the best morpheme accuracy for Natugu.

# 5 Discussion

## 5.1 Impact of training data size

Table 5 displays the evolution of the F1-scores at the morpheme levelCheck this broken down by gloss type (lexical and grammatical) for three sizes of the training dataset in Natugu (200, 500, and all 791 sentences). For both settings, the model reaches better scores for grammatical glosses, and, unsurprisingly, lexical glosses benefit more from the increase in training data. While the additional features in S2 were mostly introduced to improve the lexical gloss prediction in the small resource condition, it is noteworthy that they also help improve the prediction of grammatical labels. Similar observations were made for the other test languages.

<table><caption>Table 5: F1-scores for grammatical and lexical glosses with an increasing number of training data in Natugu.</caption><thead><tr><th rowspan="2"></th><th colspan="2">S1</th><th colspan="2">S2</th></tr><tr><th>gram</th><th>lex</th><th>gram</th><th>lex</th></tr></thead><tbody><tr><td>200</td><td>93.3</td><td>80.5</td><td>93.6</td><td>81.3</td></tr><tr><td>500</td><td>95.3</td><td>88.5</td><td>95.2</td><td>88.3</td></tr><tr><td>full</td><td>95.7</td><td>89.5</td><td>95.9</td><td>89.6</td></tr></tbody></table>

## 5.2 Number of selected features

Table 6 presents the number of active features (in thousands) selected among all features (in millions) by S1 and S2. We note here that thanks to the l₁-regularisation, most feature weights are set to 0 since less than 1% of the features are actually

active. For illustrative purposes, Appendix A lists the features with the largest weight for the Lezgi system.

<table><caption>Table 6: Number of active features (out of the total number of computed features) for each setting and language.</caption><thead><tr><th></th><th>ddo</th><th>git</th><th>lez</th><th>ntu</th><th>usp</th></tr></thead><tbody><tr><td>S1</td><td>167k (170M)</td><td>3k (0.8M)</td><td>43k (24M)</td><td>60k (39M)</td><td>132k (34M)</td></tr><tr><td>S2</td><td>174k (172M)</td><td>3k (0.8M)</td><td>46k (24M)</td><td>64k (40M)</td><td>137k (35M)</td></tr></tbody></table>

# 6 Conclusion

Assuming the lexical glosses can be aligned with words in the target translation, we repurposed a statistical machine translation system based on a globally-normalised model, akin to CRFs, that allows us to dynamically define a local set of labels for the automatic gloss generation task. Using two sets of features, our systems are compatible in low- and very low-resource settings and outperformed the baseline models according to several evaluation metrics.

We plan on further exploring feature functions on both source and target sides. Besides, since our systems rely on automatic alignments, which may contain and project some noise, we will try to remove this dependency modelling alignment as an unobserved variable in a latent variable model. Furthermore, as our submission focused on low-resource data conditions, we did not consider neural methods, which are notably data-intensive; future work would be to integrate word embeddings for better-resourced languages such as Arapaho.

Our code is available at: https://github.com/shuokabe/gloss_lost.

# Acknowledgements

This work was partly funded by French ANR and German DFG under grant ANR-19-CE38-0015 (CLD 2025). The authors warmly thank Thomas Lavergne for his help and assistance regarding the configuration and exploitation of Lost.

# References

* Diego Barriga Martínez, Victor Mijangos, and Ximena Gutierrez-Vasques. 2021. Automatic interlinear glossing for Otomi language. In *Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas*, pages 34–43, Online. Association for Computational Linguistics.
Balthazar Bickel, Bernard. Comrie, and Martin Haspelmath. 2008. The Leipzig Glossing Rules: Conventions for interlinear morpheme-by-morpheme glosses. Leipzig: Max Planck Institute for Evolutionary Anthropology, Department of Linguistics. https://www.eva.mpg.de/lingua/resources/glossing-rules.php.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Michael Ginn, Sarah Moeller, Alexis Palmer, Anna Stacey, Garrett Nicolai, Mans Hulden, and Miikka Silfverberg. 2023. Findings of the SIGMORPHON 2023 shared task on interlinear glossing. In Proceedings of the 20th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics.

Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1627–1643, Online. Association for Computational Linguistics.

John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, page 282–289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.

Thomas Lavergne, Alexandre Allauzen, Josep Maria Crego, and François Yvon. 2011. From n-gram-based to CRF-based translation models. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 542–553, Edinburgh, Scotland. Association for Computational Linguistics.

Percy Liang, Alexandre Bouchard-Côté, Dan Klein, and Ben Taskar. 2006. An end-to-end discriminative approach to machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 761–768, Sydney, Australia. Association for Computational Linguistics.

Angelina McMillan-Major. 2020. Automating gloss generation in interlinear glossed text. In Proceedings of the Society for Computation in Linguistics 2020, pages 355–366, New York, New York. Association for Computational Linguistics.

Sarah Moeller and Mans Hulden. 2018. Automatic glossing in a low-resource setting for language documentation. In Proceedings of the Workshop on Computational Modeling of Polysynthetic Languages, pages 84–93, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Xingyuan Zhao, Satoru Ozaki, Antonios Anastasopoulos, Graham Neubig, and Lori Levin. 2020. Automatic interlinear glossing for under-resourced languages leveraging translations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5397–5408, Barcelona, Spain (Online). International Committee on Computational Linguistics.

## A Feature weights

<table><caption>Table 7: Top 10 (positive) features of S2 for Lezgi.</caption><thead><tr><td>Type</td><td>Feature</td><td>Weight</td></tr></thead><tbody><tr><td>uni-gloss-start</td><td>say ∧ луг</td><td>3.22</td></tr><tr><td>bi-gloss-morph</td><td>say ∧ AOR ∧ лагъа</td><td>3.22</td></tr><tr><td>bi-gloss-morph</td><td>talking ∧ AOC ∧ гафарун</td><td>2.80</td></tr><tr><td>uni-gloss-morph-pts</td><td>. ∧ -1 ∧ . ∧ 4/4</td><td>2.75</td></tr><tr><td>bi-gloss-morph</td><td>fortress ∧ OBL ∧ къеле</td><td>2.65</td></tr><tr><td>uni-gloss-end</td><td>now ∧ ила</td><td>2.65</td></tr><tr><td>uni-gloss-start</td><td>dog ∧ киц</td><td>2.65</td></tr><tr><td>bi-gloss-morph</td><td>newspaper ∧ OBL ∧ газет</td><td>2.49</td></tr><tr><td>uni-gloss-start</td><td>girl ∧ руш</td><td>2.49</td></tr><tr><td>bi-gloss-morph</td><td>SBST ∧ PST ∧ ди</td><td>2.49</td></tr></tbody></table>

Table 7 displays the features with the largest weight in the S2 system trained on the Lezgi corpus. We can notice here that some (initial or final) character trigram features (uni-gloss-start and uni-gloss-end) are relevant, corresponding to lexemes that either typically occur with an inflexion mark: ‘лугъу’ and ‘лугъун’ for ‘say’, occurring approximately 200 times together or that combine with a prefix, as ‘гила’ and ‘игила’ for ‘now’ (around 20 co-occurrences).