Findings of the SIGMORPHON 2023 Shared Task on Interlinear Glossing
Year: 2023

Abstract:
This paper presents the findings of the SIGMORPHON 2023 Shared Task on Interlinear Glossing. This first iteration of the shared task explores glossing of a set of six typologically diverse languages: Arapaho, Gitksan, Lezgi, Natügu, Tsez and Uspanteko. The shared task encompasses two tracks: a resource-scarce closed track and an open track, where participants are allowed to utilize external data resources. Five teams participated in the shared task. The winning team Tü-CL achieved a 23.99%-point improvement over a baseline RoBERTa system in the closed track and a 17.42%-point improvement in the open track.

heading: 1 Introduction
text: Roughly half of the world’s languages are currently endangered (Seifart et al., 2018). As a result, language preservation and revitalization have become significant areas of focus in linguistic research. Both of these endeavors require thorough documentation of the language, which is crucial for creating grammatical descriptions, dictionaries, and educational materials that aid in language revitalization. However, traditional manual language documentation is a time-consuming and resource-intensive process due to the costs associated with collecting, transcribing, and annotating linguistic data. Therefore, there is a need to expedite the documentation process through the use of automated methods. While these methods can never fully replace the expertise of a dedicated documentary linguist, they have the potential to greatly facilitate and accelerate the annotation of linguistic data (Palmer et al., 2009).
Linguistic annotation involves several interconnected subtasks, including: (1) transcription of speech recordings, (2) morphological segmentation of transcribed speech, (3) glossing of segmented morphemes, and (4) translation of the transcriptions into a matrix language, such as English.
These processes result in a semi-structured output known as an interlinear gloss, as demonstrated in the Natügu example below:
(1) ma house yrkr-tx-o-kz-Ø finish-INTS-GDIR.DOWN-also-3MINIS .
. Houses were gone too.
This paper presents the findings of the SIGMORPHON 2023 Shared Task on Interlinear Glossing1, which focuses on automating step (3) of the language documentation pipeline. Notably, this shared task represents the first initiative specifically dedicated to interlinear glossing. Despite the prevalence of interlinear glossed text as a data format in language documentation, the automatic generation of glossed text remains relatively underexplored in the field of natural language processing (NLP). We hope that this shared task can help stimulate further work in automated glossing.

heading: 2 Background
text: Existing work in data driven automated glossing has utilized both traditional feature-based approaches like maximum entropy classifiers (MEMM) (Ratnaparkhi, 1996) and conditional random fields (CRF) (Lafferty et al., 2001) as well as more recent neural models like LSTM encoderdecoders (Sutskever et al., 2014) and transformers (Vaswani et al., 2017). Palmer et al. (2009) investigate active learning for interlinear glossing using the MEMM architecture. McMillan-Major (2020) incorporated translations as auxiliary supervision in a CRF glossing model. Moeller and Hulden (2018) and Barriga Martínez et al. (2021) compare traditional feature-based models and LSTM encoder-decoder models. Zhao et al. (2020) present a modified multi-source transformer model which incorporates translations as auxiliary supervision.
The current literature on automatic glossing exhibits notable gaps, as several techniques that have
1https://github.com/sigmorphon/2023glossingST
186
proven valuable for other morphology tasks have yet to be explored for glossing. There are several intriguing directions for future research, including:
1. Crosslingual training (Çöltekin, 2019; Anastasopoulos and Neubig, 2019) has shown promise for morphological inflection and could be investigated for its potential in glossing.
2. Incorporating additional noisy training data (Wiemerslage et al., 2023) can improve accuracy for low-resource inflection and could help improve the performance of glossing models as well. In the context of interlinear glossing, this data could come from large multilingual databases like ODIN (Lewis and Xia, 2010) which is automatically created with the aid of web crawling and is known to be noisy.
3. Data augmentation techniques (Liu and Hulden, 2021; Anastasopoulos and Neubig, 2019; Silfverberg et al., 2017) are now a wellestablished technique in morphological inflection and could enhance the training process for glossing models.
4. Hard attention models (Aharoni and Goldberg, 2017; Makarov et al., 2017) have delivered strong performance for several morphology tasks in low-resource settings and could also be applied to interlinear glossing.
5. Multitask training (Rama and Çöltekin, 2018) and meta-learning (Kann et al., 2020) techniques could be leveraged to enhance glossing performance.
6. Finally, pretrained language models like ByT5 (Xue et al., 2022) have demonstrated strong performance in various morphology tasks, yet their potential for interlinear glossing remains unexplored.
The submissions in this shared task explore several of these techniques, including the use of pretrained language models, data augmentation, utilization of external data, and the application of hard attention models.

heading: 3 Tasks and Evaluation

heading: 3.1 Interlinear Glossed Text
text: Interlinear Glossed Text (IGT) serves as a means to capture the syntactic and morphological charac-
teristics of words within a corpus. It is a semistructured format which lacks strict annotation standards, leading to variations in annotation practices among different annotators. These variations can be influenced by documentation requirements, adopted theoretical frameworks, and other factors (Palmer et al., 2009).
For this shared task, the data adheres to the Leipzig glossing conventions (Lehmann, 1982). The Leipzig format follows a three-line documentation style, including morphological segmentation of the input tokens, glosses of individual morphemes, and translations. Below is an example from Arapaho, one of the languages used in the shared task:
(2) nih-bii3ihi-noo 2S.PAST-eat-1S nohkuseic morning
I ate this morning.
In this example, the first line represents the morphological segmentation, the second line provides glosses for each morpheme, and the third line presents the corresponding translation.
The transcription line (nih-bii3ihi-noo nohkuseic in Example 2) gives the orthographic transcription of a sentence, phrase or utterance in the source language. The transcription may be segmented with dashes to indicate morpheme boundaries.
The gloss line ("2S.PAST-eat-1S morning" in Example 2) provides a linguistic gloss for each morpheme in the transcription line. For glossing, morphemes are grouped into two distinct categories:
1. Functional morphemes or grams include affixes and functional words which do not carry their own lexical meaning. Functional morphemes are glossed using uppercase labels like 1S (first-person singular affix) which indicate grammatical category and/or syntactic function. Portmanteau morphs, which denote multiple functions, can be glossed using compound labels like 2S.PAST. Gloss labels typically come from a fixed inventory like UniMorph (Sylak-Glassman, 2016; Kirov et al., 2018; Batsuren et al., 2022b), although conventions are not standardized and are often varied to fit the needs of the language.
2. In contrast to functional morphemes, lexical morphemes or stems are open-class words and stems which carry semantic meaning. These are glossed in lowercase using their translation
in a matrix language like English or Spanish; thus, for example, bii3ihi is glossed as eat.
The translation line (’I ate this morning.’ in Example 2) of an IGT entry provides a translation in a high-resource language such as English. The tokens in the translation are not necessarily aligned with specific words in the source language, as languages often express equivalent concepts in differing numbers of words.

heading: 3.2 The Interlinear Glossing Task
text: The objective of the shared task is to develop automated systems capable of predicting the gloss of a given input utterance, using its orthographic transcription and translation as input. The glossing task presents several key challenges, such as disambiguation of ambiguous morphemes and accurate translation of word stems. The shared task explores two distinct resource settings, referred to as tracks, which differ in terms of the supervision provided during model training and at test-time.
The Closed Track (Track 1) In the closed track, the input consists of the orthographic transcription of the target utterance, for example, nihbii3ihinoo nohkuseic (Arapaho), and its translation to a matrix language like English: ’I ate this morning’ (note the lack of morpheme boundaries in the transcription). The aim is to generate a gloss 2S.PAST-eat-1S morning. This setting poses a significant challenge since the glossing model does not have access to a morphological segmentation of the input utterance. Therefore, it must infer the number of morphemes and the identity of the component morphemes for each input word without any supervision. The closed setting draws inspiration from the work of Zhao et al. (2020), which utilizes a similar setup.
The Open Track (Track 2) In a practical language documentation setting, various types of resources can be available as auxiliary supervision when training glossing systems. These resources may include manually glossed text, morphological segmentations, dictionaries, raw text in the target language, and more. The open track aims to explore the extent of glossing performance achievable when participants are allowed to utilize auxiliary resources. In addition to the data provided in the closed track, morphological segmentations are provided in the open track. For instance, for the Arapaho example mentioned earlier, a morphological segmentation nih-bii3ihi-noo nohkuseic
would be included. Gold standard segmentations are provided both for model training and at testtime. Moreover, participants are encouraged to make use of external data resources except for additional glossed text in the target language.

heading: 3.3 Evaluation of Glossing Performance
text: We evaluate glossing performance with regard to two metrics: word-level and morpheme-level glossing accuracy. Word-level glossing accuracy is defined as the fraction of words in the test data which received a fully correct gloss like 2S.PAST-eat-1S:
wacc = Count(correctly glossed tokens)
Count(all tokens) (1)
Note that all the individual morphemes in the word have to be correctly glossed. In contrast, morpheme-level glossing accuracy is defined as the fraction of morphemes in the test data which received the correct gloss:
macc = Count(correctly glossed morphemes)
Count(all morphemes) (2)
In the closed track, where gold standard morphological segmentations are not provided, it may happen that the system predicts too few or too many glosses for an input word. This complicates computation of morpheme-level glossing accuracy. When too few morphemes are predicted, we pad the predictions with NULL morphemes until the number of morphemes corresponds to the gold standard gloss (e.g. 2S.PAST-eat → 2S.PAST-eat-NULL). When too many morphemes are predicted, we discard extra morphemes at the end of the output (e.g. 2S.PAST-eat-1S-PL→ 2S.PAST-eat-1S).
For the official shared task results, we compute accuracy over multiple languages. We then report micro average glossing accuracy across the different languages. Micro average word-level glossing accuracy is used for the official ranking of the participating submissions.

heading: 3.4 Comparison to Other NLP Tasks
text: While interlinear glossing forms a distinct and interesting NLP task in its own right, it has connections to many commonly explored NLP tasks, particularly part-of-speech (POS) tagging, lemmatization, morphological tagging2, and morphological segmentation (McCarthy et al., 2019; Cotterell and
2Also known as morphological analysis in context.
Heigold, 2017; Müller et al., 2015; Batsuren et al., 2022a). All of these tasks involve varying degrees of grammatical analysis.
Interlinear glossing is particularly strongly connected to morphological tagging as both involve morphological annotation in context. However, there are two major differences between the tasks:
1. In interlinear glossing, a morpheme-level annotation of the input sentence is generated. The output of a glossing model provides the order of various morphological elements in the input tokens, indicating the position of different affixal elements. In contrast, morphological tagging provides a more abstracted representation where the order of morphemes is lost.
2. Another difference between morphological tagging and interlinear glossing is related to the treatment of lexical elements. In morphological tagging, it is common to return the lemma of input words along with the associated grammatical information of the inflected input word. In glossing, on the other hand, it is common to annotate word forms with a translation of the input lexeme in a matrix language like English. This substantial difference between the tasks introduces elements of machine translation into the morphology task.
Following the approaches of McMillan-Major (2020) and Zhao et al. (2020), the shared task datasets provide gold standard translations of the input sentences as additional supervision during both training and test time. Thus, the task of lexeme translation involves retrieving the lemma of the correct lexemes from the provided translation.

heading: 4 Data

heading: 4.1 Languages and Glossed Data
text: Arapaho [arp] is an Algonquian language with a few hundred speakers in Wyoming, USA. It is highly agglutinating and polysynthetic, with the verb carrying the heaviest morphological load (Cowell and Moss, 2008). Polysynthesis in Arapaho includes noun incorporation, where special forms of certain nouns become part of the verb. The corpus used in this shared task contains narratives and conversation that have been documented starting in the 1880s until the present day, including a few religious texts that are translations from
English. It is written in the popular Arapaho orthography. Much of the data is available through the Endangered Languages Archive3 or the Center for the Study of Indigenous Languages of the West4.
Gitksan [git] The Gitksan are one of the Indigenous peoples of the northern interior region of British Columbia, Canada. Today, Gitksan is the most vital Tsimshianic language, but is still critically endangered with an estimated 300-850 speakers (Dunlop et al., 2018). The language has an “analytic to synthetic” morphology (Rigsby, 1986, 1989) and, unlike many Canadian Indigenous languages, it is not polysynthetic. It has a rich assortment of derivational morphemes and substantial capacity for compounding; consequently, its degree of word-complexity has been described as similar to German (Tarpent, 1987). The data used for the shared task were extracted from a paper containing three stories by the Gitksan elders Barbara Sennot, Hector Hill and Vincent Gogag (Forbes et al., 2017).
Lezgi [lez] (aka Lezgian) is a NakhDaghestanian (Northeast Caucasian) language spoken by over 500,000 speakers in Russia and Azerbaijan (Eberhard et al., 2023). The corpus used is from the Qusar dialect in Azerbaijan (Donet, 2014). It is a highly agglutinative language with overwhelmingly suffixing morphology (Haspelmath, 1993). Noun cases are formed by case-stacking which is a unique characteristic of Nakh-Daghestanian languages. Instead of a unique morpheme for each case, case-stacking composes case inflections by “stacking” sequences of case suffixes as illustrated in Table 1.
Natügu [ntu] belongs to the Reefs-Santa Cruz group in the Austronesian family. It is spoken by about 4,000 people in the Temotu Province of the Solomon Islands. It has primarily agglutinative morphology with complex verb structures (Åshild Næss and Boerger, 2008). The corpus used for the shared task contains transcribed narratives and a large written text.5
Tsez [ddo] (aka Dido) belongs to the TsezHinukh branch of the Nakh-Daghestanian family.
3https://elar.soas.ac.uk/Collection/MPI189644 4https://www.colorado.edu/center/csilw/
arapaho-language-archives 5Natqgu grammar and large text available at https://www. langlxmelanesia.com/tilp
It has about 14,000 speakers in Daghestan, Russia. It has a rich agglutinative, suffixing morphology. The corpus is part of the Tsez Annotated Corpus Project (Comrie et al., 2022; Abdulaev and Abdullaev, 2010).6
Tutrugbu [nyb] (aka Nyagbo, Nyangbo) is a Niger-Congo language with a few thousand estimated speakers in Ghana (Eberhard et al., 2023). It is a highly agglutinative language that features some reduplication (Essegbey, 2019). The corpus from which the shared task data was extracted contains a variety of spontaneous data supplemented with elicited data collected with a range of documentary techniques.7
Uspanteko [usp] (aka Uspantek) belongs to the K’ichean branch of the Mayan language family spoken by as many as 6000 speakers in the Guatemalan highlands and in diaspora communities (Bennett et al., 2016). Uspanteko is a lightly agglutinative language with complex verbal morphology and ergative-absolutive alignment (Coon, 2016). Uspanteko is unusual among Mayan languages for
6https://tsezacp.clld.org/ 7‘Unpublished Nyangbo (Tutrugbu) texts’ compiled by Dr.
James Essegby
its use of contrastive lexical tone (Bennett et al., 2022).8 The texts were collected, transcribed, translated and annotated as part of an OKMA Mayan language documentation project (Pixabaj et al., 2007) and are currently accessible via the Archive of Indigenous Languages of Latin America.9 The corpus includes oral histories, personal experience texts, and stories; preprocessing of the corpus is described in Palmer et al. (2010).

heading: 4.2 Shared Task Data
text: Shared task datasets were generated from original glossed source data in various formats (LaTeX, CLDF10 and Flex11) using dedicated conversion scripts. We aimed to make minimal changes to the original glossed data while ensuring consistent annotation practices across languages. All morpheme boundaries were converted to a unified format using hyphens ("-"), all glossed word stems were lowercased (or titlecased in the case of proper nouns) and all affix glosses were uppercased. Apart from potential changes to casing, gloss symbols were not modified. Portmanteau morphs, where morphemeboundaries cannot be identified, were glossed using a period syntax (".") as in the examples here.it.is and 2S.PAST.
An example of a glossed Arapaho sentence in the official shared task format is given in Figure 1. This entry comes from the open track (track 2), where morphological segmentations are provided. The following lines are included in the gloss:
\t the orthographic representation,
\m the morphological segmentation of the orthographic representation,
\g the gloss of the orthographic representation and
\t the English or Spanish translation.
8Tone is not, however, marked in the shared task dataset. 9https://ailla.utexas.edu
10https://cldf.clld.org/ 11https://software.sil.org/fieldworks/
The token counts in the transcription, segmentation and gloss of a given example have to match. However, the token count in the translation line is allowed to differ. Examples in the source data which did not follow this restriction were filtered out.
We split the datsets into non-overlapping training, development and test data. For languages where there was a clear division into separate texts, we aimed to use one complete text for development and testing, respectively, and the rest of the data for training. This was the case for Gitksan and Arapaho.12 For the rest of the languages, we used 80% of the sentences for training, and 10% for development and testing, respectively. Statistics on data sizes are provided in Table 2. Note that the table gives token counts, not sentence counts, and the counts do not, therefore, exactly correspond to an 80-10-10 split.
Data characteristics The shared task datasets encompass a range of diverse data conditions. The training data size, as shown in Table 2, varies from approximately 140k tokens for Arapaho to a mere 261 tokens for Gitksan, with most languages having between 2k and 15k tokens of training data. With the potential exception of Arapaho and Uspanteko, all the languages qualify as low-resourced datasets. Additional characteristics of the datasets are presented in Table 3:
1. Type-token-ratio (TTR) for most languages falls within the 20-30% range with the notable exception of Gitksan where TTR is 61.3% which is likely to be related to the very small size of the training set.
2. We compute out-of-vocabulary (OOV) rates on the test set. For most languages, OOV rates are below 30% with Gitksan once again being a notable exception with OOV rate of 79.9%. In general, these rates are high compared to typical OOV rates for English text.
3. As a further analysis, we also report morpheme-level OOV rates on the test set, which can be more illuminating for morphologically complex languages. These fall below 10% for most languages with the exception of Gitksan, where morpheme-level OOV is
12For Arapaho, text 56 is used for development and text 63 for testing. For Gitksan, we used Hector Hill’s story for development and Vincent Gogag’s story for testing.
41.2%, again due to the very small training set.
In Table 3, we also report statistics related to the morphological characteristics of the languages:
1. The average number of morphemes per word can be computed based on the morphological segmentations provided for track 2. For training data, this ranges from 1.4, for Uspanteko, to 2.0 for Tsez, meaning that many multimorphemic words can be found in all of the datasets.
2. Finally, we also compute the gloss ambiguity, that is, the average number of distinct glosses that a morpheme receives in the training data. For example, the English suffix -s is ambiguous between two readings because it can be both a number and tense marker. Glossing ambiguity can be seen as one indicator of the difficulty of a glossing task. For most of the shared task languages, it is very close to 1. The only exceptions are Gitksan (1.3) and Uspanteko (1.2), both of which contain frequent and ambiguous affixes.
The shared task datasets also provide English or Spanish translations, which can be valuable when glossing word stems. Table 4 presents statistics on how often the correct stem translation can be found in the utterance translation.13. We present separate statistics for in-vocabulary tokens, which have been observed in the training set, and for out-ofvocabulary (OOV) tokens, which are absent from the training set. The coverage ranges from 37% for Uspanteko (40% for OOV tokens) to 71% for
13To compute these statistics, the translations in the test set were first lemmatized using the Stanza toolkit (Qi et al., 2020).
Tsez (72% for OOV tokens). This demonstrates that translations are likely to contain valuable information for the glossing task, particularly for OOV tokens, which can be challenging to gloss without access to stem translations.

heading: 5 Glossing Systems

heading: 5.1 The Baseline System
text: The baseline system utilizes the RoBERTa architecture with default hyperparameters (Liu et al., 2019). The glossing task is treated as a token classification task, where words or morphemes form the input, and the IGT gloss (or gloss compound) forms the output label. In the closed track, we train wordlevel models; in the open track, where morphological segmentations are provided, morphemes form the input units to the glossing model. The baseline model is trained on the shared task training data without pretraining. We train one model for each language. For a detailed presentation of the baseline system, please see Ginn (2023).14
A transformer-based architecture is an effective choice for this task, as interlinear glossing often involves disambiguating homonymous morphemes based on context. For example, the English plural morpheme -s is spelled the same as the presenttense third-person singular verb morpheme, and the correct label must be determined from the lexical and sentence context. We decided to use a masked architecture rather than a sequence-to-sequence setup. During initial development, we also experimented with a sequence-to-sequence architecture, but this required more data to converge, and delievered inferior performance. Error analysis revealed this to be due to isolated insertions and deletions of morphemes. This is difficult to fix because there exists no a priori restriction on the morpheme count generated by the model.
The baseline system includes a number of known limitations which leave room for improvement; particularly, it can not effectively handle out-ofvocabulary words or morphemes, does not perform any segmentation in the closed track, and does not make use of part-of-speech tags or other resources in the open track. The system also does not utilize translations.
14Code for the baeline system can be found in the shared task repository https://github.com/sigmorphon/ 2023glossingST/tree/main

heading: 5.2 Participant Systems
text: Here we describe the participating systems. Table 5 provides an overview of the strategies employed by the different teams.
COATES (Coates, 2023) This system is based on the LSTM encoder-decoder architecture (Sutskever et al., 2014) and participated in the closed track of the shared task. The input to the glossing system consists of short context windows centered at the target word. Windows of width 1 and 2 are used to generate candidate predictions and the final output prediction of the model is generated using weighted voting among the output candidates.
LISNTEAM (Okabe and Yvon, 2023) This submission is a hybrid CRF-neural system and participated in the open track of the shared task. The system is a combination of two components: (1) An unsupervised neural alignment system SimAlign (Sabet et al., 2020) originally intended for machine translation, and (2) A CRF sequence labeling system Lost (Lavergne et al., 2011). The alignment system is used during training to associate word stems with lexemes in the translations. It uses cosine similarity of BERT representations (Devlin et al., 2019) to score the association between lexemes in the translation and the word stems in the gloss. Alignment allows the system to learn to pick lexemes from the translation line for stems which do not occur in the training data and thus to gloss unseen word forms. The CRF model is used to gloss the morphemes in the input sentence. The team submitted two systems LISNTEAM1 and LISNTEAM2 which differ with regard to the featurization of the CRF model.
SIGMOREFUN (He et al., 2023) This team submitted transformer-based systems and participated in the open track of the shared task. The authors experiment with the pretrained byte-level transformer model ByT5 (Xue et al., 2022) and the multilingual pretrained transformer XLM-RoBERTa (Conneau et al., 2020) fine-tuned for glossing. Interestingly, the ByT5 model falls behind the XLM-RoBERTa model in terms of glossing accuracy. To boost performance, the team incorporate additional glossed data from the ODIN database and, for Gitksan, lexemes from a Gitksan morphological analyzer (Forbes et al., 2021). The team also experiment augmenting the gold standard training sets with artificially generated glossing data. This team incorporated both translations and segmentations into
the model input using specialized prompts. The team made four submissions to the shared task SIGMOREFUN1 – SIGMORFUN4 displaying different combinations of model and data augmentation strategy.
TEAMSIGGYMORPH (Cross et al., 2023) This team participate both in the open and closed track. They investigate the performance of different input and output representations: character-based, bytebased and subword-based. For the closed track, the team used a vanilla transformer model. For the open track, they applied a BiLSTM encoderdecoder model and the ByT5 byte-level transformer model. The team accomplished stem-translation using a heuristic approach which combines translation statistics computed from the training set and copying of unseen stems, which often represent proper names. Like team SIGMOREFUN, this team also found that ByT5 underperformed compared to other model architectures.
TÜ-CL (Girrbach, 2023) This team participated both in the open and closed track of the shared task (in fact, the team also participated in this year’s SIGMORPHON inflection shared task using the same model). The system uses straight-through
gradient estimation (Bengio et al., 2013) to train a hard-attentional neural glossing model. For the closed track submission, the system induces a shallow morphological segmentation of the input text. This happens without any segmented training data which is not available in the closed track. Morpheme boundaries are assigned using the hard attention weights learned by the model. For the open track, gold standard segmentations are used. For both tracks, gloss tags and stems are then predicted for each morpheme using an MLP. This model delivers very strong performance while, surprisingly, not utilizing translations in any way.

heading: 6 Results and Discussion

heading: 6.1 Closed track (track 1)
text: The official shared task results for the closed track are presented in Table 6. Three teams participated in the closed track and two of these teams presented a complete submission for all shared task languages and beat the baseline system. Only teams with a complete submission (TÜ-CL and COATES) were eligible to participate in the official shared task evaluation. Of these two teams, TÜ-CL achieved the best micro average word-level glossing accuracy 71.30% with their second submission TÜ-CL2. Team TÜ-CL also delivering the best performance for all individual languages in track 1.
It is noteworthy that both teams TÜ-CL and COATES beat the shared task baseline by wide margins: 23.99%-points for TÜ-CL and 12.24%- points for COATES. This demonstrates that even in the resource-scarce closed track setting, large improvements in glossing accuracy are possible over a baseline transformer system. All track 1 submissions strongly outperform the baseline for Nyangbo. Likewise, we see great improvements over the baseline for Lezgi and Natügu.
Results for morpheme-level glossing accuracy
largely mirror those of word-level accuracy. Again TÜ-CL delivers the best performance for all languages. A general observation is that morphemelevel accuracies in track 1 are lower than wordlevel accuracies. This can be attributed to the fact that multi-morphemic words are often difficult to gloss correctly when the morphological segmentation is not give. A single incorrectly identified morpheme boundary will often result in several incorrectly glossed morphemes. To see why this is the case, consider the English past tense verb form walked. If the word is incorrectly analyzed as a monolithic adjective, both the stem walk and past tense marker -ed will be incorrectly glossed. This effect weighs down morpheme-level accuracy for the closed track.

heading: 6.2 Open track (track 2)
text: The official shared task results for the open track are presented in Table 7. In the open track, we got submissions from four teams, two of which presented complete submissions for all shared task languages. Both of these teams beat the baseline with regard to micro averaged word-level glossing accuracy. Similarly as in the closed track, TÜ-CL achieved the best overall performance and the best performance for most languages. For Arapaho, the SIGMORFUN team achieved the best performance and, for Natügu and Gitksan, LISNTEAM achieved the best performance. TÜ-CL beat the baseline system with regard to micro average word-level glossing accuracy by a wide margin of 17.42%- points.
Overall performance in the open track is, un-
derstandably, higher than in the closed track due to the fact that gold standard morphological segmentations were provided during training and test time, and additional resources were allowed, which some of the participants utilized. However, absolute improvement over the baseline is lower in the open track than the closed track. This may be a consequence of the fact that the learning problem in the open track is easier. It is also noteworthy that morpheme-level performance is higher than wordlevel performance for the open track, whereas the opposite is true for the closed track. This is understandable because gold standard morphological segmentations are provided and a single isolated glossing error is less likely to ruin the gloss for the complete word form in the open track.

heading: 6.3 Analysis of performance
text: We now present a more detailed analysis of the shared task results. This analysis is related to Figure 2 which presents average performance of shared task systems on the different languages and their relationship with training data size, out-ofvocabulary (OOV) rate and type-token-ratio (TTR).
Impact of training data size The size of the training set is one of the most influential factors determining the performance of natural language processing systems. This observation also holds true for the shared task results. The training data sizes vary from 261 tokens for Gitksan (git) to 139,714 tokens for Arapaho (arp). It is evident that the highest micro average word-level glossing performance in the open track is achieved for Arapaho, which benefits from the largest training set. In the closed track, Arapaho stands among the top three languages in terms of glossing accuracy, but the best performance is observed for Tsez (ddo), which has approximately 37,000 training tokens. This places it among the higher-resourced languages in the shared task. Conversely, Gitksan, with the smallest training set, consistently exhibits the lowest glossing performance. Overall, a clear trend emerges, demonstrating an improvement in glossing performance as the training data size increases.
Impact of OOV rate While out-of-vocabulary (OOV) rate computed on the test set is an important predictor of performance in tasks like morphological tagging (Müller et al., 2015), it does not seem to have a clear impact on system performance in this shared task. While the highest OOV rate and
lowest performance are attained for Gitksan, this is largely an artefact of its very small training data size. If we disregard Gitksan, the impact of OOV rate both for the open and closed track is unclear. In fact, in the open track, the best average glossing performance is attained for Arapaho which has the second-highest OOV rate. Nevertheless, Arapaho also has the largest training set. This might seem like a surprising coincidence but we must remember that Arapaho is highly morphologically complex which tends to lead to higher OOV rates.
Impact of TTR Similarly to OOV, Type-tokenratio in the training set can be seen a measure of the diversity of the training data. We would expect a higher TTR to improve glossing performance. However, according to the statistics presented in Figure 2, the trend is not very clear. While the best performance in the closed track is attained for Tsez, which has moderately high TTR, the secondbest performance is attained for Uspanteko with the lowest TTR.

heading: 7 Future Directions
text: The submissions in this shared task have explored several novel techniques that have not been previously applied to automatic interlinear glossing. Surprisingly, pretrained language models like ByT5 did not perform as well as one might expect based on their strong performance on other morphology tasks. This unexpected outcome raises the need for further investigation.
One interesting observation is that the winning submission, TÜ-CL, completely disregards the provided translations. While this could suggest that translations may not be as useful for the glossing task, we believe there is still room for improvement in this area. Incorporating large pretrained English models as a reliable source of translated text could potentially lead to additional enhancements.
Considering the availability of extensive morphological resources for many languages, such as those provided by UniMorph and similar projects, multi-task learning holds promise for interlinear glossing. Additionally, we encourage further exploration of crosslingual approaches, leveraging the ODIN database of interlinear glossed text, which despite being noisy, offers a highly multilingual resource for research purposes.

heading: 8 Conclusion
text: The 2023 SIGMORPHON Shared Task on Interlinear Glossing received submissions from five teams which presented a wealth of interesting techniques greatly expanding the field of automated interlinear glossing. The submissions achieved substantial im-
provements over a baseline RoBERTa system. The winning team TÜ-CL achieved a 23.99%-point improvement over the baseline in the closed track and a 17.42%-point improvement in the open track using a hard attention model.

heading: Acknowledgements
text: We would like to express our gratitude to the organizers of the SIGMORPHON workshop and all the participants of the shared task for their valuable contributions. We would also like to extend our sincerest thanks to the speakers and linguists who have dedicated their efforts to the development of the corpora used in this shared task. Lastly, Miikka Silfverberg wants to acknowledge the assistance provided by ChatGPT during the editing process of this manuscript.

heading: Future Work
text: There are several intriguing directions for future research, including:
1. Crosslingual training (Çöltekin, 2019; Anastasopoulos and Neubig, 2019) has shown promise for morphological inflection and could be investigated for its potential in glossing. 2. Incorporating additional noisy training data (Wiemerslage et al. , 2023) can improve accuracy for low-resource inflection and could help improve the performance of glossing models as well. In the context of interlinear glossing, this data could come from large multilingual databases like ODIN (Lewis and Xia, 2010) which is automatically created with the aid of web crawling and is known to be noisy. 3. Data augmentation techniques (Liu and Hulden, 2021; Anastasopoulos and Neubig, 2019; Silfverberg et al. , 2017) are now a wellestablished technique in morphological inflection and could enhance the training process for glossing models. 4. Hard attention models (Aharoni and Goldberg, 2017; Makarov et al. , 2017) have delivered strong performance for several morphology tasks in low-resource settings and could also be applied to interlinear glossing. 5. Multitask training (Rama and Çöltekin, 2018) and meta-learning (Kann et al. , 2020) techniques could be leveraged to enhance glossing performance. 6. Finally, pretrained language models like ByT5 (Xue et al. , 2022) have demonstrated strong performance in various morphology tasks, yet their potential for interlinear glossing remains unexplored. The submissions in this shared task explore several of these techniques, including the use of pretrained language models, data augmentation, utilization of external data, and the application of hard attention models