Towards Multilingual Interlinear Morphological Glossing
Authors: Shu Okabe
François Yvon
Venue: EMNLP 2023 Findings (2023)
Keywords: Computational Language Documentation; Interlinear Morphological Glosses; Structured Prediction;

Abstract: Interlinear Morphological Glosses are annotations produced in the context of language documentation. Their goal is to identify morphs occurring in an L1 sentence and to explicit their function and meaning, with the further support of an associated translation in L2. We study here the task of automatic glossing, aiming to provide linguists with adequate tools to facilitate this process. Our formalisation of glossing uses a latent variable Conditional Random Field (CRF), which labels the L1 morphs while simultaneously aligning them to L2 words. In experiments with several under-resourced languages, we show that this approach is both effective and data-efficient and mitigates the problem of annotating unknown morphs. We also discuss various design choices regarding the alignment process and the selection of features. We finally demonstrate that it can benefit from multilingual (pre-)training, achieving results which outperform very strong baselines.

Towards Multilingual Interlinear Morphological Glossing
Shu Okabe
François Yvon
Université Paris-Saclay & CNRS LISN, rue du Belvédère 91405 Orsay, France shu. www.ekaabbee...o imsi.
Sorbonne Université & CNRS ISIR, 5 Place Jussieu
75005 Paris, France
francois, yvon@i sir. upme. fr
Abstract
Nesis
Pono uži
ZOWD
nesi-s
Pono
uži ZOW-n
Interlinear Morphological Glosses are annota- tions produced in the context of language doc- umentation. Their goal is to identify morphs occurring LI sentence
he.0BL-CENI
three
son
bO.NPRS-PST.UNW
He had three
explicit their function and meaning, with the further support of an associated translation L2. We study here the task of automatic glossing, aiming to provide linguists with adequate tools to facil- itate this process. Our formalisation of gloss- ing latent variable Conditional Random Field (CRF), which labels the LI morphs while simultaneously aligning them L2 words. In experiments with several under-resouree lan- guages, we show that this approach both effective and data-efficient and mitigates the problem of annotating unknown morphs. We also discuss various design choices regarding the alignment process and the selection of fea- tures. We finally demonstrate that it can benefit from multilingual (pre-) trainitra, achieving re- sults which outperform very strong baselines.
Figure sample entry in Tsez: LI sentence (f), and its morpheme-segmentedd version (æ), its gloss (z)) and a 1.2 translation (z). Grammatical glosses are in small capital, lexical glosses in straight orthography.
In this paper, we study the task of automatically computing the gloss tier, assuming that the mor- phological analysis and the free L2 translation z are available. As each morpheme has exactly one associated gloss,' obvious formalisation of the task that we mostly adopt views glossing as a se- quence labelling task performed at the morpheme level. Yet, while grammatical glosses effectively constitute a finite set of labels, the diversity lexi- cal glosses is unbounded, meaning that our tagging model must accommodate open vocabulary of la- bels. This issue proves to be the main challenge of this task, especially in small training data regimes.
1 Introduction
Interlincar Morphological Gloss (IMG) (Lehmann, 2004; Bickel et al., 2008) is an annotation layer aimed to explicit the meaning and function of each morpheme in some documentation ('object') lan- guage L1, using (metaa-language L2. In compu- tational language documentation scenarios, LI is typically a low-resource language under study, and L2 well-resoureed language such English. Figure 1 displays an example IMG: the source sentence in LI is overtly segmented into a se- quence of morphemes (æ), each of which is in one-to-one correspondence the corresponding gloss sequence y. Fach unit in the gloss tier is either grammatical description (OBL for the oblique marker in Figure 1) semantic tag (son in Fig- ure 1), expressed by lexeme in L2. An idiomatic free translation z in L.2 usually also provided. y and z help linguists unfamiliar with LI to under- stand the morphological analysis in æ.
To handle such cases, assume that lexical glosses can be directly inferred from the transla- tion tier, an assumption we share with (McMillan- Major, 2020; Zhao et al., 2020). our model, we thus consider that the set of possible morpheme labels in any given sentence is the union of (i) all grammatical glosses, (ii) lemmas occurring in the target translation, and (iii) loqueetty-assocciae la- bels from the training data. This makes model hybrid between sequence tagging (because of (i) and (iii) and unsupervised sequence alignment (because of (ii)), as illustrated in Figure 2. Our implementation relies on a variant of Conditional Random Fields (CRFs) (Lafferty et al., 2001; Sut-
'The reality is slightly more complex. as illustrated by 'compound' glosses such as 'he.oBt' in Figure I, associating two descriptions to the same morpheme. In this work. such compound labels are processed just as any other purely lexical gloss during feature extraction, training and inference. This allows us to only distinguish two types of labels.

---

ton and McCallum, 2007), which handles latent variables and offers the ability to locally restrict the set of possible labels. The choice of a CRF-based approach is motivated by its notable data cata-efficieecy, while methods based on neural networks have diffi- culties handling very low resource settings- this is again confirmed by results in $5.1.
1
exp a.Chla.o)) (1)
Pe (3(()) -
Ze (2))
where {G = 1 K} are the feature functions with associated weights A |0] ... 0x] IRK and (z) is the partition function summing over all label sequences. For tractability, in linear- chain CRFs, the feature function Gj, only test local properties, meaning that each Gj. decomposes as (æ, y) E. (e(t 1t-1, L, æ) with gle ())) local feature. Training is performed by maximising the regularised conditional log-likelihood on set of fully labelled instances, where the regulariser is proportional to the & ((0) or & (110113 norm of the parameter vector. Exact decoding of the most likely label sequence is achieved with Dynamic Programming (DP); furthermore, an adaptation of the
In this work, generalise previous attempts tackle this task with sequence tagging systems based on CRFs such as (Moeller and Hulden, 2018; McMillan-Major, 2020: Barriga Martínez et al.. 2021) and makes the following contributions: (a) we introduce ($2) a principled and effective end- to-end solution to the open vocabulary problem; (b) we design, implement and evaluate several vari- ants of this solution ($3), which obtain results that match that of the best-performing systems in the 2023 Shared Task on automatic glossing ($5.1); (c) in experiments with several low-resource languages ($4), we evaluate the benefits of an additional mul- tilingual pre-training step, leveraging features that are useful cross- cooss-iinuuistccllly ($5.5). Owing to the transparency of CRF features, we also provide an analysis of the most useful features ($5.6) and discuss prospects for improving these techniques.
algorithm computes the pos- terior distribution of any y/t conditioned
Using CRFs for sequence labelling tasks has long been the best option the pre-neural era, ow- ing to (i) fast and data-efficient training procedures, even for medium-size label sets (c.g. hundreds of labels (Schmidt et al., 2013)) and higher-order label dependencies (Vieira et al., 2016), (ii) the ability to handle extremely large sets of interdependent features (Lavergne et al., 2010). They can also be used in combination with dense features computed by deep neural networks (Lample et al., 2016).
Ne
(LAT
GEN1
III
PST.UNW]
x
nesi
Pono
uzi
ZOW
=
he
had
three
soms
2.2 Augmenting labels with translations
One of the challenges of automatic glossing is the need to introduce new lexical glosses in the course of the annotation process. This requires extend- ing the basic CRF approach and incorporating a growing repertoire of lexical labels. We make the assumption [H] that these new labels can be ex- tracted from the L2 translation (z in Figure l).
(r(w)) (or y) he.OBL
son
beNNPRS
Figure 2: Tagging morphs for the L1 sentence in Fig. 1. Va represents the set of all grammatical glosses in the training data, x the words occurring the translation, Vz (zr) the set of lexical labels from the training dictio- nary, and y the reference lexical labels seen in training. During training. automatic alignments between x and are used. Dashed lines symbolise the ambiguous origin the label, possibly in both z and (x)).
Informally, this means that the grammatical label set V' now needs to be augmented with L.2 words in z or equivalently, with indices in [1 ..... |=|]. This raises two related questions: (a) how to exactly specify the set of labels y in inference and training. (b) depending on answers to question (a), how to learn the model parameters?
2 A hybrid tagging/ / alignment model
2.1 The tagging component
The core of our approach is a CRF model, the main properties of which are defined below. Assuming for now that the set of possible glosses is closed set y, our approach defines the conditional prob- ability of sequence y of T labels in y given a sequence x of T morphemes as:
In our model, we additionally consider tra source of possible lexical labels, (x)) which contains likely glosses for morphemes in æ. There are several options to design ) (æz): for instance, to include all the lexical glosses seen in training or to restrict to one or several glosses for each word

---

2t. In our experiments ($5), we select for each mor- pheme in æ the most frequently associated gloss the training corpus. y thus decomposes into a global part Ve and a sentence-dependen part Vi(a) (1...) Performing inference with this model yields values y/ that either directly corre- spond to the desired gloss or correspond to an inte- ger, in which case the (lexical) gloss at position t is Sy- Formally, the gloss labels are thus obtained as 9r ()y),, with Φ() the deterministic decoding function defined as Vy E Yc (.) : ((v) y
y/t
Ox
Ot
Sys
Ve (2())
all
0
all
0
(s1)
>>
(s2)
>
ag if 3/2 Fox
(83)
0 otherwise
# Vr.(x)
=
(*)
all
>>
at
all
Table 1: Three supervisions for the hybrid CRF model. (*) means that the correct label does not occur in the dictionary nor in the translation. To preserve reference reachability,' we augment (xr))) with the correct label.
and Vi |1...//20 [=] (b1) = 24.
Training this hybrid model is more difficult than for regular CRFs, for lack of directly observing
Vt- We observe instead Út = ((v)): while the correspondence is non-ambiguous for grammatical glosses, there an ambiguity when ğt is present in both (x))) and z, when it occurs multiple times in z. We thus introduce new, partially observed, variable of which indicates the origin of gloss le: Or = when J. E Va U ().()) and o > when y/ comes from L2 word Zoz- The full model is:
nary; (S3) assumes that the label originates from the translation only if an exact match is found.
3 Implementation choices
3.1 Aligning lexical glosses with target words
To align the lexical glosses with the L2 translation, we use SimAlign (Jalili Sabet et al., 2020), an unsu- pervised, multilingual word aligner which primar- ily computes source target alignment links based on the similarity of the corresponding embeddings in some multilingual space. Note that our task is much simpler than word alignment in bitexts, as the lexical gloss and the translation are often in the same language (e.g. English or Spanish), meaning that similarities can be computed in a monolingual embedding space. We extract alignments from the similarity matrix with Match heuristic, it gave the best results in preliminary experiments. Match views alignment as maximal matching problem in the weighted bipartite graph containing all possible links between lexical glosses and L2 words. This ensures that all lexical morphemes are aligned with exactly one L2 word.
po o|x, = Zo (xx exp 10 G( ((.x8,220))
By making the origin of the lexical label(s) explicit, we distinguish in $3.3 between feature functions associated with word occurrences in and those for word types in (x) (Täckström et al., 2013). Learning 9 with (partially observed) variables is possible in CRFs and yields a non-convex optimi- sation problem (see e.g. (Blunsom al., 2008)). In
this case, the gradient of the objective is a differ- ence of two expectations (Dyer et al., 201 eq. (1)) and can be computed with forward-backwart re- cursions. We, however, pursued another approach, which relies on automatic word alignment be- tween lexical glosses and translation ($3.1) to pro- vide proxy information for o. Assuming at 0 for grammatical glosses and unaligned lexical glosses and a > 0 otherwise, we can readily derive the supervision information o, needed training, ac- cording to the heuristics detailed in Table 1, which depend on the values of and az-
one home two khan place become be
in one
home there
is place for two kings
Figure 3: Example of SimAlign alignment between lex- ical glosses and English translation (Tsez sentence).
Three heuristic supervision schemes are in Ta- ble I, which vary on how ambiguous label sources are handled: (s1) only considers dictionary entries, which makes the processing of unknown words im- possible; (s2) only considers translations, possibly disregarding correct supervision from the dictio-
Figure displays an alignment computed with the Match method. Most alignments are trivial and associate identical units (e.g. one/'one") or morphologically related words (e.g. son/'sons') Non-trivial (correct) links comprise (khan/"kins'))
PL.c., the correct label is always part of the search space.
A Asssmiig there are fewer lexical glosses than 1.2 words.

---

which is the best option as 'khan' does not occur in the translation. A less positive case is the (er- roneous) alignment of be with 'for', which only exists because of the constraint of aligning every lexical gloss. Nevertheless, frequent lemmas such as 'be' will occur in multiple sentences, and their correct labels are often observed in other training sentences. We analyse these alignments in $5.4.
On the label side, feature functions the gloss value y and type (GRAM or LEX); for labels aligned with an L2 word, we additionally collect its PoS p¹ and its position in 2, which acts as a distortion feature." Such label features enable us to generalise alignment patterns for unknown L1 morphemes. More about features in Appendix A.
4 Experimental conditions
3.2 Implementing the hybrid CRF model
4.1 L1 languages
Following e.g. (Dyer al., 2011; I; Lavergne al..
We consider five (out of seven) languages from the SIGMORPHON 2023 Shared Task Interlinear Glossing (Ginn al., 2023): Tsez (ddo), Gitksan (git), Lezgi (lez), Natugu (ntu: surprise language), and Uspanteko (usp; target translation Spanish).? Table 2 gives general statistics about the associated datasets; a brief presentation of these languages is in Appendix B and in (Ginn et al., 2023).
2011), our implementation of the CRF model* heavily relies on weighted finite-state models and operations (Allauzen et al., 2007), which we use to represent the spaces of all possible and reference labellings on a per sentence basis, and to efficiently compute the expectations involved in the gradient, well as search for the optimal labellings and compute alignment and label posteriors.
Training is performed by optimising the pe- nalised conditional log- -ogllikeihood with a variant of gradient descent (Rprop. (Riedmiller and Braun, 1993)), with f regularisation to perform feature selection, associated with parameter value 0.5 that set during preliminary experiments and kept fixed for producing all the results below.
language
ddo
git
lez.
niu
usp
train
3,558
31
701
791
9,774
dev
445
42
88
99
232
test
445
37
87
99
633
Table 2: Number of sentences for each language
Given the size of our training data ($4.1) and typical sentence lengths, training and decoding are very fast, even with hundreds of labels and millions of features. A full experiment for Lezgi takes about 20 minutes a desktop machine; processing the larger Tsez dataset takes about 10 hours.
4.2 Pre-processing L2 translations
L2 translations are lemmatised and PoS tagged with spaCy, 8 using the en_core_web_ and es_core_news__e pipelines for English and Span- ish respectively. All lemmas in the translation are lowercased except for proper nouns.
3.3 Observation and label features
4.3 Multilingual corpus
Our implementation can handle a very large set of sparse feature functions (s. testing arbitrary propertics the input LI sequence in conjunction with either isolated labels (unigram features) or pairs of labels (bigram features). Regarding L1, we distinguish between orthographic feattrrrrppiccctereeeee. which test various properties of the morpheme string (its content, prefix, suffix, CV structure and length), and positional features, which give information about the position of the morpheme in a word; all these features can also test the properties of the surrounding morphemes (within the same word or in its ncighbours). Note that a number of these features abstract away the orthographic content, a property we exploit in our multilingual model.
We also explore multilingual pre-training by lever- aging a recent IGT corpus, IMTVault (Nordhoff and Krämer, 2022), which contains IGT examples from various publications in Language Science Press. In an attempt to capture cross-linguistic pat- terns, we only keep sentences with well-defined language code with at least 30 sentences, since the languages we study range from 31 to thousands of sentences, leaving us 173 languages. The dataset
Forr labels in Yr. (xr), we associate the most frequently found PoS tag in the training dataset via aligament. As gram- matical morphemes have no aligned target words, we use the generic label GRAM for all grammatical glosses.
"Non-aligned units (i.e. glosses from (w) and V'c) have dedicated positions (-1 and -2, respectively).
"We did not run our models on the Nyangbo (nyb) dataset, which does not include a translation tier.
'Our code is available at: https:://gitubl com/
*https: http///ppac...///////

---

is shuffled and then split into 30K training, 2K development, and 2K test datasets.
5 Experiments
5.1
Results
4.4 SimAlign settings
Table 3 reports the scores of the baselines from $4.6, as well as the best results in the Shared Task on Automatic Glossing (BEST_ST)" and the results of variants of our system on the official testsets. We only report below the word- and morpheme-level (overall) accuracy, which are the [wo official metrics of the Shared Task."! !!
Since the glosses and the translation are in the same language, we use the embeddings from the En- glish BERT (bert-base-uncad
(Devlin et al., 2019) when the L.2 language is English and mBERT ("bert-base-min www.mmllttttlllllieeeeelelelllle
for Span- ish (for Uspanteko). We stress here that our model is compatible with several target languages, SimA- lign being an off- off-the-shelf multilingual (neural) aligner.
model maj
ddo
git
lez
ntu
usp
65.3
28.1
81.2
81.5
72.8
CRF+maj
29.4
84.9
88.1
76.2
Our preliminary experiments showed that the em- beddings from the O-th layer yielded the best align- ments, especially compared to the 8-th layer, which seems to work best in most alignment tasks. A plausible explanation is that contextualised embed- dings are unnecessary here because lexical glosses do not constitute a standard English sentence (for instance, they do not contain stop words, and their word order reflects the L1 word order).
BASE_ST
75.7
16.4
34.5
41.1
76.6
BEST_ST
85.8
31.5
85.4
89.3
78.5
CRF (S1)
36.2
25.5
65.3
59.0
53.2
CRF (s2)
51.5
29.9
52.8
65.0
63.3
CRF (83)
85.6
33.6
82.8
89.1
78.9
maj
79.1
51,2
85.8
87.1
79.5
CRF+maj
51.1
88.3
92.3
82.5
BASE_ST
85.3
25.3
51.8
49.0
82.5
BEST_ST
92.0
52.4
87.6
92.8
84.5
4.5 Evaluation metrics
CRF (s1)
60.1
47.8
73.5
73.0
65.5
CRF (s2)
70.2
45.3
65.0
76.8
73.3
We use the official evaluation metries from the Shared Task: morpheme accuracy, word accuracy, BLEU, and precision, recall, and Fl-score com- puted separately for grammatical (Gram) and lexi- cal (Lex) glosses. We report the results of our best system with all metries in Appendix.
CRF (s3)
91.9
52.4
87.0
92.8
84.4
Table 3: Accuracy (overall) at the word (top) and mor- pheme (bottom) levels for baseline systems and three variants of the hybrid CRF model the test dataset. Best scores in each language & metrics are in bold. 12
4.6 Baselines
A first observation is that system (S3), which effectively combines the information available in dictionary and obtained via aligmment in an inte- grated fashion (see $2.2) greatly outperforms (s1) (only dictionary) and (s2) (only alignment), obtain- ing the best performance among our three variants. (S3) also consistently better than all the base- lines, with larger gaps when few training sentences are available (e.g. Gitksan or Lezgi). In compari- son, the BERT-based baseline suffers a large per- formance drop in very low-data settings, as also reported in (Ginn, 2023). Our CRF model also achieves competitive results compared the best system submitted to the Shared Task, especially for the word-level scores. These scores confirm
Below, we consider three baseline approaches which handle glossing sequence-labelling task: maj: a dictionary-based approach, which as- signs the majority label (grammatical and lex- ical) seen in the training dataset to a source morpheme and fails for oot-of-vocabulary morphemes;
CRF+maj: hybrid model relying on a CRF to predict grammatical glosses and unified lexical label (as in (Moeller and Hulden, 2018; Barriga Martínez et al., 202 Known lexical morphemes are then assigned lexical label according to the maj model;
BASF_ST: is the Transformer-based base- line developed by the SIGMORPHON Shared Task organisers and detailed in Ginn (2023),9
htt thub. com/si htmm//iimmorphon/2222lssss/// blob/main/ main/resultss as.uutts.m.. md.
" Full results CRF (s3) in Appendix
PCRF+maj was not computed on Tsez because the number of features is too prohibitive. For our systems, CRF (S and CRF (s2) are the lower bounds of our results.
http://ttw...ooomiioooop//p///// tree/main/baseelne.

---

that decent to good accuracy numbers can be ob- tained based on some hundreds of training sen- tences. Note, however, that annotated datasets of that size are not so easily found: in the IMTVault (Nordhoff and Krämer, 2022), only 16 languages have more than 700 sentences, which is about the size of the Lezgi and the Natugu corpora.
size (50, 200, 700, 1,000, 2,000, full) in Tsez. With 200 examples already, our model does much better than the simple baseline maj and delivers usable outputs. (S3) also has a faster improvement rate than the baseline for small training datasets (e.g. almost +7 points between 200 and 700 sentences), while maj increases by only +4 points. The return of increasing the dataset size above 1,000 is, in comparison, much smaller. While the exact num- bers are likely to vary depending on the language and the linguistic variety of the material collected on the field, they suggest that ML techniques could be used from the onset of the annotation process.
5.2 Handling unknown morphemes
Leveraging the translation in glossing opens the way to better handle morphemes that were unseen in training. Table 4 displays some statistics about unknown morphemes in test datasets. For most languages, they are quite rare, representing solely around or below 10% of all lexical glosses in the test set, with Gitksan notable outlier (Ginn et al.. 2023). Among those unseen morphemes, a signif- icant proportion (from a third in Tsez up to 70% in Gitksan) of the reference lexical gloss is not even present in the translation!? 13 (cf. 'not in L2' in Table 4). Taking this into account nuances the seemingly low accuracy for unknown lexical mor- phemes: in Uspanteko, for instance, the system reaches accuracy of 29.3 when the best achiev- able score is about 35. To have more optimistic view of our prediction, we 'approximate" lexical glosses with lemmas from the translation, using automatic alignments (e.g.. king instead of khan in Figure 3). By evaluating the unknown morphemes with their reachable labels (cf. 'align. accuracy' line), we get higher scores, such as 40.4 in Natugu.
train maj
50
200
700
1,000
2,000
full
61.0
72.4
76.7
77.6
78.7
79.1
CRF (s3)
66.9
80.6
87.5
89.2
90.7
91.9
Table 5: Morpheme-level accuracy with increasing train- ing data size in Tsez for two systems.
5.4 Analysis of automatic alignments
Our approach relies on automatic alignment com- puted with SimAlign to supervise the learning pro- cess. When using the Match method, (almost) all lexical glosses are aligned with a word in the trans- lation (cf. footnote 3). We cannot evaluate the overall alignment quality, as reference alignments are unobserved. However, we measure in Table 6 the proportion of exact matches hetween the refer- ence gloss and the lemma of the aligned word.
language
ddo
git
lez
ntu
usp
number
44
200
64
47
181
model
ddo
git
lez
ntu
usp
proportion (%)
1.02
74.9
9.65
6.16
10.9
exact match
51.3
43.2
51.2
60.6
47.8
not in L2
15
143
42
24
117
Table 6: Proportion of exact matches between the ref- erence gloss and the lemma the aligned word with SimAlign for the training datasets.
gold accuracy
18.2
12.0
4.7
29.8
29.3
align. accuracy
15.9
18.0
10.9
40.4
56.9
Table 4: Statistics about unknown lexical morphemes in testsets. We report the morpheme-level accuracy.
Overall, around half of our alignments are triv- ial and hence sure, which is more or less in line with the proportions found by Georgi (2016), albeit lower due to the marked linguistic difference be- tween LI and L2 in our case. These seemingly low scores have to be nuanced by two facts. First, they are lower bound to evaluate alignment quality since synonyms (such as khan/king) are counted wrong alignments. In some cases, inflected forms are also used as gloss (e.g., dijo/decir Uspan- teko). Second, the alignment-inducas glosses are not used as is in our experiments: they supplement
5.3 Data efficiency
An important property of our approach seems to be its data efficiency. To better document this property, we report in Table 5 the morpheme-level accuracy obtained with increasingly large training data of
We studied the exact match with any lemmas the transla- tion; composed glosses are hence always considered as absent. "Whenever possible; if the lexical gloss has no automatic alignment, we keep the reference gloss.

---

the output label with their PoS tag and their po- sition in the L2 sentence. This means that even non-exact matches can yield useful features.
to delexicalised pattern features that are relevant in IMTVault but not present at all in our datasets.
Cross-lingual study Besides, since in our stud- ied languages, Tsez and Lezgi belong to the same language family, we reiterate this methodology but with the Tsez dataset as a pre-training source to predict Lezgi glosses. This kinship is also explored through successful cross-lingual transfer in (Zhao et al., 2020). We obtain 83.3 and 87.1 for word and morpheme accuracy, which is very close to the performance with (and without) IMTVault despite containing fewer sentences.
Removing L2 stop words We carried out a complementary experiment where we filtered stop words in the 1.2 translation in Gitksan to remove a potential source of error in the alignments. The number of unaligned lexical glosses ($3.1) thus increases, which generally means reduced noise in the alignment for the Match method. Yet, us- ing these better alignments and reduced label sets in training and inference yields mixed results: +1 point in word accuracy, 2 points morpheme accuracy.
Very low resource scenarios A final experiment focuses on a very low training data scenario. Save the Gitksan corpus, the test languages already rep- resent hundreds of annotated sentences. This con- trasts with actual data conditions: in IMT'vultt for instance, only 16 languages have equivalent or more sentences than Lezgi (the second-lowest lan- guage in terms of training sentences in our study; cf. $5.1). We thus focus here (simulated) very low- resource data settings by considering only 50 sen- tences 16 selected from the training data.
5.5 Multilingual pre-training
Cross-lingual transfer techniques via multilingual pre-training (Conneau et al., 2020) are the corner- stone of multilingual NLP and have been used in multiple contexts and tasks, including morpholog- ical analysis for low-resource languages (Anasta- sopoulos and Neubig, 2019). In this section, we apply the same idea to evaluate how well such techniques can help in our context. We train the model to predict the nature of the gloss (grammati- cal or lexical) with multilingual features (see Ap- pendix A): for given morpheme, its position in the word, its length in characters, its CV skelcton," 15 and the number of morphemes in the word. Using IMul ($4.3) for this task, the model reaches around 80 of accuracy.
model
ddo
lez
ntu
usp
CRF (s3)
47.6
53.7
64.9
45.9
+ IMT
48.0
54.0
65.6
48.3
CRF (s3)
66.9
63.3
74.7
57.5
IMT
67.3
63.4
75.3
59.2
Table 8: Accuracy at the word (top) and morpheme (bottom) levels for (s3) without and with multilingual pre-training on the test dataset with 50 training sen- tences.
model
ddo
git
lcz
ntu
usp
CRF (S3)
85,6
33.6
82.8
89.1
78.9
+ IMT
85.3
33.1
83.4
89.0
79.0
Table reports the results obtained with this configuration. We observe that using the multi- lingual pre-traiming helps for all languages to a more noticeable extent than before. These three experiments confirm the potential of multilingual transfer for this task, which can help improve per- formance in very low-resource scenarios. Contrar- ily, when hundreds of sentences are available, pre- training delexicalised features proves ineffective, sometimes even detrimental.
CRF (S3)
91.9
52.4
87.0
92.8
84.4
IMT
91.8
52.8
87.1
92.5
84.5
Table 7: Accuracy (overall) at the word (top) and mor- pheme (bottom) levels for the model without and with multilingual pre-training on the test dataset.
We use these pre-trained weights to initialise the multilingual features in our (S3) system. To help feature selection, notably reduce the value of L to 0.4. Pre-training results (+ IMT) are in Ta- ble We note that pre-training has a negligible effect, except in some metrics, such as in Lezgi for the word level. We observed that the most impor- tant weights in the pre-trained model correspond
5.6 Feature analysis
One positive point in using models based on CRFs relies on access to features and their correspond-
FFlttering IMTVault with this threshold would lead to 124 languages kept.
We identify consonants and vowels based on orthography.

---

ing learnt weights. We report in Appendix E ten features with the largest weight in Natugu.
Both (Moeller and Hulden, 2018) and (McMillan-Majoor, 2020) rely on CRFs, the latter study being closer to our approach as it tries to combine post-hoe the output of two CRF models operating respectively on the L1 and 1.2 tier, where our system introduces an integrated end-to-end architecture. Zhao et al. (2020) develop an architecture inspired by multi-source neural translation models, where one source the L1 sequence, and the other the L2 translation. They experiment with Arapaho, Lezgi, and Tsez, while also applying some sort of cross-lingual transfer learning. The recent SIGMORPHON exercise (Ginn et al., 2023) is a first attempt to standardise benchmarks and task settings and shows that morpheme-level accuracy in the high 80s can be obtained for most languages considered.
Among the top 1% of active features Natugu in terms of weight, we find two features testing the gloss type b with the morpheme length: (LEX, 6+) and (LEX, 5). These indicate that longer mor- phemes are likely to have a lexical label. Such an analysis can also be relevant to weights learnt through multilingual pre-training. For instance, (LEX, 5) is among the top 10% features on the IMT- Vault dataset, suggesting a lrnss-lingual tendency longer morphemes being lexical.
6 Related work
Language documentation With the ever- pressing need to collect and annotate linguistic resources for endangered languages, the field of computational language documentation is quickly developing, acknowledged by the ComputEL. workshop."? Regarding annotation tools, recent research has focused on all the steps of language documentation, from speech segmentation and transcription to automatic word and morpheme splitting to automatic interlinear glossing.
Latent variable CRFs models Extended CRF models were proposed and used in many studies, including latent variables to represent, e-g. hidden segmentation as in (Peng et al., 2004) or hidden syn- tactic labels (Petrov and Klein, 2007). Closer to our work, (Blunsom et al., 2008; Lavergne et al., 2011) use latent structures to train discriminative statis- tical machine translation systems. Other relevant work unsupervised discriminative alignment are in Berk-KKirkpatrick et al., 2010; Dyer al., 201 1), while Nichues and Vogel (2008) use a supervised symmetric bi-dimensional word alignment model.
(Xia and Lewis, 2007; Georgi et al., 2012, 2013) use, we do, interlinear glosses to align morphs and translations, then to project syntactic parses from L2 back to L1, a technique pioncered by Hwa et al. (2005), or to extract grammatical patterns (Bender et al., 2014). Through these studies, a large multilingual database of IMGs was collected from linguistic papers, curated, and enriched with additional annotation layers (Lewis and Xia, 2010; Xia et al., 2014) for more than 1,400 languages. (Georgi, 2016) notably discusses alignment strate- gies in ODIN and trains a multilingual alignment model between the gloss and translation layers in our work, we extend multilingual training to the full annotation process. Another massive multilin- gual source of glosses is IMT'Vault, described in (Nordhoff and Krämer, 2022), studied in $5.5.
7 Conclusion
This paper presented hybrid CRF model for the automatic interlinear glossing task, which was specifically designed and tailored to work well in low data conditions and to effectively address is- sues due to out-of-vocabulary morphemes. We presented our main approach, which relies on analysing the translation tier, and discussed our main implementation choices. In experiments with five low-resource languages, we obtained accuracy scores that match or even outperform those of very strong baselines, confirming that accuracy values in the 80s or above could be obtained with a few hun- dred training examples. Using a large multilingual gloss database, we finally studied the possibility of performing cross-lingual transfer for this task.
Automatic glossing Automatic glossing was first studied in (Palmer et al., 2009; Baldridge and Palmer, 2009), where active learning was used to incrementally update an underlying tagging system focusing mainly on grammatical morphemes (lex- ical items are tagged with their PoS). (Samardžić et al., 2015) added to this an extra layer aimed to annotate the missing lexical tags, yielding a system that resembles our mRj+ baseline.
There are various ways continue this work and improve these results, such as removing the noise introduced via erroneous alignments links- either by marginalising over the 'origin' variable, by filtering unlikely alignments based on link poste-

---

rior values, or by also trying to generate alignment links for function words (Georgi, 2016; McMillan- Major, 2020). Our initial experiments along these lines suggest that this may not be the most promis- ing direction. We may also introduce powerful neu- ral representations for LI languages; while these were usually available for a restricted number of languages, recent works have shown that even low- resource languages could benefit from these tech- niques (Wang al., 2022: Adebara et al., 2023).
library. In Implementation and Application of Au- tomata: 12th International Conference, CIAA 2007, Praque, Czech Republic, July 16-18, 2007, Revised Selected Papers 12, pages 11 1233 Springer.
Antonios Anastasopoulos and Graham Neubig. 2019. Pushing the limits of low-resource morphological in- flection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNL) pages 984-996, Hong Kong, China, Association for Com- putational Linguistics.
Limitations
Jason Baldridge and Alexis Palmer. 2009. How well does active learning actually work? Time-based eval- uation of cost-reduction strategies for language docu- mentation. In Proceedings of the 2009 Conference Empirical Methods in Natural Language Processing, pages 296-305, Singapore. Association for Compu- tational Linguistics.
The main limitation comes from the small set of lan- guages (and corresponding language family) stud- ied in this work. In general, texts annotated with Interlinear Morphological Gloss are scarcely avail- able due to the time and expertise needed to an- notate sentences with glosses. However, corpora such as IMTVault (Nordhoff and Krämer, 2022) or ODIN (Lewis and Xia, 2010) or languages such as Arapaho (39,501 training sentences in the Shared Task) pave the way for further experiments.
Diego Barriga Martínez, Victor Mijangos, and Xi- mena Gutierrez- Gutierrez-Vasques. 2021. Automatic interlin- ear glossing for Otomi language. In Proceedings of the First Workshop on Natural Language Processing for Indigenous Languages of the Americas, pages 34-43, Online. Association for Computational Lin- guistics.
Moreover, another shortcoming of our work stems from the fact that we do not use neural mod- els in our work, while, for instance, the best submis- sion to the Shared Task relies on such architectures. In this sense, we have yet to compare the scala- bility of our performance to larger data. Still, one of our main focuses was to tackle glossing in very low-resource situations for early steps of language documentation and to study how to handle previ- ously unseen morphemes at the inference step.
Emily M. Bender, Joshua Crowgey, Michael Wayne Goodman, and Fei Xia. 2014. Learning grammar specifications from IGT: A case study of chintang. In Proceedings of the 2014 Workshop on the Use of Computational Methods in the Study of Endangered Longuages, pages 43-53, Baltimore, Maryland, USA. Association for Computational Linguisties.
Taylor Berg- Kirkpaiik,, Alexandre Bouchard-Côté, John DeNero, and Dan Klein. 2010. Painless un- supervised learning with features. In Human Lan- guage Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 582-590, Los Angeles, California, Association for Computational We thank the anonymous reviewers and meta- reviewer for their comments and suggestions. This work was partly funded by French ANR and German DFG under grant ANR-19-CE38-0015 (CLD 2025). The authors warmly thank Thomas Lavergne for his help and assistance regarding the configuration and exploitation of Lost.
Acknowledgements
Linguistics.
Balthazar Bickel, Bernard. Comrie, and Martin Haspelmath. 2008 The Leipzig Glossing Rules: Conventions for interlinear morpheme-by- morpheme glosses. Leipzig: Max Planck Insti- tute for Evolutionary Anthropology. Department of Linguistics. https: //wps: eva. www.:wwwwwaaa...eeee.oom de/lingua/ resources/ wwwp/rreeeegoossnnnrllreeeee
References
Phil Blunsom, Trevor Cohn, and Miles Osbore.. 2008 A discriminative latent variable model for statisti- cal machine translation. In Proceedings of ACL-08: HLT,, pages 200-208, Columbus, Ohio. Association for Computational Linguistics.
Ife Adebara, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Alcides Alcoba Inciarte. 2023. SERENGETI: Massively Multilingual Language Models for Africa.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised
Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wo- jeiech Skut, and Mehryar Mohri. 2007. Openfst: general and efficient weighted finite-state transducer

---

cross- cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High qual- ity word alignments without parallel training data using static and contextualized embeddings. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1627-1643, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long and Short Papers). pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
John D. Lafferty, Andrew McCallum, and Fermando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling quence data. In Proceedings of the Eighteenth In- ternational Conference on Machine Learning. ICML "01, page 282-289, San Francisco, CAA, USA, Morgan Kaufmann Publishers Inc.
Chris Dyer, Jonathan H. Clark, Alon Lavie, and Noah Smith. 2011. Unsupervised word alignment with arbitrary features. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 409 419, Portland, Oregon, USA. Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2023. Ethnologue: Languages of the World, 26th edition. SII. International, Dallas, Texas.
Thomas Lavergne, Alexandre Allauzen, Josep Maria Crego, and François Yvon. 2011. From n-gram- based to bas- translation models. In Proceed- ings of the Sixth Workshop on Statistical Machine Translation, pages 542-553, Edinburgh, Scotland. Association for Computational Linguistics.
Ryan Georgi. 2016. From Aari to Zulu: massively mul- tilingual creation of language tools using interlinear glossed text. Ph.D. thesis, University of Washington, Seattle, WA, USA.
Thomas Lavergne, Olivier Cappé, and François Yvon. 2010. Practical very large scale CRFs. In Proceed- ings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 504-513. Uppsala. Sweden. Association for Computational Linguistics.
Ryan Georgi, Fei Xia, and William Lewis. 2012. Im- proving dependency parsing with interlinear glossed text and syntactic projection. In Proceedings of COL- ING 2012: Posters, pages 371 3-1-380, Mumbai, India. The COLING 2012 Organizing Committee.
Christian Lehmann. 2004. Interlinear morphemic gloss- ing. In Morphologie. Ein internationales Hand- buch zur Flexion und Wortbildung. volume 17 of Handbücher zur Spruch- und Kommunikationswis senschaft, pages 1834-1857. Berlin & New York: W. de Gruyter.
Ryan Georgi, Fei Xia, and William D. Lewis. 2013. Enhhanedd and portable dependency projection algo- rithms using interlinear glossed text. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 106-311,880 Sofia, Bulgaria. Association for Com- putational Linguistics.
William D. Lewis and Fei Xia. 2010. Developing ODIN: A multilingual repository of annotated language data for hundreds of the world's languages. Journal of Literary and Linguistic Computing (LLC), 25(3)339- 319.
Michael Ginn. 2023. SIGMORPHON 2023 shared task of interlinear glossing: Baseline model. arXiv preprint arXiv:2303.1 14234.
Michael Ginn, Sarah Moeller, Alexis Palmer, Anna Stacey, Garrett Nicolai, Mans Hulden, and Miikka Silfverberg. 2023. Findings the SIGMORPHON 2023 shared task on interlinear glossing. In Pro- ceedings of the 20th SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology. pages 186-201, Toronto, Canada. Association for Computational Linguistics.
Angelina McMillan-Major 2020. Automating gloss generation in interlinear glossed text. In Proceedings of the Society for Computation in Linguistics 2020. pages 355-366, New York, New York. Association for Computational Linguistics.
Sarah Moeller and Mans Hulden. 2018. Automatic glossing in a low- low-resource setting for language doc- umentation. In Proceedings of the Workshop on Computational Modeling of Polysynthetic Languages, pages 93, Santa Fe, New Mexico, USA. Associa- tion for Computational Linguistics.
Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Nat. Lang. Eng., 113):311-325

---

Jan Nichues and Stephan Vogel. 2008. Discriminative word alignment via alignment matrix modeling. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 18-25, Columbus, Ohio. Association for Computational Linguistics.
Tim Vieira, Ryan Cotterell, and Jason Eisner. 2016. speed-accurage tradeoffs in tagging with variable- order CRFs and structured sparsity. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 1973-1978, Austin, Texas, Association for Computational Linguistics.
Sebastian Nordhoff and Thomas Krämer. 2022. IMT- Vault: Extracting and enriching low-resource lan- guage interlinear glossed text from grammatical de- scriptions and typological survey articles. In Proceed- ings of the 8th Workshop Linked Data in Linguis- tics within the 13th Language Resources and Eval- uation Conference, pages 17-25, Marseille, France. European Language Resources Association.
Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2022. Expanding pretrained models to thousands more languages via lexicon-hased adaptation. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pupers), pages 863-877, Dublin, Ireland. Association for Computational Linguistics.
Alexis Palmer, Taesun Moon, and Jason Baldridge. 2009. Evaluating automation strategies in language documentation. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Lan- guage Processing, pages 36-44, Boulder, Colorado. Association for Computational Linguistics.
Fei Xia and William Lewis. 2007. Multilingual struc- tural projection across interlinear text. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguisties; Proceedings of the Main Conference, pages 452-459, Rochester, New York. Association for Computational Linguistics.
Fuchun Peng, Fangfang Feng, and Andrew McCallum. 2004. Chinese segmentation and new word detection using conditional random fields. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 562-568, Geneva, Switzerland. COLING.
Fei Xia, William Lewis, Michael Wayne Goodman. Joshua Crowgey, and Emily M. Bender. 2014. En- riching ODIN. In Proceedings of the Ninth Inter- national Conference on Language Resources and Evaluation (LREC'14), 14), pages 3151-3157, Reykjavik, Iceland. European Language Resources Association (ELRA).
Slav Petrov and Dan Klein. 2007. Discriminative log- linear grammars with latent variables. In Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc.
Xingyuan Zhao, Satoru Ozaki, Antonios Anastasopou- los, Graham Neubig, and Lori Levin. 2020. Auto- matic interlinear glossing for under-resouree, lan- guages leveraging translations. In Proceedings of the 28th International Conference on Computarional Linguistics, pages 5397-5408, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
Martin Riedmiller and Heinrich Braun. 1993. direct adaptive method for faster backpropagation learning: the RPROP algorithm. In JEEE International Confer- ence on Neural Networks, volume 1, pages 586-591.
Tanja SamardZić, Robert Schikowski, and Sabine Stoll. 2015. Automatic interlinear glossing two-level sequence classification. In Proceedings of the 9th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH), pages 68-72, Beijing, China. Association for Computational Linguistics.
A Model features
The input side æ is the LI morpheme sequence; from each Fı, from which we also deduce the fol- lowing features: its position / within the word coded as a numerical value (from 0 to n) for com- plex words, or 'F' for free morphemes, its length in characters, its 3-char prefix and suffix (d and respectively), the number of morphemes in the word, and its CV skeleton di (Consonant and Vowels) based on the orthography of the mor- pheme. Figure 4 displays an example of input and the associated features, while Table 9 illustrates the output label format according to the origin of the gloss.
Christoph Schmidt, Oscar Koller, Hermann Ney. Thomas Hoyoux, and Justus Piater. 2013. Using viseme recognition to improve a sign language trans- lation system. In Proceedings of the 10th Interna- rional Workshop on Spoken Language Translation: Papers, Heidelberg, Germany.
Charles Sutton and Andrew McCallum. 2007. An in- troduction to conditional random fields for relational learning. In Lise Getoor and Ben Taskar, editors, Introduction to Statistical Relational Learning, chap- ter 4, pages 93 127. MIT Press.
With all these inputs to predict the output labels, we compute unigram and bigram feature functions, detailed in Table 13.
Oscar Täckström. Dipanjan Das, Slav Petrov, Ryan Mc- Donald, and Joakim Nivre. 2013. Token and type constraints for cross-lingual offt-of-speech tagging. Transactions of the Association for Computational Linguistics, 1:1-12.
Besides, we add features that test propertics be- tween the source and translation: a copy feature to

---

Input source
features
outputs
features
position (in word)
length
first 3
last 3
copy
position
CV skele-
reference
GRAM
PoS
copy trg cf
positiio
morpp, o
letters d
letters
SEC
SFC pa
ton d!
gloss y
LEX
tag
tg po
nest
D
4
nes
esi
0
1/4
CVCV
he.0RI.
LEX
PRON
0
1/4
1
1
5
0
1/4
C
GENI
GRAM
GRAM
-1
-2
F Pono
F
5
1'o
0n0
0
2/4
CCVCV
three
LEX
NUM
0
3/4
F
021
u22
VCV
son
LEX
NOUN
0
70W
70W
Figure 4: Example of input, outputs, and associated features to Lost for the Tsez reference sentence of Figure 1.
Origin
lions) for s3. We note here that thanks to the 41- regularisation term, most feature weights are set to O and less than 1% of the features are actually retained.
Pe
GENI
GRAM
king
LEX
NOUN
khan
LEX
NOUN
10
Mr. 0. 10
khan
LEX
NOUN
10
ddo
git
lez
ntu
usp
Table 9: Example of output features extracted from each label set, using the example Figure The reference (ye, or) is the training supervision.
Active s3 Total
196k
4k
51k
73k
151k
188M
1M
26M
44M
29M
Table 10: Number of selected features among all com- puted features for the s3 system in each language.
handle glosses that literally appear in the L2 sen- tence (namely, proper nouns) and distortion fea- ture which tests the difference in relative positions between the source morpheme and the (possible) lexical label, whenever Oz > O.
D Full results
Table 11 displays the results of the s3 system with all metrics presented in $4.5. Two scores of accuracy are computed for both morpheme and word levels: an overall (Ovr) value and sentence- averaged value (Avg).
Brief language presentation
We describe below the studied languages.
Tsez (ddo) is Nakh-Daghestanian language spoken in the Republic of Dagestan in Russia.
E Example of learnt features
Gitksan (git) is a Tsimshian language spo- ken on the western coast of Canada (British Columbia).
Table 12 displays 10 features with the largest weight in the s3 system in Natugu. Here, we ignore trivial features for numbers or punctuation signs that also have high weight values. The ID of the feature refers to Table 13.
Lezgi (lez) is a Nakh-Daghestanian language spoken in the Republic of Dagestan in Russia and in Azerbaijan.
Natugu (ntu) is an Austronesian language spo- ken in the Solomon Islands.
Uspanteko (usp) is a Mayan language spoken in Guatemala.
According to Ethnologue (Eberhard et al., 2023), the two Nakh-Daggestaniin languages have be- tween 10K to IM speakers, while the other three have fewer than 10K users.
C Number of active features
Table 10 presents the number of active features (in thousands) selected among all features (in mil-

---

Morpheme Acc.
Word Acc.
BLEU (Morpheme)
Lex
Gram
Ovr: 91.9
Ovr: 85.6
P: 91.7
P: 92.1
Tsez (ddo)
81.3
R: 90.7
R: 93.0
Avg: 91.9
Avg: 85.6
Fl: 91.2
Fl: 92.5
Ovr: 52.4
Ovr: 33.6
P: 23.0
P: 76.7
Gitksan (git)
19.2
27.7.
R: 68.4
Avg: 53.1
Avg: 33.1
Fl: 24.8
Fl: 72.3
Ovr: 87.0
Ovr: 82.8
87.2
P: 86.7
Lezgi (lez)
73.1
R: 87.5
R: 86.2
Avg: 86.4
Avg: 82.0
Fl: 87.3
Fl: 86.5
Ovr: 92.8
Ovr: 89.1
90.1
P: 95.5
Natugu (ntu)
82.8
R: 89.8
R: 95.8
Avg: 92.6
Avg: 88.7
F1: 89.9
Fl: 95.6
Ovr: 84.4
Ovr: 78.9
P: 78.1
P: 89.9
Uspanteko (usp)
69.4
76.3
R: 91.8
Avg: 79.3
Avg: 74.6
Fl: 77.2
Fl: 90.8
Table 11: Full results with s3 system the IGT Shared Task test dataset.
ID
Feature
Weight
25
be A 1/4 A ngi A 3/4
2.91
9
PREP A NMLZI
2.49
25
market 4/4 maket 2/4
2.49
8
body riq
2.36
25
tree A 3/4 A no A 1/4
2.29
7
SUBR A kx
2.28
3
SUBR A kx
2.28
25
catechist A 4/4 A katkis A 2/4
2.28
8
take A twz
2.28
25
cat A 3/4 A mu A 4/4
2.28
Table 12: 10 features with the largest weight in Natugu.

---

ID
Feature
Test
Example (cf. Figure 4 5)
uni-gloss
(g: 9)
PST.UNW
2 bi-gloss
(g: = 9) A 1(gi-1 = g))
(be.NPRS, PST.UNW)
3 uni-gloss-morph
1 = 9) 1 1(m, m²
(PST.UNW, n)
4 uni-gloss-be cesss-bi-mmt gloss-bi-morph
1(9: 9) 1 1(m4-1 m²' 1mm, m)
(PST.UNW, ZOW, n)
5
uni-gloss-position
1 9) A I(4 -t)
(PST.UNW. I)
6
uni-gloss-length
1 = g) 1[4 =!)
(PST.UNW: I)
7
uni-gloss-start
1(g: g) 1(d4
(PST.UNW.
8
uni-gloss-end
1(9: = g) A 1(G 6)
(PST.UNW, n)
9
bi bi-gloss-morpe
1(9: = g) A 1 = g') A 1(m, = m)
(be.NPRS, PST.UNW, n)
10
"uni/bi-br bin
1(4 b) (A1(6,-1 = D))
GRAM ((LEX, GRAM))
11
uni/bi-pos pos
1(a = p) (Ai(p:-1 = D'))
GRAM ((VERB, GRAM))
12
uni-bin-morph
1(. = b) A 11mm = m)
(GRAM, n)
13
sumi-bin-possition/lnnghh
1(b 6) A 1(;: f)/1(4 = 1]
(GRAM, (GRGM, 1)
14
uni-bin-sttarteend
1 (4 б) A 1(4 d)/1(e; e)
(GRAM, n) (GRAM, n)
"uni-bin-be
16 = b) A 1(t
A A1(t--15
(GRAM. 0, 1)
16
bi-bin-gloss
1(9: g) A 1(b,, B)
(LEX, PST.UNW)
17
bi-gloss-bin
1(b = b) A A1(g-11 = d')
(PR.NPRS, GRAM)
uni-pos-morph
1(;: p) A 1(m; m)
(GRAM, n) (VERB, PST.UNW)
19
bi-pos-gloss
1 g) d(; - 7/)
20
bi-gloss-pos
1(p
1(8
(be.NPRS, GRAM)
21
uni-pos-start/end
1(p = p) A 1(4 -d/(G=
(GRAM, n) (GRAM, n)
22
uni-copy-trg
1(c4 = ct)
-1
23
uni-copy-trg-see
1(d, = ct) A 1(cs; = c8)
(-1. 0)
24
uni-posi-ts
1(pt, pt) L(ps, = ps)
(-2, 4/4)
25
morphhss
1 9) [pl, pl)
(PST.UNW, -2, 4/4)
A1(mm = m) A 1(ps; pps)
26
uni-gloss-delex
1 = g) A (dl; dl)
(PST.UNW, C)
27
*uni-bin-delex
144 = b) A 1(d,; dl)
(GRAM, C)
28
uni-pos-deles
1(p, P) 1(dl, dl)
(GRAM, C)
29
suni-bin-bi-delex
1hh 6) A (dd;-11- dl') ) A 1(dl di)
(GRAM, CVC, C)
30
bbi-bin-bi-delex
1 - 83) 144- (b)=b)
(LEX, PST.UNW, CVC, C)
A1(14;11 1 dl') ^ 1 dl)
31
-umi-bin-rel-morp--poiiie
1(4 = b) A 1(4 1) (mml: ml)
(GRAM. 1)
Table 13: Unigram and bigram features for our submissions: features about the main gloss label on top, those involving the two other general outputs, and other additional features at the bottom. Star-marked feature names indicate multilingual features also used for pre-training.


Reviews:
[{"review_id": "ex1IdyRXbk", "reviewer": "Reviewer_H7gr", "summary": "", "questions": "The drop in performance for BERT needs to be explained, given that it is so much worse than CRF. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\nOn multilinguality, our goal was to see to what extent learning cross-linguistic features was possible, i.e. delexicalised features such as morpheme length or CV skeleton. By learning such features on a larger (and available) corpus, very low-resource languages may benefit, as in the last part of 5.5. \n\nA peculiarity of the task of automatic glossing is that cross-corpus and cross-lingual learning is made difficult by the annotation variability despite a general framework such as the Leipzig Glossing Rules: while lexical glosses (in English) vary depending on the topic and domains in each corpus, we also found that many grammatical glosses are in fact language-specific\n\nThe BERT baseline is not ours but is provided by the shared-task organisers. As it seems, the base model (RoBERTa) is too complex to be efficiently fined-tuned with the small amount of data for most test languages. See also the discussion in Section 3.1 of https://arxiv.org/pdf/2303.14234.pdf. We will make sure to better explain this drop in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "  1. a well designed experiment for the task\n  2. an extension of a known CRF mechanism and its comparison to this baseline and to a BERT baseline. \n", "reasons_to_reject": "  1. The task is fairly niche, which should not preclude its publication.\n  2. I am not convinced about the multilingual aspect. From the title I expected a zero-shot transfer mechanism, for example, training on French, testing on Italian or German.\n", "questions_to_authors": "The drop in performance for BERT needs to be explained, given that it is so much worse than CRF. ", "paper_topic_and_main_contributions": "The paper reports a CRF model aimed at predicting interlinear morphological glosses for both lexical and grammatical features.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wQygXYt3Ez", "reviewer": "Reviewer_fF76", "summary": "", "questions": "- Why did you prefer CRF rather than a neural model? Why didn't you compare the results with a neural approach?\n- In Section 1, by \"open vocabulary problem\", do you mean that the number of possible combinations of morphological features (OBL, GEN1, etc.) is too large? In fact, I think that this is not exactly an open vocabulary issue. Although the number of combinations may be high, it is not impractical to consider all (as opposed to, e.g., the number of derived words in a morphologically rich language, which is nearly infinite due to properties like recursion).\n- What do you mean exactly by \"such cases are considered as lexical glosses\" in footnote 1? It should be stated more clearly.\n- At the end of Section 2.1, you mention two problems and also the labels GRAM and LEX. I couldn't see what these problems are for and what these labels are exactly.\n- The part \"In our model, ... because of (ii)\" is not clear, I cannot catch what you mean. Please rewrite more clearly.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\nQ1: The reason behind the choice of CRF stems from previous studies (Moeller and Hulden, 2018; Barriga Martínez et al., 2021), where CRF outperformed neural methods (RNN, LSTM, and biLSTM), mainly due the latter needing more training data, which proves to be a challenge in the case of very low-resource languages. The data efficiency of our model is discussed in section 5.3 (line 402). \nThat being said, the Shared Task on automatic glossing provides several competitive neural alternatives evaluated on the same data. One of them, based on Transformers (RoBERTa), is already reported as BASE_ST in our paper (see lines 349-351); several others are documented in (Ginn et al, 2023). We will stress this point again in the final version.\n\nQ2: The open vocabulary problem in this paper concerns lexical glosses. As you rightfully point out, the set of grammatical labels is fixed for a given language, but the set of lexical labels is unbounded. Lexical labels seen in the training data are often not sufficient to cover all the lexical labels occurring in the evaluation data (out-of-vocabulary issue), which is why we use the translation to identify new possible labels (lines 49-54). We will emphasise this issue to make it clearer in the final version.\n\nQ3: By 'such cases are considered as lexical glosses' in footnote 1, we wanted to state our decision on handling 'compound' glosses such as he.OBL (lexical for he, and grammatical for OBL). Because of its lexical part, we considered these kinds of glosses as merely lexical, and no specific treatment has been done. This way, we just consider two types of glosses. We will explain this in more detail in the final version of the paper.\n\nQ4: The end of section 2.1 will be clarified in the final version. The two sets of labels denote simplified versions of the task, where we for instance just predict the gloss category (GRAM or LEX), but not its value.\n\nQ5: The discussion at the end of page 1 expresses that our model brings together two approaches. On the one hand, the set of grammatical glosses (i) and labels from the dictionary (iii) give us a fixed set, as in a standard sequence labelling task, while, on the other hand, predicting lemmas from the target sentence (ii), through their indices, makes it closer to an alignment problem. We will better explain this in the final version of the paper.\n\nWe finally note an issue with reproducibility: as we will release the code (footnote 4) and only experiment with publicly available data, it is not clear which details are missing for reproducibility.\n\nThank you for pointing out typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper focuses on a less-studied problem and shows the usefulness of the proposed approach on several very low-resource languages. As a novel contribution, translation of the given sentence is used and the open vocabulary problem (a large number of morphological feature combinations in the output) is addressed. A detailed set of ablation experiments were given including very low-resource scenarios and a feature analysis.", "reasons_to_reject": "- It is not clear why the authors preferred a CRF model which, as the authors stated, is a good model in the \"pre-neural era\" and why they didn't compare the results with an LSTM- or transformer-based neural model. This seems as a weakness of the paper.\n- Although the paper is generally well-written, there are some parts that are difficult to understand and that makes following the paper difficult Some of them are mentioned below.", "questions_to_authors": "- Why did you prefer CRF rather than a neural model? Why didn't you compare the results with a neural approach?\n- In Section 1, by \"open vocabulary problem\", do you mean that the number of possible combinations of morphological features (OBL, GEN1, etc.) is too large? In fact, I think that this is not exactly an open vocabulary issue. Although the number of combinations may be high, it is not impractical to consider all (as opposed to, e.g., the number of derived words in a morphologically rich language, which is nearly infinite due to properties like recursion).\n- What do you mean exactly by \"such cases are considered as lexical glosses\" in footnote 1? It should be stated more clearly.\n- At the end of Section 2.1, you mention two problems and also the labels GRAM and LEX. I couldn't see what these problems are for and what these labels are exactly.\n- The part \"In our model, ... because of (ii)\" is not clear, I cannot catch what you mean. Please rewrite more clearly.", "paper_topic_and_main_contributions": "The paper proposes an approach for interlinear morphological glossing (IMG) of a sentence that also makes use of a translation of the sentence. This version of the IMG problem aims at, given the morphemes of a sentence in a low-resource language and its translation in a high-level language, determining the morphological analysis/glossing of the sentence. The paper views the problem as a hybrid problem formed of two parts which are sequence labeling and sequence alignment. A CRF model is used and it is adapted to the problem as taking two input parameters and outputting the gloss sequence. Experiments were conducted on five low-resource languages by using English as the high-resource language. Ablation studies were performed.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "one-to-one correspondence the -> one-to-one correspondence with the\nthe feature function Gk only test -> the feature function Gk only tests\nL2 translation are\nCRF model the test dataset -> CRF model on the test dataset"}}, {"review_id": "tyGRHtd0wX", "reviewer": "Reviewer_v4Qx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n- We strive to improve the clarity of our work. It would be of tremendous help if you could point us out some technical issues where you feel more detail would be helpful.\n\n- For the languages we studied, which contain very morphologically-rich languages such as Lezgi, Natugu or Tsez, the set of grammatical labels is around a few hundred. This size of labels is still manageable with linear-chains CRFs (see e.g. (Mueller et al., 2013; Lavergne and Yvon, 2017), which develop CRFs with very large sets of labels (up to several hundreds).\n\n- EMNLP is about empirical methods in NLP, which is what we report; so it is not immediately clear to us where a more linguistic-oriented presentation would be needed. It would greatly help us to improve the paper if the reviewer could expand their comment.\n\nWe finally note an issue with reproducibility: as we will release the code (footnote 4) and only experiment with publicly available data, it is not clear which details are missing for reproducibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Personally, I like the topic.", "reasons_to_reject": "- The technical issues are not well presented.\n- For heavily inflexional or agglutinative languages, the possible labels may be huge. Is the CRF a good option?\n- The whole work is organized in an engineering-oriented manner, while the topic should be linguistic-oriented. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work focuses on automatic interlinear morphological glossing. An approach by CRF is proposed and several low-resource languages are taken into experiments and discussions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the concept of automatic glossing. The authors leverage not only the morpheme sequence of the target sentence but also its translation to determine the appropriate glossing. The authors introduce a latent variable Conditional Random Field (CRF) model, incorporating traditional symbol-based features. The authors place particular emphasis on addressing low-resource scenarios. They assert that their CRF model outperforms neural models.

This paper is well-written and the experiments were consistent with the research objectives. While it would not reach a large audience, this paper is worthy of presentation at the main conference.
Decision: Accept-Findings