An Ensembled Encoder-Decoder System for Interlinear Glossed Text
Year: 2023

Abstract:
This paper presents a submission to Track 1 of the 2023 SIGMORPHON shared task on interlinear glossed text (IGT) (Ginn et al., 2023). There are a wide amount of techniques for building and training IGT models (see Moeller and Hulden, 2018; McMillan-Major, 2020; Zhao et al., 2020). We describe the system’s ensembled sequence-to-sequence approach, perform experiments, and share the submission’s test-set accuracy. We also discuss future areas of research in low-resource token classification methods for IGT.

heading: 1 Introduction
text: This paper is a system demonstration for our submission to the 2023 SIGMORPHON shared task on interlinear glossed text (Ginn et al., 2023). We focused on the closed track of the task, where only the input sentence, output gloss, and translation are provided in training data. This was more restrictive than the open track, in which more information was available, such as morphological segmentations or part-of-speech tags.

heading: 1.1 Interlinear Glossed Text
text: Interlinear glossed text (IGT) is a form of linguistic data annotation which highlights the grammatical properties of a corpus of text. IGT is not standardized and varies from annotator to annotator (Palmer et al., 2009), but typically uses three lines for each sentence of text. The data provided in the shared task follow the Leipzig glossing conventions (Comrie et al., 2008), in which the first line contains a transcription in an “object language,” i.e. the language of study; the second line is a morphemeby-morpheme annoatation of the sentence (called a “gloss”); and the third line is a direct translation.
(1) Ap VER yukwhl IPFV-CN ha’niisgwaa’ytxw. INS-on-rest
But it was Sunday.
Ex. 1 shows an example in Gitksan from the task’s training data. In the gloss, the functional morphemes are referred to as “grams” and the lexical morphemes are “stems,” as per Zhao et al. (2020).

heading: 1.2 Related Work
text: Moeller and Hulden (2018) used a character-level system that combined a Support Vector Machine for recognizing grams and stems with a Conditional Random Fields labeller for assigning output grams to input characters, using a BIO-tagging convention (Ramshaw and Marcus, 1995). They also trained a character-level LSTM encoder-decoder on the BIO-tagged data.
McMillan-Major (2020) uses an ensembled system in which two CRF models focus on the source text and gloss, and translation text and gloss, respectively.
Zhao et al. (2020) use a transformer-based encoder-decoder system in which the encoder is multi-sourced: the source text and the translation are encoded separately and then combined in a single attention mechanism.

heading: 1.3 Baseline Model
text: The IGT shared task baseline model (Ginn, 2023) is a transformer-based token classification system. The authors found that a sequence-to-sequence model required more data to converge and performed worse when compared to the token classification approach.

heading: 2 Methods
text: The system was based on an encoder-decoder model using the LSTM architecture (Sutskever et al., 2014). It used ensembling and data augmentation as a method to counteract the relatively lower performance of encoder-decoder models as highlighted in the previous section. The system was implemented with Fairseq (Ott et al., 2019) and trained on a single Nvidia GeForce MX350.
217
Fairseq has built-in support for transformers as well as LSTMs, but the former requires more resources to train. The GPU used for this project did not have sufficient memory for training a convergent transformer model, and so the LSTM architecture was chosen instead.

heading: 2.1 Representing target glosses
text: The source language text was represented as a sequence of characters, and we experimented with several approaches for representing the gloss as a target alphabet. Initially, the output gloss was also represented as a sequence of characters. Later, we used a token-based output alphabet. See Table 1 for examples.
The shared task dataset includes translated stems in its glosses. We experimented with representing the stems with a special token instead, and a model for generating stems from the translation, but chose to use the token-based output with the original stems in the final results. Development-set results can be found in Table 3.

heading: 2.2 Word-level training examples
text: Instead of giving the system an entire sentence to gloss as one example, the system was trained with word-level examples, which included tokens on either side of the “target” word for added context. Since the output gloss contains the same number of tokens as the input sentence, training and in-
ference can be performed on the word-level, and sentence-level results can be created from a simple concatenation of word-level results. The number of tokens on either side of the “target” became a hyperparameter, and we found that a word-window of two tokens on either side gave the best results for a single model. See Table 2 for results.

heading: 2.3 Ensembling and voting
text: The final form of the system was a combination of a model trained on a window size of one, and another trained on a window size of two. During inference, Fairseq provides a negative log-likelihood (NLL) score for the model’s predictions. A final output token was chosen by finding the smallest NLL score for either model’s predictions. Figure 1 depicts the ensembling and voting process.

heading: 2.4 A model for predicting stems
text: We experimented with an additional sequence-tosequence model for generating stems using the gloss and the translation text. The full translation was used as an input sequence, and just the stems from the gloss would be used as an output sequence. The input and output sequences were represented with a character-level alphabet.
The system used a simple technique for adding stems to the final model predictions: During the combination of word-level results into sentencelevel outputs, the system replaced the special stem tokens with predictions from the stemmer model in the order that each sentence-level stemmer result was generated. If the model generated too many stems, the rightmost outputs would be left out, and
if there were too few, at least one special stem token would remain. Figure 2 represents a glossing system working with the stemmer model.
This stemmer model was prototypical and we found that it did not have an effect on overall performance or stem F1 score.

heading: 2.5 Evaluation
text: We used the shared task baseline model’s evaluation script, which calculates a variety of metrics, including an overall BLEU score, stem F1, precision, and recall, as well as word- and morpheme-level accuracy.

heading: 3 Results
text: The final system consisted of two word-level sequence-to-sequence models, trained with a word window size of one and two, respectively. The input alphabet consisted of characters and the output
alphabet was token-level. The models were trained with an inverse square root learning rate scheduler, early stopping, and the Adam optimizer. Models trained on all languages except for Gitksan used a batch size of 128. Since the Gitksan training dataset was just 31 examples long, it used a batch size of 64 instead.
See Table 4 for results across all languages in the shared task training data.

heading: 3.1 Analysis
text: For most of the languages, the system performed better than the baseline in terms of word-level and morpheme-level accuracy. However, the relative performance varied by language: the system’s word-level accuracy for Arapaho is 15% lower than the baseline, while the same metric for Nyangbo is 72% higher.
We hypothesized that these results could have
been caused by differences in morpheme-to-word ratios across the languages. Since the system was trained on word-level examples, a lower ratio would suggest longer sequences for training — Wu et al. (2021) points out that transformer models perform better than RNNs on longer sequences.
For each training dataset, we calculated the average morpheme-to-word ratio and found that Nautugu and Uspanteko have the joint highest ratios of 0.83, while Arapaho and Nyangbo are lower with 0.72 and 0.75, respectively. The language with the lowest ratio was Tsez, at 0.6.
There seems to be a weak trend: The model under-performed or was at par with the baseline for languages with low ratios. For languages with a higher ratios, the model performed better than the baseline, with an exception of Uspanteko.
From this analysis, we conclude that training data size and morpheme-to-word ratio alone cannot explain the model’s under-performance for Arapaho and over-performance for Nyangbo.

heading: 4 Conclusion
text: This was a system demonstration of our submission to the 2023 SIGMORPHON shared task on interlinear glossed text. While the system was not the best-performing of all the submissions, it nonetheless performed consistently better than the baseline model in terms of word- and morpheme-level testset accuracy.
The system was relatively inexpensive to train, as it was built on a single CUDA-enabled laptop. This could be an advantage of the LSTM-based architecture: when Wu et al. (2021) introduced the transformer architecture to character-level transduction tasks, the authors noted that transformer-based performance depended on finely-tuned hyperparameters and longer training times.
However, non-encoder-decoder systems for the ILG task still show lots of promise, especially for small datasets. Further research can be done to examine the effect of ensembling and data augmentation on CRF or LSTM-based token classification systems.
More work can be done on the stem generation system as well: a linguist-created inflectional database like the one described in Oliver et al. (2022) could algorithmically recognize stems and look up translations. Also, an upstream word alignment model, such as one of the IBM models described in Brown et al. (1993), could help with the
construction of a stemmer system. We hope this demonstration will lead to future work on low-resource systems for automatic ILG, in terms of both computation and dataset size.

heading: 5 Limitations
text: Due to time constraints, it was not possible to perform a satisfactory grid search on the large combinations of training hyperparameters, preprocessing techniques, and stem approaches. It is possible that a more optimal system is possible, but we were unable to find it.

heading: 6 Acknowledgements
text: Special thanks to Miikka Silfverberg, Naomi Liu, the organizers of the shared task, and the anonymous reviewers.

heading: Future Work
text: We hope this demonstration will lead to future work on low-resource systems for automatic ILG, in terms of both computation and dataset size