Tü-CL at SIGMORPHON 2023: Straight-Through Gradient Estimation for Hard Attention
Year: 2023

Abstract:
This paper describes our systems participating in the 2023 SIGMORPHON Shared Task on Morphological Inflection (Goldman et al., 2023) and in the 2023 SIGMORPHON Shared Task on Interlinear Glossing. We propose methods to enrich predictions from neural models with discrete, i.e. interpretable, information. For morphological inflection, our models learn deterministic mappings from subsets of source lemma characters and morphological tags to individual target characters, which introduces interpretability. For interlinear glossing, our models learn a shallow morpheme segmentation in an unsupervised way jointly with predicting glossing lines. Estimated segmentation may be useful when no ground-truth segmentation is available. As both methods introduce discreteness into neural models, our technical contribution is to show that straight-through gradient estimators are effective to train hard attention models.

heading: 1 Introduction
text: This paper describes our systems participating in the SIGMORPHON–UniMorph Shared Task on Typologically Diverse and Acquisition-Inspired Morphological Inflection Generation (Goldman et al., 2023) and the SIGMORPHON 2023 Shared Task on Interlinear Glossing. For morphological inflection, we participate in part 1, and for interlinear glossing we mainly target the closed track.
Morphological Inflection is the task of predicting the correct inflected form given a lemma and set of morphological tags. An example from the Italian dataset in the shared task is
votare (“to vote”) V;IND;FUT;NOM(1,PL)−−−−−−−−−−−−−→ voteremo.
The organisers of the shared task provide train, validation and test splits for 26 languages. In the case of Hebrew, 2 datasets are provided. Train splits contain 10K (lemma, tags, form) triples, validation and test splits contain 1K triples.
Interlinear glossing is the task of predicting glossing lines, which is a sequence of morphological tags, including lexical translations for each token, on the sentence level given the surface text and optionally a translation. An example of interlinear glossing taken from the train portion of the Gitksan dataset in the shared task is:
(1) Iin CCNJ-1.I dip 1PL.I gidax ask guhl what-CN wilt. LVB-3.II
“And we asked what he did.”
The organisers of the shared task provide train, validation and test splits for 7 typologically diverse languages. Dataset sizes differ for each language. Furthermore, the shared task features a closed track, where only surface text and a translation is available for each sentence, and an open track, where canonical morpheme segmentation and POS tags are provided as additional information.
Especially when the main focus of training machine learning models is scientific discovery, even the notoriously good performance of deep neural models (Jiang et al., 2020) may not be satisfactory. Instead, models should also yield insights into what they learn about the data. However, clear and interpretable explanations are often hard to derive from models by post-hoc analysis, although many methods exist (Holzinger et al., 2020; Burkart and Huber, 2021; Rao et al., 2022). On the other hand, self-interpretable models, i.e. models whose calculations directly reveal discrete information, are generally hard to train with gradient methods and do not reach the same effectiveness as fully continuous models (Niepert et al., 2021).
Therefore, in this work we aim at narrowing the gap between inherently interpretable models and fully continuous deep sequence-to-sequence models by demonstrating the effectiveness of straightthrough gradient estimators in optimising discrete intermediate representations by gradient methods.
171
As applications, we construct a model type for morphological inflection that shows, without ambiguity, which subset of lemma characters and tags causes the prediction of a form character. Our proposed model for interlinear glossing enriches the given surface text with shallow morpheme segmentation.
Our main contributions are: (1) We show the effectiveness of straight-through gradient estimators for learning hard attention; (2) We present a model for morphological inflection that unambiguously shows which subset of lemma characters and tags lead to the prediction of a form character; (3) We present a model that learns shallow morpheme segmentation jointly with interlinear glossing in an unsupervised fashion.

heading: 2 ST Optimization of Hard Attention
text: We discuss hard attention as mappings of the following form: Let k ∈ N be the number of target positions (e.g. the number of decoder positions in a encoder-decoder sequence-to-sequence model), and X ∈ Rn×d the matrix containing d-dimensional feature vectors of n source elements (e.g. learned embedding vectors). Each target element yi, i ∈ {1, . . . ,K} is calculated as a sum of source element encodings, formally yi = ∑ j∈→i xj where xj is the jth row vector in X and →i⊆ {1, . . . , n} is the set of source elements aligned to target position i. Note that a source element may be aligned to multiple target elements, i.e. appear in →i for different i.
This mapping can be calculated by a matrix multiplication ξ ·X = Y ∈ Rk×d, where columns of ξ ∈ {0, 1}k×n are the multi-hot encodings of index sets (→i)i∈{1,...,K}. Formally, this means
ξi,j = { 1 if j ∈→i 0 if j /∈→i
We assume ξ is a sample from a underlying categorical distribution where we can compute the marginals ξ̂i,j that specify the probability
ξ̂i,j = Pr[j ∈→i]
of j being included in →i. For example, in the case of dot-product attention, we have z ∈ Rk×n the matrix product of decoder states and encoder states. Then, we obtain ξ̂ by softmax over rows, and ξ by sampling from the categorical distributions defined by rows of ξ̂. At test time, argmax is used instead of sampling.
The main problem is how to side-step sampling during gradient-based optimization, because sampling is not differentiable. One solution is the so-called straight-through estimator (Bengio et al., 2013; Jang et al., 2017; Cathcart and Wandl, 2020) which means using ξ for the forward pass, i.e. when computing model outputs, but using ξ̂ for backpropagation, i.e. when computing gradients of model parameters w.r.t. the loss.
However, gradients of X are affected by the discreteness of ξ as well, because ξi,j = 0 also means xj does not receive gradients from yi. Therefore, when using straight-through gradient estimation, we should use ξ̂ when computing gradients of X. Formally, for some differentiable function f that is applied to Y, we set
∂f(ξ ·X) ∂ξ̂ = XT ∂f(Y) ∂Y
∂f(ξ ·X) ∂X =
( ∂f(Y)
∂Y
)T · ξ̂,
which can be implemented as
Y = ξ̂ ·X− sg ( (ξ̂ − ξ) ·X ) , (1)
where sg is the stop-gradient function (van den Oord et al., 2017) which behaves like the identity during forward pass, but has 0 partial derivatives everywhere.

heading: 3 Applications
text: In this section, we describe how to apply the method from Section 2 to sequence transduction (Section 3.1) and sequence segmentation (Section 3.2). We keep formulations more general than necessary for the shared tasks, because we want to highlight that the methods apply to similar problems as well.

heading: 3.1 Sequence Transduction
text: Sequence Transduction means transforming an input or source sequence s1:n = s1, . . . , sn into an output or target sequence t1:m = t1, . . . , tm. Successful model types for this tasks are neural encoder-decoder networks with attention (Bahdanau et al., 2015). These models use an encoder which computes contextual source symbol representations s1, . . . , sn and a decoder which computes autoregressive target symbol representations t1, . . . , tm. Entries of the attention matrix ξ̂ are
dot products1 of source representations and target representations, normalised to a categorical distribution over source symbols for every target symbol. Output symbols are predicted from the concatenation of the respective previous autoregressive target representation with the weighted sum of source symbol representations, where weights correspond to probabilities of the respective attention distribution. In terms of interpretability, this type of model has two problems:
Soft Attention The role of soft attention (i.e. using ξ̂ directly) with regard to explaining model predictions is not entirely understood (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Therefore, we want to replace soft attention with hard attention, whose interpretability is undisputed. We replace soft attention with hard attention by sampling source elements from rows of ξ̂ during training. The sampled index sets are used to discretise ξ̂ into ξ. We enable end-to-end training through Equation (1).
Contextual Representations Contextual symbol encodings represent information about the whole sequence, not just the encoded symbol. In deep models, it is therefore not clear what information actually is encoded (Meister et al., 2021). For this reason, we want to use non-contextual symbol embeddings for prediction and use contextual symbol encodings only for computing ξ̂.
However, only selecting one single source symbol by hard attention and then not using any contextual information is not sufficient for successful transduction. For example, in the case of morphological inflection discussed here, predictions have to take morphological tags and surrounding characters into account when transducing a source character. Therefore, we use two attention heads computing different kinds of attention:
1. Softmax-normalised attention ξ̂symbol to select a single symbol to transduce.
2. Sigmoid-normalised attention ξ̂cond to select multiple symbols as conditions. In this case, the sigmoid function σ is applied to every dotproduct of encoder states and decoder states individually, yielding a Bernoulli distribution for every combination. ξcond is the result of
1There are different ways to calculate unnormalised attention scores (Luong et al., 2015; Brauwers and Frasincar, 2023), but without loss of generality we restrict the discussion to dot-product attention.
sampling from each Bernoulli distribution. At test time, we round to 0 or 1 instead of sampling to ensure deterministic predictions.
Predictions are computed from the combined context vectors, formally
Ysymbol = ξsymbol· Xembed Ycond = ξcond· Xembed pj(• | s1:n) = MLP([ Ysymbolj ,Ycondj ]) (2)
where • is a placeholder to indicate distributions over the target alphabet, pj is the distribution for the jth target symbol, and Xembed is the matrix containing non-contextual source symbol embeddings.
In this formulation, the decoder is still autoregressive, but is only involved in computing attention scores, not predictions any more. Therefore, it is entirely transparent which source symbols are responsible for which predictions. Also, the condition vector is a sum of equally weighted noncontextual symbol embeddings. The only nontransparent computation are the attention scores. Formally, the model learns a mapping S×2S×N → T where S is the source alphabet, N are the natural numbers (to account for multiplicities of symbols), and 2S×N indicates the power set. T is the target alphabet. The attention mechanism selects the contextually appropriate arguments for this mapping. A more detailed description of the concrete model architecture is in Appendix B.
Of course, the increased transparency limits the expressivity of the model. One problem is that gradient signals for encoder and decoder are insufficient, because their only remaining role is to compute attention matrices. Therefore, we train sequence transduction models in a multi-task setting, using the interpretable mechanism described above together with the typical mechanism, i.e. predicting the next target symbol from decoder state and combined contextual source symbol encodings. However, we use the same attention matrices in both cases. Predictions of type one-to-many (e.g. converting a single morphological tag to a suffix consisting of multiple characters) are also problematic: For each single target symbol, a different source symbol or condition is required. Possible solutions are augmenting the target alphabet with symbol ngrams (Liu et al., 2017) or allowing for local non-autoregressive predictions (Libovický and Helcl, 2018). However, we leave exploration of such methods to future work. Finally, condition
vectors Ycond are insensitive to order due to summing being a commutative operation. This problem can be mitigated by positional encodings, but we do not observe improvements in preliminary experiments and do not explore this option here.

heading: 3.2 Sequence Segmentation
text: We combine hard attention with Structured Attention proposed by Kim et al. (2017). In particular, we consider the case of sequence segmentation and propose an end-to-end trainable interlinear glossing model (for the closed tack, where this information is not given) that first performs shallow morphological segmentation2 on input words and then predicts the gloss label for each morpheme. Note that the method is also applicable to other tasks that require sequence segmentation and further processing of resulting segments, such as joint Sandhi segmentation and morphological parsing in Sanskrit (Li and Girrbach, 2022). In contrast to Kim et al., our segment encodings respect a particular sampled segmentation due to hard attention, and do not represent expected feature values.
Encoder Model Given a sentence as input to the glossing model, we first apply a character level encoder such as BiLSTM (Hochreiter and Schmidhuber, 1997; Graves and Schmidhuber, 2005), to compute contextual character representations on the sentence level. Then, we continue processing on the word level and denote a word w by its characters w = s1, . . . , sn. Each word consists of a sequence of characters that are represented by contextual features computed in the previous step. For each character at position i, we predict a Bernoulli distribution parametrised by probability p
seg i ∈ (0; 1) that indicates whether the corresponding character is the last character of a (shallow) morpheme segment in our case. We also adopt the method by Raffel et al. (2017) to add Gaussian noise to unnormalised scores during training to encourage discreteness of segmentation probabilities.
Furthermore, each word is paired with the number of morphemes in the word. According to Leipzig Glossing Conventions (Comrie et al., 2008, Rule 2), the number of morphemes in a word is given by the number of hyphen-separated labels assigned to a word. During inference, the number
2Shallow morphological segmentation means only segmenting the surface string. Contrast this to canonical segmentation, which also restores a latent canonical form of present morphemes (Kann et al., 2016).
of labels and therefore morphemes is not given. In this case, a straightforward solution is to start a new morpheme whenever the segmentation probability exceeds a certain threshold τ . However, we found trivial solutions for τ like 12 not to work well, while learning to predict the number of morphemes in a word from the max-pooled character representations by a MLP works well in our case. Therefore, we adopt the latter option and leave exploration of the former method to future work. In cases where we have no information about the number of segments during training, marginalising the number of segments still remains as option.
Computing Marginals Character-level segmentation scores psegi have to be converted to the attention matrix ξ̂ by marginalising all segmentations. For each source element si, marginalisation computes the marginal probability of having a morpheme boundary at si. Adopting the terminology of Section 2, morphemes correspond to target elements and characters to source elements. Each source element is aligned to exactly one target element and the alignment is monotonic. This means each source element can only be aligned to the same target element as the immediately preceding source element or alternatively it can be aligned to the next target element. Accordingly, we compute marginals, i.e. distributions over targets for each source element, by the forward-backward algorithm, same as Kim et al. (2017). The forward recursion is given by α1,1 = 1 and
αi,j = αi−1,j · (1− psegi−1) + αi−1,j−1 · p seg i−1
for i > 1, j ≥ 1 where psegi ∈ (0; 1) is the predicted segmentation probability of the ith source element. Note that the first source element is always part of the first segment. Backward scores are computed as βn,k = 1 and
βi,j = βi+1,j · (1− psegi,j ) + βi+1,j+1 · p seg i,j
for i < n and j ≤ k. Finally, marginals are given by ξ̂i,j =
αi,j ·βi,j αn,k
. In practise, computations are performed in log-space.
Training For discretising segmentations, it is most convenient to simply choose the maximum likelihood segmentation, which corresponds to starting new segments at the k − 1 indices with maximum segmentation probabilities. The corresponding target segment representations are computed by Y = ξ · X, where X is the matrix of
source element representations. Note that we use the discrete assignments ξ for computing segment representations and Equation (1) for training.
In the case of interlinear glossing, distributions over labels for each morpheme are computed by a MLP taking morpheme representations, i.e. rows in Y, as input. Loss, then, is the cross-entropy between predicted distributions over labels and ground-truth labels. Note that in this case, and in contrast to Section 3.1, we compute morpheme representations from contextualised character representations, and not from non-contextual embeddings, because we think that only sets of characters of shallow morpheme segments are not sufficient to compute the semantic information necessary for glossing, especially the correct translations.
This has two consequences, first of all the model is not transparent (i.e. interpretable), and character encodings may be “fuzzy” in the sense that information is locally spread across multiple characters which may obscure precise morpheme boundaries. A similar effect has been shown for sparse attention by Meister et al. (2021). We leave exploring more interpretable models similar to the model described in Section 3.1 and biasing models towards more precise morpheme segmentation to future work. In this work, our main focus is to provide shallow morpheme segmentation as additional predictions, not to build an entirely interpretable glossing model.

heading: 4 Evaluation
text: Here, we evaluate the methods presented in Section 3 by participating in the shared task on morphological inflection and in the shared task on interlinear glossing. Technical details of the experimental setup and hyperparameters are in Appendix A.

heading: 4.1 Baselines
text: For the morphological inflection shared task, the organisers provide a neural and a non-neural baseline. For the interlinear glossing shared task, the organisers provide a transformer-based neural baseline. Furthermore, we add a CTC-based sequence labelling model (Graves et al., 2006) as baseline. The CTC model encodes the source sentence on the character level by a BiLSTM encoder and predicts a label or blank from each character. Here, we exploit that the number of labels is the same as the number of morphemes, and each word has at least as many characters as morphemes.

heading: 4.2 Data Representation
text: For a detailed description of the shared task data, refer to the respective shared task overview papers. In the case of morphological inflection, we convert (lemma, tags, form) triples to (source, target) pairs by removing all punctuation from the tags and prepending the remaining sequence of tags to the lemma characters. Special pre- and postprocessing is applied to Japanese in order to eliminate Kanji, see Appendix C.
In the case of interlinear glossing, no modification is necessary for the closed track. However, for the open track, we replace the source text by hyphen-separated morphemes. We assume they contain more information than the original unsegmented text, and the unsegmented text does not add any information when the segmented morphemes are available. In this case, we also do not learn shallow segmentation, but predict a single label from each morpheme. The CTC baseline flexibly learns alignments of labels to characters in both cases. In both cases, we approach interlinear glossing as a sequence labelling problem.

heading: 4.3 Results
text: Morphological Inflection In Table 1, we report macro-averaged test set accuracy and edit distance. Full results for all languages achieved by our model and the baselines are in Appendix D. For clarity, we only report results of the best system for every participating team. Results show that our interpretable model loses on performance compared to more flexible neural models, such as the Transformer baseline. On 22 of 27 languages, the neural baseline beats our model. However, results also show that introducing interpretability does not have catastrophic consequences regarding performance. With some advantages in macro-averaged scores, our model performs roughly on par with the nonneural baseline, beating it on 14 of 27 languages. In summary, these results suggest that introducing interpretability to neural models causes some decrease in performance, but having neural interpretable models still gives better results than having interpretable non-neural models.
To illustrate patterns learned by our models, we show examples of the selected source symbols and condition symbols. The first example in Figure 1 is taken from the French (validation) dataset, namely the target inflection is
juger “to judge” V;COND;NOM(1,PL)−−−−−−−−−−−−→ “jugerions”.
In this case, the prediction is correct. We can see how the model first selects characters of the stem to copy. Here, few if any other source symbols are selected as conditions. Then, for predicting the inflection suffix “-ions”, the model selects tag symbols both to transduce and as conditions.
Next, we consider an example from the Italian dataset, namely
estraniarsi “to alienate onself” V;IND;PRS;NOM(1,PL)−−−−−−−−−−−−−→ “ci estranieremmo”.
The corresponding selected symbols and conditions are shown in Figure 2. This example shows an interesting non-monotonic pattern, namely moving the reflexive pronoun “si” to the front and changing it to the correct number and person, in this case 1st plural. The model correctly captures this, as we can see from the selected transduction symbols (left side). Also, the model learned which conditions to select for changing the “s” in “si” to “c”. After this transform, the model copies the stem by selecting stem characters as transduction symbols and conditions. Finally, the model generates the inflectional suffix by selecting mainly tags as transduction characters and conditions.
Interlinear Glossing In Table 2, we report macro-averaged word level and morpheme level test set accuracies. Both our additional CTC baseline and our morpheme-segmentation model, henceforth referred to as “morph”, compare favourably to the transformer baseline. Furthermore, our morph model achieves better performance than competing models on track 1, where the unsupervised learning of morpheme segmentation is relevant, which shows that learning additional linguistically relevant structures can improve performance by injecting useful inductive biases in the model. Furthermore, we again conclude that
using discrete structure as intermediate representations does not necessarily decrease performance catastrophically. Instead, it seems helpful in this case. Finally, we note that translations are not necessary for current glossing models to achieve strong performance, since we do not use them.
We also show examples of shallow segmentation learned by the morph model. We focus only on the segmentation, because this is the main contribution of our model. Note that all segmentations were learned in an unsupervised way for track 1 alongside the main objective, i.e. predicting the glossing line. For track 1, morphological segmentation is not given, unlike for track 2.
Because our model learns shallow segmentations, while ground-truth segmentations provided for track 2 are canonical segmentations, we can not conduct a quantitative analysis. Therefore, we restrict the analysis to anecdotal qualitative analysis of 2 example segmentions predicted by our models.
First, we consider a prediction for the following Natugu example:
(2) Nedr Ne-dr ne-dr rlilrdr r-li-lr-dr r-li-r-dr doa doa doa nzpwxng nz-pwx-ng nz-pwx-ngq . . .
“The two of them had four children.”
Morphemes are separated by hyphens “-”. The predicted segmentation is in the second line, and the ground-truth segmentation is in the third line. The predicted segmentation differs from the groundtruth, because it copies characters and therefore can not change capitalisation, and two morphemes (“lr” → “r” and “ng” → “ngq”) are normalised in the ground-truth segmentation, so that they differ from their surface form.
Next, we consider a Uspanteko example:
(3) i i i tiyuq t-iyuq ti-yu’ sol sol sol ji’ ji’ ji’ tren t-r-en t-r-en tib’ek t-ib’e-k ti-b’e-k
“Y llega solo asi hace se va.”
Again, the predicted segmentation is in the second line. In two cases the vowel “i” is assigned to the wrong morpheme, but the predicted glossing line (not shown) is still correct. In this case, incorrect segmentation is apparently not a problem for the subsequent classification of morphemes, but in other cases this may cause problems. Here, we
can see that contextualised character encodings can bypass the discrete morpheme segmentation step.

heading: 5 Related Work
text: Much recent work in character-level transduction aims at making models both more interpretable and stronger by using sparse (Peters and Martins, 2019, 2020) or hard attention (Aharoni and Goldberg, 2017; Wu et al., 2018; Wu and Cotterell, 2019; Makarov and Clematide, 2018b,a). Modifying the attention mechanism is necessary, because for soft attention, which does not realise hard alignment decisions, the relation of model outputs and attention weights is not fully understood (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Hard attention does not suffer from this problem, because it realises hard alignment decisions, so that we exactly know which information influences the output of attention. Especially if the main purpose of models is to gain insights into the data and not mainly to achieve better performance on modelling it, interpretability of models is crucial.
However, working with hard attention is notoriously difficult, because it introduces nondifferentiability into models. This means that the sophisticated machinery developed for gradientbased optimisation of deep neural models fails in this case. The most popular but also most expensive approach to train hard attention mechanisms is marginalising all alignments (Yu et al., 2016; Raffel et al., 2017; Wu et al., 2018; Wu and Cotterell, 2019) or approximating the marginalisation (Shankar et al., 2018). Different approaches to approximate gradients instead of having exact gradients by marginalisation are using reinforcement learning (Xu et al., 2015; Makarov and Clematide, 2018a), reparametrising discrete distributions or working with continuous relaxations (Jang et al., 2017; Maddison et al., 2017), and perturbation based gradient approximators (Niepert et al., 2021). While all methods to use hard attention with gradient-based optimisation come with
problems, we find straight-through gradient estimators (Bengio et al., 2013) very effective to compute informative discrete intermediate representations even for complicated attention scenarios. Similar results have been reported for other fields as well, both practically (Sahoo et al., 2022) and in theoretical analysis (Yin et al., 2019). Therefore, we propose two models, one for interpretable sequence transduction and one for joint segmentation and segment classification, based on straight-through gradient estimators.

heading: 6 Conclusion
text: In this work, we propose to optimise hard attention as building block in deep neural networks, which in principle is non-differentiable, by straight-through gradient estimation. In particular, we describe applications to interpretable sequence-to-sequence models and sequence segmentation models. We evaluate our approaches on two important tasks in computational morphology, namely morphological inflection and interlinear glossing which are shared tasks of the ACL SIGMORPHON 2023 workshop. Our approaches achieve good results in morphological inflection despite of constrained expressivity compared to fully differentiable models, and strong results in interlinear glossing. These results provide encouraging evidence that learning interpretable intermediate representations by deep neural network does not necessarily lead to intolerable sacrifices in performance. We hope that future work can benefit from these insights by combining interpretable representations and the generalisation abilities of neural models for scientific discovery.

heading: Acknowledgements
text: We thank the organisers of both shared tasks for their efforts and for their help during training and evaluation phases. Also, we thank Çağrı Çöltekin for his feedback on a draft version of this paper and Leonard Salewski for a helpful discussion in the early phase of this project.

heading: A Experimental Setup
text: Here, we describe our experimental setup, including hyperparameters, in detail. Also note that our code is available on GitHub. Code for morphological inflection is here: https://github.com/ LGirrbach/sigmorphon-2023-glossing. Code for interlinear glossing is here: https://github. com/LGirrbach/sigmorphon-2023-inflection. All models are implemented in PyTorch (Paszke et al., 2019) and pytorch_lightning.3
In all cases, models are optimised using the AdamW optimizer (Loshchilov and Hutter, 2019) without weight decay, learning rate is 0.001, and all other parameters are the PyTorch defaults. We use
3https://github.com/Lightning-AI/ lightning/
an exponential decay learning rate scheduler which multiplies the learning rate by factor γ after each epoch. We tune γ for each combination of model and language. Furthermore, we clip gradients with absolute value greater than 1.
Models are trained exclusively on the training splits provided by the shared task organisers. After each training epoch, generalisation performance is estimated by performance on the validation split. In the case of Morphological Inflection, our main metric is normalised edit distance (lower is better). In the case of Interlinear Glossing, our main metric is accuracy of correctly predicted glossing lines (higher is better). If performance does not improve for 3 epochs, training is stopped. Only the best model checkpoint, i.e. the checkpoint 3 epochs before ending training, is retained.
A.1 Hyperparameter Tuning For each combination of language and model, we optimise hyperparameters independently. The tuned hyperparameters and their corresponding value ranges are in Table 3. Note that other than stated there, the minimum batch size for Morphological Inflection models is always 4. Also, minimum batch size for Arapaho and Uspanteko languages (Interlinear Glossing) is 16. The maximum batch size for Arapaho and Uspanteko is 128. The maximum batch size for Gitksan (Interlinear Glossing) is 16.
In each case, we sample 50 sets of hyperparameters using the optuna library (Akiba et al., 2019). For each sampled set, a model is trained and the performance on the validation set is recorded. After sampling 50 sets, the set that resulted in the best performance on the validation set is saved. For training models for submission of results to the shared tasks and all other analyses, we exclusively use hyperparameter that were found best in this hyperparameter study. Since we tune parameters for many models, we do not report the results here. However, they are available in our GitHub repositories.
A.2 Main Evaluation For submitting results to the shared tasks and further analyses, we retrain 5 models for each combination of model type and language. The models use the hyperparameters from the tuning study described in Appendix A.1. However, they have different initialisations and may therefore perform differently. For submitting results, we select the
model the model with best performance on the validation set. Having multiple copies of the same model type trained with the same hyperparameters is useful, because especially in low-resource scenarios different initialisations can have a relevant impact on the generalisation performance. On the one hand, it is necessary to estimate this variance to see how robust models are, on the other hand this helps mitigating spurious effects in the analyses.

heading: B Sequence Transduction Model: Detailed Description
text: Here, we provide a more detailed description of the sequence transduction model (see Section 3.1).
For training, we have two paired sequences s = s1, . . . , sn and t = t1, . . . , tm, representing the source and target sequence, respectively. The goal is to maximise the likelihood of transducing the source sequence s to the target sequence t.
The first step is to represent all symbols in both sequences by high-dimensional non-contextual vectors, i.e. embeddings. Thus, we arrive at embedded sequences s = s1, . . . , sn and t = t1, . . . , tm, with si, t ∈ Rd. Here, boldfaced lowercase variables represent vectors. Furthermore, we denote the n×d matrix with source symbol embeddings as rows as S ∈ Rn×d and we denote the m×d matrix with all target symbol embeddings as T ∈ Rm×d. Note, that embeddings of source symbols and target symbols are optimised independently of each other, i.e. they are not paired.
In the next step, we encode the source sequence by a bidirectional LSTM and arrive at a contextual representation hsi for each source symbol with index i, 1 ≤ i ≤ n. Likewise, we encode the target sequence using a unidirectional LSTM and denote the autoregressive encoding of the symbol with index j, 1 ≤ j,≤ m as htj . Formally, we can
write
hsi = BiLSTM(si | s1, . . . , sn) (3) htj = LSTM(tj | t1, . . . , tj−1) (4)
and representing all contextualised representations as matrices:
Hs = BiLSTM(S) (5)
Ht = LSTM(T) (6)
So far, this formulation is the same as conventional LSTM-based encoder-decoder models. However, the attention mechanism is different. According to the descriptions in Section 2 and Section 3.1, we define attention matrices as follows:
Z = Hs · (Ht)T (7) ξ̂symbol = softmax(Z) (8)
ξ̂cond = σ(Z) (9)
where Z ∈ Rn×m represents the unnormalised dot-product attention scores, softmax is applied to columns of Z and the sigmoid function σ is applied elementwise. In fact, it seems counterintuitive to use the same unnormalised scores in both attention heads, but this works well in practise. From these attention matrices, we obtain the discretised alignment matrices ξsymbol and ξcond by sampling columnwise one-hot vectors in the case of ξsymbol and elementwise values ∈ {0, 1} for ξcond. Then, we calculate context vectors as
Ysymbol = ξ T symbol · S (10)
Ycond = ξ T cond · S (11)
Note, that here we use the discretised alignment matrices ξsymbol and ξcond in place of the real-valued matrices ξ̂symbol and ξ̂symbol. Because discretisation is non-differentiable, we instead use a way of calculation that enables straight-through gradient estimation, namely Equation (1):
Ysymbol = ξ̂ T symbol · S
− sg ( (ξ̂symbol − ξsymbol)T · S ) (12)
Ycond = ξ̂ T cond · S
− sg ( (ξ̂cond − ξcond)T · S ) (13)
Finally, we predict a distribution over target symbols by a MLP from the concatenation of Ysymbol and Ycond according to Equation (2):
pj(• | s1, . . . , sn) = MLP([Ysymbolj ,Ycondj ]) (14) where pj(• | s1, . . . , sn) indicates the distribution over target symbols at prediction position with index j. Here, the concrete ground truth target symbol is tj .
To train the model, we use the typical supervised objective, namely minimising the cross-entropy between predicted categorical distributions over target symbols and the one-hot encoded ground truth target symbol at each prediction position. This also means we use teacher forcing for sequential predictions. During inference, t is predicted in a step-wise fashion by greedy decoding.
However, note that in addition to the setup described above, we also use the typical attention mechanism in a multi-tasking fashion, but only during training. We do not use contextualised source symbol representations for inference. The reason why we still use the typical loss as auxiliary loss is that the gradient signal on Hs and Ht is weak when their only role is to calculate Z. In this case, we calculate
C = ( softmax ( Hs · (Ht)T ))T ·Hs (15)
where softmax is applied to columns, and then predict distributions over target symbols as
pauxj (• | s1, . . . , sn) = MLP(C) (16)
so that we can also calculate the cross-entropy loss on pauxj (• | s1, . . . , sn) and differentiate model parameter w.r.t. the sum of both losses during training. Note, again, that this is only to stabilise training by optimising the contextual representations, and does not affect the model architecture for inference.

heading: C Preprocessing of Japanese
text: The Japanese writing system includes 4 alphabets, namely Latin characters (Romaji), Chinese characters (Kanji), and two syllabic scripts (Katakana and Hiragana) that were derived from Chinese characters by simplification and standardisation. The Japanese dataset provided with the shared task contains Kanji, Hiragana, and Katakana. Kanji constitute a problem for character-level models, because they are effectively an open set. The number of Kanji taught in Japanese schools is already ≈ 2000,
to which variants, obsolete Kanji, and special Kanji only used in names may be added. Therefore, a model for Japanese language data should have the possibility to process previously unseen Kanji.
In the case of morphological inflection, however, this problem may be ignored, because Kanji are never altered in inflection. Instead, inflectional suffixes are expressed in Hiragana. Kanji may still appear in stems, and have therefore be dealt with.
We apply the following preprocessing: We replace all Kanji by a special placeholder symbol K that is the same for every Kanji. This applies to both source lemmas and target forms. In the case of successful prediction of target lemmas, the number of Kanji in the target form is the same as the number of Kanji in the source lemma. Therefore, we copy Kanji from the source by replacing the predicted placeholders. The order of Kanji in source and target does not change. In case of predicting fewer placeholders in the predicted form than there are Kanji in the source lemma, which is an error, we copy as many Kanji as there are placeholders in the predicted form from left to right. In case of predicting more placeholders in the predicted form than there are Kanji in the source lemma, which also is an error, we leave additional placeholders unchanged, i.e. we do not change the predicted placeholder symbol, but still copy as many Kanji as possible from left to right.

heading: D Full Results
text: Morphological Inflection In Table 5, we report the official test set accuracies achieved by our model for all languages. For comparison, we also show results of the neural and the non-neural baseline. Likewise, we report official test set edit distances in Table 6.
These results show some interesting patterns. For example, the neural baseline performs worst on Japanese, which we assume is an effect of the alphabet (see Appendix C). Therefore, our proposed pre- and postprocessing for Japanese is important. Alternatively, a copy mechanism could be used.
Another trend is that our model performs strongly on semitic languages (e.g. afb, amh, arz, heb, heb_unvoc), especially compared to the nonneural baseline. Here, non-concatenative morphology gives our model an advantage, because usually individual transforms do not involve multiple characters, such as suffix ngrams. Remember that our model can only predict exactly one form character
from each combination of transduction lemma character and condition set. This shows that our model is able to capture complex patterns, but may not be optimal to generate extensive surface transforms.
Interlinear Glossing In Table 4, we report official test set accuracies achieved by our models for all languages. In most cases, our Morph model is superior to the CTC model, which confirms the benefit of morpheme segmentations for the task of interlinear glossing. The same conclusion is supported by the stark differences between track 1 and track 2. In track 2, performance is generally much better, especially when looking at morpheme level accuracies. Remember that we only use the ground-truth morpheme segmentation as additional information in track 2. We conclude that research in morpheme segmentation will be useful for computational models for interlinear glossing as well.

heading: Future Work
text: We hope that future work can benefit from these insights by combining interpretable representations and the generalisation abilities of neural models for scientific discovery