TAMS: Translation-Assisted Morphological Segmentation (2024)

Abstract: Canonical morphological segmentation is the process of analyzing words into the standard (aka underlying) forms of their constituent morphemes. This is a core task in endangered language documentation, and NLP systems have the potential to dramatically speed up this process. In typical language documentation settings, training data for canonical morpheme segmentation is scarce, making it difficult to train high quality models. However, translation data is often much more abundant, and, in this work, we present a method that attempts to leverage translation data in the canonical segmentation task. We propose a character-level sequence-to-sequence model that incorporates representations of translations obtained from pretrained high-resource monolingual language models as an additional signal. Our model outperforms the baseline in a super-low resource setting but yields mixed results on training splits with more data. Additionally, we find that we can achieve strong performance even without needing difficult-to-obtain word level alignments. While further work is needed to make translations useful in higher-resource settings, our model shows promise in severely resource-constrained settings.

heading: 1 Introduction
text: Morphological segmentation is the task of breaking words into morphemes, the smallest semantic units of a language. Morphemes can merge and change during word formation, and the precise morphological composition of a word is often obfuscated in its surface form. Segmentation can thus take two forms: surface/linear segmentation and canonical segmentation, which divides a word into the “canonical” forms of its morphemes (cf. Figure 1). One important motivation for automated canonical segmentation is to expedite the process of linguistic analysis, including the creation of Interlinear Glossed Text (IGT). IGT is a form of morphological annotation that typically
adheres to the Leipzig glossing format (Lehmann, 1982), a linguistic representation wherein each line of the target text is broken up into a transcription line, a gloss line (morphological annotation), and a translation line. IGT is a crucial resource in endangered language documentation work, but it is costly and time-consuming to generate. The task of morphological segmentation is a key component in glossing, and automated canonical segmentation could aid in this process. Prior work has shown that automated methods have the potential to assist language documentation and revitalization (Palmer et al., 2009; Moeller et al., 2020; Moeller and Hulden, 2021; Chaudhary et al., 2022; Ahumada et al., 2022).
Neural models have been shown to perform well on the task of canonical segmentation (Kann et al., 2016; Ruzsics and Samardžić, 2017), but the success of these models has been restricted by the availability of annotated segmentation data. IGT, though a limited resource, is one important source of training data for canonical segmentation. Until now, primarily the transcription and gloss lines of IGT have been used as input to segmentation models, while the translations have been overlooked. Moreover, in real-world language documentation settings, translated data is often much more available than morphologically analyzed data, making it practically attractive as an additional input. A typical language documentation pipeline begins with transcription, followed by translation, followed only then by morphological analysis. This com-
6752
monly results in a setting where a linguist will have a wealth of translation data but comparatively little morphological segmentation data. Here we consider how supervised canonical segmentation methods can be improved by leveraging this underutilized supplemental data source. Also, because they are typically into more widely-spoken languages, translations provide the opportunity to make use of pretrained models that likely have much higherquality representations than any model available for the low-resource target language.
Our work is inspired in part by Zhao et al. (2020) who experiment with leveraging translations for the task of automatic interlinear glossing (i.e., generating the gloss line of IGT). They use a multi-source word-level transformer to jointly model the transcription and translation sequences, and outperform previous baselines. Their work shows promise for the utility of translation data in morphological analysis tasks. However, they assume the presence of well-segmented data and state that "proper segmentation remains a challenge and that the creation of segmentation tools is a valuable endeavor." Our work endeavors to address the segmentation issue.
We treat canonical segmentation as a sequenceto-sequence problem and use a character-level pointer-generator LSTM (See et al., 2017) to map each surface form word to its segmented, canonicalized form. We experiment with Tsez and Lezgi, two Northeast Caucasian languages, and Arapaho, a Plains Algonquian language. We leverage existing sentence-level English translations present in the IGT data from the SIGMORPHON 2023 Shared Task on Interlinear Glossing (Ginn et al., 2023) and create two datasets of word-level transcription–translation alignments: automatically with awesome-align1(Dou and Neubig, 2021), and manually according to conventions described in §4.3.2. We then embed these translations with BERT (Devlin et al., 2019) and experiment with incorporating them into our baseline model’s encoder and decoder. We also analyze the impact of training set size on the efficacy of our approach by limiting our training split to simulate varied levels of resourcedness. Finally, we analyze the effect of automatic vs. manual word-alignments on a subset of the Tsez data. 2 Based on poor automatic alignment performance, we introduce an additional model variant (TAMS-CLS) which incorporates
1Licensed under the BSD 3-Clause License. 2https://github.com/lgessler/tama/blob/master/
data/tsez/wat/Tsez%20Train%20101-310.json
the translation only at the sentence level. We find that although gold alignment does lead to better performance with our approach, we still see improvement over the baseline with automatic alignments in some cases. In the extremely low data setting (n=100), our approach outperforms the baseline by an average of 1.99 percentage points and as high as 2.87 points, even with poor quality automatic alignments. In many cases, across training set sizes, TAMS-CLS is the highest performing model configuration, which suggests that we may not even need word-level alignments to achieve performance improvements with TAMS. These results are promising in the context of the extremely resource constrained language documentation setting. However, in higher data settings, incorporating translations may or may not be beneficial. Our results also suggest that aligning translations may not be necessary to see improvements over the baseline.

heading: 2 Related Work
text: Modeling Morphological Segmentation Recent work on neural methods for canonical segmentation primarily focuses on LSTMs. Kann et al. (2016) use a bidirectional RNN encoder-decoder with a neural reranker. Mager et al. (2020) adapt this work to the low-resource setting and find that the pointer-generator network vastly improves over the performance of the LSTM canonical segmentation model for this setting. Recent work has also used the transformer for canonical segmentation: Moeng et al. (2021) test several sequence-to-sequence models for the task and find that the transformer performs the best on 4 Nguni languages.
Morphological Information within Embeddings Previous work has suggested that distributional similarity is an informative cue for morphology (Yarowsky and Wicentowski, 2000; Schone and Jurafsky, 2001), and that static word embeddings encode some morphological information (Musil, 2021; Soricut and Och, 2015). Other work has suggested that BERT embeddings could encode grammatical and morphological information (Nastase and Merlo, 2023; Jawahar et al., 2019). BERT embeddings have also been used for part-of-speech tagging (Tsai et al., 2019; Singh et al., 2021; Mohseni and Tebbifakhr, 2019). We aim to leverage the morphological cues intrinsic to pretrained embeddings of English translations to improve our segmentation models.

heading: 3 Incorporating Translations into Morphological Segmentation Models
text: Here, we present our method for using translations as a source of additional signal for morphological analysis. We first align the word forms in our target language to relevant word forms in the translation. We then obtain embeddings of these word-forms from a high-resource language model. We describe several approaches for turning these embeddings into a fixed-length representation and for incorporating them as input to the segmentation model.

heading: 3.1 Encoder–Decoder Networks
text: The most common architecture for morphological analysis is the neural encoder–decoder architecture with attention (Bahdanau et al., 2015). An encoder– decoder network estimates the probability of an output sequence y = y1, . . . , yT ′ in terms of an input sequence x = x1, . . . , xT by decomposing the output sequence’s joint probability using the chain rule of probability, where yt is conditioned on all previous output items and some representation of the input sequence vt computed using a function g:
p(y1, . . . , yT ′) = T ′∏
t=1
p(yt|vt, y1, . . . , yt−1)
vt = g(x, y1, . . . , yt−1)
For morphological tasks, this architecture is commonly implemented by treating words as character sequences and using RNNs for the encoder and the decoder. The encoder is responsible for producing representations of x which are useful for the decoder, and the decoder is responsible for producing conditional probability distributions for making a prediction for y, ŷ.

heading: 3.2 Translation Assistance
text: The translations of textual data from low-resource languages are usually written in a high-resource language such as English or Spanish. High-resource languages have very high-quality pretrained language models (PLMs) and therefore rich word representations available to them, and we hypothesize that incorporating this signal into the process of morphological segmentation may be helpful. For example, information from the high-resource language might help the segmenter resolve lexical ambiguities.
Here, we propose several related methods for incorporating information from a high-resource translation into an RNN-based encoder–decoder morphological segmenter. For clarity, we will concretely consider a unidirectional LSTM-based encoder, though our approach is trivially applicable to other RNN-based encoder–decoder architectures. We refer to the network’s embedding and hidden dimension sizes as emb and hid, respectively.
Consider a translated sentence with X , Y , and R. X = x1, . . . ,xn is a sequence of unsegmented words. Y = y1, . . . ,yn is the sequence of the corresponding segmented words. R = r1, . . . , rm is the sequence of words in the translation. We use a PLM to obtain a dense representation for each translation word, d = d1, . . . , dm. We additionally refer to any sentence-wide representation (such as BERT’s [CLS] token) that the PLM might produce as d0. We refer to the PLM’s hidden representation size as hPLM.
Alignment A preliminary step is to produce alignments between source and translation words like so, where align represents an aligner’s decision on whether two words are aligned:
A = {⟨xi, rj⟩|xi ∈ X ∧ rj ∈ R ∧ align(xi, rj)}
The aligner is assumed to be external to the segmentation system.
Translation Representation For each word x, we now have some aligned translation word representations dalign = da, . . . , db. We next produce v, a fixed-length representation of dalign which will be of length emb. We investigate three different strategies for producing this representation which differ in how they treat the sentence-wide representation d0. The intuition behind including the CLS token in our representation is that it may allow us to capture sentence-level dynamics better than wordlevel alignments alone. For the CLS-None strategy, we discard d0 and average pool dalign before using a model parameter Wtrans ∈ RhPLM×emb to project the vector from the hidden size of the PLM to the embedding size of the model:
v = avg(da, . . . , db)Wtrans
The CLS-Avg strategy is identical to CLS-None except that d0 is included in the average:
v = avg(d0, da, . . . , db)Wtrans
For the CLS-Concat strategy, we first average the aligned words like in CLS-None, but we introduce two model parameters, Wtrans,Wcls ∈ RhPLM× 1 2 emb, where Wtrans is applied to the averaged words and Wcls is applied to d0, and their concatenation is used as the final fixed vector:
v1 = avg(da, . . . , db)Wtrans
v2 = d0Wcls
v = v1 ⊕ v2
Incorporation Strategies After we have computed v, we need to incorporate it into the encoder and/or the decoder’s process. We consider four different strategies for incorporation, described next.
For Concat, we double the model’s input size to 2× emb and concatenate v to the input embedding at each time step in the LSTM. Concat-Half is identical to Concat, except the model’s input size is held constant, with character embeddings and v sharing the dimensions equally. The model’s character embedding module and Wtrans above have their output dimensions halved accordingly. For Init-State, assuming that there is some integer z such that z× emb = hid, we initialize the LSTM’s hidden state as z concatenations of v to itself, ⊕z 1 v. For Init-Char, we modify the LSTM’s input sequence so that v appears first, as if it were the embedding of a character.
All strategies are applicable to either the encoder or the decoder; we experiment with most combinations, except for Init-Char in the decoder.

heading: 4 Data
text: To perform our experiments, we use IGT data from the SIGMORPHON 2023 Shared Task on Interlinear Glossing (Ginn et al., 2023) in three languages: Lezgi, Tsez, and Arapaho. All of the data is licensed under CC BY-NC 4.0. Each word in this IGT format has a surface form, a canonical segmentation, morpheme-level glosses, and an English translation for the sentence the word appears in. An example is shown in Figure 2, which also provides a sample of our manual alignment process.

heading: 4.1 Languages
text: We experiment with three languages: Arapaho, Lezgi, and Tsez. All three languages present interesting modeling challenges given the complexity of their morphological processes. In addition, the experiments cover different types of difficult mor-
phology, as the two language families represented are typologically distinct.

heading: 4.1.1 Tsez
text: Tsez [ddo] belongs to the Tsezic subgroup, which is part of the larger Nakh-Daghestanian language family. Its morphology is highly agglutinative and suffixing. Tsez has complex nominal case morphology that allows multiple case suffixes to modify a single word, and there are around 250 possible combinations of these case suffixes. In terms of verbal morphology, Tsez separates verbs into four groups depending on the final segment of the stem, which affects the surface representation of the composite morphemes, including five possible indicative tense-aspect suffixes (Comrie and Polinsky). Tsez also has a rich set of converbs that are derived from the verb stem through multi-step morphophonological processes. We consider Tsez to be our development language and conduct all hyperparameter tuning on Tsez.

heading: 4.1.2 Lezgi
text: Lezgi [lez] belongs to the Lezgic branch within the Nakho-Daghestanian language family. Like, Tsez, Lezgi is a highly agglutinative language with a largely suffixing morphology. Lezgi morphology is predominantly inflectional and nouns are inflected for number, case, and localization. There are 18 nominal cases in Lezgi, 14 of which are locative (Haspelmath, 1993). Morphologically, verb stems are divided into three groups – Masdar, Imperfective, and Aorist stems – which impact the inflectional suffixes they can take on. Three distinct verb forms can be derived from the Masdar stem, nine from the Imperfective, and five from the Aorist (Haspelmath, 1993). Several additional secondary verbal categories particularly relating to mood can be achieved via suffixing on the verb. Given its close phylogenetic relationship to Tsez, we consider Lezgi to be an in-family test language.

heading: 4.1.3 Arapaho
text: Arapaho [arp] is an Algonquian language that is highly agglutinating and polysynthetic. Noun stems can be inflected for plurality, obviation, vocative, and locative cases through suffixing. Nouns also necessarily belong to either animate or inanimate gender, and gender impacts the surface representation of many inflectional markers. Arapaho nouns also participate in derivational morphology, and modified nouns can be derived from indepen-
dent nouns or verbs. Arapaho verbal morphology is even more complex. In terms of inflectional morphology, verb stems can be divided into four different classes that each take different markers for person, number, and obviation. Verbs can also be broken up into four different orders– affirmative, non-affirmative, conjunct, and imperative– which also impact the inflectional morphemes. Arapaho derivational verbal morphology is extensive, and unique verb forms can be derived through processes of prefixation, suffixation, denominalization, reduplication, and noun incorporation. We consider Arapaho to be particularly interesting as an out-of-family test language because of its rich morphology that is notably distinct from that of Tsez.

heading: 4.2 Preprocessing
text: The transcription and translation lines are not always pretokenized in these datasets (e.g., punctuation sometimes appears next to words), and it is necessary to tokenize the data for this reason. We use HuggingFace transformers’ (Wolf et al., 2020) BertPreTokenize pretokenizer for this purpose, with some additional processing to make language-specific corrections. (In Arapaho, for example, the apostrophe character ' represents a consonant, and it should not be separated from words it appears in.) After processing, we verify that there are equal numbers of surface and canonical forms, and we discard any sentences for which this is not true. Finally, we initialize our training instances by finding all unique pairs of surface and canonical forms at the word level and choose one randomly if there is more than one occurrence of it in the corpus. Both surface and canonical forms are NFD normalized. Our full datasets consist of 53800 words in Arapaho, 10952 words in Tsez, and 2060 words in Lezgi. In our experiments, the Arapaho dataset is downsampled to 16666 words, to bring its size closer to the other two datasets.

heading: 4.3 Word Alignment
text: To facilitate canonical segmentation on the word level, we preprocess our dataset by aligning words in the transcription line to their corresponding word(s) in the translation line. We experiment with two alignment methods: automatic and manual.

heading: 4.3.1 Automatic Alignment
text: We automatically align with awesome-align (Dou and Neubig, 2021), which extracts word alignments
from multilingual BERT and does not require training data for application to a new target language. We use awesome-align’s default hyperparameters except for the following: we use softmax extraction, set the softmax threshold to 5e-6, and set batch size to 32. After alignment, we split the instances for our main experiments into train, development, and test sets at a 60/20/20 ratio.

heading: 4.3.2 Manual Alignment
text: We manually create gold alignments for the Tsez data according to the general principles laid out in Melamed (1998). Melamed (1998) provides conventions for navigating complications that arise when translation is not literal, such as omissions, phrasal correspondence and idioms, amongst other linguistic nuances. Additionally, we define several language-specific principles outlined in Appendix A that address unique difficulties in mapping Tsez grammatical constructions to English. Figure 2 shows an example sentence in Tsez that we have manually aligned. Our full gold-aligned dataset consists of 1419 words, which we divide into 500- word training and test sets, and a 419-word development set.

heading: 5 Experiment
text: We experiment with the strategies described in §3.2 for incorporating information from translations into our morphological segmentation model. We proceed by first tuning our exact approach on the development split of a single language, Tsez, by exhaustively considering every combination of encoder, decoder, and CLS token translation incorporation strategies. We then apply our topperforming model to the test splits of all three languages. All experiments are performed on NVIDIA A100 GPUs, and all model implementations were
based on those provided by Yoyodyne.3

heading: 5.1 Translation Vectors
text: We use bert-base-cased (Devlin et al., 2019) to generate contextual word embeddings of the translations of each word in our aligned dataset. We then generate fixed-length translation vectors by averaging the embeddings of each word-piece in the translation sequence that was aligned to the word under consideration, as described in §3.

heading: 5.2 Evaluation Metrics
text: We employ three common metrics for evaluation. The first is whole-word accuracy, indicating the proportion of words that were segmented entirely correctly. To get a better picture of subword-level errors, we also use character-level edit distance. Finally, we use the modified F1 score outlined in Mager et al. (2020) to calculate the F1 score at the morpheme level. We consider precision to be the proportion of morphemes in the prediction that also occur in the gold, and recall to be the proportion of morphemes in gold that also occur in the prediction.

heading: 5.3 Model and Hyperparameters
text: In preliminary experiments, we conduct a hyperparameter search in order to determine which model architectures, model sizes, and optimization methods are most effective for these datasets.
Baselines and Settings for Hyperparameter Tuning Our baseline models for this task perform canonical segmentation without taking translations into account. The architectures we consider are a Transformer, a pointer-generator LSTM with a bidirectional encoder, and an attentive LSTM. Details are outlined in Appendix B. As it would be prohibitively expensive to do this in every experimental setting, we limit the scope of this search to baseline models on the Tsez datasets on three training split sizes: 100, 500, and 6572.
Architectures The pointer-generator LSTM (See et al., 2017) performs better in all settings than either of the two other model architectures, and we therefore adopt it for all subsequent experiments. Moreover, this choice is supported by evidence from Mager et al. (2020) that the pointer-generator is well suited to the low-resource canonical segmentation task.
3https://github.com/CUNY-CL/yoyodyne
The pointer-generator LSTM differs from a regular LSTM encoder-decoder in that it has a pointer network (Vinyals et al., 2015), which allows the model to copy over specific characters in the input sequence to the output sequence. The decoder assesses the probability of copying an element from the input to the output rather than generating it, then computes the probability distribution of the output at each time step by combining the probability distribution across the output vocabulary with the attention distribution over the input characters. The weights, indicating the probability of generation or copying, are determined by a feedforward network.
Data Size Matters The results of hyperparameter tuning are very similar for the 500-sentence and full data settings, but vary notably for the 100- sample setting. In all subsequent experiments, we train our 100 training sample models with one set of hyperparameters and all other models with a separate set.
In the 100-sample setting, we use a batch size of 16, two encoder and two decoder layers, an embedding size of 512 and a hidden size of 1024. We set dropout to 3.662 × 10−1, learning rate to 2.411× 10−4, and train for up to 607 epochs.
For all other experimental settings, we use a batch size of 64, one encoder and one decoder layer, an embedding size of 1024, and a hidden size of 2048. We set dropout to 2.212×10−1, learning rate to 8.056× 10−4, and train for up to 627 epochs.
Both settings use the Adam optimizer and the ReduceLROnPlateau scheduler.4

heading: 5.3.1 Alignment Considerations
text: Unfortunately, awesome-align does not produce alignments comparable to our gold-alignments. As reported in Table 2, awesome-align achieves an F1 score of only 0.1735 on our set of 500 gold aligned Tsez samples. Though we cannot assess the alignment quality of the automatic alignments we produced for Lezgi or Arapaho, we take this as an indication that our automatic alignments are of dubious accuracy, and this may adversely affect the performance of our approach. To address this possibility, we experiment with removing alignment from our approach entirely. We treat the cls-token embedding as a sentence-level representation of our translation input and use this in place of word-level
4https://pytorch.org/docs/stable/generated/ torch.optim.lr_scheduler.ReduceLROnPlateau.html.
embeddings. We then incorporate the embeddings as normal. We call this configuration CLS-Only.

heading: 6 Results and Discussion
text: We treat Tsez as our development language and perform a search over all translation incorporation strategies using the Tsez development set to determine the overall highest performing configuration, which we call TAMS. This configuration consists of Init-State in the encoder, Concat-Half in the decoder, and CLS-Concat as the CLS strategy. We additionally consider a variation we call TAMSCLS, which is identical to TAMS except that it employs the CLS-Only strategy described in §5.3.1. Further details of our translation incorporation strategy search are included in §6.3.

heading: 6.1 Test Languages
text: We apply our final TAMS model configuration to the automatically aligned test sets in each of our three languages: Tsez (development language), Lezgi (in-family test language), and Arapaho (outof-family test language). To simulate varying levels of data availability, we experiment with models trained on 100, 250, and 500 training samples in addition to experimenting with the full training set. For each of the train-limits, we report average metrics over five randomly chosen subsets of the full training set. There was a relatively high level of variance in the performance of our models, which should be considered when interpreting
our results. The standard deviation of accuracy measures ranges between 1.11-2.33 across all settings. Full standard deviation metrics are reported in Appendix B.
Results are shown in Table 1. In most cases, we see that edit distance, F1, and accuracy are roughly in agreement, so we focus on accuracy as our main evaluation metric. We find that on average our TAMS and/or TAMS-CLS outperforms the baseline in every train data setting and almost every metric. In general, performance gains are highest on the lower-resource settings, while on the higherresource settings, performance improvements are slight if present at all. In the n=100 setting, TAMS outperforms the baseline by an average of 1.99 percentage points, suggesting that our model is most beneficial in truly low-data settings. We also see consistent gains on Arapaho, which indicates that with further work, our model could be useful for polysynthetic languages. This is particularly exciting considering the relative difficulty of segmenting polysynthetic languages.
Without assessments of alignment quality for each of our test sets, we cannot properly analyze the impact of poor alignments on TAMS, however we see that TAMS-CLS often outperforms TAMS. In fact, TAMS-CLS is often the highest performing model, which suggests that alignments are not strictly necessary to see performance gains from translation incorporation. We consider this a promising finding because of the relative difficulty of sourcing alignments.

heading: 6.2 Manual vs. Automatic Alignment
text: To directly compare the influence of automatic alignment on performance, we additionally train a TAMS model on gold-aligned Tsez data (Gold) and compare it to a TAMS model trained on equiv-
alent data aligned with awesome-align. While we still report the average performance across 5 models trained on distinct training sets of 100 samples, the training sets we average here are distinct from the ones in §6.1) so performance metrics cannot be directly compared. To emphasize this point, we call the awesome-aligned setting Awesome-Gold. We report the performance of each model in Table 3. We find that TAMS-CLS is the highest performing model, suggesting that gold alignment data might not be necessary to see the best possible performance from TAMS.
It is possible that the granularity of information TAMS learns from alignments may in fact detract from its ability to generalize. This would be a positive result because expert alignment is costly, and without the need for good alignments, training a TAMS model becomes more feasible for a broader range of languages. However, further experimentation on more diverse languages is necessary to draw any conclusions to this end.

heading: 6.3 Supplementary Analysis of Translation Incorporation Strategies
text: In this section we provide details of the search that led to the final model configuration investigated above. We treat translation incorporation strategy as a hyperparameter and test all combinations on the Tsez development set to determine the optimal approach.
We take the average whole-word accuracy for configuration across all training split sizes on the Tsez development set using the CLS-None strategy. The full results of this search are shown in Table 4. From this search, we find that the overall highest performing strategy configuration is Init-State in the encoder and Concat-Half in the decoder. This configuration outperforms the baseline by an average of 1.58 percentage points and is the top
performer on the 100 and 500 train sample settings. Only in the highest train data setting does the baseline outperform our top configuration.
For the 100 train sample setting, Concat in the encoder and Concat-Half in the decoder, the second highest performing configuration overall, was the top configuration. With this in mind, we perform a second search over the CLS strategies on these two configurations, shown in Table 5. From this, we find our final TAMS configuration: InitState in the encoder, Concat-Half in the decoder and CLS-Concat as our CLS Strategy.

heading: 7 Conclusion
text: We present a novel method for incorporating information from translations into a morphological segmentation model to support low-resource canonical segmentation. Using Tsez as a development language, we determine our best-performing model (TAMS), which uses a fixed-length representation of the translation in two ways: to initialize the hidden state in the encoder (Init-State) and to concatenate to the input at each time step in the decoder (Concat-Half). Our model is most beneficial in the super low-resource setting (n=100), where it outperforms the baseline by 1.99 percentage points on average across three morphologically complex languages. And although we only tune our model on the Tsez development set, we also see performance gains for Arapaho, a typologically and morphologically distinct polysynthetic language. This promising result suggests that TAMS could be beneficial for a wide range of languages. We believe TAMS will be the most beneficial in the language documentation setting, where extreme resource constraints are realistic and often expected.
The findings of our TAMS-CLS experiment are especially promising because they indicate that we may be able to see benefits from incorporating translations in canonical segmentation even without word-level alignments. This opens up TAMS to many more languages which may not have highquality word-level alignments available.
However, our results are more mixed in higherresource settings, which indicates that there is still more work to be done to determine whether translations are a valuable addition to canonical segmentation models in cases where more data is available. Overall, canonical segmentation for morphologically complex languages remains a challenging task, but we believe that this work indicates that
translations should be explored further as an additional data resource. There are several avenues for future work we wish to highlight. A first possible improvement strategy could be to experiment with providing more explicit information instead of or in addition to translations, such as the POS tags of the aligned English words. Second, it would be interesting to see whether our results can be reproduced on other languages and with other PLMs. Third, there may be other ways to use translation-based representations with LSTMs.
Limitations
Due to data availability, we experimented only on two language families, Northeast Caucasian and Algonquian, but ideally we would have tested on more language families. We cannot concretely say that our models would perform equivalently on a more diverse set of languages. Another limitation was in the exhaustiveness of our hyperparameter search. Ideally, we would have searched each possible CLS token strategy with each possible configuration of translation incorporation strategies but
we were unable to due to the GPU hours that would have been required.

heading: 8 Acknowledgements
text: We thank the anonymous reviewers for their useful suggestions and feedback, as well as the LECS and NALA Labs at the University of Colorado Boulder. We would like to extend our gratitude to Bernard Comrie and Sarah Moeller for their invaluable assistance with interpreting our datasets. Parts of this work were supported by the National Science Foundation under Grant No. 2149404, “CAREER: From One Language to Another.” Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. This work utilized the Blanca condo computing resource at the University of Colorado Boulder. Blanca is jointly funded by computing users and the University of Colorado Boulder.

heading: A Manual Word Alignment Directives
text: A.1 High-level Directives • Align English prepositions with the Tsez word with the equivalent case marker.
• If some grammatical information is expressed in one language but not the other, behave as if it were expressed in the corresponding phrase. In such cases, the extra word (‘the’, for example) should be aligned to the head of the corresponding phrase.
• But, if the definiteness is expressed on a modifier of the noun head (like an attributive adjective), then align the article to the modifier bearing the definiteness information instead.
A.2 Lower-level Directives • Pronominal subjects that are not expressed in Tsez and are expressed in English should have the
English subject aligned to the head predicate in Tsez.
• If in English you have a PP and in Tsez you have a relative clause where the subject position has the gap, align the preposition introducing the PP with the relative clause’s predicate
• If a quotative verb like ‘say’ is used a variable amount of times in one language compared to the other, then align all instances of the quotative verbs together, so long as the quoted material they’re all referring to is identical.
• Always align the expletive ‘there’ with the existential verb in Tsez. And if there is an adverbial ‘there’-equivalent in Tsez, do not align it with anything in English unless there really is an adverbial, non-expletive ‘there’ or similar in English
• For articles in English, if there’s something very close to an article in Tsez (‘a’, or ‘this’, or . . . ), then prefer aligning the English article to the similar word instead of the noun.

heading: B Hyperparameter Tuning
text: We conduct hyperparameter tuning for the baseline Tsez models without translation using random search with the full training set of 6572 words. We then took the top two architectures and performed a hyper-parameter sweep with 100 and 500 training samples to simulate a lower-resource setting. The top performing models for each architecture and training set size are outlined in Tables 8, 7, 6. The search space of architecture specific hyperparameters is outlined in Table 9 and Table 10 and the search space of optimization parameters is outlined in Table 11. All models used Adam optimization. We report whole-word accuracy on the development set.
C Standard Deviation of TAMS Performance

heading: Future Work
text: There are several avenues for future work we wish to highlight. A first possible improvement strategy could be to experiment with providing more explicit information instead of or in addition to translations, such as the POS tags of the aligned English words. Second, it would be interesting to see whether our results can be reproduced on other languages and with other PLMs. Third, there may be other ways to use translation-based representations with LSTMs. Limitations
Due to data availability, we experimented only on two language families, Northeast Caucasian and Algonquian, but ideally we would have tested on more language families. We cannot concretely say that our models would perform equivalently on a more diverse set of languages. Another limitation was in the exhaustiveness of our hyperparameter search. Ideally, we would have searched each possible CLS token strategy with each possible configuration of translation incorporation strategies but
we were unable to due to the GPU hours that would have been required