CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models
Authors: Benjamin Minixhofer
Jonas Pfeiffer
Ivan Vulić
Venue: EMNLP 2023 Main (2023)
Keywords: segmentation
multilinguality
tokenization
compound
compounds
TL;DR: Current LLMs are bad at splitting compound words into their parts; we introduce a dataset to evaluate this deficiency and multiple ways to fix it.

Abstract: While many languages possess processes of joining two or more words to create compound words, previous studies have been typically limited only to languages with excessively productive compound formation (e.g., German, Dutch) and there is no public dataset containing compound and non-compound words across a large number of languages. In this work, we systematically study decompounding, the task of splitting compound words into their constituents, at a wide scale. We first address the data gap by introducing a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We then use this dataset to evaluate an array of Large Language Models (LLMs) on the decompounding task. We find that LLMs perform poorly, especially on words which are tokenized unfavorably by subword tokenization. We thus introduce a novel methodology to train dedicated models for decompounding. The proposed two-stage procedure relies on a fully self-supervised objective in the first stage, while the second, supervised learning stage optionally fine-tunes the model on the annotated Wiktionary data. Our self-supervised models outperform the prior best unsupervised decompounding models by 13.9% accuracy on average. Our fine-tuned models outperform all prior (language-specific) decompounding tools. Furthermore, we use our models to leverage decompounding during the creation of a subword tokenizer, which we refer to as CompoundPiece. CompoundPiece tokenizes compound words more favorably on average, leading to improved performance on decompounding over an otherwise equivalent model using SentencePiece tokenization.

CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models
Benjamin Minixhofer' Jonas Pfeiffer Ivan Vulić
University of Cambridge
Google DeepMind
Abstract
Easy Compounds
Hard Compounds
100
100
While many languages possess processes of joining two or more words to create compound words, previous studies have been typically lim- ited only to languages with excessively produc- tive compound formation (e.g.. German, Dutch) and there is public dataset containing com- pound and non-compound words across a large number languages. In this work, we system- atically study decompounding, the task of split- ting compound words into their constituents, at wide scale. We first address the data gap by in- troducing a dataset of 255k compound and non- compound words across 56 diverse languages obtained from Wiktionary. We then this dataset to evaluate an arra of Large Language Models (LLMs) on the decompounding task. We find that I.LMs perform poorly, especially on words which are tokenized unfavorably by subword tokenization. We thus introduce novel methodology to train dedicated models for decompounding. The proposed two-stage procedure relies fully sule-supe ob- jective in the first stage, while the second, super- vised learning stage optionally fine-tunes model the annotated Wiktionary data. Our self-supervised models outperform the prior best unsupervised decompounding models by 13.9% accuracy average. Our fine-tuned models outperform all prior (language-specific) decompounding tools. Furthermore, our models to leverage decompounding during the creation subword tokenizer, which we refer to CompoundPiece. CompoundPiece tokenizes compound words more favorably on average, leading to improved performance on decompounding over otherwise equivalent model using SentencePiece tokenization.
Oura (81+82)
80
80
Ours (S1+82)
Ours (81)
Accurccy
(S1)
28
0.3
#shoss
Rshots
FLAN UL2 208 -
FLAN X5XXL
FLAN ASANTSXA XL
FLAN
Figure 1: In-context learning performance of LLMs on compound segmentation vs. our method ($5).
Constituents
Subwords
Hard Compound
wind
sur surfing fing
winds
ur
1ng
Easy Compound
spell binding
spell
bind
ing
Figure Examples of easy and hard compounds w.r.t the T5 tokenizer (also used by FLAN UL.2 20B).
2018). Historically, it has also been widely used as preprocessing step for other NLP tasks, e.g. for information retrieval (Monz and De Rijke, 2002; Braschler and Ripplinger, 2004), automatic speech recognition (Adda-Decker and Adda, 2000) and machine translation (Koehn and Knight, 2003).
Decompounding can come two similar yet dif- ferent task formats: (i) compound segmentation and (ii) compound normalization (Ziering and van der Plas, 2016). Compound segmentation is the task of segmenting word into its compound constituents, while preserving its surface form (e.g. bridesmaid + brides maid). Compound normalization is the task of recovering the base form of each compound constituent (e.g. bridesmaid + bride maid).'
1 Introduction
Decompounding is the task of separating compound words into their single word constituents. Decom- pounding is used in user-facing tools such dic- tionaries and morphological analyzers (Altinok,
Most prior work on decompounding has focused on the few languages with excessively productive
'In morphological segmentation, segmentation and normal- ization are also referred to as surface-level segmentation and canonical segmentation, respectively (Cotterell et al., 2016).
Equal senior authorship.

---

compound formation such Finnish, German and Swedish (Kochn and Knight, 2003; Shapiro, 2016; Riedl and Biemann, 2016). However, compound words occur in a large, diverse number of lan- guages (Vogel and Scalise, 2010). Yet, datasets which annotate compounds with their segmented or normalized form sparsely exist, even in languages with high compound usage. As the first contribu- tion of this work, aim address this issue by introducing dataset of 255k compound words and their normalized form as well as non-compound words covering 56 languages obtained from Wik- tionary (www. wiktionary.orr org).
poundPiece tokenizer achieves 5.5% improved performance on compound normalization over an otherwise equivalent SentencePiece model.
Contribations. 1) We introduce dataset for de- compounding of 255k words across 56 languages obtained from Wiktionary. 2) We show that a byte-level language model can efficiently decom- pound words via a two-stage training framework, whereas current subword-basee based LLMs fall short. 3) We present a way to improve subword tok- enization by performing compound segmentation during creation of the tokenizer. 4) We make our code. models and dataset publicly available at gi titthub.com ctmub..cm/bbbi ccom/biing www.inniiooreeeeee wwwpho/fe/ccooouppiicccc.oo
Using our dataset, we then find that large lan- guage models (LLMs), which typically rely subword-based tokenization (Sennrich al., 2016; Kudo and Richardson, 2018), struggle with decom- pounding, as illustrated in Figure 1. Performance is especially low for compounds where subword boundaries do not coincide with compound con- stituent boundaries; we term compounds with this property 'hard' compounds (Figure 2).
Related Work
Decompounding, Early work in decompounding used word frequency lists along with manually specified suffixes (e.g., connective -s-) to seg- ment and normalize German compounds (Langer, 1998; Koehn and Knight, 2003). Subsequently, multiple submissions to the Morpho Challenge morphological segmentation (Kurimo et al., 2010) explicitly or implicitly made use of compound seg- mentation (Lignos, 2010; Virpioja et al., 201 I). Later work replaced the fixed list of suffixes used in Koehn and Knight (2003) by learned morpho- logical operations from parallel corpora (Macherey et al., 2011) or from pre-lemmatized corpora of non-compound words (Ziering and van der Plas, 2016). Another branch of work added more linguis- tic knowledge in the form black- and white-lists to the paradigm of Koehn and Knight (2003), re- sulting in J'WorSSpliterr" (German) and nl-splitter (Dutch); this has only been done for couple of languages due to its knowledge-intensivee nature. CharSplit (Tuggener, 2016) achieves high perfor- mance for German by relying on the frequency of character n-grams appearing within the compound.
In order to create more effective decompound- ing model, we then formulate compound segmenta- tion and normalization as sequence-to-sequence learning task (Sutskever et al., 2014) and train a byte-level ByT5 model (Xue et al., 2022) using a two-stage framework. In the first stage, we use a novel self-supervised to learn compound segmentation without any la- beled data. In the second stage, turn the model into a compound normalization model via super- vised training on our Wiktionary data, In addition, we introduce a procedure to predict the segmenta- tion of any compound word based on its normalized form, effectively making compound segmentation a subtask of normalization. Finally, we demonstrate that decompounding has real-world applications by investigating compound segmentation for lan- guage model tokenization. We apply compound segmentation as pretokenization during training of SentencePiece tokenizer (Kudo and Richard- son, 2018), which results fewer hard compounds while incurring no extra cost during training and inference of the language model (i.c. the only extra cost occurs during creation of the tokenizer).
prechentreiitiion objective
While the approaches above use (at most) light supervision, there exist supervised approaches which learn directly from an annotated corpus of compounds and their constituents, along with op- tional auxiliary signals (Biemann et al., 2008; Al- fonseca et al., 2008). In contrast, SECOS (Riedl and Biemann, 2016) is fully unsupervised and hunguage-agnosta method achieving competitive performance by using word embeddings along with word frequencies for semantic compound segmen-
Our Stage 1 models outperform the best prior unsupervised models by 13.9% accuracy on aver- age, while our (supervised) Stage models outper- form all prior Linguage-speciffe decompounding tools. Furthermore, a model trained with a Com-
"git comm//aniielnaeerr//vrtsstttttee thub. http:////ooniiiiooff////aos-eo--

---

16
Festive (Compounti Negetive CompnCa
167
101
Figure 3: Number of positive and negative examples across languages in the Wiktionary dataset.
Word
Constituents
Language
terms in languages with highly productive com- pound formation beyond German and Dutch, such as Finnish and Greek (Pollatsek et al., 2000; Lindén and Pirinen, 2009; Koliopoulou, 2014; Shapiro, 2016; Virkkunen et al., 2018). to the best of our knowledge there exist no public datasets for decom- pounding in these languages (and beyond).
(same) + (ancestor)
Persian
(sibling)
akiratis
akis (eye) rutas (circle)
Lithuanian
(horizon)
ueeaap
mex (fimit) apu (distance)
Kazukh
(bordor)
Abentcuer
None
German
(adventure)
zissuane
Linguistically Informed Tokenization. Various studies have tried augmenting replacing the "lin- guistically uninformed' subword-tokenizers used in contemporary LMs (Devlin al., 2019; Raf- fel et al., 2020, inter alia) such as SentencePiece (Kudo and Richardson, 2018) and BPE (Sennrich et al., 2016) with linguistic knowledge. Using man- ually constructed morphological analyzers before applying BPE (Pan et al., 2020) or after generation (Matthews et al., 2018) has led to improvements. but is limited by the availability (and quality) of morphological analyzers across many languages. Unsupervised morphological segmentation has not shown consistent improvements (Zhou, 2018; Sal- eva and Lignos, 2021; Domingo et al., 2023); see Mielke et al. (2021) for additional discussion.
eiss (cash) wald (stream)
Gujarati
(cashflow)
Figure 4: Example words the Wiktionary dataset.
tation. Our method improves over SECOS in the unsupervised case and provides a unified alter- native to prior language-specific decompounding tools via additional training on labelled data.
Relation to Morphological Segmentation. De- compounding can be seen as a special case of mor- phological segmentation (Batsuren et al., 2022a). However, a large amount of work in morphological segmentation focuses derivational and inflec- tional morphology (Cotterell et al., 2016; Faruqui al., 2016; Cotterell et al., 2018; McCarthy et al., 2019; Goldman et al., 2022), which is reflected by datasets such as UniMorph (Batsuren et al., 2022b) and MorphyNet (Batsuren et al., 2021) annotating inflectional and derivational affixes, but not com- pound constituents. The SIGMORPHON-209 Shared Task (Batsuren et al., 2022a, SMST 2022) breaks this pattern by providing dataset for seg- mentation into compound constituents in addition inflectional and derivational affixes. We improve on the SMST 2022 dataset by broadening coverage from to 56 languages, as well as handling nega- tives (i.e., compouddo more carefully ($3.1).
Methodology
3.1 Dataset Construction
We use words categorized as compound terms on Wiktionary to create a dataset for decompound- ing. The information on Wiktionary allows asso- ciating compound terms with their corresponding normalized constituents. Since Wiktionary only annotates the top-level split, we recursively split constituents into their smallest parts by checking if the top-level constituents are themselves com- pound words. Many prior decompounding tools do not evaluate performance on negative examples (i.e. non-compound words; Kochn and Knight, 2003; Riedl and Biemann, 2016; Tuggener, 2016) since most prior datasets do not contain any (Henrich
Decompounding Datasets. Besides the SMST 2022 dataset, datasets for decompounding include AuCoPro (van Zaanen al., 2014) for Dutch and Afrikaans, and the GermaNet dataset for Ger- man (Henrich and Hinrichs, 2011). 1). Although there is a significant amount of work studying compound
For instance, highwayman is segmented into hi highway man instead of high way man.

---

and Hinrichs, 2011; van Zaanen et al., 2014). It is not trivial obtain negative examples from Wik- tionary since a large amount of compound words are not categorized as such, leading to many false negatives. We solve this issue by using all normal- ized compound constituents as negative examples, since by definition the compound constituents can also appear their own as non-compound words. Note that this way of obtaining negative examples is biased against words which never occur inside compounds; however, we found this to be a rather weak bias (Appendix E). We include every lan- guage with at least 100 words, leading to a dataset which covers 56 languages. The number of train- ing examples is shown in Figure 3, example words in Figure 4. We select up to 1,000 words (but at most 50% of total words) in every language eval- uation data. See Appendix A for further details concerning the dataset.
Input
Word x: akiratis (horizon)
Norm, constituents c: (akis (eye), ratas (circlej
Find optimal segmentation
5g
82
C(S)
0
kiratis
3
3
ak
iratis
2
2
aki
ratis
akir atis
2
akira tis
2
akirat is
3
akirati
Output
Segmentation s*: (aki, ratis)
Figure 5: Turning compound normalization into seg- mentation by minimizing edit distance ($3.3).
than the non-hyphenated form. 5
We strip all words of hyphens and train a seq2seq LM to predict the original (hyphenated) form of each word. We introduce logit bias b added to the logit of the token representing a hyphen to skew generation towards or away from hyphenation at inference time. Training on this data enables ef- fective compound segmentation without relying on human annotations, as demonstrated later in $5.
3.2 Two-Stage Training
To overcome the problem of data scarcity low- resource languages, we introduce a two-stage train- ing procedure for creating dedicated decompound- ing models. In Stage I, we train on the self- supervised objective of restoring hyphenation in words extracted from large- scale Web corpus, leading to a self-supervised compound segmenta- tion model. In Stage 2, we fine-tune the model on compounds and their normalized constituents from an annotated corpus in a supervised fashion, turning it into a compound normalization model.
Stage 2: Supervised Compound Normalization. In the second stage, we improve upon the Stage model by additional training on labeled data, where the inputs are individual compounds, and the target is to predict the normalized constituents of each compound, separated by a hyphen. Training ex- clusively on compound normalization allows using data from the collected Wiktionary dataset, which contains compound terms along with their normal- ized constituents across many languages, but does not contain compound segmentation annotations.
Stage 1: Supe--uee Compound Segmenta- tion. This stage is motivated by the fact that hyphen characters can be seen as a high-precision, low- recall indicator of compound constituent bound- aries, in the same way that newline characters are high-precision, low-recall indicator of sentence boundaries (Minixhofer et al., 2023). We use this natural segmentation into compound constituents to create a compound segmentation model with- out requiring any labeled data. First, we obtain all words containing hyphen plus an equivalent amount of non-hyphenated words from corpus of unannotated text. Hyphens primarily have two uses: (1) as compound boundary and (2) to indi- cate the word continues on the next line. We only want retain hyphens when they function as com- pound boundaries, so we filter the instances of (2) by discarding all words where the hyphenated form of the word occurs x S e 6 times less frequent
3.3 Turning Normalization into Segmentation
Considering the scarcity of annotated compound segmentation data, it infeasible to train a multi- lingual model directly on segmentation. Thus, we introduce method to predict a segmentation given the normalized constituents. Let æ be word of length n. In addition, we have k normalized com-
Consider for example the hyphen-as compound-boundary in side-experiment and the hyphen-as-nnn
in experi -eents. #espert-rents will be considerably than #slde-eweerinents #stceesper inents ir was chosen from {e-4, 44, - -6 by manual inspection in preliminary experiments.
#esparinents

---

pound constituents { ) (e.g. predicted by the Stage model). Our aim to find bound- aries r = (ro..... rk). ro = 0, rk = n giving rise to the segmentation s - (=fro- :
(a) among the bright flowerbeds,
(b) among the br ight
Flowerbeas
xfrx-1 nx]}- We approach this problem by minimizing the edit distance of each segment to its corresponding normalized constituent. This leads to an optimiza- tion problem where the cost C(s) indicates the total edits needed to turn all segments into their corresponding normalized constituents:
(c)
among the br ight Flowerbesd
Figure 6: (a) no pretokemization, (b) pretokenization by splitting whitespace, (c) our pretokenization.
We hypothesize that hard compounds may im- pair language model performance due to the non- trivial relation of subwords to the compound word. In contrast, in easy compounds the word is nat- urally decomposed into its constituents. The in- creased difficulty of hard compounds is apparent on the requence-to-sequeee tation task: for an easy compound, all tokens can be copied to the output (only the special separa- tor tokens must be inserted). On the other hand, for hard compounds, the tokens change, requiring knowledge of the characters within each token.
(($) - Σ [(s ci).
Here, L an edit distance metric such as Lev- enshtein distance (Levenshtein et al., 1966). The optimal segmentation s* is the segmentation with the minimal cost: s* arg min, C(s).
compound segmen-
In case of ties, prefer segmentations with higher edit cost for segments with lower indices due to the preference for languages in our training set for suffixation over prefixation (Hammarström, 2021). 6 There is total of (e"1) possible segmen- tations, so solving the optimization problem via enumeration of all solutions is only feasible for short words (Figure 5). We introduce more effi- cient search algorithm which is capable of quickly finding the optimal segmentation of long words by enumerating candidates in order of lower bound on the edit distance in Appendix B. This method can be used to turn the normalization predictions a model into segmentation. We also use it on the ground- -ruth normalization from Wiktionary, mak- ing it possible to approximate compound segmen- tation performance by comparing the segmentation corresponding to the ground-truth normalization to the segmentation produced by the model normal- ization predictions.
Tokenizers where every possible constituent boundary is token boundary trivially do not give rise to any hard compounds. This includes character-level (Clark al., 2022; Tay et al., 20226) as well as byte-level tokenizers (Xue et al., 2022). However, many contemporary language models use subword-based tokenizers to increase efficiency (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020). We propose modification to subword tok- enization reduce the number of hard compounds while keeping the efficiency advantages.
Subword tokenizers typically segment text into pre-tokens (c.g. by splitting on whitespace) be- fore applying their subword tokenization algo- rithm (Mielke et al., 2021). We propose modifying pretokenization by applying compound segmenta- tion in addition to splitting modification is only done during creation of the to- kenizer, thus incurring no additional cost once the tokenizer has been created. We refer to tokenizers created in this way as CompoundPiece tokeniz- ers. The modified pretokenization tries to create more subwords which do not span compound con- stituent boundaries, thus decreasing the fraction of hard compounds (Figure 6). It aims to turn the dual-route model for computing the meaning of complex (compound) words proposed by Hofmann et al. (2021) into a single-route model which al- ways computes the meaning of compounds from their constitutent subwords, and never stores a com- pound word as a single subword.
whitespace. This
3.4 Reducing Hard Compounds
We define hard compounds relative to particu- lar tokenizer compound words where the con- stituent boundaries do not coincide with token boundaries set by the tokenizer. More formally, a compound word made up constituents and subwords hard if the constituent boundaries r {ro. } are not a subset of the token boundaries t = = {lo,..,, 4 i.e. r t.
"E.g., given x = br bridesmaid, e = (bride, maid], J. we prefer the segmentation !brides, m over (bride, smaid) although their cost is equal.

---

4 Experimental Setup
Languages. For clarity of presentation, we present results on Danish, German, English, Spanish, Esto- nian, Greek, Persian, Finnish, Hungarian, Kazakh, Latvian, Dutch, Polish and Swedish as a linguisti- cally diverse subset of languages with productive compound formation in the main paper. For the full evaluation across all languages, see Appendix C.
4.1 Data
We obtain Stage data by selecting all words con- taining a hyphen from a subset of the mC4 corpus (Xue et al., 2021) which results in -25M hyphen- ated words. As negative examples, we choose the n most common words from mC4 such that there is equivalent amount of non-hyphenated and hy- phenated words in every language. Regarding the Stage 2 data, see Section $3.1 before.
Results and Discussion
Main compound segmentation results are shown in Table 1. For the self-supervised models, we choose the logit bias b 3 to bias generation towards hyphenated words. 8 ByT5 outperforms subword- based models by a large margin with an absolute 8.9% improvement over the best subword-baa based model after Stage training, and a 3.7% improve- ment after Stage training. Comparing models not trained on any annotated data, the self-supervised ByT5 outperforms SECOS on 13 out of 14 lan- guages, and by 13.9% on average.
4.2 Training
We train decompounding model using two-stage framework ($3) covering 56 languages. We use ByT5 (Xue et al., 2022) as our main pretrained model and the main starting point since directly ingests Unicode bytes instead of using subword tokenization, leading to zero hard compounds. We compare our approach against the subword-based T5 (Raffel et al., 2020), Flan-T5 (Chung et al., 2022) and mT5 (Xue et al., 2021) trained with the same two-stage framework. We use t5x (Roberts al., 2022) for training with batch size of 5 312 and maximum sequence length of 64 tokens, oth- erwise matching T5 pretraining (Raffel et al., 2020). The setup is the same for Stage and Stage 2.
We further compare against language-specite and supervised methods in Table 2. Our ByT5- based model outperforms all prior methods on ev- ery dataset. Since GermaNet tests compound head segmentation (i.e., even if word contains multiple constituents, it is only split into head and a modi- fier) we count example correctly segmented if either the first constituent matches the modifier or the last constituent matches the head.
4.3 Evaluation
Metrie. We measure performance via averaged ac- curacy, i... the ratio of examples which are entirely correctly segmented or normalized.
Evaluating LLMss Decompounding. We also evaluate in-context learning performance of multi- ple LLMs on compound segmentation. We use T5 models with 770M, 3B and 11B parameters (Raffel et al., 2020) as well as the UL2 model with 20B parameters (Tay et al., 2022a) since all of them use the same tokenizer, enabling performance compar- isons hard compounds across LLMs. We use the model versions fine-tuned on the Flan dataset col- lection (Chung al., 2022), matching our prompt to the style of instructions in the Flan collection (Appendix D). Zero- to 16-shot results are shown Figure 7. Although the LLMs perform non-trivially well on easy compounds, performance is close to zero (<3%) on hard compounds. Intriguingly, UL2 20B performs worse than Flan T5 XXL (1IB), re- versing the trend seen other tasks (Tay et al., 2022a). All the LLMs perform considerably worse than our ByT5- model; see Figure
Datasets. Besides our new Wiktionary evalua- tion subset, we use the established datasets for particular languages: GermaNet (Henrich and Hin- richs, 2011) 1), AuCoPro for Dutch (van Zaanen et al., 2014) as well the subset containing compound-only words across 6 languages from the SIGMORPHON 2022 Shared Task (Batsuren et al., 2022a).?
Baselines. We use SECOS as the main unsuper- vised baseline, as well as CharSplit, JWS and nl- splitter as baselines using different amounts of su- pervision. For the SIGMORPHON 2022 Shared Task dataset, we compare against the task winner, DeepSPIN-3 (Peters and Martins, 2022).
We do not include words containing derivational or inflec- tional aflixes since the type of morpheme is not specified, so it is not possible to distinguish between derivational/intlccioon. affixes and compound constituents. We also do not include ront words since we found from manual inspection that >10% of root words are mislabeled, likely due to the difficulty of obtaining negative examples from Wiktionary ($3.1).
"Chosen among the set [0, 1, 2, 3, 44 to maximize perfor- mance on the English validation data.

---

Macro
de
en
C8
et
fa
hu
Iv
pl
SW
Arg.
SECOS
30.0
66.5
41.2
29.0
23.4
5.3
1.4
53.1
38.8
5.0
13.9
46.8
22.2
32.2
29.2
T5
55.3
56.1
85.9
69.8
29.0
0.0
0.0
31.6
48.6
16.9
29.6
44.9
36.1
53.1
39.8
Flan-T5
58.5
71,0
37.0
0.0
33.0
53.4
41.7
44.8
40.3
56.5
42.9
SI
mTS
25.8
38.8
79.7
58.3
18.6
21.6
3.9
24.1
18.8
45.0
20.2
23.0
32.9
21.9
30,9
P
ByT5
75.6
76.0
91.3
77.2
51.6
40.9
20.9
52.7
70.0
75.9
41.7
57.2
51.8
64.8
60.5
TS
86.3
96.0
95.4
82.5
0.0
98.2
18.3
78.0
69.6
Flan-T5
86.6
95.3
95.5
83.2
80.9
0.0
98.3
87.3
16.5
68.2
93.6
77.4
89.4
69.5
S1+S2
mT'
87.1
94,1
95.4
82.3
83,2
73.1
62.1
97.1
90.4
86.7
76.7
93.4
84.1
90.0
85,4
ByT5
92.2
96.6
97.8
87.1
92.6
86,1
76.6
98.8
97.2
91.7
84.8
97.5
91.2
94.3
91.7
SECOS
96.1
86.6
93.8
97.4
98.6
99.7
100
88.2
95.5
100
100
94.1
96.9
97.3
96.0
88.5
91.8
91.7
88.7
82.3
100
100
82.2
93.8
74.0
87.4
83.7
90.6
91.8
89.0
Flan-TS
92.1
89,9
100
82.9
72.9
87.0
87.0
90.4
SI
mTs
92.7
92.8
90.9
92.3
89.9
95.3
99.3
88.2
98.0
88.0
95.9
89.1
94.5
94.8
93.0
N
ByT5
89.0
89.7
88.4
81.5
76.0
95.7
97.3
77.6
87.1
72.1
87.7
80.3
91.4
87.8
85,8
TS
93.3
94.5
97.8
100
95.4
91,1
97.4
Flan-T5
94.1
95.5
97.9
95.9
95.8
100
100
96.7
98.6
92.6
96.7
97.5
97.1
96.7
96.8
S1+$2
mT5
93.8
96.2
99.2
97.4
97.9
96.3
98.7
94.1
98.6
96.9
98.1
96.7
97.9
97.3
97.1
ByT5
95.2
96.2
98.3
98.8
97.9
97.3
97.3
95.4
99.7
99.2
98.9
97.9
99.0
97.6
97.8
SECOS
53.5
72.4
53.9
63.2
56.0
60.9
52.2
58.4
59.0
50.7
61.0
58.1
57.8
53.6
57.9
67.1
66.5
87.3
79.3
52.1
59.0
51.5
39.3
64.7
44.4
61.2
54.2
62.1
65.8
61.0
Flan-T5
69.1
68.3
80.5
56.6
59.0
51.5
40.6
67.0
44.2
66.5
54.9
64.2
62.9
S1
mTs
49.6
54.6
82.4
75.3
49.5
65.1
53.1
33.8
47.0
65.7
61.6
38.8
62.3
45.9
56.0
All
ByT5
80.4
80.0
90.6
79.4
62.2
73.2
60.3
56.5
76.1
74.1
66.9
62.7
70.7
72.4
71.8
TS
95.6
90.2
59.0
51.5
97.8
53.4
84.6
87.6
83.5
Flan-T5
89.3
95.4
96.1
89.6
87.3
59.0
51.5
98.1
91.3
53.2
83,7
94.5
86.8
91.8
83.4
S1+S2
mT5
89.5
94.7
96.3
89.8
89.6
86.8
80.9
96.6
93.3
91.6
88.4
94.2
90.7
92.4
91.1
ByT5
93.3
96.5
97.9
92,9
94.9
92.7
87.3
98.3
98.1
95.3
92.5
97.6
94.9
95.4
94.8
Table I: Accuracy on compounds (Positives=PP) non-compound words (Negatives=:) and across all examples. We report scores SECOS as baseline, as well as Stage 1 training only (S1) and Stage plus Stage 2 training (S1+S2).
Postives (Casy)
Postives (farnt)
Negatives
20
20
20
XXL
16
2
15
Figure 7: Few-shot in-context learning performance of L.LMs on easy positives, hard positives, negatives and across all examples. Hard negatives the same across all LLMs since they use the same tokenizer.
Reducing Hard Compounds via Compound- Piece. To evaluate our method reducing the number hard compounds in subword-based lan- guage models ($3.4), we train CompoundPiece models in two configurations: (i) multilingual to- kenizers across all 56 languages and (ii) separate monolingual tokenizers for every language. For the multilingual tokenizers, we sample languages with p(I.) x | where pT..) is the probability sampling text from a language with [L] texts as in prior work (Conneau et al., 2020). We use a subsample of 10M texts from the mC4 corpus (Xue et al., 2021) with = 0.2. The vocabulary size is 250k for the multilingual and 32k for the monolin-
gual tokenizers, following prior work (Rust et al., 2021; Conneau et al., 2020).
We use our fine-tuned ByT5 model for train- time pretokenization into compound constituents and SentencePiece (Kudo and Richardson, 2018) with Unigram LM (Kudo, 2018) as the subword tokenization applied after pretokenization. As a baseline, we train SentencePiece tokenizers with pretokenization into words (split by whitespace) on the same data. Table shows the percentage of hard compounds for every tokenizer. Compound- Piece reduces the number of hard compounds from 27.1% - 9.7% on average in the monolingual case. In the multilingual case, there is a less marked

---

Monolingual
Multilingual
Segmentation Normaizztiin
Word
Segmentation
Segmentation
Token Origin
P
All
P
N
All
.tt es, sk, gbo: yo, mg, fr at: id. hu, la
JWS
83.7
83.7
53.4
53.4
tugboat tug, boat
tu, gbo, at
CharSplit
95.1
95.1
GermaNet
SECOS
83.6
83.6
_inddst: da ate: it, et,
ByT5 (S1+S2)
97.9
97.9 79.6
79.6
mindstate _indd state
_mindst, ate
JWS
59.7 97.6 70.8 43.2 97.6
_coa: gl.
Ours (de)
CharSplit
84.7 29.5 68.6
coatrack _coat, rack
coa, track
track: hu, th, da
SECOS
66.5
72.4
ByT5 (S1+52)
96.6 96.2
96.5
89.8
96.2
91.7
Table 4: Example compound words which easy
nl-splitter
74.5
74.5 67.1
67.1
for the monolingual but hard for the multilingual Com- poundPiece tokenizer. "_" indicates whitespace.
AuCoPro-nl
SECOS
59.7
59.7
ByT5 (S1+52)
91.7
91.7 76.2
76.2
nl-splitter
61.2 96.7
69.7 47.0 91.2
57.6
Segmentation
Normalization SPM-T5 CPM-TS
Ours (ul)
SECOS
46.8 94.1
58.1
Language
SPM-T5
CPM-T5
ByT5
97.5 97.9
97.6
87.8 97.9
90.2
Danish
77.8
77.7
65.5
69.1
SMST 2022
D
88.6
88.6 87.3
87.3
German
81.0
80.7
61.5
63.8
ByT5 (S1+52)
92.5
92.5 88.6
88.6
English
84.9
85.8
82.9
84.0
Spanish
75.2
74.7
50.1
55.2
Estonian
78.6
84.5
55.1
61.3
Table Comparison against supervised and rule-based baseline models. We use the subset of compound-only words from the Sigmorphon Shared Task (SMST) 2022 data which covers 7 languages (Batsuren et al., 2022a).
Greek
70.6
70.0
47.1
57.8
Persian
58.2
61.2
46.6
58.1
Finnish
72.8
74.1
59.0
59.6
Hungarian
76.2
76.9
73.3
76.2
Kazakh
72.9
75.7
59.0
74,4
Latvian
75.2
69.1
53.5
57.3
Multilingual
Monolingual SPM
Dutch
78.2
80.7
60.9
64.9
I Language
SPM (mTS)
SPM
CPM
CPM
Polish
65.8
65.6
42.6
46.7
Swedish
76.2
77.3
61.0
65.6
Danish
15.5
16.5
12.4
24.7
5,9
German
9.9
10.3
8.2
14.6
1.8
Macro Arg.
74.6
75.3
58.4
63.9
English
7.5
8.2
4.6
6.8
3.7
Spanish
29.0
24.9
18.7
14.2
10.3
Table 5: Accuracy of our multilingual T5 modelss trained with SentencePiece (SPM-T5) and CompoundPiece
Estonian
25.5
35.4
7.2
Greek
39.9
33.6
23.1
28.9
14.9
(CPM-T5) segmentation and normalization.
Persian
38.6
46.1
37.2
70.9
41.8
Finnish
25.1
25.1
20.3
10.3
5.1
Hungarian
13.8
17.1
10.1
26.1
3.7
Kazakh
14.4
13.7
9.0
28.4
4.0
Segmentation
Normalization
Latvian
20.2
23.8
16.1
47.5
11.7
Dutch
12.8
15.4
10.2
17.2
3.3
P
N
All
P
N
All
Polish
45.7
33.1
33.6
Swedish
13.9
17.7
12.5
21.3
5,4
ByT5 (S1)
50.8
82,5
66.6
28.5
82.5
55.2
-hyphen filtering
53,8
62.3
58.9
30.3
62.3
Macro Avg.
22.3
23.2
16.5
27.1
9.7
ByT5 (S1+S2) SS
80.9
98.0
89.8
58.2
97.8
78.5
79.3
97.3
88.6
56.8
97.1
77.4
Table 3: Percentage of hard compounds after segmenta- tion with different tokenizers. SPM (mTS) is the Sen- tencePiece tokenizer used by mT5 (Xue al., 2021). SentencePiece (SPM) and CompoundPiece (CPM) tok enizers are trained on text in all 56 languages (Multilin- gual) and for every language separately (Monolingual).
Table 6: Ablation studies on not filtering hyphens-as- new/ine-indicator and on skipping Stage 1 training.
common tokens in other languages is likely the lead cause for the increased number of hard compounds in the multilingual tokenizers. It could potentially be solved by adjusting token probability based on the input language; leave this to future work.
improvement of 23.2% - 16.5%. This may be because tokens from different languages interfere with the segmentation of any given word. We test this hypothesis by computing plausible token ori- gins for tokens in the multilingual tokenizer. This is done by checking which monolingual tokeniz- ers also contain the token in their vocabulary, and ordering the result by unigram token probability. Examples are shown in Table 4. Interference from
To more thoroughly evaluate our tokenization, train multilingual T5 models using Sentence- Piece and CompoundPiece. We the same sam- pling ratio (a 0.2) of mC4 for creating the tokenizer, but instead use subset of 500M texts. We match the architecture and the pretraining setup of the base model, but train for a total

---

-65.5B tokens." We evaluate the model on the de- compounding task. Results are shown in Table 5.
and Sustainable Language Technology for a Truly 2022)). Multilingual World' (no 221137; 2022-). Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC).
Ablation Studies. We quantify the impact of the most significant design choices of our model in Table 6. Although filtering hyphens-as-newiing as-newline- indicator ($4.1) removes only 300k words (<1 %) from the pretraining data, it increases performance on negatives by a large margin. Removing Stage training (i.c., fine-tuning directly on the Wiktionary data instead) consistently decreases performance.
We thank Sebastian Ruder and Srini Narayanan for helpful feedback on draft of this paper.
References
Martine Adda-Decker and Gilles Adda. 2000. Mor- phological decomposition for in german. In Workshop on Phonetics and Phonology in ASR, Saar- brücken, Germany, pages 129-143.
6 Conclusion
We systematically investigated word decompound- ing tasks of compound segmentation and normal- ization on wide scale and in multilingual contexts. To this end, we introduced a dataset of 255k words including compounds and non compouudssds across 56 languages from Wiktionary, which allowed us to evaluate performance of LLMs on decompound- ing. We found that current LLMs' performance is limited due to hard compounds which arise when subword token boundaries do not coincide with compound constituent boundaries. We then intro- duced dedicated models for decompounding which use byte-level tokenization to entirely avoid hard compounds. Finally, we used our decompounding models to create novel CompoundPiece tokenizers, keeping the efficiency advantages of subword tok- enization while strongly decreasing the amount of hard compounds; this increases the performance of CompoundPiece models over comparable Senten- cePiece models on the decompounding tasks.
Enrique Alfonseca, Slaven Bilac, and Stefan Pharies. 2008. Decompounding query keywords from com- pounding languages. In Proceedings of ACL-08: HLT. Short Papers, pages 253-256, Columbus, Ohio. Association for Computational Linguisties.
Duygu Altinok. 2018.
Demorphy. german lan- guage morphological analyzer. arXiv preprint arXiv:1803.0090.
Khuyagbaatar Batsuren, Gábor Bella, Aryaman Arora, Viktor Martinovic, Kyle Gorman, Zdenëk Zabokrt- sky, Amarsanaa Ganbold, Šárka Dohnalová. Magda Ševěíková. Katefina Pelegrinová, Fausto Giunchiglia, Ryan Cotterell, and Ekkaternaa Vylomova. 2022a. The SIGMORPHON 2022 shared task morpheme seg- mentation. In Proceedings of the 19th SIGMOR- PHON Workshop on Compulational Research in Pho- netics, Phonology, and Morphology, pages 103-116, Seattle, Washington. Association for Computational Linguistics.
Khuyagbaatar Batsuren, Gábor Bella, and Fausto Giunchiglia. 2021. MorphyNet: a large multilingual database of derivational and inflectional morphology. In Proceedings of the 18th SIGMORPHON Workshop Computational Research in Phonetics, Phonology, and Morphology, pages 39-48, Online, Association for Computational Linguistics.
Limitations
Although self- supervieed training Stage allows for decompounding without any annotated training data, Stage training is limited to languages with sufficient entries in Wiktionary: this excludes ex- tremely low-resource languages. Furthermore, due to computational constraints have not trained larger models using CompoundPiece tokenization; hence we are unable report on its benefits at larger scales and on tasks besides decompounding.
Khuyagbaatar Batsuren, Omer Goldman, Salam Khal- ifa, Nizar Habash, Witold Kieraš, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yusti- nus Ghanggo Ate, Maria Ryskina, Sabrina Mielke. Elena Budianskaya, Charbel El- Khaissi, Tiago Pi- mentel, Michael Gasser, William Abbott Lane, Mohit Raj, Matt Coler, Jaime Rafael Montoya Samame, Delio Siticonatzi Camaiteri, Esaú Zu- maeta Rojas, Didier López Francis, Arturo Once- vay, Juan López Bautista, Gema Celeste Silva Vil- legas, Lucas Torroba Hennigen, Adam Ek, David Guriel, Peter Dirix, Jcan-Philippe Bernardy, An- drey Scherbakov, Aziyana Bayyr-ool, Antonios Anastasopoulos, Roberto Zariquiey, Karina Sheifer. Sofya Ganieva, Hilaria Cruz, Ritván Karahóğa, Stella Markantonatou, George Pavlidis, Matvey Plu- garyov, Elena Klyachko, Ali Salehi, Candy An- gulo, Jatayu Baxi, Andrew Krizhanovsky, Natalia
A Acknowledgements
Ivan Vulié is supported by a personal Royal So- ciety University Research Fellowship 'Inclusive
I of mT5's IIT tokens, chosen in line "This equates to Ie with our computational resources.

---

Krizhanovskaya, Elizabeth Salesky, Clara Vania, Sar- dana Ivanova, Jennifer White, Rowan Hall Maud- slay, Josef Valvoda, Ran 7migrod, Paula Crarnowska, Irene Nikkarinen, Aelita Salchak, Brijesh Bhatt, Christopher Straughn, Zoey Liu, Jonathan North Washington, Yuval Pinter. Duygu Ataman, Marcin Wolinski, Totok Suhardijanto, Anna Yablonskaya, Niklas Stoehr, Hossep Dolatian, Zahroh Nuriah, Shyam Ratan, Francis M. Tyers, Edoardo M. Ponti, Grant Aiton, Aryaman Arora, Richard J. Hatcher, Ritesh Kumar, Jeremiah Young. Daria Rodionova, Anastasia Yemelina, Taras Andrushko, Igor Marchenko, Polina Mashkovtseva, Alexandra Serova, Emily Prud'houd hommeaux, Maria Nepomni- ashchaya, Fausto Giunchiglia, Eleanor Chodroff, Mans Hulden, Miikka Silfverherg, Arya D. Me- Carthy, David Yarowsky, Ryan Cotterell, Reut Tsar- faty. and Ekaterina Vylomova. 2022b. UniMorph 4.0: Universal Morphology. In Proceedings of the Thirteenth Language Resources and Evaluation Con- ference, pages 855, Marseille, France. European Language Resources Association.
Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Ryan Cotterell, Christo Kirov, John Sylak Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. Mc- Carthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Ja- son Eisner, and Mans Hulden. 2018. The CoNLLL SIGMORPHON 2018 shared task: Universal mor- phological reinflection. In Proceedings of the CONLL-SIGMORPHON 2018 Shared Task: Univer- sal Morphological Reinflection, pages 1-27, Brussels. Association for Computational Linguisties.
Ryan Cotterell, Tim Vieira, and Hinrich Schütze. 2016. A joint model of orthography and morphological seg- mentation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 664 -649, San Diego, California. Association for Computational Linguisties.
Chris Biemann, Uwe Quasthoff, Gerhard Heyer, and Florian Holz. 2008. ASV toolbox: modular collec- tion language exploration tools. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Language Resources Associa- tion (ELRA).
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41888 Minneapolis, Minnesota. Association for Computational Linguistics.
Martin Braschler and Bärbel Ripplinger. 2004, How ef- fective is stemming and decompounding for german text retrieval? Information Retrieval, 7:291-316.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vone Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Miguel Domingo, Mercedes
Alexan- dre Helle, Francisco Casacuberta, and Manuel Her- ranz. 2023. How much does tokenization affect neural machine translation? In Computational Lin- guistics and Intelligent Text Processing: 20th Inter- national Conference, CICLing 2019, La Rochelle, France, April 7-13, 2019, Revised Selected Papers, Part I, pages 545-554. Springer.
Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. 2016. Morphological inflection genera- tion using character sequence to sequence learning. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 634 643, San Diego, California. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling insruucioonfiintume language models. arXiv preprint arXiv:22100 20. 11416.
Omer Goldman, David Guriel, and Reut Tsarfaty. 2022. (un)solving morphological inflection: Lemma over- lap artificially inflates models' performance. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics ( (Volume 2: Short Papers), pages 864-688860 870, Dublin, Ireland. Association for Computational Linguistics.
Jonathan H. Clark, Dan Garrette, Tulia Turc, and John Wicting, 2022. Canine: Pre-training an efficient tokenization-free encoder for language representa- tion. Transactions of the Association for Computa- tional Linguistics, 10:73-91.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco
Harald Hammarström. 2021. Measuring prefixation and suffixation in the languages of the world. In

---

Proceedings of the Third Workshop on Computational Typology and Multilingual NLP, pages $1-89, Online. Association for Computational Linguisties.
Vladimir I Levenshtein et al. 1966. Binary codes capa- ble of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710. Soviet Union
Yijie Han, Bhagirath Narahari, and H-A Choi. 1992. Mapping a chain task to chained processors. Infor- mation Processing Letters, 44(3):141-44.
Constantine Lignos. 2010. Learning from unseen data, In Proceedings of the Morpho Challenge 2010 Work- shop, pages 35-38. Helsinki.
Verena Henrich and Erhard Hinrichs. 2011. Determin- ing immediate constituents of compounds in Ger- maNet. In Proceedings of the International Confer- ence Recent Advances in Natural Language Process- ing 2011, I, pages 420-422 426, Hissar, Bulgaria. Associa- tion for Computational Linguistics.
Krister Lindén and Tommi Pirinen. 2009. Weighted finite-state morphological analysis of Finnish com- pounding with HFST-LEXC. In Proceedings of the 17th Nordic Conference of Computational Linguistics (NODALIDA 2009), pages 89-95, Odense, Denmark. Northern European Association for Language Tech- nology (NEALT).
Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Deriva- tional morphology improves BERT"s s interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 3594-3608 Online, Association for Computational Linguistics.
Klaus Macherey, Andrew Dai, David Talbot, Ashok Popat, and Franz Och. 2011. sanguage-independee compound splitting with morphological operations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1395-1404, Portland, Oregon, USA. Association for Computational Lin- guistics.
Philipp Koehn and Kevin Knight. 2003. Empirical meth- ods for compound splitting. In 10th Conference of the European Chapter of the Association for Compu- tational Linguistics, Budapest, Hungary. Association for Computational L Linguistics.
Fredrik Manne and Tor Sorevik. 1995. Optimal par- titioning of sequences. Journal of Algorithms. 19(2):235-249.
Austin Matthews, Graham Neubig, and Chris Dyer. 2018. Using morphological knowledge in open- vocabulary neural language models. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 1435-1445, New Orleans, Louisiana. Association for Computational Linguisties.
Maria Koliopoulou. 2014. Issues of modern greek and german compounding: a contrastive approach. Jour- nal of Greek Linguistics, 14(1):117-125.
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Milka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019. The SIGMORPHON 2019 shared task: Morphological analysis in context and cross- lingual transfer for inflection. In Proceedings of the 16th Workshop on Computational Research in Pho- netics, Phonology, and Morphology, pages 229-244, Florence, Italy. Association for Computational Lin- guisties.
Taku Kudo and John Richardson. 2018 SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational I. Linguistics.
Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky. Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoit Sagot, et al. 2021. Between words and characters: A brief history of vocabuocabulary modeling and tokenization in nlp. arXiv preprint
Mikko Kurimo, Sami Virpioja, Ville Turunen, and Krista Lagus. 2010. Morpho challenge 2005-2010: Evaluations and results. In Proceedings of the 11th Meeting of the ACL Special Interest Group on Com- putational Morphology and Phonology. pages 87- 95, Uppsala, Sweden. Association for Computational Linguistics.
Benjamin Minixhofer, Jonas Pfeiffer, and Ivan Vulić. 2023. Where's 's the point? self- self-supervised multilin- gual punetuation-agneee
sentence segmentation. In Proceedings of the 61st Anmual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 7215-7235, Toronto, Canada. Association for Computational Linguistics.
Stefan Langer. 1998. Zur morphologie und semantik von nominalkomposita In Tagungsband der 4. Kon- ferenz zur Verarbeitung natürlicher Spruche (KON- VENS), pages 83-97. Bonn.

---

Christof Monz and Maarten De Rijke. 2002. Shallow morphological analysis in monolingual information retrieval for dutch, german, and italian. In Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001 Darmstadi, Germany, September 3-4, 2001 Revised Papers, pages 262-277. Springer.
Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages aages31188 3113-3135, Online. Association for Computational Linguistics.
Jonne Saleva and Constantine Lignos. 2021. The ef- fectiveness of morphology-ave aware segmentation in low-resource neural machine translation. In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguisties: Student Research Workshop, pages 164-174, Online. Association for Computational Linguistics.
Yirong Pan. Xiao Li, Yating Yang. and Rui Dong. 2020. Morphological word segmentation on agglutinative languages for neural machine translation. arXiv prepriat Xixx:000100055:
Ben Peters and Andre F. T. Martins. 2022. Beyond char- acters: Subword-lewe morpheme segmentation. In Proceedings of the 19th SIGMORPHION Workshop on Computational Research Phonetics, Phonology, and Morphology, pages 131-138, Seattle, Washing- ton. Association for Computational Linguisties.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany, Association for Computational Lin- guistics.
Alexander Pollatsek, Jukka Hyönä, and Raymond Bertram. 2000. The role of morphological con- stituents in reading finnish compound words. Journal of Experimental Psychology: Human perception and performance, 26(2):820.
Naomi Tachikawa Shapiro. 2016. Splitting compounds with ngrams. In Proceedings of COLING 2016, the 26th International Conference on Computational Lin- guisties: Technical Papers, pages 630 640, Osaka, Japan. The COLING 2016 Organizing Committee.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140))1-67.
Robyn Speer. 2022. rspeer/wordfreq: v3.0.
Ilya Sutskever, Oriol Vinyals, and Quee V Le. 2014. Sc- quence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
Martin Riedl and Chris Biemann. 2016. Unsupervised compound splitting with distributional semantics ri- vals supervised methods. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 617-622, San Diego, California. Association for Computational Linguis- tics.
Yi Tay, Mostafa Dehghani. Vinh Q Tran, Xavier Gar- cia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng. Neil Houlsby, and Donald Metzler. 2022a. Unify- ing language learning paradigms. arXiv preprint arXiv:2205.0513:
Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022b. Charformer: Fast character transformers via gradient- based subword tokenization. In International Con- ference on Learning Representations.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang. Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Mare van Zee, Jacob Austin, Se- bastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdh- ery, Jasmijn Bastings, Jannis Bulian, Xavier Gar- cia. Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel. Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin- Maitin-Shepard, Noah Fiedel, Mark Omernick, Bren- nan Sacta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022. Scal- ing up models and data with t5x and seqio. arXiv preprint arXiv:2203.1 17189.
Don Tuggener. 2016. Incremental coreference resolu- lion for German. Ph.D. thesis, University of Zurich.
Menno van Zaanen, Gerhard van Huyssteen, Suzanne Aussems, Chris Emmery, and Roald Eiselen. 2014. The development of Dutch and Afrikaans language resources for compound boundary amalysis. In Pro- ceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14). pages 1056-1062, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Päivi Johanna Virkkunen, Juraj Simko, Heini Henriikka Kallio, and Martti Tapani Vainio. 2018 Prosodic features of finnish compound words. In Proceed- ings of the 9th International Conference on Speech Prosody 2018 International Speech Communications Association.
Phillip Rust, Jonas Pfeiffer, Ivan Vulié, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? the monolingual performance of multilin- gual language models. Proceedings of the 59th Annual Meeting of the Association for Computational

---

Sami Virpioja, Ville T. Turunen, Sebastian Spiegler. Os- kar Kohonen, and Mikko Kurimo. 2011. Empirical comparison of evaluation methods for unsupervised learning of morphology. Traitement Automatique des Langues, 52(2):45-90.
A Dataset Statistics
Statistics for the training and validation splits of the Wiktionary dataset are shown in Table 7.
B Efficient Segmentation Algorithm
Irene Vogel and Sergio Scalise. 2010. Cross- disciplinary issues in compounding. Cross- Disciplinary Issues Compounding, pages 1-390.
Pseudocode of the brute-force algorithm to turn normalization into segmentation is shown Al- gorithm 1. Since enumerating all possible seg- mentations is only feasible for short words ($3.3) we introduce a more efficient algorithm (Algo- rithm 2) where candidate segmentations are or- dered such that segmentations with constituents closest in length the corresponding normalized constituents uppear first. Assuming insertions and deletions both have a cost of one (as the case in standard Levenshtein distance), constituents are thus sorted in increasing order of a lower bound on edit distance. The procedure can stop once the lower bound on edit distance reaches the cost of the best solution found far since by that point it is impossible for better solution to be found. tiairtttoo----segmmettttttoo
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharam Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5; Towards token-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Computational Linguis- rics, 10:291-306.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou. Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Giulio Zhou. 2018 Morpholagical zerv-shot neural machine translation. Ph.D. thesis, Master's thesis, University of Edinburgh.
Note that the problem related to sequence partitioning (Manne and Sorevik, 1995; Han et al., 1992) where the aim is to find a partition of sequence such that the maximum cost across partitions of some cost function is minimized. However, since our goal is to find the partitioning with the minimum aggre- gated cost, algorithms for conventional sequence partitioning are not applicable.
Patrick Ziering and Lonneke der Plas. 2016. To- wards unsupervised and language-inddependen com- pound splitting using inflectional morphological transformations. In Proceedings of the 2016 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Techmologies, pages 644-653, San Diego, Cal- ifornia, Association for Computational Linguisties.
C Results for All Languages
Segmentation accuracy for all languages is shown in Tables 8-11.
D LLM Prompts
The prompt used for LLM evaluations ($5) is shown in Figure 8. The prompt was chosen among 10 prompts to maximize performance T5 Large. For 2- to 16-shot results, provide 50% positive (compound) and 50% negative (non- compound) examples in a random order.
Flan
E Quantifying Negative Collection Bias
We conduct an experiment to measure the extent of the bias against words which do not occur in- side compounds in our data collection methodol- ogy ($3.1). In particular, we quantify the bias against long non-compound words, which usually would not occur inside compounds. We took a

---

Zero-shot:
Data: Compound a; norm. constituents Result: Optimal segmentation s*.
{word)
k ||c|| t |
Hyphenate the above word.
rg 0, I' <-
Ans
best _est__cotttte f- 00
n-shot:
for Ti. nn- E (t do
[n]
(example_00
Compute C(s)
/* see $3.3 */
if C(s) 8 hesst t bost_cost < C(s) end end
< bost bost_cost _ost then
Hyphenate the above word. Ans (example 0_hppennatedd
{ @example_x m
8 best
A Algorithm 1: Naïve brute-force segmentation.
Hyphenate the above word.
Ans (example (example___hypheeeleelleeeeeeeee byphenatte)))
Data: Compound æ, norm. constituents c.
{word)
Result: Optimal segmentation s*
k |c]. n < acc
To + 0, rk f n
Hyphenate the above word.
Ans:
best__ost 4- 0c
Figure Prompts used evaluate LLM in-context learning compound segmentation performance.
/* A is the total di difference in length of the normalized constituents to the word.
*/
random sample of 500 words each from word fre- queney lists in English and German (Speer, 2022), manually removed compound words, and com-
A n- E, |c|
lower
lower_bound bound
A
pared the length statistics of this (unbiased) sample of non-compounds to our non-compound dataset.
while lower_ < best_cost do k,
offsets x
While words in our non-compound dataset are indeed shorter average (6.0 vs. 6.7 chars for English, 6.7 vs. 7.1 chars for German), with less than one character length difference on average, there only a weak length bias in data collection.
E
x lower bownr_
x
}}
lower bound + lower lower_bound + for 01...., E offsets do
Tl...., Tk-1
lcil + --,, Σ0 [.|
We also found qualitatively that our non- compound dataset contains a wide variety of words since compounding is typically process that can occur for many different root words.
Compute 8, C(8) /* see 53.3 */
if C(s) < bost_cost then
best 8
best cost C(s)
end
end end
best
Algorithm 2: Segmentation by enumerating candidates in order of increased lower bound on edit distance.

---

Training #Negative
Validation
Language iso
#Positive
Total
#Positive
#Negative
Total
Afrikaans
af
326
193
519
322
197
519
Azerbaijani
78
97
175
85
89
174
Belarusian
be
32
47
79
40
38
78
Bulgarian
bg
71
89
160
68
92
160
Bengali
bn
301
334
635
304
331
635
Catalan
220
218
438
218
437
Czech
388
358
746
392
354
746
Welsh
cy
308
273
581
299
580
Danish
da
2145
1298
3443
German
de
20743
7846
28589
708
1000
Greek
292
508
208
507
English
en 22896
6480
29376
Esperanto
20 1097
849
559
Spunish
cs
433
834
417
417
834
Estonian
349
315
664
288
664
Basque
cu
200
98
199
Persian
268
314
582
300
582
Finnish
69948
13314
83262
848
152
1000
French
149
135
284
148
283
Western Frisian
fy
92
85
177
90
86
176
Irish
ga
332
322
654
328
325
653
Galician
70
79
149
80
69
149
Gujarati
gu
227
279
506
221
285
506
Hehrew
he
29
34
63
18
44
62
Hindi
hi
472
569
1041
478
522
1000
Hungarian
hu
5238
3162
8400
644
356
1000
Armenian
hy
872
745
1617
509
491
1000
Indonesian
id
26
45
71
32
38
70
Icelandic
2333
1603
3936
592
408
1000
Italian
452
352
804
437
366
803
Georgian
kn
137
156
293
149
143
292
Kazakh
kk
244
292
536
278
258
536
Kirghiz
ky
39
45
84
39
44
83
Latin
la
450
410
860
452
407
859
Lithuanian
It
65
94
1.99
76
83
159
Latvian
Iv
244
249
493
223
269
492
Malagasy
mg
35
42
77
32
45
77
Macedonian
mk
75
94
169
79
90
169
Malayalam
ml
318
435
753
331
421
752
Maltese
35
36
36
35
Dutch
15184
5258
20442
761
239
1000
Panjabi
pu
24
34
58
19
39
58
Polish
556
1184
523
477
1000
Portuguese
pt
40
53
44
Romanian
533
268
S33
Russian
718
1471
1000
Slovak
26
25
Albanian
sq
237
127
236
Swedish
SV
4172
13055
Tamil
710
516
Telugu
909
507
493
1000
Thai
4287
2754
7041
386
1000
Turkish
295
582
Ukrainian
uk
281
572
277
295
572
Yiddish
yi
162
218
380
176
203
379
Yoruba
yo
349
312
661
348
312
660
Total
164713
58757
223470
17539
13938
31477
Table 7: Statistics of the Wiktionary datasct.

---

Macro
be
bg
bn
cy
da
de
el
Arg.
SECOS
7.4
4.1
20.2
30.0
66.5
5.3
41.2
29.0
47.5
64.7
20.0
14.7
0.0
61.2
30.6
41.1
55.3
56.1
0.0
85.9
65.3
69.8
43.7
FLAN TS
52.8
69.4
17.5
16.2
0.0
59.8
36.0
43.8
58.4
58.5
0.0
89.1
67.6
71.0
45.7
SI
MTS
22.7
34.1
20.0
10.3
14.5
50.7
28.8
36.5
25.8
38.8
21.6
79.7
44.0
58.3
34.7
N
ByTS
64.9
70.6
29.4
34.5
68.5
48.2
49.8
75.6
91.3
82.3
77.2
61.0
TS
83.9
75.3
22.5
35.3
0.0
70.8
68.9
60.2
86.3
96.0
0.0
95.4
78.7
82.5
61.1
FLAN T5
84.8
74.1
75.3
67.3
59.5
86.6
95.5
77.6
83.2
S1+S2
mT5
83.5
85,9
70.0
76.5
79.6
71.7
76.8
57.9
87.1
94.1
73.1
95.4
78.2
82.3
79.4
BYT5
90.4
89.4
80.0
79.4
91.1
81.7
85.5
72.9
92.2
96.6
86.1
97.8
89.8
87.1
87.1
SECOS
100
99.1
100
96.1
86.6
99.7
93.8
97.4
87.8
84.3
100
80.5
91.8
100
91.7
83.9
88.7
85.5
FLAN T5
90.4
85.4
71.1
73.9
100
92.2
79.9
74.4
88.5
92.1
100
91.3
86.2
89.9
86.8
S1
mT5
95.4
91.0
81.6
93.5
95.8
93.6
96.3
82.9
92.7
92.8
95.3
90.9
90.2
92.3
91.7
P
ByT5
87.8
86.5
93.1
84.9
87.9
70.5
89.0
95.7
88.4
76.6
81.5
T5
91.9
97.8
94.7
96.7
100
97.2
94.6
93.6
93.3
94.5
100
98.3
97.7
97.8
96.3
FLAN T5
90.9
96.6
94.7
95.7
100
95.4
93.5
96.1
94.1
100
97.9
97.3
95.9
S1+$2
mT5
92.4
100
100)
100
97.0
95.9
97.7
95.7
93.8
96.2
96.3
99.2
98.0
97.4
97.1
BYTS
93.4
98.9
100)
100
97.3
97.2
98.0
95.7
95.2
96.2
97.3
98.3
98.0
98.8
97.5
SECOS
60.6
51.5
58.0
53.5
72.4
60.9
53.9
63.2
62.8
74.7
41.0
52.1
74.6
54.3
57.2
59.0
87.3
73.5
79.3
64.2
FLAN T5
67.1
77.6
43.6
49.4
52.1
76.0
56.8
58.6
69.1
68.3
59.0
89.6
75.8
80.5
66.0
SI
mTS
50.3
63.2
50.0
58.1
56.9
72.1
60.9
59.0
49.6
54.6
65.1
82.4
64.4
75.3
61.6
All
BYT5
73.6
78.7
51.3
65.0
76.7
67.0
59.8
80.4
80.0
73.2
90.6
79.8
79.4
72.1
TS
86.9
86.8
57.7
70.6
52.1
84.0
81.1
76.4
88.8
95.6
59.0
96.1
87.1
90.2
79.5
FLAN T5
87.1
85.6
57.7
69.4
52.1
85.4
79.8
77.2
89.3
95.4
59.0
96.1
86.3
89.6
79.3
S1+S2
m'T5
86.9
93.1
84.6
90.0
88.7
83.8
86.7
76.2
89.5
94.7
86.8
96.3
86,9
89.8
88.1
BYTS
91.5
94.3
89.7
91.2
94.3
89.5
91.4
84.0
93.3
96.5
92.7
97.9
93.4
92.9
92.3
Table 8: Accuracy languages af-es.
Macro
ct
cu
fa
fi
fr
fy
gl
gu
he
hi
hu
hy
Avg.
SECOS
23.4
4.1
1.4
53.1
11.9
2.5
38.8
T5
29.0
28.6
0.0
31.6
31.9
53.3
69.8
50.0
0.0
0.0
0.0
48.6
0.0
34.4
26.9
FLAN T5
37.0
31.6
0.0
33.0
31.9
58.9
70.1
51.2
0.0
0.0
0.0
53.4
40.6
29.1
SI
N
mTS
18.6
18.4
3.9
21.5
24.4
59.8
38.8
59.3
22.2
39.7
18.8
4.3
12.5
26.2
BYTS
51.6
42.9
20.9
52.7
44.4
52.2
76.8
$2.5
79.6
38.9
66.5
70.0
10.2
50.0
50.7
TS
77.7
38.8
0.0
98.2
48.1
84.4
83.2
65.0
0.0
0.0
0.0
89.1
0.0
46.9
45.1
FLAN TS
80.9
41.8
0.0
98.3
49.6
86.7
81.4
60.0
0.0
0.0
0.0
87.3
0.0
46.9
45.2
S1+$2
mTS
83.2
50.0
62.1
97.1
48.9
81.1
82.6
62.5
85.1
44.4
81.8
90.4
77.2
40.6
70.5
BYT5
92.6
58.2
76.6
98.8
62.2
91.1
88.7
67.5
90.0
33.3
88.9
97.2
85.1
53.1
77.4
SECOS
98.6
100
100
88.2
97.3
95.7
95.5
TS
82.3
85,1
100
82.2
94.6
87.2
82,2
82.6
100
100
100
93.8
100
81.6
90.8
FLAN T5
82.3
87.1
100
82.9
98.0
87.2
76.0
94.2
100
100
100
91.6
100
76.3
91.1
SI
P
89,9
95.3
90.7
88.3
95.7
99.4
98.0
97.6
78.9
ByT5
76.0
71.3
97.3
77.6
91.9
88.4
77.8
71.0
94.7
95.5
98.3
87.1
92.5
57.9
84.1
95.1
98.0
100
95.4
97.3
100
98.8
95.7
100
100
100
99.2
100
100
98.5
FLAN T5
95.8
96.0
100
96.7
97.3
100
97.5
95.7
100
100
100
98.6
100
100
98.4
S1+S2
m'T5
97.9
97.0
98.7
94.1
98.6
97.7
98.2
100
97.2
97.7
99.0
98.6
97.4
100
98.0
ByTS
97.9
97.0
98.6
100
98.8
100
97.9
100
99.2
99.7
98.2
SECOS
56.0
52.8
52.2
58.4
56.5
45.6
$9.0
T5
52.1
57.3
51.5
39.3
64.7
69.9
76.0
65.1
56.3
71.0
52.2
64.7
49.1
60.0
59.2
FLAN T5
56.6
59.8
51.5
40.6
66.4
72.7
73.0
71.1
56.3
71.0
52.2
67.0
49.1
60.0
60.5
S1
AlI
49.5
54.3
60.1
56.8
74.0
81.0
70.9
47.0
50.1
48.6
BYT5
62.2
57.3
60.3
56.5
69.3
69.9
77.3
61.1
88.1
79.0
83.1
76.1
50.6
54.3
67.5
85.2
68.8
91.0
79.2
56.3
92.7
75.7
FLAN T5
87.3
69.3
51.5
98.1
74.6
93.2
89.4
76.5
56.3
71.0
52.2
91.3
49.1
75.7
74.0
S14$2
mTS
89.6
73.9
80.9
96.6
74.9
89.2
90.4
79.9
91.9
82.3
90.8
93.3
87.1
72.9
85.3
ByTS
94,9
77.9
81.3
95.5
93.7
82.6
94.5
94.3
98.1
91.5
78.6
89,2
Table Accuracy on languages et-id.

---

Macro
ka
kk
ky
la
mg
mk
ml
pu
Arg.
SECOS
32.5
5.0
5.3
13.9
46.8
41.4
42.6
0.0
16.9
29.0
21.1
29.6
31.2
15.2
0.0
19.4
44.9
0.0
21.9
FLAN TS
45,4
48.5
0.0
17.6
12.8
33.0
28.9
41.7
34.4
17.7
0.0
25.0
44,8
0.0
25.0
SI
MTS
26.4
26.8
21.5
45.0
30.8
21.9
11.8
20.2
25.0
24.1
18.4
25.0
23.0
42.1
25.8
N
ByTS
65.9
75.9
64.1
33.6
25.0
41.7
37.5
33.5
27.8
57.2
68.4
TS
78.7
68.6
0.0
18.3
23.1
59.3
65.8
69.1
53.1
49.4
0.0
41.7
94.0
0.0
44.4
FLAN
78.7
68.9
0.0
61.7
63.2
68.2
40.6
41.7
93.6
42.9
S1+S2
mT5
82.9
68.9
82.6
86.7
79.5
61.9
60.5
76.7
53.1
72.2
68.0
52.8
93.4
63.2
71.6
BYT5
90.5
81.2
83,9
91.7
84.6
73.5
80.3
84.8
65.6
88.6
83.7
S8.3
97.5
78.9
81.7
SECOS
97.0
100
99.8
100
94.1
83.1
88.8
77.3
80.3
81.9
87.4
100
91.4
83.7
84.6
FLAN T5
80.1
91.3
99.3
72.9
86.4
82.8
86.7
87.0
75.6
66.7
100
85.7
87.0
100
85.8
S1
P
mT5
90.0
92.1
97.2
88.0
95.5
84.5
85.5
95.9
84.4
90.0
96.9
97.1
89.1
100
91.9
ByT5
82.1
83.6
72.1
86.4
56.0
71.1
87.7
57.8
72.7
94.3
80.3
T5
96.1
96.4
100
91.1
97.7
94.8
92.8
97.4
88.9
93.3
100
94.3
97.5
100
95.7
FLAN T5
95.6
96.2
97.7
95.6
95.2
96.7
91.1
100
97.1
97.5
S1+$2
mTS
95.6
97.3
98.6
96.9
100
96.1
96.4
98.1
82.2
96.7
98.3
88.6
96.7
100
95.8
BYTS
97.1
97.8
99.3
99.2
100
98.5
94.0
98.9
91.1
98.9
98.8
100
97.9
100
98.0
SECOS
61.9
50.7
50.1
61.0
58.1
58.4
63.6
44.4
48.2
53.3
52.8
61.2
53.2
43.8
54.9
54.2
67.2
FLAN T5
59.6
68.0
48.6
44.2
51.8
56.6
59.1
66.5
58.4
43.8
56.0
54.9
54.9
67.2
56.4
SI
mTS
52.3
56.5
58.6
65.7
65.1
51.6
50.3
61.6
59.7
59.2
62.4
60.6
38.8
81.0
58.8
All
BYT5
72.5
68.6
75.0
74.1
75.9
44.2
49.1
66.9
49.4
62.7
55.5
60.6
62.7
89.7
64.8
TS
85.8
81.3
49.0
53.4
62.7
76.1
79.9
84.6
74.0
72.8
56.0
67.6
94.8
67.2
71.8
FLAN T5
85.6
81.3
48.6
60.2
77.8
79.9
83.7
70.1
72.8
56.0
69.0
94.5
67.2
71.4
S1+S2
m'T5
88.1
81.8
90.4
91.6
90.4
78.1
79.2
88.4
70.1
85.2
85.0
70.4
94.2
87.9
84.4
BYTS
93.2
88.8
91.4
95.3
92.8
85.3
87.4
92.5
80.5
94.1
92.2
78.9
97.6
93.1
90.2
Table 10: Accuracy on languages is-pa.
Macro
TO
TU
sk
SV
ta
tc
th
tr
uk
Avg.
SECOS
22.2
9.4
7.8
35.9
32.2
7.7
T5
36.1
30.2
51.9
22.3
12.0
29.4
53.1
0.0
0.0
0.0
28.1
17.7
0.0
12.9
21.0
FLAN T5
40.3
47.2
55.6
25.6
16.0
31.2
56.5
0.0
0.0
0.0
34.8
22.0
16.1
24.7
SI
N
mTS
32.9
20.8
47.8
15.8
16.0
22.9
21.9
19.6
40.2
9.3
15.2
13.0
36.4
10.9
23.0
BYTS
51.8
45.3
61.2
31.0
36.0
34.9
64.8
46.1
61.7
27.0
32.6
35.7
50.6
18.7
42.7
TS
78.0
49.1
63.8
50.3
48.0
52.3
89.6
0.0
0.0
0.0
67.4
37.5
0.0
19.0
39.6
FLAN TS
77.4
49.1
64.9
50.1
60.0
45.9
89.4
0.0
0.0
0.0
66.
36.1
0.0
19.0
39.9
S1+$2
mTS
84.1
39.6
65.7
77.9
60.0
45.0
90.0
61.0
81.9
83.7
71.6
76.9
76.1
31.3
67.5
BYT5
91.2
56.6
73.5
91.3
72.0
56.0
94.3
73.6
84.4
90.6
82.3
86.6
83.0
48.0
77.4
SECOS
96.9
97.7
95.8
92.1
97.3
100
TS
90.6
86.4
90.9
66.1
89.7
80.3
91.8
100
100
99.5
87.5
68.5
100
86.2
88.4
FLAN T5
90.4
88.6
91.3
67.1
86.2
81.1
92.4
100
100
99.7
89.3
72.2
100
85.3
88.8
SI
P
94.5
81.8
96.1
94.8
94.4
87.1
99.5
ByT5
91.4
79.5
86.8
67.3
86.2
83.5
87.8
63.2
95.7
89.6
81.2
72.2
92.1
85.6
83.0
98.1
97.7
95.8
95.9
96.6
93.7
96.7
99.8
100
100
95.6
97.3
100
93.9
97.2
FLAN T5
97.1
97.7
97.7
95.9
96.6
95.3
96.7
100
100
100
96.3
98.6
100
97.1
97.8
S1+S2
m'15
97.9
97.7
95.8
98.6
100
99.2
97.3
94.6
97.8
97.2
98.5
99.7
97.0
97.8
97.8
ByTS
99.0
97.7
97.6
97.6
96.9
98.2
99.3
98.0
98.1
98.0
SECOS
57.8
49.5
51.6
63.6
53.6
50.8
T5
62.1
55.7
71.3
43.9
53.7
56.8
65.8
51.6
49.3
38.4
55.8
43.9
53.6
47.6
53.5
FLAN T5
64.2
66.0
73.4
46.1
53.7
58.1
68.3
51.6
49.3
38.5
60.2
47.9
53.6
48.8
55.7
S1
AlI
62.3
48.5
71.1
59.3
62.3
45,9
58.2
68.9
51.2
70.2
49.7
BYT5
70.7
60.8
73.9
48.9
63.0
61.0
72.4
54.9
78.5
51.2
55.2
54.5
72.8
50.3
62.0
87.6
74.1
74.6
91.9
51.5
68.4
53.6
54.4
67.7
FLAN T5
86.8
71.1
81.2
72.7
79.6
72.5
91.8
51.6
49.3
38.6
80.2
68.4
53.6
55.9
68.1
S14$2
mTS
90.7
66.0
80.7
88.1
81.5
74.2
92.4
78.3
89.7
88.9
84.2
88.6
87.3
62.7
82.4
BYTS
94,9
75.3
78.4
95.4
85.6
93.2
91.0
71.7
87.5
Table 11: Accuracy on languages pl-yo.


Reviews:
[{"review_id": "bHQec8hWJO", "reviewer": "Reviewer_X8iv", "summary": "", "questions": "In line 326, you mentioned \"computed ground-truth segmentation\". This looks contradictory: a computed piece of data cannot be \"ground-truth\" unless it is derived deterministically from some other source of ground truth. Does this mean \"computed segmentation that corresponds best to ground-truth normalisation from Wiktionary\"?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback!\n\nAlthough we did not run extrinsic evaluation on the usual NLU tasks, we do provide extrinsic results on Compound Segmentation and Normalisation in Table 5 (which are downstream tasks used e.g. in online dictionaries).\n\nThe problem with faithfully running a more thorough extrinsic evaluation is that due to our limited computational budget we have not been able to train a new T5 model from scratch which can compete with the prior T5 models (c.f. Footnote 10). We did now run additional extrinsic experiments on WikiANN NER* obtaining 17.2% F1 score with the SentencePiece model and 24.6% F1 score with our CompoundPiece model. This is promising, but it is not clear how this improvement translates to more fully trained models. \n\nWe hope to follow up with work on efficiently transferring models trained with SentencePiece (or another tokenizer) to CompoundPiece. This would allow more meaningful extrinsic evaluation on a limited budget, but would have been out of scope for this work.\n\nTo answer your question:\n\n> In line 326, you mentioned \"computed ground-truth segmentation\". This looks contradictory: a computed piece of data cannot be \"ground-truth\" unless it is derived deterministically from some other source of ground truth. Does this mean \"computed segmentation that corresponds best to ground-truth normalisation from Wiktionary\"?\n\nThis is correct, we will change the wording to reflect this fact.\n\n*Training a multilingual model jointly on the train set of all languages, and macro-averaging F1 score of the test set of each language.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses a well-defined and important task, proposes a resonable set of methods for solving it, and achieves meaningful results, most noticeably beating ad hoc language-specific models. Additionally, a new dataset for training decompounding models is proposed.", "reasons_to_reject": "The main drawback of the paper is the complete lack of extrinsic evaluation. Better performance on compound segmentation should lead to better downstream performance, and without checks of this kind it is not evident if we should bother with retraining the models using CompoundPiece (as opposed to any of the other approaches to improving multilingual tokenisation that have been proposed in recent years). The rather pedestrian performance of the CompoundPiece on reducing the number of hard compounds in the multilingual setting makes one wonder if the proposed method generalises well: in some languages and orthographies hyphens are normally not used at all, which is an evident impediment. Generally, while the basic compound-splitting model is presented thoroughly and looks like a \"finished product\", its usefulness is rather narrow, while the merits of CompoundPiece remain harder to ascertain.", "questions_to_authors": "In line 326, you mentioned \"computed ground-truth segmentation\". This looks contradictory: a computed piece of data cannot be \"ground-truth\" unless it is derived deterministically from some other source of ground truth. Does this mean \"computed segmentation that corresponds best to ground-truth normalisation from Wiktionary\"?", "paper_topic_and_main_contributions": "This paper proposes a two-stage approach for splitting compounds into constituent parts. The first stage consists of training a character/byte based model (ByT5 is used in the paper) in a self-supervised way over words written with hyphens as positive examples and words without hyphens as negative (non-compound) examples. Hyphens without a following new-line symbol are used as a \"high precision, low recall\" marker of compound-constituent boundary. The fact that many compounds are written both with and without hyphens (cf. fine-tuning vs. finetuning) helps the model to generalise. The second stage consists of further training this model in a supervised way on the new dataset based on Wiktionary. \n\nIn order to compare character-based and subword-based models, the authors use their model to make an additional pass over the pre-training dataset for subword-based langauges models, such as mT5, at the tokenisation stage. The standard SentencePiece tokeniser is applied to texts where compounds have been separated and as a result subword tokens, ideally, do not span the compound-constituent boundary. The resulting tokeniser is called CompoundPiece.\n\nThe authors show that their ByT5-based model achieves better performance on the task of compound segmentation and compound normalisation (segmentation + lemmatisation) than language-specific models from the Sigmorphon Shared Task 2022. From among the models trained by the authors, ByT5 has the best results on average, outperforming subword-based models.\n\nA particular problem addressed by the authors is the issue of hard compounds: the compounds where the constituent boundaries do not coincide with the boundaries of subwords used by the respective tokeniser (e.g., when the word \"windsurfing\" is tokenised as \"_winds_-ur-f-ing\"). One of the aims of CompoundPiece is to reduce the number of hard compounds in training datasets. The results on this task are decent in the monolingual setting (the percentage of hard compound falls from 27.1% to 9.7%) but not great in the multilingual setting (23.2% vs. 16.5%).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8fNhJDpMFH", "reviewer": "Reviewer_WAo7", "summary": "", "questions": "-- Did you spot any interesting cross-linguistic trends in your results? e.g., by language families?\n\n-- Are the developed algorithmic methods relevant for other morphological tasks or other segmentation tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your encouraging feedback! Regarding your questions:\n\n> Did you spot any interesting cross-linguistic trends in your results? e.g., by language families?\n\nWe did not thoroughly check cross-linguistic trends or cross-lingual transfer, this would be an interesting area for follow-up work.\n\n> Are the developed algorithmic methods relevant for other morphological tasks or other segmentation tasks?\n\nThe normalisation-to-segmentation algorithms are relevant to any problem of finding subsequences with minimal total edit distance to a second set of sequences of equal length. This could for example also be useful to compute a surface-level segmentation from a canonical segmentation [1]. We will clarify this in the paper.\n\n> It would be helpful to include a table in the appendix with all the nitty-gritty details of implementation/optimization (e.g., hyperparameters, packages).\n\nWe will make sure to include such a table in the final version.\n\n[1]: https://aclanthology.org/N16-1080/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- The paper addresses an important task, that is or should be part of the basic analysis pipeline of many languages.\n-- The paper extends the scope of the task to a diverse set of languages, which is especially commendable given the limited scope of languages explored in previous work. The resulting dataset is large: 255K compound words and I am sure it will be useful in future work.\n-- The paper proposes a nice algorithmic method for self-supervised learning of segmentation, using a nice trick based on the use of hyphens.\n-- Sound experimental setup, and good results over an extensive set of settings.\n-- The paper is well structured and well presented.", "reasons_to_reject": "I don't see any major risks in publishing it. One possible minor risk is that the method presented in the work is that I do not think it will likely to make a wide impact outside the scope of the task it addresses. The paper could have potentially made a contribution to the linguistic literature but does not include much discussion of the linguistic implications of its results.\nThe same goes for technical contribution: while the method appears sound and effective, little discussion is made as to whether the effectiveness of the method is likely to make an impact for other tasks or sub-fields.", "questions_to_authors": "-- Did you spot any interesting cross-linguistic trends in your results? e.g., by language families?\n\n-- Are the developed algorithmic methods relevant for other morphological tasks or other segmentation tasks?", "paper_topic_and_main_contributions": "Compounding is pervasive in many of the world's languages, but most computational work on analyzing compounds has focused on a handful of languages. This paper studies the task of decompounding, namely analyzing compounds to their constituent parts. The work makes several contributions in this respect: (1) it compiles a benchmark of compounds in 56 diverse languages, by mining them from Wikitionary; (2) it defines two variants of the task of decompounding - one defined as segmentation and the other as extracting the stems; (3) it sets a benchmark in terms of the performance of existing models on this task, and (4) it proposes a new self-supervised method for compound segmentation.\n ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "-- It would be helpful to include a table in the appendix with all the nitty-gritty details of implementation/optimization (e.g., hyperparameters, packages)."}, "extra_text": {}}, {"review_id": "vW4pdD2MJL", "reviewer": "Reviewer_8bjC", "summary": "", "questions": "In footnote 5 you highlight the problem of obtaining suitable non-compounds.  This may indeed be a problem as the non-compounds you select (if I understand correctly) also occur as part in the compounds.\n\nI don't understand your point 1 on line 261. In many languages, hyphens are never used to indicate a compound boundary.\n\nOn line 386 you mention you use n most common words.  Typically the most common words are shorter than the compounds.  How does this influence the results?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for very much your feedback and for bringing up the valid concern about non-compound data collection being strongly biased towards short words.\n\nWe ran an additional experiment to quantify if such a bias exists (and to what extent): We took a random sample of 500 words each from word frequency lists in English and German [1], manually removed compound words, and compared the length statistics of this (unbiased) sample of non-compounds to our (biased) non-compound dataset.\n\nWhile words in our non-compound dataset are indeed shorter (avg. 6.0 vs 6.7 characters for English, avg. 6.7 vs 7.1 characters for German), we would argue that with <1 character length difference on average, there is only a weak length bias in data collection. We will add this result to the paper to mitigate the concern.\n\nWe also found qualitatively that our non-compound dataset contains a wide variety of words since compounding is typically a process that can occur for many different root words.\n\nTo address your remaining questions:\n\n> I don't understand your point 1 on line 261. In many languages, hyphens are never used to indicate a compound boundary.\n\nWe found that hyphens are used to indicate compound boundaries in all 56 languages which we investigated. This may not be formally correct in all languages, but it is present in every one of them in text from the internet.\n\n> On line 386 you mention you use n most common words. Typically the most common words are shorter than the compounds. How does this influence the results?\n\nThis is a practical choice purely for Stage 1 to reduce noise. Since n is very large (avg. >400k per language) this does not particularly favour short words. We will clarify this in the paper.\n\n[1]: via https://github.com/rspeer/wordfreq", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Compounding is an active process in many languages. This introduces difficulties for instance related to dictionary development, spelling checking, etc.\nThe data collection contains interesting data that can also serve the basis for more linguistically oriented research.", "reasons_to_reject": "The process of creating the data collection has limitations. In particular, the process of finding non-compounds has a strong bias (if I understand correctly a bias towards short words).  Furthermore, assumptions are made related to the use of hyphens (see below).", "questions_to_authors": "In footnote 5 you highlight the problem of obtaining suitable non-compounds.  This may indeed be a problem as the non-compounds you select (if I understand correctly) also occur as part in the compounds.\n\nI don't understand your point 1 on line 261. In many languages, hyphens are never used to indicate a compound boundary.\n\nOn line 386 you mention you use n most common words.  Typically the most common words are shorter than the compounds.  How does this influence the results?\n", "paper_topic_and_main_contributions": "This paper creates a multi-lingual data collection that contains information on compounds and their boundaries. Using the data collection, a number of systems are developed that aim to automatically identify the compound boundaries.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary (adapted from Reviewer 8bjC): This paper creates a multi-lingual data collection that contains information on compounds and their boundaries. Using the data collection, a number of systems are developed that aim to automatically identify the compound boundaries. The dataset and thus the evaluation are performed on a much wider set of languages than previous work, which tended to focus on compound-heavy languages. Wiktionary and the vocabulary of a large-scale web corpus are used as data sources.

The soundness (3/3/4) and excitement (3/4/4) scores suggest the paper is overall sound and there is excitement for its content. The discussion period identified minor clarifications that the authors should make in revisions, and that there are some desirable follow-up experiments to be run that were not feasible due to computational resource limitations. Reviewer X8iv stated that there are efficient ways to run these resources and that resource limitations should not be a barrier; the authors may want to address this in a discussion or limitations section.

As reviewer X8iv points out, there is no extrinsic (i.e. a downstream, non-decompounding task) evaluated. As reviewer WAo7 points out, it’s not clear that this approach will have any impact outside of the decompounding task.

However, this paper provides both a much larger dataset than was available through previous work, and substantial experiments, including showing that their approach even outperforms hand-tuned per-language models (Table 2). The reviews suggest that this is a substantial contribution, even if perhaps in a narrow area that does not yet have demonstrated downstream impact.

A historical note for the authors: you have correctly identified work such as Koehn and Knight (2003) in the early 2000s as the first widely known statistical unsupervised decompounding approaches. What is less widely known is that there was a series of unsupervised (and later adding semi-supervised) morphological decomposition shared tasks known as Morpho Challenge, running from 2005-2010 [1]. While they did not separately evaluate decompounding, it was key to performance, and [2] explicitly integrates Koehn and Knight’s objective into a morphological analyzer lead to then state-of-the-art results in Finnish. Whether you choose to discuss or review this work is of course, entirely up to you.

[1] http://morpho.aalto.fi/events/morphochallenge/
[2] C. Lignos, Learning from Unseen Data. Proceedings of Morpho Challenge 2010. https://aaltodoc.aalto.fi/bitstream/handle/123456789/827/isbn9789526033303.pdf?sequence=1&isAllowed=y#page=37
Decision: Accept-Main