Effects of sub-word segmentation on performance of transformer language models
Authors: Jue Hou
Anisia Katinskaia
Anh-Duc Vu
Roman Yangarber
Venue: EMNLP 2023 Main (2023)
Keywords: Language modeling
Natural language processing
Morphological segmentation
TL;DR: We compare GPT and BERT models trained with two unsupervised algorithms for morphological segmentation---Morfessor and StateMorph---against the statistical segmentation algorithm BPE

Abstract: Language modeling is a fundamental task in natural language processing, which has been thoroughly explored with various architectures and hyperparameters.  However, few studies focus on the effect of sub-word segmentation on the performance of language models (LMs).  In this paper, we compare GPT and BERT models trained with the statistical segmentation algorithm BPE vs. two unsupervised algorithms for morphological segmentation — Morfessor and StateMorph.  We train the models for several languages — including ones with very rich morphology — and compare their performance with different segmentation algorithms, vocabulary sizes, and model sizes.
The results show that training with morphological segmentation allows the LMs to:
(1) achieve lower perplexity,
(2) converge more efficiently in terms of training time,
and
(3) achieve equivalent or better evaluation scores on downstream tasks.  
Lastly, we show that (4) LMs of smaller size using morphological segmentation can perform comparably to models of larger size trained with BPE — both in terms of (1) perplexity and (3) scores on downstream tasks.  Points (2) and (4) impact on sustainability, since they reduce the model cost; and while 2 reduces cost only in the training phase, 4 does so also in the inference phase.

Effects of sub-word segmentation on performance of transformer language models
Jue Hou," "t Anisia Katinskaia," Anh-Duc Vu," Roman Yangarber
Department of Computer Science
i Department of Digital Humanities University of Helsinki, Finland first. last@helsiteiiii
Abstract
2015). Another solution is to use an extensive dic- tionary (Jean al., 2015; Luong et al., 2015). This approach is problematic for morphologically-rin languages, such as Finnish or Russian. The word- based approach results in huge vocabulary, and still very many OOV tokens, due to the possibly rare morphological variants (Sennrich et al., 2016).
Language modeling is fundamental task in natural language processing, which has been thoroughly explored with various architectures and hyperparameters. However, few studies fo- cus on the effect of sub-word segmentation on the performance of language models (L.Ms). In this paper, we compare GPT and BERT models trained with the statistical segmentation algo- rithm BPE vs. two unsupervised algorithms for morphological segmentation- and StateMorph. We train the models for sev- eral languages-including ones with very rich morphology-ant compare their performance with different segmentation algorithms, vocab- ulary sizes, and model sizes. The results show that training with morphological segmentation allows the LMs to: 1. achieve lower perplexity, 2. converge more efficiently in terms of training time, and achieve equivalent or better evalu- ation scores downstream tasks. Lastly, show 4. that LMS of smaller size using mor- phological segmentation can perform compara- bly to models of larger size trained with BPE- both in terms of (1) perplexity and (3) scores on downstream tasks. Points 2 and 4 impact on sustainability, since they reduce the model cost; and while reduces cost only in the train- ing phase, 4 does also in the inference phase.
At the other extreme is character-based tokeni- zation vation-treating each character as separate to- ken; but this leads to long sequences, and non- meaningful tokens, which inevitably increase the computational complexity (Libovický et al., 2022). Hence, the most common approach is in the middle-"sub-woote tokenization. segmenter is an algorithm that splits words into sub-word units, between word- and character- based tokenization.
The most common choice for segmenter is the "Byte-Pait Encoding" (BPE) algorithm, origi- nally designed for simple data compression (Gage, 1994). Currently most state-of-the-arr neural lan- guage models, such as GPT (Radford et al., 2019) and BERT (Devlin et al., 2018 18), use it as the seg- menter. The idea behind BPE and its variants is to compute the frequencies of all segments, to it- eratively merge the most frequent consecutive pair of segments into one segment, and add it to the vocabulary of tokens. This yields a compression of the data, and also way to represent the data using a finite vocabulary of segment tokens-much smaller than by using entire word tokens.
1 Introduction
A key component required to train language mod- els is the tokenizer. One "basic" naive approach word-based lokenization-splittee
BPE works well in training language models, as it reduces the size of the vocabulary while still preserving some frequent words in the language as tokens. However, BPE has several problems. It is greedy algorithm, and gives a crude approxi- mation to the "true" linguistic structure of words, that is, morphemes. A morpheme is linguistically meaningful segment of a word-it carries either semantic or syntactic content, and cannot be split further into smaller segments. Also, morpheme retains its meaning in different contexts.
the input into words, and assembling training sequence as the list of word tokens. The problem with this ap- tokens (OOV) during
proach is oft-of- inference- -ic. tokens not seen during training. One solution pick set of most frequent words, and replace all other words with special [UNK] to- ken (Bahdanau et al., 2015; Sutskever et al., 2014). This works well only when the unknown vocab- ulary is small (Cho et al., 2014; Bahdanau et al.,

---

The goal of this work is to determine whether we can improve the performance of language models by using segmenters that are more sophisticated than BPE linguusissl motivated and morpho- logically informed. For example, BPE may de- cide to segment the English word baking into two frequent segments: ba-king. However, we may consider bak-ing a "better" segmentation: root mor- pheme bak and suffix morpheme ing. It stands to reason that "knowing" about morphology- having more meaningful segments in its vocabulary- should help the LM predict data better. To be fair, the LM might be able to "recover" from the naïve segmentation, and still achieve excellent perfor- mance on various tasks, but it will have to do so at a higher cost!-in terms of more parameters and longer training- which degrades sustainability.
Park al. (2021) trained models with several segmentation algorithms, including BPE and Mor- fessor (Creutz and Lagus, 2002), on corpus of Bible verses in 92 languages. They evaluate the models according to surprisal (negative log- likelihood) per verse. Their results show that Mor- fessor segmentation yields much lower surprisal per verse than character or BPE segmentation for the overwhelming majority of the tested languages.
Bostrom and Durrett (2020) compare the output of BPE and Unigram tokenization (Kudo, 2018 on English and Japanese. Comparing with gold- standard morpheme boundaries, they found that Un- igram tokenization segments words more closely to morphological references, while BPE greedily merges sub-wout words according to their frequency, even the sub-word pair is not semantically mean- ingful. They experimented with evaluating lan- guage models on downstream tasks, and pre-trained models from scratch with various segmentation methods. They found that Unigram tokenization outperforms BPE on both English and Japanese.
Which segmentation algorithm yields a better language model? We explore the "goodness" of a LM through four research questions:
RQ1: compared to BPE, does morphological segmentation help LMs achieve lower perplexity?
RQ2: compared BPE, does morphological segmentation help L.S converge faster?
Toraman et al. (2022) conduct comprehensive study to evaluate how tokenization affects the per- formance of Turkish LMs. Besides BPE and its variant WordPiece (Wu et al., 2016), they apply morphological analysis tool for Turkish, and use as sub-word segmenter. Similarly to (Bostrom and Durrett, 2020), they conducted experiments by evaluating six downstream LM tasks. Although BPE and WordPiece achieve the best performance, LM with morphology-basee segmentation achieve comparable results. They also point out that LM with bigger vocabulary size can generally per- form better. However, large vocabulary increases the computational complexity.
RQ3: compared to BPE, does morphological seg- mentation allow LMs to achieve similar (or better) performance on downstream tasks?
RQ4: compared BPE, does morphological segmentation allow us to reduce model size without compromizing model performance?
The paper is orgamized as follows: Section 2 briefly reviews prior work on language modeling with morphological features, and the segmentation methods. Section 3 introduces the language mod- els and the downstream tasks used for evaluation. Section 4 presents the results of our experiments. Scction concludes and discusses future work.
2.1 Sub-word segmentation
2 Prior Work
We briefly introduce the three segmentation algo- rithms we use in this work. We leave out the math- ematical details, and refer the reader to the original papers for further information.
Sub-word tokenization is well-studied topic in natural language processing. Several approaches are proposed to segment words into sub-word units, (Batsuren et al., 2022; Peters and Martins, 2022: Minixhofer et al., 2023). However, few papers have studied the effect of sub-word segmentation on the performance of neural language models.
BPE: is one of most popular sub-word segmen- tation algorithms (Gage, 1994). It is greedy algo- rithm, that starts from a character-based "segmen- tation" (tokenization) of each word, then iteratively sorts all adjacent pairs of segments by frequency of co-occurrence, and merges the most frequent ad- jacent pair. The merged pair is added the vocab- ulary, until the vocabulary reaches a pre-selected maximum size. At this point, BPE stops merging.
Hofmann al. (2021) discuss how sub-word segmentation affects BERT's interpretation of com- plex words. They conduct series of semantic probing tasks and show that applying morphologi- cal segmentation is beneficial for language models.

---

We use Google's SentencePiece implementation of BPE.
to compare language models under similar condi- tions: with same lexicon size. In the following. we denote normal StateMorph by SM, and StateMorph with a pruned lexicon by SMp.
is an unsupervised algorithm for morphology induction (Creutz and Lagus, 2002). Based on the Minimum Description Length (MDL) principle (Rissanen, 1998; Grünwald, 2007), it fa- vors lexicons with fewer and shorter sub-words. It learns the model parameters with MAP estima- tion using a two-part cost function: prior cost and a corpus cost. The corpus cost encodes the probability of the corpus given the lexicon and seg- mentation.
Morfessor:
3 Methods
To evaluate the impact of the segmentation on the language models, we conduct several experiments. For each LM-BERT and GPT, we perform four evaluations, corresponding to the four research questions. Each model is trained with the sub- word lexicon resulting from the three segmentation algorithms: BPE, Morfessor, and StateMorph.
The original Morfessor has been extended in sev- eral ways (Grönroos al., 2020, 2014); however, the later implementations are included the of- ficial Python library distribution. Therefore, we use the Morfessor 2.0 implementation (Virpioja et al., 2013) of the baseline method. We emphasize that in the Morfessor baseline that we use, vocabulary size fixed, determined by the training (not customiz- able). Exploring other variants with configurable vocabulary sizes is left for future work.
3.1 Training the language models
RQI asks: which of the segmentation algorithms yields a better language model-in terms of per- plexity? In the information-theoren sense, this is the definitive measure of the model's "goodness": perplexity tells how well the model is able to pre- diet the data. Thus, in thcoretical terms, model with lower perplexity is better model.
StateMorph: is based on the MDL principle similarly to Morfessor, but is different in several respects (Nouri and Yangarber, 2017). It tries to model the morphological structure of the language using states in finite-state network. Each state learns to emit segments ("morphs") of certain kind, e.g., verb stems, noun suffixes, etc. It also uses MDL-based two-part objective: the cost of the complete data-i.e., the segmented corpus- is the cost of the model, plus the cost of the data given the model. The cost of the model consists of three components: the cost of the morph lexicon, the cost of transitions between states, and the cost of emitting morphs from states. Unlike Morfessor, Stape uses simulated annealing optimize the search, which makes learning much slower. We use the baseline implementation released with the original paper.! It decides on its own optimal lexi- con size, according to the optimized objective.
We also keep track of how many steps each LM takes to converge, to answer RQ2: does a smarter segmentation help the LM learn faster.
Hyperparameters used in training follow the same settings as for BERT base in (Turc et al., 2019), except we use a smaller feed-forwwrd/filter size to reduce the model size. We describe all settings and hyperparameters in detail in Appendix A.
We used a smaller instance size- 256, half of what typically used for BERT. This is due to limited computational resources and exceptionally large vocabulary size for some LMs. We were not able to train with a conventional batch size. As the instance size is smaller than regular and resources are limited, we would like to focus only experimenting with the effect of segmentation and avoid spare resources on harder side-tasks such as next-sentence prediction. Therefore, we did not include it as a part BERT pre-training. We aim for sentence-level LMs (rather than larger context). In future work, we can repeat this on bigger models properly, with more computational resources.
In our experiments, we also use a variant of State- Morph segmentation, where we configure the de- sired lexicon size. Similarly to Morfessor baseline, we cannot control the size of StateMorph's lexicon during training; however, we can prune the lexicon after training, by simply dropping the least frequent morphs. The resulting segmenter will tend to seg- ment a word with more frequent sub-words (even if the overall cost may be higher). But this allows us
Data: the corpora used to train the language models are as follows. The Finnish corpus con- tains data from two major Finnish news agencies: Helsingin Sanomat (HS) and YLE.² The corpus contains around 17M instances. For Russian, we use the Taiga corpus (Shavrina and Shapovalova,
hhtp::/hlp.cs.hel httpe/siiii..immmpp....oe
MctaShare: Yle Finnish News Archive

---

2017). The corpus contains 66M instances. The English corpus comes mainly from the English Wikipedia dump from 2020-05-01. We add our own news data, privately crawled from the Internet. The corpus contains 61 M instances. For Turkish, we use the OSCAR corpus (Abadji et al., 2022). The corpus contains 20M instances. Each training instance is composed of sentences.
3.3 Downstream tasks
confirm that language RQ3 asks whether models with lower perplexity yield comparable (or better) performance on downstream tasks. Thus we also evaluate performance of the models on practical applications. We consider two types of tasks: classification vs. generative tasks.
Finnish: we use two topic classification tasks. and Part-of-Speech (PoS) tagging. which is a sequence-labeling task. Topic classification is based on two corpora: In-domain and Out-of- domain. For the In-domain task, we use the YLE corpus, same as for training the segmentation algo- rithms and LMs. We classify documents into four topics: Sport, Politics, Science, and Culture. For the Out-of-domain task, we use the dataset from Ylilauta' with topics: Sport, Politics, Science, and Drikk We use the same instance size as in pre-training. For each corpus, we use 10k, 1k, and Ik for training, validation, and testing, respectively. The PoS tagging task based on Finnish-TDT from UD: Universal Dependency (Nivre et al., 2020).
Pre-processing: For each language, we train all segmentation algorithms with the same word list, extracted from the training corpus. We lower- case all words to assure that the segmentation of given word the same regardless of its case. How- ever, the language models should handle mixed- case input. Additional technical details about the pre-training phase are given in Appendix A.1.
To make the comparison fair, we make sure that the experiments use the same vocabulary size. As mentioned above, we are not able to customize the lexicon size for Morfessor directly. To customize the lexicon size for StateMorph, we prune the lex- icon by frequency of the morph. Therefore, set the lexicon size for BPE manually to match the lexicon size produced by Morfessor and State- Morph. We leave the exploration of the effeet of Staph variants and Morfessor variants with adjustable lexicon size for future work.
For the generative task, use paraphrase dataset from (Kanerva al., 2021). We use 21k for training, 2.6k for validation, and 2.6k for testing. Russian: For classification, we use topic clas- sification based on the Lenta.ru corpus; part-of- speech tagging, based on Russian-SynTTggRu from UD; and linguistic acceptability (LA) task, us- ing RuCoLa (Mikhailov et al., 2022), with GPT and BERT models. For the generative task, we use paraphrase generation with cu-paraphrase-NME Leipzig dataset.
We should clarify that we do not aim for optimal lexicon size in terms of LM pre-training or any of the downstream tasks; it is only to make the comparison fair. Searching for the optimal lexicon size would require exorbitant compute resources, and is likely highly language-specific. This is not our goal here. The sizes of the resulting lexicons are shown Appendix A.
We explore PoS tagging only with BERT, as GPT is left-to-right LM, which prevents the model's 's access the "future" during training, while PoS tagging may require inference from the entire sur- rounding context, not only the left side. Therefore. GPT may not be suitable for PoS tagging.
3.2 Model size
RQ4 asks whether a smarter segmentation will al- low us to build language model that has equiv- alent better performance with smaller model size measuued in terms of the number of learned model parameters. This is a crucial question for sustainability, since the large language models con- sume vast amounts computing resources-bor in the training and in the inference phases.
We explore the paraphrase generation task only with GPT models, as we did not include next- sentence prediction in pre-training our BERT-and therefore it may not be suitable for generative tasks. We leave this for future work.
Experiments and results
We configure our small models similarly to BERT mestium in (Ture et al., 2019); more details about hyperparameters are in Appendix A.
This section presents series of experiments that we conduct to address the research questions.
MMetaShae: Ylilauta Corpus HuggingFace: nu-praphnass-NMMTHLippgg
'HuggingFace Wikipedia datasets

---

Morfessor
Morfesser
60
bpe_ 233k
60
bpe_ b49_ 149K
149K
50
bpe_78 78k
bpe_ 70k
SM
40
SM
44
Smp
30
20
20
750
1250
(a) Finnish GPT
(b) Finnish BERT
bpe 516k
bpe 516k
60
bpe 489k
60
bpe 489k
Morfessor
Morfessor
50
50
SM
SM
bpe 170k
bpe bpe_170k 170k
SMp
SMp
30
20
20
200
400
600
800
250
500
750
1000
1250
1500
Number batches (k)
Number of batches
(c) Russian GPT
(d) Russian BERT
Figure Learning curves for pre-training Finnish and Russian LMs with different segmentation methods.
Morfessor
Morfessor
bpe.
25
bpe 86k
SM
30
30
bpe. 111k
bpe_79K 79k
SMp
SMp
bpe. 52k
bpe 43k
25
20
20
15
10
15
100
200
300
500
600
700
100
200
300
400
Number of batches (k)
Number of batches
(a) English GPT
(b) Turkish GPT
Figure 2: Learning curves for pre-training English and Turkish GPT with different segmentation methods.
4.1 Pre-Trainie Tee-Triinng with different segmentations
phological segmentation yields lower perplexity. There are only few exceptions: BERT with Mor- fessor (FI and RU) and GPT with Morfessor (EN and TR); but perplexity very near to the BPE counterpart. This suggests models with morpho- logical segmentation yield overwhelmingly better perplexity, compared to using BPE, with the same vocabulary size (RQ1).
To explore the behavior of the different segmenta- tion methods, we pretrain array of LMs, GPT and BERT. We visualize the learning curves of Finnish and Russian models in Figure/ 1. Solid lines indicate BPE, lines with dashes indicate mor- phological segmentation; line colors code the vo- cabulary sizes. The same experiments and Turkish, GPT only, are shown Figure 2.
English
Regarding the impact of segmentation the learning progress: the number of steps to reach convergence (first column in both tables) higher for most models with BPE, several times higher for some. This suggests that the models with mor- phological segmentation are also more efficient
We show the final perplexity, and the number of steps to reach convergence Finnish and Russian in Table 1, English and Turkish in Table 2.
The curves and the Tables show that with the same vocabulary size-for for almost all models, mor-

---

GPT
BERT
Segmenter
Size
steps (k)
PPL
Segmenter
steps (k)
PPL.
steps (k)
PPL.
bpe (78k)
Base
757
23.69
bpe (233k)
593
38.17
1426
14.18
Finnish
SMp
Small
542
21.22
Morfessor
578
38.09
1339
15.20
Finnihh
bpe (149k)
664
30.16
1690
9.75
bpe
Small
833
29.23
SM
419
19.71
521
7.23
SMp
Base
724
16.65
bpe (78k)
757
23.69
1522
bpe (170k)
Base
660
24.97
SMp
724
16.65
584
5.92
Russian
SMp
Small
651
23.39
bpc (488k)
762
35.01
888
12.96
bpe
Small
1029
28.99
Morfessor
6019
30.43
591
13.43
SMp
Base
816
19.08
Russian
bpc (516k)
843
34.96
840
13.40
363
29.66
11.47
Table 3: Pre-training results for GPT language models
bpe (170k)
660
24.97
1663
6.97
of different sizes with BPE vs. SMp. compared blue.
SMp
816
19.08
Base models the as in Table 1.
Table 1: Pre-training results for Finnish and Russian LMs with different segmentation methods. Vocabulary size (in parentheses) applies to models each box.
the base LMs have 189M parameters in Finnish, 354M parameters in Russian-which is 43% and
GPT
28% bigger respectively." This further confirms RQ4: morphological segmentation can help reduce the model size, and improve the sustainability the models in the training and inference phases.
Segmenter
steps (k)
PPI.
bpe (15lk)
624
19.82
Morfessor
594
20.05
English
bpe (177k)
579
20.33
SM
465
18.93
4.3 Fine-tuning for downstream tasks
bpe (107k)
739
17.92
We do not try to optimize the fine-tuning settings for the specific downstream tasks, since we do not aim for state-of-the-arr performance. Rather our goal to explore the impact of segmentation on the performance of LMs on downstream tasks. We run each downstream task experiment three times, and report the mean and standard deviation (a) of the resulting scores. We evaluate classification task performance with 3 metries: Accuracy, F1, and Matthews Correlation Coefficient (MCC).
SMp
531
17.55
bpe (92k)
402
13.53
Morfessor
393
14.39
Turkish
bpe (85k)
474
13.04
SM
426
11.07
bpe (60k)
381
12.93
SMp
432
10.67
Table 2: Pre-training results for English and Turkish GPTs with different segmentation methods. Vocabulary size (in parentheses) applies to models each box.
Topic classification (Finnish): Table 4 shows in-domain and out-of-domain topic classification for both LM types. The performance of the LMs is overall quite close. The best performing model the In-domain task is BERT with SMp segmen- tation, with an average accuracy of 92.6%. This around I % higher than its corresponding BPE- segmented LM. The best model on the Out-of- domain task is GPT with SM segmentation, with av- erage accuracy of 79.3%. This is about 3% higher than its corresponding BPE-segmented LM.
terms of training time (RQ2).
4.2 Pre- Fre-Training with different model sizes
We next explore RQ4: model size. We train GPT models above for Finnish and Russian- with morphological segmentation (SMp) and with BPE, but in smaller size, i.c., with smaller number of parameters. In this experiment, we use models with the smallest vocabulary: 78k for Finnish and 170k for Russian.
All models achieve very good scores on In- domain data, and relatively reasonable scores on Out-of-domain. We apply the t-test to compare the morphologically-ssgmeen
Figure 3 shows the learning curves, and Table 3 shows the final perplexity at convergence for mod- els of different sizes. We see that smaller LMs with morphological segmentation yield better perplex- ity compared to larger LMs with BPE segmenta- tion. The smaller LMs have 132M parameters in Finnish, and 276M parameters in Russian, whereas
LMs with their corre- sponding BPE-segmented LMs. Only BERT with SM shows a significantly worse performance than
"Details about the numhers of parameters for all models are given in Appendix A, Tables 10 and 11.

---

bpe_ 73k small
bpp 170k small
60
bpe_ 75k
60
bpe 170k
SMp small
SMp small
SMp
50
SMp
44
20
20
600
1000
(a) Finnish GPT
(b) Russian GPT
Figure Learning curves pre-training language models with different model sizes.
In-domain (YLE corpus)
Out-of-domall (Ylilauta corpus)
Accuracy
Fl-measure
MCC
Accuracy
Fl-measure
MCC
Avg.
0
Avg.
Avg.
Avg.
Avg.
Avg.
bpe (233k)
90.0
1.4
85.8
2.0
86.8
1.8
75.8
0.3
70.3
2.0
68.1
0.6
morfessor
89.4
1.0
85.0
1.4
85.7
1.6
78.4
2.4
72.2
3,4
71.2
3.3
bpe (149k)
87.9
1.2
84.2
1,2
84.1
1.6
77.2
2.2
70.8
3.3
69.7
3.5
GPT
SM
90.1
2.1
86.2
3.1
87.0
2.7
79.3
3.6
74.4
4.9
72.9
4.8
bpe (78k)
89.1
2.8
84.1
2.5
85.4
1.5
77.5
2.9
71.1
3.6
70.1
3.9
SMp
90.1
1.5
86.5
1.8
86.7
2.4
77.6
1.5
71.9
1.8
70.2
2.3
SMp-sm
88.4
2.8
84.8
3.5
84.9
3.6
75.5
2.5
69.4
4.3
67.3
3.4
bpe-sm
88.9
2.5
83.7
2.8
85.0
3.5
75.0
2.3
68.5
2.9
66.4
3.3
bpe (233k)
90.5
2.5
89.7
1.0
87.4
1.6
72.5
3.7
67.1
4.7
64.1
4.9
morfessor
90.2
1.1
89.6
2.8
87.0
3.5
75.4
1.5
69.5
1.8
67.5
2.2
BERT
bpe (149k)
91.8
1.3
91.3
1.0
89.3
1.6
75.4
3.7
70.7
4.3
67.5
4.8
SM
89.9
0.2
89.2
0.3
86.6
0.3
76.2
3.2
71.2
2.7
68.5
3.6
bpe (78k)
91.4
1.5
90.7
1.4
88.6
2.1
77.0
3.3
71.3
2.5
69.7
4.0
SMp
92.6
0.2
91.9
0.2
90.1
0.3
74.1
4.3
69.3
4.4
66.0
6.1
Table 4: Fine-tuning for Finnish topic classification.
BERT with BPE (149k), with v-value of 0.03. All other t-tests for all comparison pairs in all metrics do not show a significant difference performance, with p-values all greater than 0.05. This suggests that the Finnish LMs with different segmentations perform comparably on topic classification after fine-tuning.' This includes smaller-size models (SMp-sm and BPE-sm). GPT with SMp-sm, which is smaller in size, is comparable with GPT with BPE (78k), which is of regular size. The minimum v-value of their t-test for all metries is 0.2, while the maximum p-value is 0.41. This further suggests potential benefits for sustainability.
Ylilauta, with similar fine-tuning settings.
Topic classification (Russian): Table 5 shows the performance each LM after fine-tuning. Overall, all models achieve very good performance on this task. The GPT model with SMp achieves the best overall performance, with average curacy of 91.4%, about 1 1% higher than the corre- sponding model with SM. As in Finnish, the small GPT model with SMp-sm is slightly better (1%) than base GPT with BPE (170k).
Among the BERT models, BERT with SM achieves the best performance overall with 87.3% accuracy. This is relatively 8% higher in accu- racy and F1 than the corresponding model with BPE (516k), and about 11% higher in MCC. Com- pared Finnish, the difference between models with morphological segmentation and BPE is larger (though not significantly) in accuracy and FI. To
For reference, Virtanen et al. (2019) reach 90.6% accuracy on classifying YLE data, and 79.8% on
"We use an unpaired (-test with unequal variance assump- tion. The null hypothesis is: a morphologiially-ssggeee 1M performs comparably to its BPE-scgmented counterpart; the alternative hypothesis is: one LM outperforms the other.

---

Accuracy
Fi
MCC
Accuracy
MCC
Avg.
Avg.
Avg.
Avg.
0
Avg.
0
bpe (488k)
89.8
85.5
86.2
bpe (488k)
55.0
1.9
17.7
6.9
Morfessor
89.3
0,9
84.6
0.4
85.8
1.1
Morfessor
52.5
1.9
15.1
5,5
bpc (516k)
91.4
0.5
88.1
1.2
88.7
0.5
bpe (516k)
48.8
2.9
11.0
6.6
GPT
SM
90.5
0.5
86.1
0.4
87.4
0.7
bpe (170k)
88.5
1.8
83.5
2.2
84.5
2.5
GPT
SM
52.2
1.7
15.9
3.6
SMp
91.1
0.7
87.1
1.9
88.2
0.9
bpe (170k)
52.7
6.5
6.2
8.4
SMp-sm
89.4
2.2
84.4
2.0
85.7
3.0
SMp
56.0
8.3
2.4
0.5
bpc-sm
88.6
1.7
83.6
1.7
84.8
2.2
SMp-sm
49.0
3.2
8.2
7.7
bpe (488k)
87.0
2.8
82.1
3.3
82.7
3.7
bpe-sm
56.6
5.6
6.8
9.7
Morfessor
82.7
4.3
77.6
4.3
77.0
5.9
bpe (488k)
57.1
2.9
15.8
3.3
BERT
bpe (516k)
81.2
3,9
76.9
4.1
75.2
4.8
Morfessor
60.5
6.8
10.6
9.3
SM
87.3
2.0
83.2
2.5
83.1
2.7
bpc (170k)
1.9
82.5
1.9
82.8
BHRT
bpc (516k)
55.7
2.2
14.6
8.3
SMp
83,5
4.4
77.9
4.3
77.8
5.6
53.7
2.4
16.4
3.1
bpc (170k)
54.8
5.0
11.7
2.1
Table 5: Fine-tuning tone-tuning for Russian topic classification.
SMp
52.1
4.1
14.8
4.6
Accuracy Avg.
MCC
Table 7: Fine-tuning for Russian linguistic acceptability
Avg.
bpe (233k)
95.0
94.0
0.1
SM and SMp achieve the best performance, but this is only slightly better than the performance of BERT with BPE (149k) and BPE (78k), while BERT with Morfessor worse by 1%. Rus- sian, LMs with morphological segmentation have generally the same performance as LMs with BPE (less than 0.1 %% difference). Overall, performance is very close this downstream task.
Morfessor
94.0
92.9
0.0
Finnish
bpe (149k)
95.0
0.0
94.0
0.0
SM
95.4
0.0
94.5
0.0
bpe (78k)
95.3
0.1
94.5
0.1
SMp
95.4
0.0
94.5
0.0
bpe (488k)
97.6
0.0
97.2
0.0
Morfessor
97.5
0.0
97.1
0.0
Russian
bpc (516k)
97.7
0.0
97.3
0.0
For comparison, Virtanen al. (2019) achieve 98.23% accuracy for Finnish with fine-tuning on the same dataset; others achieve 97.8% accuracy for Russian on this task.9
SM
97.6
0.0
97.2
0.0
bpe (170k)
98.0
0.0
97.7
0.0
SMp
97.9
0.0
97.5
0.0
Linguistic acceptability: Table 7 shows the re- sults on Russian. This is very difficult task, with accuracy only around 50-60%. Overall, the BERT model with Morfessor segmentation achieves the best accuracy, not significantly better than the cor- responding BPE, which p-value of their t-test is 0.25. BERT and GPT achieve relatively close performance. The small GPT model with SMp seg- mentation performs worse than regular GPT with BPE (170k), in relative terms by 7% on accuracy, but better on MCC, by 32%.
Table 6: Fine-tuning for Finnish PoS tagging.
confirm this, we apply the t-test, comparing the morphological BERTs with their corresponding BPE-segmented BERTs, for accuracy and F1.
The average t-test p-values on morphological BERT BPE BERT are 0.1 and 0.09, for accuracy and F1, respectively; the p-values for the t-tests in Finnish are 0.45 and 0.42. This indicates more sig- nificant difference between morphological BERT vs. BPE BERT in Russian than in Finnish. But the performance morphological BERTs still comparable with BPF BERTs in terms of accuracy and F1, as their p-values are over 0.05.
For comparison, GPT-3 in (Mikhailov al.. 2022) reaches 55.8% accuracy this task, while BERT reaches 75.9% accuracy.
Paraphrasing: Table shows the results of periments on paraphrase generation, another ex- tremely challenging task, even for the human. The evaluation metric chrl from machine transla- tion (Popović, 2017, 2015), uses the F-score statis- tic for character n-gram and word n-gram matches.
Public leaderboards show state-of-the-ard accu- racy of about 96% this task.*
PoS tagging: Table shows the results for Finnish and Russian. All models achieve very good accuracy and MCC. The BERT models with
"Kaggle leaderboard, text classification (RU).
"XLMM-RBERTT base, UD POS tagging: Russian

---

average chrF++
0
We explore four research questions: does mor- phological segmentation help LMs- 1: reach lower perplexity; 2: learn and converge faster; 3: perform at least as well on downstream tasks; 4: perform at least as well with smaller model size.
bpe (233k)
28.9
0.5
Morfessor
29.6
0.5
bpe (149k)
30.8
1.2
Finnish
SM
31.5
0.5
bpe (78k)
32.8
0.6
We show that LMs trained with morphological segmentation reach much lower perplexity (except Morfessor, which has the largest vocabulary) than LMs trained with BPE (RQ1). ). We also show that LMs trained with morphological segmentation con- verge faster than LMs trained with BPE (RQ2).
SMp
31.9
1.4
SMp-sm
27.2
0.3
bpe-sm
28.2
2.1
bpe (488k)
24.0
0.1
Morfessor
25.8
0.7
bpe (516k)
26.2
0.2
Russian
SM
26.0
0.6
We evaluate the performance of language mod- els on several downstream tasks (RQ3). The results show that the performance of LMs with morpholog ical segmentation (including smaller LMs) is com- parable to models with BPE. While performance on the topic classification tasks is quite convincing, we meet several challeges with other downstream tasks. The tasks are so high-level and so difficult, and the baseline performance so low, that the gains from "smarter" segmentation may not be easy to demonstrate directly. The languages we work with have a paucity of "standard" datasets for downstream tasks with sufficient labeled data. In the future, we will investigate more languages and more downstream tasks. However, the theoretical results from RQ1, 2 and 4 are convincing.
bpe (170k)
22.6
0.8
SMp
28.9
0.0
SMp-sm
22.2
0.8
bpe-sm
22.6
0.3
Table 8: Fine-tuning for paraphrase generation
The Finnish models and most of the Russian models show fairly similar performance. The per- formance of the Russian models with SMp segmen- tation is relatively better than the models with BPE segmentation by 28%. The small Russian models with SMp segmentation is 2% below the regular model with BPE (170k), while the small Finnish model with SMp-sm segmentation is 21% below the regular model with BPE (78k), in relative terms.
To investigate the impact of segmentation on model size (RQ4), pre-train smaller version of each LM with State for each language. We show that-for a fixed vocabulary size small LMs with StateMorph segmentation have lower perplexity than regular-sized LMs with BPE seg- mentation.
5 Conclusions and future work
We have explored the impact of intelligent seg- mentation algorithms on the performance of lan- guage models. We experiment with four languages: Finnish, Russian, Turkish, and English, with an in-depth investigation of the first two. 10 We train two language mPT and BERT-and com- pare the statistical segmentation algorithm, BPE, with two morphological segmentation algorithms: Morfessor and StateMorph.
This suggests that morphological segmentation can reduce the size of language models, and im- prove the sustainability of LMs (RQ4). Sustain- ability is impacted by RQ2, but even more so by RQ4, since RQ2 affects only the training, whereas RQ4 affects training and inference.
We aim to show that LMs trained on a vocabulary based on morphological information are better than LMs trained on "naïve" sub-word segments pro- duced by BPE. Although BPE does not explicitly model morphology, it will inevitably stumble into discovering some morphemes as well becasse many morphemes also happen to be frequent sub- words. This makes BPE tough baseline beat.
In future work, we plan to experiment with more morphological segmentation algorithms, a broader range of languages, and more types of language models, such as Transformer-y XL. (Dai et al., 2019) and XLNet (Yang al., 2019). As final point, the smarter segmentations that we have tested have much room for improvement-e.ge we can expect that supervised or rule-based morphological seg- mentation will be still better than the unsupervised segmentation that we have tested so far.
'"The languages are chosen as representatives of their re- spective aub-famliiss-Finn--uge and Slavio-which have very rich morphology, both verbal and nominal, certainly among the richest among the European languages.

---

Acknowledgements
Ryan Cotterell, and Ekaterina Vylomova. 2022. The SIGMORPHON 2022 shared task on morpheme seg- mentation. In Proceedings of the 19th SIGMOR- PHON Workshop on Computational Research in Pho- netics, Phonology, and Morphology, pages 103-116, Seattle, Washington.
This research was supported in part by BusinessFin- land (Grant 42560/31/2020), and by grant from the Helsinki Institute for Information Technology (HIIT). We are grateful for assistance from Javad Nouri.
Kaj Bostrom and Greg Durrett. 2020. Byte pair encod- ing is suboptimal for language model pretraining. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 4617-4624, Online.
Limitations
We acknowledge that this work has several limi- tations. First, we use only two language model architectures, GPT and BERT. Second, we use only two languages, Finnish and Russian, for each ar- chitecture. Third, we acknowledge our selection of segmentation algorithms is limited; other segmen- tation algorithms exist, both supervised or unsuper- vised. We plan to investigate other languages and the impact of different segmentation algorithms as well as different LM architectures and settings in future work.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-devoder for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1724 1734, Doha, Qatar.
Mathias Creutz and Krista Lagus. 2002. Unsupervised discovery of morphemes. arXiv preprint cs/0205057.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language mod- els beyond a fixed-length context. arXiv preprint 1XXi::901.02280
We also acknowledge that the size of the lexicon is a factor which impacts the performance of the language model. Due to limited computational resources, we experimented with a limited choice of lexicon sizes, where some of them may not be optimal. We plan to investigate further the effects of lexicon size. Lastly, we explored only limited number of downstream tasks, which may not reveal the complete picture about the performance of a language model.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language under- standing. arXiv preprint arXiv: 1810.04805.
Philip Gage. 1994. A new algorithm for data compres- sion. C Users Journal, 12(2):23-38.
Stig-Arne Grönroos, Sami Virpioja, and Mikko Ku- nimo. 2020. Morfessor EM+ Prune: Improved sub- word segmentation with expectation maximization and pruning. arXiv preprint urXiv:2003.0313
Ethics Statement
This data used this work is mostly open data, or data used with explicit permission from its pub- lisher. We do not use any private data, nor any data that is not allowed to be used for research purposes.
Atig-Arme Grönroos, Sami Virpioja, Peter Smit, and Mikko Kurimo. 2014. Morfessor FlatCat: An HMM- based method for unsupervised and semi-supervised learning of morphology. In Proceedings of COLING, the 25th International Conference on Computational Linguistics, pages 1177-1185.
References
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a Cleaner Document- Oriented Multilingual Crawled Corpus. arXiv e- prints, page TtV.:2001006442
Peter Grünwald. 2007. The Minimum Description Length Principle. MIT Press.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Deriva- tional morphology improves BERT's s interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing, pages 3594 3608, Online.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA.
Khuyagbaatar Batsuren, Gábor Bella, Aryaman Arora, Viktor Martinovic, Kyle Gorman, Zdenčk Zabokrt- sky. Amarsanaa Ganbold, Šárka Dohnalová, Magda Ševčíková, Kateñina Pelegrinová, Fausto Giunchiglia,
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large tar- get vocabulary for neural machine translation. In

---

Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1-10, Beijing, China.
Ben Peters and Andre F. T. Martins. 2022. Beyond char- acters: Subword-les morpheme segmentation. In Proceedings of the 19th SIGMORPHION Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 131-138, Seattle, Washing- ton.
Jenna Kanerva, Filip Ginter, Li-Hsin Chang, liro Ras- tas, Valtteri Skantsi, Jemina Kilpeläinen, Hanna-Man Mari Kupari, Jenna Saarni, Maija Sevón, and Otto Tarkka. 2021. Finnish paraphrase corpus. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 288-298. Reykjavik, Iceland.
Maja Popovié. 2015. chrF: character n n-gram F-score for automatic MT evaluation. In Proceedings of the tenth workshop on statistical machine translation. pages 392-395.
Maja Popović. 2017. chrF++; words helping character n-grams. Proceedings of the second conference on machine translation, pages 612-618.
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics, pages 66-75, Melbourne, Australia.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9
Jindfich Libovicky, Helmut Schmid, and Alexander Fraser. 2022. Why don't people use character-leve machine translation? In Findings of the Association for Computational Linguistics: ACL 2022, pages 2470-2485 Dublin, Ireland.
Jorma Rissanen. 1998. Stochastic complexity in statisti- cal inquiry, volume 15. World scientific.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics, pages 1715-1725, Berlin, Germany.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-base neural machine translation. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal.
Tatiana Shavrina and Olga Shapovalova. 2017. To the methodology of corpus construction for machine learning: <<Taiga> syntax tree corpus and parser. Pro- ceedings of "Corpora-2017" Conference, pages 78- 84.
Vladislav Mikhailov, Tatiana Shamardina, Max Ryabinin, Alena Pestova, Ivan Smurov, and Ekate- rina Artemova. 2022. RuCoLA: Russian corpus of linguistic acceptability. In Proceedings of the 2022 Conference on Empirical Methods Natural Lan- guage Processing. pages 5207-5227, Abu Dhabi, United Arab Emirates.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27.
Benjamin Minixhofer. Jonas Pfeiffer, and Ivan Vulić. 2023. CompoundPiece: Evaluating and improving decompoundigg performance of language models.
Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinuç. and Oguzhan Ozcelik. 2022. Impact of tokenization on language models: An analysis for Turkish. arXiv preprint arXiv:2204.08832
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfih Language Resources and Evaluation Conference, pages 4034-4043, Marseille, France. European Language Resourees Association.
Tulia Ture, Ming- Ming-Weii Wei Chang, Kenton Lee, and Kristina Toutamova. 2019. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, (bs/1908.08963
Sami Virpioja, Peter Smit, Stig-Ame Grönroos. Mikko Kurimo, et al. 2013. Morfessor 2.0: Python imple- mentation and extensions for Morfessor Baseline. Technical report, Aalto University.
Javad Nouri and Roman Yangarber. 2017. Learning morphology of natural language as finite-state state gram- mar. In SLSP: the 5th International Conference on Statistical Language and Speech Processing, pages 44-57. Springer.
Antti Virtanen, Jenna Kanerva, Rami Io,, Jouni Luoma. Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. CORR, abs/1912.07076.
Hyunji Hayley Park, Katherine Zhang, Coleman Ha- ley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021. Morphology Matters: A multilingual language modeling analysis. Transactions of the Association for Computational Linguistics, 9:261-270
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato,

---

Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young. Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans-
Segmenter
Size
Voc (K)
#Param (M)
Morfessor Base
233
467
bpe
Base
233
473
SM
Base
149
315
bpe
Base
149
315
lation. CoRR, hos/1609.05
SMp
Base
78
188
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
SMp
Small
78
132
bpe
Base
78
189
bpe
Small
78
134
Table 10: Segmentation of Finnish data. Corresponding vocabulary sizes (thousands tokens), and model size (millions of parameters)
Pre-Training details
Base
Small
Segmenter
Size
Voc (K)
#Param (M)
Transformer blocks (L)
12
8
Morfessor
Base
487
921
Self-attention heads (A)
12
8
bpe
Base
488
923
Hidden size (H)
768
512
SM
Base
518
979
feed-forward/filter
1024
512
bpe
Base
516
974
Table Hyperparameters for GPT/BERT models.
SMp
Base
171
355
SMp
Small
171
276
We pre-train all language models with the same settings for the optimizer. We apply Adam op- timizer, mostly with the default parameters from Torch, except we use (0.9, 0.998) for beta val- ues. We use batch size of 100 for most Russian and Finnish models, except Russian GPT with Morfes- sor, SM, and BPE with the corresponding vocabu- lary size. We are not able to fit Russian models and data into the GPU memory during training, there- fore we compensate by training with batch size 50, and accumulate gradient for 2 steps achieve the same effect as batch size 100. We use batch size 300 for all English and Turkish GPTs to speed up the pre-training process. We use the same learning rate (5e-5) for all models, except for the Finnish GPT models, which uses larger learning rate (1e- 4). We use different number of validation steps depending model type: 300 for the GPT mod- els, and 1200 for BERT models. We use a larger number of validation steps, because the random masking mechanism decrease the actual tokens for validation, as compared to GPT models. We use EarlyStop with patience of 10 and = 10-5,
bpe
Base
170
354
bpe
Small
170
275
Table 11: Segmentation of Russian data. Corresponding vocabulary sizes (thousands tokens), and model size (millions of parameters)
model on a GPU cluster for the first 36 hours. Each node in the cluster is equipped with 4 Nvidia A100- 40G GPUs and two AMD Rome 7H12 CPUs with 64 cores each. We all GPUs and 64 cores for each job. We then continue the pre-training on a bigger cluster, until the models converge, where each cluster node is equipped with AMD MI250x GPU modules. Each GPU module has a AMD EPYC "Trento" CPU and two GPU dies with 64GB of HBM2 memory, which makes 8 GPUs overall in one node. We request the same number of GPUs and CPU cores for each model, in the jobs run the first cluster. For BERT models well as English and Turkish GPT models, we pre-train only on the second cluster, with the same settings as for the GPT models.
Table 9 shows the hyperparameters used for pre- training GPT and BERT models. Tables 10 and 11 show the vocabulary size of different segmenta- tions, and their corresponding overall number of parameters, when pre-training Finnish and Russian language models, respectively.
A.1 Segmentation and capitalization
A technical point of difference between BPE and the other segmenters: BPE distinguishes tokens ap- pearing word-initially vs. elsewhere, with special symbol " " to designate whitespace, for example, ba ba-king". Morfessor and StateMorph do not distin- guish in their output whether a morpheme appears
We pre-train train in two stages for all of the Finnish and Russian GPT models. We first treitrain each

---

initially medially, 11
Therefore perform segmentation follows: pre-segment all words in lower case, then convert the tokens back to the original case, mark all tokens appearing word-initially with "_" (as BPE does), and collect the exact vocabulary for pre-training language models.
B Fine-tuning details
We follow a similar training process as pre-training. We conduct all fine-tuning tasks on the second clus- ter, which also used in pre-training, with the same resources as in pre-training. We use the same opti- mizer (AdamW), and same optimizer parameters in pre-training. We apply the same learning rate (5e-5), and the same batch size of 50; we use 50 rather than 100 in pre-training, so that all models can be fine-tuned uniformly. We uniformly accu- mulate gradient for 5 steps.
"hee do model the beginning or end of a word when learning to segment.


Reviews:
[{"review_id": "FqfyBZyiKB", "reviewer": "Reviewer_VFFv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n_A comprehensive study would cover more languages and algorithms._\n\nTrue, ideally this would be tested on many languages.\nWe present work on two languages, so far, because:\n  * We are a small academic team, not a large corporation, and we have limited compute resources.\n  * Even for two languages, we already had to train and run *hundreds of models*.\n  * We follow the _zero-one-infinity rule_: If the phenomenon is observed in two or more highly distinct cases, it is probably of significance (though not guaranteed).\n  * For many (other) morphologically rich languages, resources for training unsupervised models also are limited; resources for training downstream tasks are even more limited.\n  * Our languages are quite representative: they have very different morphological structure — one agglutinative (FIN), one fusional (RUS).\n  * In fact, we are in the process of testing our hypotheses also on English and Turkish. If the paper is accepted, we will include those results in the additional space provided.\n\n_Cover more algorithms:_ \n\nWe would be happy to test more segmentation algorithms.  Could you please provide an indication which algorithms we should test?\n\nWhat our paper presents is:\n * 99.9% of all work involving language models uses BPE.\n * Our initial experiments (on two quite different languages) find that BPE is sub-optimal:\n     - It causes the model to waste parameters to compensate for the naive segmentation.\n     - Based on the initial experiments, we observe that morphologically aware models could be at least 30-40% smaller (!)\n     - (a deeper exploration is needed, which may show even bigger gains.)\n * This means that the standard approach produces models that are slower *and* burn more electricity.\n\nWe do not claim that the study is comprehensive in regard to the number of languages.  Our claim is this is an _important problem_ to investigate further, with more languages and tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Sub-word segmentation algorithms are important in language modeling and more studies are needed.", "reasons_to_reject": "A comprehensive study would cover more languages and algorithms. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores the impact of segmentation algorithms on language models. Experiments were conducted on two morphologically rich languages, Finnish and Russian. They tested GPT and BERT models trained on different segmentation algorithms, morphological (Morfessor or StateMorph) against a statistical one (BPE). Results shows that language models trained with a morphological segmenter converge faster than those trained with BPE. Besides that, smaller models trained with morphological segmenters are comparable to larger ones trained with BPE.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HG8toRd3fm", "reviewer": "Reviewer_HPcL", "summary": "", "questions": "One question would be why the authors do not refer to the sub-word segmentation algorithm included in Sentencepiece (Kudo et al.), whcih appears to be widely used in recent years.\n\nIt is not especially clear which version of the Morfessor is used. Would it be possible to estimate what the more advanced methods would confer in terms of improvement (as far as I understand, StateMorph is better than the Morfessor variant used)?\n\nLine 255: \"Due to limited computational resources, we used 256\na smaller instance size\" - could the authors elaborate on this? I suppose they could refer to the material available in the Appendix. \n\nLine 286: \"We set the lexicon size for BPE\n287 manually—to match the lexicon size produced by\n288 the morphological segmentation algorithms. Sizes\n289 of the resulting lexicons are shown below.\" - Did the two morphological-based  algorithms (Morfessor/StateMorph) lead to similar lexicon sizes?  What if BPE had been used to determine the lexicon size for all 3 algorithms; in such a case would the results have changed substantially to make BPE a more competitive method?\n\nLine 367: \"the number of steps to reach conver- 367\ngence is higher for most models with BPE, sev- 368\neral times higher for some.\" - indicate which entries in the Table this statement corresponds to, so as to aid readability.\n\nLine 415: \"The overall variation is less than 5%, 415\nwhich suggests that Finnish LMs with different seg- 416\nmentations perform comparably on topic classifi- 417\ncation after fine-tuning.\" - have the authors checked if this variation of less than 5% is not statistically significant?\n\nLine 440: \"the difference between models with mor441\nphological segmentation and BPE is larger (though\n442 not significantly) in terms of accuracy and F1.\" - it would be useful if the authors could discuss the question of statistically significant (or non-significant) differences.\n\nIt  would be useful to determine the acronym MCC (I assume it corresponds to the Matthews correlation coefficient).\n\nA minor point: \"Finnish and Russian. These languages are chosen\n493 as representatives of their respective sub-families—\n494 Slavic and Finno-Ugric\" - change the sequence of the last two terms, at line 494 to agree with the order in line 492. \n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.  Some responses to your questions:\n\n**** _One question would be why the authors do not refer to the sub-word segmentation algorithm included in Sentencepiece (Kudo et al.), which appears to be widely used in recent years._\n\nIn our implementation, we do indeed use Google's latest version of the Sentencepiece library, which implements BPE.  We did not mention it in the paper, but we should (and will do so, if the paper is accepted).\n\n**** _It is not especially clear which version of the Morfessor is used.  Would it be possible to estimate what the more advanced methods would confer in terms of improvement (as far as I understand, StateMorph is better than the Morfessor variant used)?_\n\nWe have reported the use of Morfessor 2.0 implementation (Virpioja et al., 2013) of the baseline method.  You are correct, the experiments show that StateMorph is better than this Morfessor variant.\nWe should indeed test the more recent Morfessor variants, and we could expect more competitive performance.  Some of the later variants would allow us to control the size of the segment lexicon — the fixed lexicon size is a limitation of the baseline method.\nThe standard distribution offers Morfessor version 2.0.6.\n\nTherefore, yes, we would expect improvements from later and more sophisticated variants of Morfessor.  We are in fact testing Morfessor EM+Prune (https://aclanthology.org/2020.lrec-1.486/). We have found version 2.0.7 from one of the authors of this paper, and are comparing the performance with the other algorithms (to be included in the final paper, if accepted).\n\n\n**** _Line 255: \"Due to limited computational resources, we used a smaller instance size\" - could the authors elaborate on this? I suppose they could refer to the material available in the Appendix._\n\nYou are correct, Appendix (A) mentions this, in line 777; at line 798 we describe our resources.  Unfortunately, instances of bigger size do notf fit into our GPUs.  We should (and will) make this more clear in the main text, Section 3.1.\n\n\n**** _Line 286: \"We set the lexicon size for BPE manually—to match the lexicon size produced by the morphological segmentation algorithms. Sizes of the resulting lexicons are shown below.\" - Did the two morphological-based algorithms (Morfessor/StateMorph) lead to similar lexicon sizes? What if BPE had been used to determine the lexicon size for all 3 algorithms; in such a case would the results have changed substantially to make BPE a more competitive method?_\n\nYes, this is a very important delicate point.\nThe lexicon size can be controlled for BPE.  We could indeed go in the opposite direction and allow BPE to determine the lexicon size X that is somehow \"optimal\" — then StateMorph could be pruned to the same size X.  Also, Morfessor EM+Prune could be restricted to size X, and then we could compare the three models in that setting.\n\nWe should emphasize more clearly that at _every lexicon size_ that we explored, BPE always turned out to be inferior.  Therefore, given all the results collected so far, we would expect BPE to be inferior for other settings of the lexicon size.\n\nFinding the \"optimal\" size for BPE — in terms of perplexity — would require an extensive search: as the lexicon size decreases, we expect the perplexity to improve, until some critical point, after which it will again grow.\n\nTo answer the question \"where is BPE's *OWN* optimum vocabuary size\", we would need to determine that for each language, and for each dataset.\n\nWe will also provide a more clear explanation that our morfessor variant is *NOT* customizable w.r.t. the vocabuary size.  Other variants are, and will be tested.\n\n\n**** _Line 367: \"the number of steps to reach convergence is higher for most models with BPE, several times higher for some.\" - indicate which entries in the Table this statement corresponds to, so as to aid readability._\n \nYes, this should (and will) be clarified — it refers to the first column \"Steps (k)\" in Table 1.\n\n**** _Line 415: \"The overall variation is less than 5%, which suggests that Finnish LMs with different segmentations perform comparably on topic classification after fine-tuning.\" - have the authors checked if this variation of less than 5% is not statistically significant?_\n\nYes, the statement in the paper may be a bit confusing.  We should clarify it to state:\n\n\"For the In-Domain measures (Accuracy, F1, MCC), for 3 out of the 6 tested conditions, the morphological segmentation is better than BPE.  For the Out-Of-Domain measures, for 5 out of the 6 tested conditions, morphological segmentation performs better (by within 5 percentage points). From this we conclude that the models — morphological and BPE — perform comparably overall.\"\n\nWe will also check and report the statistical significance, as you suggest, though that was not the intent in the original statement.  But it's a good idea.\n\n\n**** _Line 440: \"the difference between models with morphological segmentation and BPE is larger (though not significantly) in terms of accuracy and F1.\" - it would be useful if the authors could discuss the question of statistically significant (or non-significant) differences._\n\nThese statements are making general comparisons, to say that the performance is comparable.\nIn the final paper, we will provide more careful calculations of statistical significance.\n\n\n**** _It would be useful to determine the acronym MCC (I assume it corresponds to the Matthews correlation coefficient)._\n\nYes, we will add a clarification that this MCC does stand for that.\n\n\n**** _A minor point: \"Finnish and Russian. These languages are chosen 493 as representatives of their respective sub-families— Slavic and Finno-Ugric\" - change the sequence of the last two terms, at line 494 to agree with the order in line 492._\n\nAbsolutely  :) \n\n(Thank you for spotting the unfortunate reversal.)\n\nThank you also for spotting and pointing out the _typos_, we will of course fix those!  Very helpful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The article contains extensive experiments on comparing the different sub-word segmentation algorithms. \n\nThe description of the problem is clear and a set of 4 Research Questions (RQ1 to RQ4)  are set early in the manuscript. \n\nAlso, in each of the downstream tasks discussed in the experimental section,  the authors provide details on the performance of dedicated systems to compare these to their own experiments. \n\nTo my understanding the experiments are thorough and well defined. \n\nEach experimental run  is carried out 3 times which is commendable. \n\nAlso, the sustainability aspect has been looked into by investigating smaller models.\n\nThe linguistic quality is good as well as the method of exposition of material. Also, the system parameters are provided in the Appendix, to aid reproducibility.\n", "reasons_to_reject": "There are a few weaknesses in the manuscript, but I believe this is a sound and interesting paper that would be of use to the EMNLP community and probably also beyond. ", "questions_to_authors": "One question would be why the authors do not refer to the sub-word segmentation algorithm included in Sentencepiece (Kudo et al.), whcih appears to be widely used in recent years.\n\nIt is not especially clear which version of the Morfessor is used. Would it be possible to estimate what the more advanced methods would confer in terms of improvement (as far as I understand, StateMorph is better than the Morfessor variant used)?\n\nLine 255: \"Due to limited computational resources, we used 256\na smaller instance size\" - could the authors elaborate on this? I suppose they could refer to the material available in the Appendix. \n\nLine 286: \"We set the lexicon size for BPE\n287 manually—to match the lexicon size produced by\n288 the morphological segmentation algorithms. Sizes\n289 of the resulting lexicons are shown below.\" - Did the two morphological-based  algorithms (Morfessor/StateMorph) lead to similar lexicon sizes?  What if BPE had been used to determine the lexicon size for all 3 algorithms; in such a case would the results have changed substantially to make BPE a more competitive method?\n\nLine 367: \"the number of steps to reach conver- 367\ngence is higher for most models with BPE, sev- 368\neral times higher for some.\" - indicate which entries in the Table this statement corresponds to, so as to aid readability.\n\nLine 415: \"The overall variation is less than 5%, 415\nwhich suggests that Finnish LMs with different seg- 416\nmentations perform comparably on topic classifi- 417\ncation after fine-tuning.\" - have the authors checked if this variation of less than 5% is not statistically significant?\n\nLine 440: \"the difference between models with mor441\nphological segmentation and BPE is larger (though\n442 not significantly) in terms of accuracy and F1.\" - it would be useful if the authors could discuss the question of statistically significant (or non-significant) differences.\n\nIt  would be useful to determine the acronym MCC (I assume it corresponds to the Matthews correlation coefficient).\n\nA minor point: \"Finnish and Russian. These languages are chosen\n493 as representatives of their respective sub-families—\n494 Slavic and Finno-Ugric\" - change the sequence of the last two terms, at line 494 to agree with the order in line 492. \n\n", "paper_topic_and_main_contributions": "The article discusses the effects of different sub-word segmentation algorithms to transformer-based language models.\n\nThe authors compare three segmentation algorithms, namely BPE, Morfessor and StateMorph, using  two morphologically complex languages, Finnish and Russian. \n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "not aware of missing references."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 249: \"answer RQ1: does a smarter 249\nsegmentation help the LMs learn faster.\" - should there be a question mark at the end?\n\nLine 483: \"The small Russian models\n484 with SMp segmentation is 2% below the regular\n485 model with BPE (170k),\" - check agreement of verb to subject."}}, {"review_id": "MnU3u5F16g", "reviewer": "Reviewer_knhq", "summary": "", "questions": "Line 257, why doesn't it include the next sentence prediction? What does it have to do with instance size 256? This setting may have an effect on the experimental results.\n\nLine 394, the performance of each segmentation method on fine-tuning is also important.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n_The authors study only two morphologically rich languages—Finnish and Russian. It is unclear whether the results apply to other languages. This would prevent the widespread use of these two alternative segmentation methods._\n\n(Similarly to our answer to review VFFv —)\nIdeally, this would be tested on many languages. We present work on two languages, so far, because:\n* We are a small academic team, not a large corporation, and we have limited computing resources.\n* Even for two languages, we already had to train and run hundreds of models.\n* We follow the zero-one-infinity rule: If the phenomenon is observed in two or more highly distinct cases, it is probably of significance (though not guaranteed).\n* For many (other) morphologically rich languages, resources for training unsupervised models also are limited; resources for training downstream tasks are even more limited.\n* Our languages are quite representative: they have very different morphological structures — one agglutinative (FIN), and one fusional (RUS).\n* In fact, we are in the process of testing our hypotheses also on English and Turkish. If the paper is accepted, we will include those results in the additional space provided.\n\nOverall, we do not claim that the study is comprehensive in regard to the number of languages as well as segmentation algorithms. Our claim is this is an important problem to investigate further, with more languages and tasks. \n\nThe goal of our paper is:\n * to present our investigations so far, \n * draw attention to this important issue, and\n * to open the discussion and further exploration.  \nIt is quite expensive (in computational resources) to add each additional language.\n\n\n*** Questions For The Authors:\n\n**** _Line 257, why doesn't it include the next sentence prediction?_\n\nWe are concerned that with the _smaller instance size_ — to which we are constrained by our GPU resources — make it place limitations on the next-sentence prediction task.\nIn future work (when we have more computational resources) we agree, one should test with training on this task, as well.  However, when training with this additional task, we can expect that _both_ models will perform stronger as a result — both BPE _and_ morphologically aware segmentation.\n\nWe should note that: \n1. *Our* goal is not to beat state-of-the-art results on BERT/GPT\n2. For our experiment and our goals:\n    - we are not aiming to compare our model(s) with models trained in \"stronger\" settings.\n    - we want to make sure that the models are A. comparable *among themselves*, B. trainable in reasonable time.\n        - thus we trained them on somewhat simpler task. However, among themselves these models are comparable.\n        - the next-sentence prediction task is not necessary _for comparability_.\n    - can be explored in future work.\n\n\n**** _What does it have to do with instance size 256? This setting may have an effect on the experimental results._\n\nIn line 257: _\"Due to limited computational resources, we used a smaller instance size---256, half of what is typically used for BERT.\"_ ⇒  We did not want to \"waste\" model parameters on a harder task.  Rather we focus only on the segmentation, and the effect of segmentation on the end result.\n\nWe aimed for a more pure experiment - attainable by/available to more researchers: smaller model size.\n\nIn future work, we (and others) can repeat this on bigger models, with more computational resources.\n\n\n**** _Line 394, the performance of each segmentation method on fine-tuning is also important._\n\nWe do not aim for optimal hyper-parameters for the different segmentation methods, since that would make them not comparable among themselves.  We fix the hyper-parameters, and compare the performance on downstream tasks _across the segmentation models_.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors demonstrate that Morfessor and StateMorph are more effective than BPE for small model size settings. This is very useful for practical applications.", "reasons_to_reject": "The authors study only two morphologically rich languages—Finnish and Russian. It is unclear whether the results apply to other languages. This would prevent the widespread use of these two alternative segmentation methods.", "questions_to_authors": "Line 257, why doesn't it include the next sentence prediction? What does it have to do with instance size 256? This setting may have an effect on the experimental results.\n\nLine 394, the performance of each segmentation method on fine-tuning is also important.", "paper_topic_and_main_contributions": "This paper compares the performance of two unsupervised sub-segments Morfessor and StateMorph with BPE on training language models and downstream tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper presents a detailed study and analysis of different sub-word segmentation algorithms and their impact on convergence, generalization and efficiency. Reviewers unanimously found the paper to be sound and exciting with few editorial and experiemental design issues.
Decision: Accept-Main