Splintering Nonconcatenative Languages for Better Tokenization
Authors: Bar Gazit, Shaltiel Shmidman, Avi Shmidman, Yuval Pinter
Accepted at: acl25

Splintering Nonconcatenative Languages for Better Tokenization

Bar Gazit$^{3\lambda}$    Shaltiel Shmidman$^{\delta}$    Avi Shmidman$^{\chi\delta}$    Yuval Pinter$^{3\lambda}$
$^3$Department of Computer Science, $^3$Data Science Research Center $^\chi$Bar-Ilan University
Ben-Gurion University of the Negev Ramat Gan, Israel
Be’er Sheva, Israel $^\delta$DICTA, Jerusalem, Israel
{bargazi@post,uvp@cs}.bgu.ac.il shaltieltzion@gmail.com avi.shmidman@biu.ac.il

Abstract

Common subword tokenization algorithms like BPE and UnigramLM assume that text can be split into meaningful units by concatenative measures alone. This is not true for languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. We present SPLINTER, a pre-processing step which rearranges text into a linear form that better represents such nonconcatenative morphologies, enabling meaningful contiguous segments to be found by the tokenizer. We demonstrate SPLINTER’s merit using both intrinsic measures evaluating token vocabularies in Hebrew, Arabic, and Malay; as well as on downstream tasks using BERT-architecture models trained for Hebrew.

1 Introduction

Large language models (LLMs) have become pivotal in natural language processing (NLP), offering extensive utility across diverse applications. Central to constructing an LLM is producing basic input units from the text sequence, for which subword tokenization is still the standard approach, using methods such as byte-pair encoding (BPE) (Sennrich et al., 2016), WordPiece (Schuster and Nakajima, 2012), and UnigramLM (Kudo, 2018). However, subword tokenizers exhibit diminished effectiveness in nonconcatenative languages (NCLs) such as Hebrew and Arabic (Klein and Tsarfaty, 2020). While tokenizers assume linear segmentation of words, NCLs’ units of meaning are typically intertwined within words, and cannot be separated linearly, as the root letters are not adjacent (Khaliq and Carroll, 2013). A Hebrew example is provided in Table 1. In the Hebrew word לְאָוֹד ‘la’avod ‘to work’, the root letters are ‘אוֹד’, ‘א’, and ‘ל’, placed in an infinitive morphological template manifested by the locations of ‘ל’ and ‘א’. This characteristic forces linear tokenizers to split words into morphologically-incoherent tokens, losing the downstream models’ ability to generalize across various forms of the same lemma, and eventually reducing model performance when applied to a large variety of tasks such as text generation and translation (Keren et al., 2022; Levi and Tsarfaty, 2024; Shmidman et al., 2024).

We present SPLINTER, a statistical algorithm for linearizing NCL text through rearranging the text sequence by iteratively pruning characters from words, with the intent of isolating characters representing template forms. The manipulated text can then be input into any ordinary linear tokenizer for processing as usual, adapting the NCL data into the morphologically-concatenative input BPE and its like expect. We show that vocabularies and models trained over SPLINTER-processed text outperform those starting from raw NCL text on both intrinsic and extrinsic measures in Hebrew, Arabic, and Malay.1

<table>
  <tr>
    <th>עברית על המנורה.</th>
    <th>‘I worked on the presentation.’</th>
  </tr>
  <tr>
    <th>הhard work paid off.</th>
    <th>‘The hard work paid off.’</th>
  </tr>
</table>

Table 1: Examples of Hebrew text (read from right to left) exemplifying its nonconcatenative morphology. The root נִבְר ‘work’ appears in both sentences, but in (a) it comprises a linear segment of the text whereas in (b) it is broken by the templatic character ו.

2 Tokenizing with Splinters

When designing our approach, our goal was to create a relinearized sequence for words in NCL languages, while adhering to several constraints. One constraint is that the transformation must be reversible, ensuring that the new representation can always be converted back to the original text. In

1https://github.com/MeLeLBGU/Splintering.
Standard Flow:

Input        Tokenizer        Embedding
Original text   Segmented text

Splinter Flow:

Input        SPLINTER        Tokenizer        Embedding
Original text   Splintered text   Segmented text

Figure 1: Overview of a Hebrew language model pipeline using the example ‘to work’: standard flow vs. incorporating SPLINTER. This figure is a real example for the differences in tokenization using a BPE tokenizer with a vocab size of 2000. Gray boxes are ordered from right to left.

addition, we aimed to develop a tool that would integrate smoothly with existing tokenizers, ensuring seamless adoption without requiring modifications in their implementation. Additionally, the method should be applied only to the intended languages, without affecting the entire text. We also considered that the method should be adaptable to two distinct use cases: (1) models trained primarily in an NCL language, where the majority of the vocabulary belongs to that language (e.g., DictaBERT (Shmidman et al., 2023) has a large vocabulary size of 128K); and (2) large-scale multilingual LLMs, where most tokens are allocated to English and only a small portion is left for the NCL language (e.g., GPT-4o, which has approximately 2.3K tokens allocated for Hebrew). Thus, the method should be effective across different vocabulary sizes.

We develop SPLINTER as a pre-tokenization step designed to address the challenges of subword tokenization in NCLs. The core idea of the algorithm is that in certain NCLs, many words are formed by embedding root letters into specific morphological templates. For instance, the Hebrew word לְמִיּוֹר is derived from the root מִיּוֹר placed in the template _1_5. Since there are far fewer templates than roots, template letters tend to appear in specific positions within a word more consistently than root letters do (e.g., for לְמִיּוֹר: “0:ל”;“3:י”). Empirically, we observed that in Hebrew and Arabic, when a word is longer than 3 characters, there is always at least one deletion that, when applied, transitions the word to a different, existing template while preserving the same root.2 For instance, the word לְמִיּוֹר turns into לְמִיּוֹר when the template letter 1 is removed. By repeating this process iteratively, the word eventually reduces to only its root letters. This method can be seen as a way to isolate the root letters from the template.

In this method, NCL words are transformed into a sequence of single-letter reductions, with the goal of rearranging them to better align with existing subword tokenizers. This is achieved by expanding the language’s alphabet to include not only its original letters but also single-letter reductions, where each reduction consists of a letter paired with the index from which it was removed.3 From another perspective, this method explores the impact of expanding a language’s alphabet to enhance word representation. We illustrate the high-level application of SPLINTER into the NLP pipeline in Figure 1.

2.1 SPLINTER Reduction Map Creation

As mentioned above, the SPLINTER pre-tokenization step processes a word by iteratively applying single-letter reductions. To achieve this, we must determine which reduction to perform at each iteration. We propose an algorithm that analyzes a given corpus and generates a mapping, where the keys represent word lengths and the

2However, not every letter’s deletion will necessarily result in a valid template. For example, removing 1 from לְמִיּוֹר would produce the non-word לְמִיּוֹר.

3In practice, we found that using negative indices for the latter half of a word’s characters both aligns well with the suffixing nature of certain morphemes and reduces the resulting alphabet by 15%. Not shown in our examples for simplicity. Further low-level details are provided in Appendix A.
values are ordered lists of reductions, ranked from most to least frequent.

Converting the Corpus to Unigram Frequencies
The algorithm begins by processing a corpus in the target language, which undergoes several preprocessing steps. First, all diacritics are removed from the text (for Hebrew only). The text is then split into words using the following regex pattern:
\.\s|\n|-|.|:"|\(\)
Next, words that appear fewer than 10 times in the corpus are discarded, along with any words containing letters from other languages. Additionally, we normalized final and non-final Hebrew letters to maintain consistency in root-based word connections. Specifically, all final letters were replaced with their non-final forms, and vice versa when applicable (i.e., in cases where a non-final form occurs in the final position of a word, mostly in borrowed words like קetchup ‘ketchup’). This adjustment helps preserve morphological relationships, such as between הולך holex ‘(he) is walking’ and הולכים holxim ‘(they) are walking’, both derived from the root הלך ‘walk’. After this transformation, these words become הולך and הולכים, respectively, both clearly retaining the root הולך. The reverse transformation ensures that distinctions between different word groups are still maintained, keeping the original text recoverable.4 Finally, each remaining word is assigned its frequency count in the corpus, resulting in a unigram dictionary mapping words to their respective frequencies.

Scoring the Reductions
With the unigram dictionary constructed, we group the words by their length and iterate over them. For each word \( w_i \) of length \( k \geq 4,\)5 we evaluate all possible single-character reductions \( \{r_0, r_1, \ldots, r_{k-1}\} \), assigning each reduction a score equal to the frequency of the resulting word \( w_i^{r_j} \) in the corpus, giving lower scores to rare words, and zero score if the resulting word is not in the dictionary. For example, given the word \( w_j = \text{lמ"ד} limud ‘studying’, we examine these possible reductions: \( r_0 = "0:ב" \), \( r_1 = "1:נ" \), \( r_2 = "2:מ" \), \( r_3 = "3:י" \), \( r_4 = "4:ד" \), which produce the following respective words: \( w_j^{r_0} = \text{lמ"ד} \), \( w_j^{r_1} = \text{lמ"ד} \), \( w_j^{r_2} = \text{lיד"מ} \), \( w_j^{r_3} = \text{lיד"מ} \), \( w_j^{r_4} = \text{lיד"מ} \), and score each of

4Arabic also has final and non-final letter forms, but their selection occurs deterministically based on context. From a Unicode perspective, both forms share the same underlying character, eliminating the need for manual conversion.
5We examine words with at least four characters since most Semitic roots contain three characters.

![Intersection percent between Vanilla BPE and BPE + SPLINTER by vocabulary size for Hebrew, Arabic, and Malay. Vocabulary size is presented on a logarithmic scale.](page_820_164_579_377.png)

Figure 2: Intersection percent between Vanilla BPE and BPE + SPLINTER by vocabulary size for Hebrew, Arabic, and Malay. Vocabulary size is presented on a logarithmic scale.

the reductions the frequency of the resulting word. The score is summed for each word length \( k \), per each reduction \( r \), so as we iterate through the words, we build a map that tracks, for each word length, which reductions produced valid words, along with their scores. The resulting map is then sorted so that reductions are ranked from most to least frequent.

Following this step, the map is used as a starting point for a second iteration over the corpus. For each word of length \( k \), we attempt to apply reductions from the map, traversing it by descending frequency score. The first successful reduction that produces an existing word is recorded in a new frequency counter, using the same scoring method as in the first iteration, while all other possible reductions for that word are ignored. This step ensures that the most frequent reduction is prioritized, aligning with our goal of removing template letters first, as they tend to appear more frequently in consistent locations than individual root letters. Pseudocode for the full map creation algorithm is available in Appendix B.

2.2 Tokenizing with Splinters
With the list of reductions for each word length now sorted from highest to lowest scores, we apply it during inference for each word we encounter in a corpus or in task data, using a simple heuristic inspired by beam search to produce high-quality relinearizations. We build a scored selection tree
<table>
  <tr>
    <th>Tokenizer</th>
    <th>Type</th>
    <th>Cognitive plausibility</th>
    <th>Rényi efficiency</th>
    <th>Tokens per word</th>
    <th>4+ token words</th>
    <th>1-char tokens</th>
    <th>Distinct Neighbors</th>
  </tr>
  <tr>
    <td rowspan="2">BPE</td>
    <td>Vanilla</td>
    <td>0.157</td>
    <td>0.524</td>
    <td><b>1.146</b></td>
    <td>0.53%</td>
    <td>6.00%</td>
    <td>2674</td>
  </tr>
  <tr>
    <td>SPLINTER</td>
    <td><b>0.179</b></td>
    <td><b>0.527</b></td>
    <td>1.165</td>
    <td>0.98%</td>
    <td>6.81%</td>
    <td><b>2640</b></td>
  </tr>
  <tr>
    <td rowspan="2">UnigramLM</td>
    <td>Vanilla</td>
    <td>0.151</td>
    <td><b>0.505</b></td>
    <td><b>1.162</b></td>
    <td>0.56%</td>
    <td>9.42%</td>
    <td>2440</td>
  </tr>
  <tr>
    <td>SPLINTER</td>
    <td><b>0.171</b></td>
    <td>0.485</td>
    <td>1.176</td>
    <td>1.00%</td>
    <td>12.46%</td>
    <td><b>2308</b></td>
  </tr>
</table>

Table 2: Intrinsic benchmark results for Hebrew on a vocab size of 128K. The tokenizers were evaluated using the HeDC4 corpus. Bold values indicate better performance between Vanilla and SPLINTER.

where the root is the full character sequence of the word scored at 1.0, and at every node select the \( b \) highest-scoring applicable reductions according to the list, keeping the product of reduction scores so far as scores in the next level of nodes. Once either the word length reaches the minimum of 3, or the depth of the tree reaches \( d \), the reduction that started the highest-scoring path is selected and applied, and the process restarts with the new reduced word.\footnote{We set \( b = d = 3 \).}

As described above, each reduction is encoded as a new composite character. This transforms the word into a new representation as a sequence of this enriched alphabet, consisting of the original characters with the addition of the composite ones, which is then used as input for standard subword tokenization methods. From the perspective of the tokenizer and the entire language model, both training and inference operate on this relinearized sequence represented by this new alphabet. In generation mode, a character sequence over the new alphabet is decoded back into reductions, which are then applied sequentially to construct words.

\section*{3 Intrinsic Evaluation}

To evaluate the effects of using SPLINTER as a pre-processing step before tokenization, we trained multiple tokenizers on raw text from nonconcatenative languages and their SPLINTER-treated counterparts. We examined the performance of both the bottom-up BPE tokenization algorithm, which works by iteratively merging tokens based on corpus co-occurrence statistics, and the top-down UnigramLM approach, which starts with a very large vocabulary and iteratively removes from it tokens which contribute minimally to the corpus’s likelihood. We train over a wide range of vocabulary sizes in order to assess the utility of SPLINTER in multiple scenarios: from multilingual models which can allocate roughly 1,000 tokens for a given language, to dedicated monolingual models with room for two orders of magnitude more tokens. We selected three languages for our experiments: Hebrew and Arabic are Semitic languages featuring root-template morphology as discussed above, while Malay is an Austronesian language rich in circumfixes and infixes—morphemes which break either the stem or the affix when forming the composite inflection. We computed the SPLINTER operations for each language using the reductions map generated from the November 2023 Wikipedia dump of the respective language. The Wikipedia dump sizes were 1.9GB for Hebrew, 3.0GB for Arabic, and 0.4GB for Malay, and were downloaded using the Hugging Face “datasets” library. We trained the tokenizers on the same Wikipedia dump used for SPLINTER training, using Google’s SentencePiece library with default settings, except for the tokenizer type (UnigramLM or BPE) and vocabulary size (800, 1K, 2K, 10K, 32K, 64K, 128K).

We qualitatively examine the differences between the Vanilla tokenizer and the SPLINTER-enhanced one.\footnote{All the examples in this paragraph are actual tokenizer outputs, comparing Vanilla BPE with SPLINTER BPE, both using a vocabulary size of 2K tokens.} Figure 1 illustrates such differences: the Vanilla BPE tokenizer linearly segments the word לְמִי ‘to work’ into “ל” “מ” “י” and “לִי”, thereby splitting the root מִי across two tokens. In contrast, the non-linear BPE + SPLINTER tokenizer separates the word into two tokens: one for the template letters “3:1;0:5” and another for the root “לִי”. A similar pattern is seen in the noun בֵּשֶׂם xišuv ‘computation’, where the Vanilla BPE tokenizer linearly segments it into “ב” and “שֶׂם”, while BPE + SPLINTER produces a token for template letters “1:;3:1” and another token to the root “שֶׂם”. Likewise, for the definite adjective הַפָּסִית ha-pašut ‘the simple.sg.masc’, the Vanilla tokenizer segments it into three tokens: “הַ”, “פָּסִית” and “ת”, whereas BPE + SPLINTER separates it into “0:;7,3:1” and the root “פָּסִית”.
<table>
  <tr>
    <th>Vocab size</th>
    <th>Type</th>
    <th>Cognitive plausibility</th>
    <th>Rényi efficiency</th>
    <th>Tokens per word</th>
    <th>4+ token words</th>
    <th>1-char tokens</th>
    <th>Distinct Neighbors</th>
  </tr>
  <tr>
    <td rowspan="2">128K</td>
    <td>Vanilla</td>
    <td>0.157</td>
    <td>0.524</td>
    <td><b>1.146</b></td>
    <td>0.53%</td>
    <td>6.00%</td>
    <td>2674</td>
  </tr>
  <tr>
    <td>SPLINTER</td>
    <td><b>0.179</b></td>
    <td><b>0.527</b></td>
    <td>1.165</td>
    <td>0.98%</td>
    <td>6.81%</td>
    <td><b>2640</b></td>
  </tr>
  <tr>
    <td rowspan="2">64K</td>
    <td>Vanilla</td>
    <td>0.181</td>
    <td>0.565</td>
    <td><b>1.224</b></td>
    <td>0.75%</td>
    <td>7.05%</td>
    <td>4272</td>
  </tr>
  <tr>
    <td>SPLINTER</td>
    <td><b>0.206</b></td>
    <td><b>0.567</b></td>
    <td>1.248</td>
    <td>1.43%</td>
    <td>8.35%</td>
    <td><b>4188</b></td>
  </tr>
  <tr>
    <td rowspan="2">32K</td>
    <td>Vanilla</td>
    <td>0.201</td>
    <td>0.610</td>
    <td><b>1.336</b></td>
    <td>1.12%</td>
    <td>8.76%</td>
    <td>5754</td>
  </tr>
  <tr>
    <td>SPLINTER</td>
    <td><b>0.223</b></td>
    <td><b>0.612</b></td>
    <td>1.365</td>
    <td>2.04%</td>
    <td>10.66%</td>
    <td><b>5631</b></td>
  </tr>
  <tr>
    <td rowspan="2">10K</td>
    <td>Vanilla</td>
    <td>0.196</td>
    <td>0.690</td>
    <td><b>1.606</b></td>
    <td>2.28%</td>
    <td>13.26%</td>
    <td>5652</td>
  </tr>
  <tr>
    <td>SPLINTER</td>
    <td><b>0.226</b></td>
    <td>0.687</td>
    <td>1.651</td>
    <td>3.86%</td>
    <td>16.56%</td>
    <td><b>5555</b></td>
  </tr>
  <tr>
    <td rowspan="2">2K</td>
    <td>Vanilla</td>
    <td>0.149</td>
    <td>0.760</td>
    <td><b>2.137</b></td>
    <td>7.44%</td>
    <td>25.90%</td>
    <td>1855</td>
  </tr>
  <tr>
    <td>SPLINTER</td>
    <td><b>0.207</b></td>
    <td>0.756</td>
    <td>2.270</td>
    <td>11.57%</td>
    <td>33.32%</td>
    <td><b>1815</b></td>
  </tr>
  <tr>
    <td rowspan="2">1K</td>
    <td>Vanilla</td>
    <td>0.109</td>
    <td><b>0.774</b></td>
    <td><b>2.436</b></td>
    <td>13.33%</td>
    <td>34.47%</td>
    <td>925</td>
  </tr>
  <tr>
    <td>SPLINTER</td>
    <td><b>0.184</b></td>
    <td>0.763</td>
    <td>2.713</td>
    <td>22.82%</td>
    <td>48.59%</td>
    <td><b>895</b></td>
  </tr>
  <tr>
    <td rowspan="2">800</td>
    <td>Vanilla</td>
    <td>0.102</td>
    <td><b>0.779</b></td>
    <td><b>2.543</b></td>
    <td>16.03%</td>
    <td>37.70%</td>
    <td>734</td>
  </tr>
  <tr>
    <td>SPLINTER</td>
    <td><b>0.182</b></td>
    <td>0.762</td>
    <td>2.890</td>
    <td>28.70%</td>
    <td>54.37%</td>
    <td><b>705</b></td>
  </tr>
</table>

Table 3: Intrinsic benchmark results for Hebrew using BPE tokenizer with different vocabulary sizes. The tokenizers were evaluated using the HeDC4 corpus. Bold values indicate better performance between Vanilla and SPLINTER.

For direct evaluation of the tokenizer vocabularies, independent of further language model architecture and training, we follow the analytical procedures collected in Uzan et al. (2024), adding pairwise comparative measures from other sources as well.

Vocabulary overlap First, we validate that SPLINTER provides models with vocabularies that are different enough from raw-text tokenizers. Hypothetically, if many common words are learned in full as single tokens from raw text, there is no need for a special pre-processing step to account for an edge case. However, in Figure 2 we show that this is not the case. In all three languages, the maximal vocabulary size of 128K stays at an intersection level below 75%, with the slope of added shared token rate declining to near constant. Moreover, even if we assume a linear extrapolation rate, the intersection rate would only exceed 85% at a vocabulary size of around 780K, which is exceptionally large and is not used even in SOTA English-dominated LLMs like GPT-4o. We note that we used a generous calculation for the intersection rate, as not all tokens in SPLINTER-enhanced tokenizers can be directly compared to those in a regular tokenizer. For instance, a token representing the reduction “0:5;3:1” cannot be linearly converted into a standard token. To make the comparison as permissive as possible, we applied the reductions within the token (e.g., converting “0:5;3:1” into “ה””) and counted it towards the intersection if the resulting token existed in the Vanilla tokenizer’s vocabulary.

As a result, the actual intersection percentage may be significantly lower than reported. We conclude that SPLINTER-enhanced tokenizers produce substantially different vocabularies at all stages of the vocabulary creation process.

Cognitive plausibility We use the metric introduced in Beinborn and Pinter (2023) to measure the correlation of the tokenizer output with the response time and accuracy of human performance on a lexical decision task. This metric is based on the hypothesis that an effective tokenizer encounters difficulty with character sequences that are also challenging for humans, and vice versa. We use the Hebrew cognitive plausibility dataset (HeLP; Stein et al., 2024) to evaluate both the BPE and UnigramLM tokenizers. Each tokenizer was compared across seven vocabulary sizes, with the Vanilla tokenizer evaluated against the SPLINTER-enhanced tokenizer. Following Uzan et al. (2024), we report the average of the absolute value correlation scores across the four linguistic setups (word/nonword × accuracy/response time). Higher scores mean better correlation with human performance.

As shown in Table 2 and Table 3, SPLINTER-enhanced tokenizers consistently correlate better with human lexical processing patterns, across all vocabulary sizes in both BPE and UnigramLM. These results suggest that downstream language models trained on SPLINTER output would reach higher scores in morphological segmentation tasks, which we evaluate in §4. We note that, unlike the following metrics, cognitive plausibility does not
<table>
  <tr>
    <th>Language</th>
    <th>Vocab size</th>
    <th>Type</th>
    <th>Rényi efficiency</th>
    <th>Tokens per word</th>
    <th>4+ token words</th>
    <th>1-char tokens</th>
    <th>Distinct Neighbors</th>
  </tr>
  <tr>
    <td rowspan="2">Hebrew</td>
    <td>128K</td>
    <td>Vanilla</td>
    <td>0.509</td>
    <td><b>1.119</b></td>
    <td><b>0.57%</b></td>
    <td><b>6.01%</b></td>
    <td>1463</td>
  </tr>
  <tr>
    <td></td>
    <td>SPLINTER</td>
    <td><b>0.511</b></td>
    <td>1.134</td>
    <td>0.92%</td>
    <td>6.61%</td>
    <td><b>1460</b></td>
  </tr>
  <tr>
    <td rowspan="2">Arabic</td>
    <td>128K</td>
    <td>Vanilla</td>
    <td>0.427</td>
    <td><b>1.134</b></td>
    <td><b>0.55%</b></td>
    <td><b>7.54%</b></td>
    <td>1444</td>
  </tr>
  <tr>
    <td></td>
    <td>SPLINTER</td>
    <td><b>0.430</b></td>
    <td>1.158</td>
    <td>1.07%</td>
    <td>7.84%</td>
    <td>1520</td>
  </tr>
  <tr>
    <td rowspan="2">Malay</td>
    <td>128K</td>
    <td>Vanilla</td>
    <td>0.471</td>
    <td><b>1.088</b></td>
    <td><b>0.55%</b></td>
    <td><b>4.45%</b></td>
    <td>337</td>
  </tr>
  <tr>
    <td></td>
    <td>SPLINTER</td>
    <td><b>0.479</b></td>
    <td>1.135</td>
    <td>1.56%</td>
    <td>5.98%</td>
    <td>354</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>2K</td>
    <td>Vanilla</td>
    <td>0.756</td>
    <td><b>2.150</b></td>
    <td><b>14.65%</b></td>
    <td><b>29.23%</b></td>
    <td>1215</td>
  </tr>
  <tr>
    <td></td>
    <td>SPLINTER</td>
    <td><b>0.770</b></td>
    <td>2.724</td>
    <td>28.79%</td>
    <td>43.37%</td>
    <td><b>1055</b></td>
  </tr>
</table>

Table 4: Intrinsic benchmark results using BPE tokenizer for Hebrew, Arabic and Malay on a vocab sizes of 2K and 128K. The tokenizers were evaluated using the Wikipedia corpus of their respective language. Bold values indicate better performance between Vanilla and SPLINTER.

focus on the tokenizer’s effectiveness as a text compression tool, offering a different perspective. Additional UnigramLM results provided in Appendix C.

Token distribution statistics  We collected distributional data for the various tokenizers using the following corpora: the HeDC4 corpus (Shalumov and Haskey, 2023) was used in Hebrew experiments looking into vocabulary size and tokenizer type (BPE vs UnigramLM), with a 10% shuffled sample (seed = 42) taken from its original 45GB corpus. For Hebrew’s cross-linguistic comparison with Arabic and Malay, we used the respective November 2023 Wikipedia dump of each of the languages.8 Based on these corpora, we report the Rényi efficiency score (Zouhar et al., 2023), as well as several other surface statistics. We report the Hebrew-specific results in Table 2 and Table 3, and crosslinguistic results in Table 4. Rényi efficiency has been proposed as an indicator of downstream task performance, such as BLEU scores in machine translation. This metric penalizes token distributions that are overly skewed toward either very high- and/or very low-frequency tokens. However, a recent study (Cognetta et al., 2024) suggests that this metric can be manipulated to produce higher scores while degrading actual performance. This highlights the importance of using multiple indicators from different perspectives to make an informed assessment of a tokenizer’s potential impact on downstream tasks.

8Prior to these experiments, we conducted a preliminary token distribution statistics evaluation of SPLINTER on a small-sized pre-modern Hebrew corpus (Gershuni and Pinter, 2022), yielding results consistent with those observed later on the other corpora. Details can be found in Appendix D.

Rényi efficiency scores for Hebrew were consistent across both the HeDC4 and Wikipedia corpora, and in general show minimal differences between SPLINTER tokenizers and Vanilla tokenizers across most tokenization settings. In BPE, the Vanilla tokenizers achieved slightly better results at lower vocabulary sizes (≤2K), while in UnigramLM, the trend was reversed, with SPLINTER tokenizers achieving slightly higher scores at lower vocabulary sizes (≤2K), and the Vanilla tokenizers achieving slightly higher scores at larger vocabulary sizes (≥10K). In Arabic and Malay, SPLINTER’s results were again very close to Vanilla’s, with SPLINTER tokenizer scores slightly higher in both large and small vocabulary sizes. The overall Rényi efficiency results suggest minimal impact on token distribution, with SPLINTER sometimes slightly improving it and other times slightly reducing efficiency.

For further intrinsic evaluation, we examined three corpus-level indicators of tokenizer compression efficiency: the average number of tokens per word (also known as subword fertility), the percentage of words tokenized into four or more tokens, and the percentage of single-character tokens. All three serve as indicators of compression efficiency, with lower values generally indicating better compression.

Across all corpora, tokenizer types, vocabulary sizes, and languages, the results consistently show that adding SPLINTER to the tokenizer reduces its compression efficiency. This result is important, as in generative LMs, for instance, less efficient compression requires more iterations to generate the
![Distinct neighbor counts for top 200 tokens in BPE and SPLINTER + BPE for a window of 2 on each side, vocabulary size 128K, HeDC4 corpus.](page_153_120_1142_496.png)

Figure 3: Distinct neighbor counts for top 200 tokens in BPE and SPLINTER + BPE for a window of 2 on each side, vocabulary size 128K, HeDC4 corpus.

same text, leading to higher computational costs. Therefore, this trade-off should be considered when applying SPLINTER in an LLM tokenizer. That being said, Schmidt et al. (2024) found that tokenization should not be viewed principally from the compression perspective. Improved compression does not always correlate with better downstream task performance, and in some cases, it may even degrade it. This again emphasizes the importance of considering multiple perspectives when assessing tokenization quality.

Contextual coherence The next aspect we examine is the contextual coherence (Yehezkel and Pinter, 2023) of the tokens produced by each tokenizer, as measured by the number of distinct neighbors each token encounters within a window of \( k \) tokens from each side (we choose \( k = 2 \)). This measurement is motivated by the main downstream application scenario of vocabularies—contextualized embeddings in language models. The fewer contexts a token appears in, the more likely a model is to learn a meaningful embedding for it over a corpus, as it offers better differentiation between token environments. Figure 3 displays the number of neighbors for the top 200 tokens in each tokenizer as ranked according to this quantity. We present the average number of distinct neighbors across the vocabulary in Tables 2, 3, and 4 as a measure of efficiency—not for text compression, but for downstream LM training. As shown in these tables, SPLINTER-based tokenizers consistently produce fewer distinct neighbors than Vanilla in Hebrew, regardless of vocabulary size, tokenization algorithm, or corpus. However, when evaluated on Wikipedia corpora, which were also used for training the tokenizers themselves, the differences were less pronounced. SPLINTER achieved a lower average number of distinct neighbors only at the 2K vocabulary size, while at 128K, its results were nearly identical to the baseline in Hebrew, and in Arabic and Malay, the baseline outperformed SPLINTER. These mixed results suggest that the impact of SPLINTER on downstream tasks may depend on vocabulary size, with its advantages being more evident in smaller vocabularies dedicated to an NCL language, while in larger vocabulary sizes, the baseline tokenizer may perform better.

4 Language Modeling with Splinters

As noted above, standard subword tokenizers are suboptimal for nonconcatenative languages such as Hebrew and Arabic. Thus, we proceed now to evaluate SPLINTER’s impact on downstream NLP tasks for Hebrew. Specifically, we evaluate Prefix Segmentation and Syntactic Parsing (Sade et al., 2018; Bareket and Tsarfaty, 2021; Zeldes et al., 2022), and Question Answering (Cohen et al., 2023).

We begin with the open DictaBERT BERT-base model (Shmidman et al., 2023), which delivers current SOTA-level performance on the aforementioned tasks (Shmidman et al., 2024) in Hebrew. We then pre-train a new BERT-base model using the same corpus and training parameters as DictaBERT, with the only modification being
<table>
  <tr>
    <th>Model</th>
    <th>QA (F1)</th>
    <th>QA (EM)</th>
    <th>Syntax (LAS)</th>
    <th>Seg (Acc)</th>
  </tr>
  <tr>
    <td><i>Train</i></td>
    <td>27K examples</td>
    <td></td>
    <td>82K words</td>
    <td>1.2M words</td>
  </tr>
  <tr>
    <td><i>Dev</i></td>
    <td>1.5K examples</td>
    <td></td>
    <td>7K words</td>
    <td>—</td>
  </tr>
  <tr>
    <td><i>Test</i></td>
    <td>1.5K examples</td>
    <td></td>
    <td>7K words</td>
    <td>307K words</td>
  </tr>
  <tr>
    <td>DictaBERT</td>
    <td>72.9</td>
    <td>63.6</td>
    <td><b>89.0</b></td>
    <td>99.1</td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>74.4</b></td>
    <td><b>65.4</b></td>
    <td><b>89.0</b></td>
    <td><b>99.3</b></td>
  </tr>
</table>

Table 5: Performance comparison of the existing SOTA Hebrew BERT (DictaBERT) with our newly-pretrained SPLINTER-based Hebrew BERT, across three downstream Hebrew NLP tasks.

the SPLINTER-processed tokenization. We then fine-tune the new base model for the aforementioned tasks, and evaluate performance vis-a-vis fine-tuning the original DictaBERT. We follow the same task parameters defined by Shmidman et al. (2024) and Shmidman et al. (2023) for the three tasks. We present the results in Table 5, noting that these should be viewed as lower-bound estimates, since we retained the original DictaBERT training parameters and only tuned the SPLINTER pre-tokenization step. With additional hyperparameter tuning of the language model itself, performance could potentially improve further.

Results Regarding both of the Question-Answer benchmarks, we find that the SPLINTER-based model provides a substantial boost in performance. We believe that this indicates that the SPLINTER-based tokenization provides the model with a substantially stronger ability to process and understand the message of a Hebrew text, and thus to achieve superior performance on this high-level textual challenge. On the sentence-level task of syntactic parsing, the performance of the SPLINTER-based model is essentially the same as the existing DictaBERT model, indicating diminished advantage for manipulation at the character level. However, for the nearly-saturated task of labeling prefix segmentation at the character level, SPLINTER-based tokenization provides over 20% reduction in errors, highlighting the effectiveness of the data-driven pattern-finding algorithm it employs.

Example 1: Question-Answer Task An illustrative example in which SPLINTER’s tokenization architecture allows it to succeed where DictaBERT fails is the following question from the QA corpus, regarding the date of a certain archaeological excavation. The relevant part of the input text is comprised of the following three sentences (provided here in English translation):

“In 1913 he purchased the tract of land that covers most of the eastern slope of the City of David in Jerusalem, and persuaded the French Jewish archaeologist Raymond Weill to conduct excavations there. This was done in response to the scandalous excavation of Montagu Parker in the City of David in 1911. Rothschild returned and financed another season of excavations, in 1923–1924, under Weill’s direction.”

The following question is then posed to the system: “In what year were the first excavations conducted in the City of David?”

DictaBERT incorrectly answers 1923 (as per the third sentence), while SPLINTER correctly answers 1911 (as per the second sentence). DictaBERT’s failure to pull the correct answer from the second sentence likely stems from the fact that in the original Hebrew of that sentence, the words “excavation of Montague Parker” are phrased using the singular form “excavation”, with a suffixed possessive pronoun (חפירה xafirato). This word differs from the non-suffixed plural term “excavations” used in the question (חפירות xafirot). Crucially, the difference between the terms is not just the suffixed 1o at the end, but also a letter from the middle of the base term. It is precisely discrepancies such as these that SPLINTER aims to solve. It stands to reason that DictaBERT’s incorrect answer stems from its inability to see the connection between these two terms; thus, it was unable to understand the relevance of the second sentence, and instead took its incorrect answer from the subsequent sentence, which includes an exact match for the term “excavations”. In contrast, thanks to its new tokenization architecture, SPLINTER recognizes the connection between the two disparate terms and correctly answers “1911”.

Example 2: Segmentation Task An analysis of the results on the prefix segmentation task serves to underscore the relative advantage of SPLINTER
over DictaBERT when it comes to concatenations of multiple proclitics.

The Hebrew language allows prepositions, conjunctions, relativizers, and definite articles to be prepended as proclitics. In the simple case a single proclitic is prepended to a given word; such cases are well attested throughout the corpus, and such combinations are often tokens in and of themselves in the DictaBERT vocabulary. However, Hebrew also allows concatenations of multiple proclitics, resulting in a string of 2–5 letters prepended to a single word. This results in over 100 possible permutations of proclitic letters that can be prepended to a Hebrew word; however, any given instance of a multiletter prefix on a given word tends to be sparsely attested and thus not included as an independent token within the DictaBERT vocabulary.

Indeed, a review of the segmentation task results reveals numerous cases of words with multi-letter prefixes where DictaBERT cautiously removes too few letters, anticipating the usual case of short and simple prefixes, while SPLINTER correctly segments the full set of concatenated proclitics.

Actual examples of such from the test corpus include the following Hebrew words: שְׁלֵמִיִּם ‘and that his students.masc’ (2-letter prefix); שְׁכָנֶה ‘that are of carbohydrates’ (2-letter prefix); שְׁמוֹרִים ‘and that are being interviewed.masc’ (2-letter prefix); וּמִקְדָּשׁ ‘and from a length’ (2-letter prefix); וּבַתְּרֵי ‘while in one.masc’ (3-letter prefix). In all of these cases, DictaBERT’s segmentation is one letter short, while SPLINTER correctly identifies the boundary between the proclitics and the primary word. Indeed, none of these words appear in DictaBERT’s vocabulary.

In a few more unusual cases, we find the converse: DictaBERT removes too many letters for the prefix. This happens with the words מִיּוֹ ‘and from bids’ (2-letter prefix) and בִּקְרֵי ‘that protruded.pl’ (1-letter prefix). These two words, too, are not found in DictaBERT’s vocabulary, and in its struggle to parse the word, DictaBERT ends up incorrectly assuming a concatenation of an additional proclitic beyond the ones actually present. SPLINTER handles both correctly.

Our review of the results on the segmentation task thus highlights how SPLINTER’s architecture allows it to successfully cope with the possibility of multi-proclitic concatenations even when the specific combination of prefix+word is seldom tested in naturally occurring Hebrew corpora, and not found within the vocabulary of the model.

5 Conclusion

In this work, we introduced SPLINTER, a novel pre-processing method for subword tokenizers designed to improve downstream performance on nonconcatenative languages (NCLs). By applying an iterative reduction process, SPLINTER restructures words in a way that better aligns with existing subword tokenizers. Our approach was designed with key constraints in mind: ensuring lossless transformation, compatibility with existing tokenization frameworks, and applicability across different vocabulary sizes and model types, whether for an NCL, a single-language LM, or a multilingual model based on English with a limited number of tokens allocated for nonconcatenative languages.

Through intrinsic evaluations, we demonstrated that SPLINTER-enhanced tokenizers exhibit distinct vocabulary distributions compared to Vanilla tokenizers. Cognitive plausibility metrics indicated that SPLINTER improves alignment with human-like lexical processing, while our analysis of compression-related metrics revealed that SPLINTER trades off slight reductions in compression efficiency for potentially better linguistic representation.

Our downstream evaluation highlights SPLINTER’s impact, particularly on higher-level NLP tasks such as question answering and on character-critical tasks such as prefix segmentation. The intermediate syntactic level appears to be less affected by the nonconcatenativity of Hebrew text.

In future work, we will extend the downstream evaluation to Arabic and other Semitic languages, as well as more languages exhibiting non-templatistic nonconcatenative phenomena. Additionally, we plan to evaluate the performance of a large multilingual generative model on various tasks after incorporating SPLINTER, examining its effectiveness in a broader linguistic context.

Limitations

Rearranging text in order to improve representation of nonconcatenative features is a hard high-level problem, and we believe our work is a first step towards remedying this inherent mismatch between modeling and language data. However, our concrete algorithm is still not universally-applicable, as shown by the difference between results on Semitic
languages and on Malay. Primarily, we attribute this to the property where each single-character pruning action must result in a valid corpus word, mostly limiting the scope of linearization to templatic morphology rather than also including infixation and circumfixation.

In addition, the increase in performance comes at the cost of less efficient token sequences, as found in our fertility analysis. Overcoming this tradeoff is important for lowering the costs of running LLMs on low-resource languages, already lagging behind their high-resource counterparts.

Acknowledgments

This research was supported in part by the Israel Science Foundation (grant No. 1166/23). The work of the third author has been funded by the Israel Science Foundation (grant No. 2617/22) and by the European Union (ERC, MiDRASH, Project No. 101071829; Principal investigators: Nachum Dershowitz, Tel-Aviv University; Judith Olszowy-Schlangen, EPHE-PSL; Avi Shmidman, Bar-Ilan University, and Daniel Stoekl Ben Ezra, EPHE-PSL), for which we are grateful. Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.

We thank the reviewers for their comments. We thank Craig Schmidt for comments on earlier versions. Ibraheem Abo Shakra helped with implementation and evaluation of the Arabic models.

References

Dan Bareket and Reut Tsarfaty. 2021. Neural modeling for named entities and morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9:909–928.

Lisa Beinborn and Yuval Pinter. 2023. Analyzing cognitive plausibility of subword tokenization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4478–4486, Singapore. Association for Computational Linguistics.

Marco Cognetta, Vilém Zouhar, Sangwhan Moon, and Naoaki Okazaki. 2024. Two counterexamples to tokenization and the noiseless channel. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16897–16906, Torino, Italia. ELRA and ICCL.

Amir Cohen, Hilla Merhav-Fine, Yoav Goldberg, and Reut Tsarfaty. 2023. HeQ: a large and diverse Hebrew reading comprehension benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13693–13705, Singapore. Association for Computational Linguistics.

Elazar Gershuni and Yuval Pinter. 2022. Restoring Hebrew diacritics without a dictionary. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1010–1018, Seattle, United States. Association for Computational Linguistics.

Omri Keren, Tal Avinari, Reut Tsarfaty, and Omer Levy. 2022. Breaking character: Are subwords good enough for mrls after all? arXiv preprint arXiv:2204.04748.

Bilal Khaliq and John Carroll. 2013. Induction of root and pattern lexicon for unsupervised morphological analysis of Arabic. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1012–1016, Nagoya, Japan. Asian Federation of Natural Language Processing.

Stav Klein and Reut Tsarfaty. 2020. Getting the #life out of living: How adequate are word-pieces for modelling complex morphology? In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 204–209, Online. Association for Computational Linguistics.

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics.

Danit Yshaayahu Levi and Reut Tsarfaty. 2024. A truly joint neural architecture for segmentation and parsing. arXiv preprint arXiv:2402.02564.

Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018. The Hebrew Universal Dependency treebank: Past present and future. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 133–143, Brussels, Belgium. Association for Computational Linguistics.

Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. 2024. Tokenization is more than compression. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 678–702, Miami, Florida, USA. Association for Computational Linguistics.

Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5149–5152. IEEE.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.

Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. arXiv:2304.11077.

Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art bert suite for modern hebrew.

Shaltiel Shmidman, Avi Shmidman, Moshe Koppel, and Reut Tsarfaty. 2024. MRL parsing without tears: The case of Hebrew. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4537–4550, Bangkok, Thailand. Association for Computational Linguistics.

Roni Stein, Ram Frost, and Noam Siegelman. 2024. Help: The hebrew lexicon project. Behavior Research Methods, 56(8):8761–8783.

Omri Uzan, Craig W. Schmidt, Chris Tanner, and Yuval Pinter. 2024. Greed is all you need: An evaluation of tokenizer inference methods. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 813–822, Bangkok, Thailand. Association for Computational Linguistics.

Shaked Yehezkel and Yuval Pinter. 2023. Incorporating context into subword vocabularies. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 623–635, Dubrovnik, Croatia. Association for Computational Linguistics.

Amir Zeldes, Nick Howell, Noam Ordan, and Yifat Ben Moshe. 2022. A second wave of UD Hebrew treebanking and cross-domain parsing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4331–4344, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Vilém Zouhar, Clara Meister, Juan Gastaldi, Li Du, Minmay Sachan, and Ryan Cotterell. 2023. Tokenization and the noiseless channel. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5184–5207, Toronto, Canada. Association for Computational Linguistics.

A Alphabet Encoding Implementation Details

To support the expanded set of characters introduced by SPLINTER, which includes the original alphabet along with symbols representing single-letter reductions, we needed an alphabet capable of handling a large number of unique symbols. In Hebrew, this expanded alphabet comprised 252 characters, while for Arabic and Malay, it grew to 400 and 460 characters, respectively. Since the Unicode character sets for these languages do not offer enough distinct symbols, we mapped each new character to a unique Chinese character, leveraging the large character set available in the Chinese writing system. This was done as a workaround, as attempts to use the Unicode Private Use Areas (PUA) with the SentencePiece library were unsuccessful. This approach allowed the tokenization process to remain seamless from the language model’s perspective, as it processed the input as Chinese text, effectively encoding the original text.

B SPLINTER Algorithm

Pseudocode for the SPLINTER map creation algorithm is presented in Algorithm 1.

C SPLINTER UnigramLM results

Intrinsic benchmark results for Hebrew using UnigramLM tokenizer on HeDC4 corpus are presented in Table 6.

D SPLINTER pre_modern results

Intrinsic benchmark results for Hebrew using BPE tokenizer on pre_modern corpus (available at https://github.com/elazarg/hebrew_diacritized/tree/master/pre_modern), are presented in Table 7.
Algorithm 1 High-level algorithm for training SPLINTER.

function TRAINSPLINTER(corpus)
    freqMap ← GetWordFrequenciesFromCorpus(corpus)
    reductions ← InitializeEmptyReductionsMap()
    for length ← 4 to maxWordLength do
        for word in freqMap[length] do
            for position in word do
                permutation ← GetWordWithoutLetter(word,position)
                if permutation ∈ freqMap[length − 1].keys then
                    reduction ← Reduction(position,word[position])
                    frequency ← freqMap[length − 1][permutation]
                    reductions[length][reduction]+= frequency
                end if
            end for
        end for
    end for
    sortedReductions ← sortReductionsByScoreDesc(reductions)
    selectedReductions ← InitializeEmptyReductionsMap()
    for length ← 4 to maxWordLength do
        for word in freqMap[length] do
            for reduction in sortedReductions[length] do
                Extract (position, letter) from reduction
                if word[position] == letter then
                    permutation ← GetWordWithoutLetter(word,position)
                    if permutation ∈ freqMap[length−1].keys then
                        frequency ← freqMap[length − 1][permutation]
                        selectedReductions[length][reduction]+= frequency
                        Break
                    end if
                end if
            end for
        end for
    end for
    Return selectedReductions
end function
<table>
  <tr>
    <th>Vocab size</th>
    <th>Type</th>
    <th>Cognitive plausibility</th>
    <th>Rényi efficiency</th>
    <th>Tokens per word</th>
    <th>4+ token words</th>
    <th>1-char tokens</th>
    <th>Distinct Neighbors</th>
  </tr>
  <tr>
    <td rowspan="2">128K</td>
    <td>Vanilla</td>
    <td>0.151</td>
    <td><b>0.505</b></td>
    <td><b>1.162</b></td>
    <td><b>1.00%</b></td>
    <td><b>9.42%</b></td>
    <td>2440</td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>0.171</b></td>
    <td>0.485</td>
    <td>1.176</td>
    <td>0.56%</td>
    <td>12.46%</td>
    <td><b>2308</b></td>
  </tr>
  <tr>
    <td rowspan="2">64K</td>
    <td>Vanilla</td>
    <td>0.180</td>
    <td><b>0.522</b></td>
    <td><b>1.243</b></td>
    <td><b>0.88%</b></td>
    <td><b>11.50%</b></td>
    <td>3907</td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>0.194</b></td>
    <td>0.495</td>
    <td>1.261</td>
    <td>1.65%</td>
    <td>16.21%</td>
    <td><b>3640</b></td>
  </tr>
  <tr>
    <td rowspan="2">32K</td>
    <td>Vanilla</td>
    <td>0.191</td>
    <td><b>0.526</b></td>
    <td><b>1.363</b></td>
    <td><b>1.46%</b></td>
    <td><b>14.41%</b></td>
    <td>5322</td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>0.208</b></td>
    <td>0.496</td>
    <td>1.391</td>
    <td>2.74%</td>
    <td>21.48%</td>
    <td><b>4931</b></td>
  </tr>
  <tr>
    <td rowspan="2">10K</td>
    <td>Vanilla</td>
    <td>0.177</td>
    <td><b>0.536</b></td>
    <td><b>1.663</b></td>
    <td><b>3.62%</b></td>
    <td><b>21.26%</b></td>
    <td>5267</td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>0.213</b></td>
    <td>0.517</td>
    <td>1.713</td>
    <td>6.53%</td>
    <td>31.63%</td>
    <td><b>5064</b></td>
  </tr>
  <tr>
    <td rowspan="2">2K</td>
    <td>Vanilla</td>
    <td>0.136</td>
    <td>0.590</td>
    <td><b>2.250</b></td>
    <td><b>11.65%</b></td>
    <td><b>33.40%</b></td>
    <td>1824</td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>0.196</b></td>
    <td><b>0.618</b></td>
    <td>2.424</td>
    <td>20.70%</td>
    <td>51.04%</td>
    <td><b>1776</b></td>
  </tr>
  <tr>
    <td rowspan="2">1K</td>
    <td>Vanilla</td>
    <td>0.127</td>
    <td>0.618</td>
    <td><b>2.604</b></td>
    <td><b>21.22%</b></td>
    <td><b>43.92%</b></td>
    <td>917</td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>0.185</b></td>
    <td><b>0.659</b></td>
    <td>2.877</td>
    <td>32.48%</td>
    <td>63.21%</td>
    <td><b>881</b></td>
  </tr>
  <tr>
    <td rowspan="2">800</td>
    <td>Vanilla</td>
    <td>0.126</td>
    <td>0.629</td>
    <td><b>2.730</b></td>
    <td><b>25.33%</b></td>
    <td><b>47.98%</b></td>
    <td>726</td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>0.177</b></td>
    <td><b>0.673</b></td>
    <td>3.060</td>
    <td>37.71%</td>
    <td>68.34%</td>
    <td><b>693</b></td>
  </tr>
</table>

Table 6: Intrinsic benchmark results for Hebrew using UnigramLM tokenizer with different vocabulary sizes. The tokenizers were evaluated using the HeDC4 corpus. **Bold** values indicate better performance between Vanilla and SPLINTER.

<table>
  <tr>
    <th>Vocab size</th>
    <th>Type</th>
    <th>Rényi efficiency</th>
    <th>Tokens per word</th>
  </tr>
  <tr>
    <td rowspan="2">128K</td>
    <td>Vanilla</td>
    <td>0.583</td>
    <td><b>1.315</b></td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>0.586</b></td>
    <td>1.349</td>
  </tr>
  <tr>
    <td rowspan="2">64K</td>
    <td>Vanilla</td>
    <td>0.608</td>
    <td><b>1.406</b></td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td><b>0.609</b></td>
    <td>1.448</td>
  </tr>
  <tr>
    <td rowspan="2">32K</td>
    <td>Vanilla</td>
    <td><b>0.636</b></td>
    <td><b>1.514</b></td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td>0.629</td>
    <td>1.575</td>
  </tr>
  <tr>
    <td rowspan="2">10K</td>
    <td>Vanilla</td>
    <td><b>0.677</b></td>
    <td><b>1.744</b></td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td>0.662</td>
    <td>1.815</td>
  </tr>
  <tr>
    <td rowspan="2">2K</td>
    <td>Vanilla</td>
    <td><b>0.730</b></td>
    <td><b>2.124</b></td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td>0.711</td>
    <td>2.266</td>
  </tr>
  <tr>
    <td rowspan="2">1K</td>
    <td>Vanilla</td>
    <td><b>0.751</b></td>
    <td><b>2.347</b></td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td>0.725</td>
    <td>2.608</td>
  </tr>
  <tr>
    <td rowspan="2">800</td>
    <td>Vanilla</td>
    <td><b>0.756</b></td>
    <td><b>2.427</b></td>
  </tr>
  <tr>
    <td><b>SPLINTER</b></td>
    <td>0.726</td>
    <td>2.757</td>
  </tr>
</table>

Table 7: Rényi efficiency and tokens per word results for Hebrew using BPE tokenizer with different vocabulary sizes. The tokenizers were evaluated using the pre_modern corpus. **Bold** values indicate better performance between Vanilla and SPLINTER.

Reviewer paper summaries: The paper proposes a method for dealing with templatic morphology before subword tokenisation Hebrew, Arabic and Malay. The methods should allow the usual tokenisers like BPE to split words into roots vs. templates. Compared to a vanilla BPE tokenisation, the proposed method scores better on cognitive plausibility, but it increases the number of tokens per word. When it comes to the evaluation on downstream tasks, the results are available only for Hebrew and they show an improvement on QA, syntactic and morphological parsing.
Summary of strengths: Given the particularity of templatic morphology in the context of subword tokenisation, developing specific approaches is a valuable goal. The proposed solution is potentially useful and it might improve model performance on languages with templatic morphology.
Summary of weaknesses: 1. The main weakness of this paper is its unclear description of the main pre-processing algorithm, starting with the following sentence: "The core idea of the algorithm is that in certain NCLs, many words are formed by embedding root letters into specific morphological templates." (088-091) It is not clear how the words are examined (134) and how the reductions are classified into valid and not valid (138). Figure 1, which illustrates the method, does not point to any reductions, while these seem to be crucial.  

2. Another important problem is the evaluation that is potentially biased: cognitive plausibility and downstream task performance are reported only for Hebrew and not for the other two languages without providing any explanations. Given the inconsistency of the other measures, we cannot draw any conclusion about the usefulness of the method beyond Hebrew. 

3. There is no meaningful comparison with the related work. The paper doesn't even have a separate section on related work and the proposed algorithm is compared to a baseline without mentioning any other, potentially alternative methods for dealing with templates before tokenisation. 

4. The substance would be more appropriate for a short paper.
Comments, suggestions and typos: I would strongly encourage revising this paper to improve:
- clarity
- generalisation
- comparison to the related work.
Overall assessment scores: 2