## 5 Modeling Units  
Speech processing is usually performed at the level of utterances, with words as tokens. In written form, an utterance is called a segment. For this reason, we will refer to segments rather than utterances in the continuation of the article.  
Using words as modeling units in a segment is not always the best choice for highly inflected languages with many different word forms, as it produces large vocabularies and, in the case of small datasets, a high OOV (out-of-vocabulary) rate. Conversely, using words, the modeling sequences are shorter, which is an advantage for approaches that have difficulties modeling long-range dependencies. Both perspectives are important in terms of memory and time complexity. Words as modeling units seem to be a good choice with very large corpora.  
The character could be an alternative token as it also generalizes well across languages that share the alphabet. It also solves the problem of word tokenization. Its OOV rate is zero. However, sequences of characters in a segment are very long, which harms the performance of many approaches commonly used for normalization. The solution to this problem could be using shorter sequences instead of segments. \(N\)-grams were used in our research. An \(N\)-gram is a sequence of \(N\) neighboring words.  
Any substring of a word could also be a potential alternative unit, which lies between a word and a character. Using substrings presents a balance between generalization and specificity. The idea of data-driven word-splitting approaches is to keep the frequently used words as single tokens and to split rare words into smaller units called subwords. Subwords could be characters or sequences of two or more characters, depending on the vocabulary size and splitting method.  
There are several word-splitting algorithms or implementations, e.g., BPE [29], unigram-based [30], and Morfessor [31]. In previous research [32], we compared different word-splitting methods in another sequence-to-sequence task and found that splitting based on unigram probabilities [30] gives comparable or slightly better results than other methods.  
Using unigrams, the probability of a sequence of \(M\) subword units \(\mathbf{x}=(x_{1},x_{2}\ldots x_{M})\) is modeled as the product of the individual units' probabilities in this sequence:  
\[P(\mathbf{x})=\prod_{i=1}^{M}p(x_{i}),\]  
where \(p(x_{i})\) is the probability of the \(i\)-th unit. The optimal word splitting \(x^{*}\) is then defined as:  
\[x^{*}=\operatorname*{arg\,max}_{x\in S(X)}P(\mathbf{x}),\]  
where \(S(X)\) is the set of all possible splitting for the training text \(X\). It can be determined using the Viterbi algorithm. An implementation of this algorithm is available in the tool SentencePiece [33]. This tool creates a vocabulary of subword units to cover all text in the training data. The user chooses the vocabulary size.  
Table 2 presents different splittings of example segments obtained in our experiments. The numbers in parentheses indicate the subword vocabulary size.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Units** & **Segment** \\ \hline Words & skupni imenovalec zapisanega knijżnega jezika \\ Translation to English & the common denominator of the written literary language \\ Subwords (5000) & \_skup ni \_imenoval ec \_zapisan ega \_knijżn ega \_jezika \\ Subwords (1000) & \_skup ni \_ime no va le c \_za pis a nega \_k ni ji z nega \_jezika a \\ Subwords (500) & \_skup ni \_i me no va le c \_za pis a nega \_k n ji z nega \_jezika a \\ Characters & \_skupni \_i me no va le c \_z apisan e ega\_knjiż nega \\  & \_je z i k a \\ \hline Words & dolgo smo ziveli v utvari \\ Translation to English & we lived in an illusion for a long time \\ Subwords (5000) & \_dolgo \_smo \_zivel i \_v\_u tva ri \\ Subwords (1000) & \_dolgo \_smo \_ziv e li \_v\_u tva ri \\ Subwords (500) & \_dolgo \_smo \_ziv e li \_v\_u tva ri \\ Characters & \_dolgo\_smo \_ziv e li \_v\_utvari \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison between different splittings.