## 2 Related Work on Diacritics Restoration
### Classical Approaches
#### 2.1.2 Statistics-Based Approaches  
In addition to the rule-based approaches, another group that effectively solves the diacritics restoration problems is based on corpus statistics. These methods, in turn, can be further divided into a character-level and a word-level. The word-level approaches are considered to be a more accurate solution, but they typically rely on expensive resources (i.e., monolingual texts to train language models, dictionaries, etc.) that do not cover the non-standard language forms. All of this makes them more language-dependent and, at the same time, less suitable for less-resourced languages. On the other hand, character-level approaches are able to more effectively cope with out-of-vocabulary words and, therefore, can be used to diacritize non-normative language texts (such as posts on social networks, forums, internet comments, etc.) in which the omitted diacritics problem is especially apparent. The majority of word-level statistical approaches are based on pre-trained probabilistic \(n\)-gram language models [14]. The \(n\)-gram language models are trained on large monolingual corpora and give a probability of encountering a particular sequence of \(n\) words in a text. The robustness of \(n\)-gram models directly depend on the size and variety of the training data. The higher the order \(n\) of the \(n\)-gram model is, the lower perplexity it has, and the better it is at language modeling. Yet, high orders of \(n\) require a vast amount of data for training and, as a side effect, inflicts sparseness, which leads to zero conditional probabilities. The models are usually based on the closed-world assumption, where words not found in the language model do not exist. Therefore, smoothing mechanisms become especially important in coping with unseen words (typically assigning non-zero probabilities). Larger \(n\)s are more cumbersome to store and compute, and are typically less beneficial for languages with a free word order in a sentence; rare combinations make language models very sparse, less robust, and they, therefore, require pruning. Since longer sequences are less probable, word-level diacritization approaches often allow for back-off or interpolation procedures. The authors of [15] successfully applied their language modeling method to the lowercased Slovak texts. The method compares the surrounding context of the target (undiacritized) word with the related \(n\)-grams (with \(n=4\)). In this way, the method considers three preceding and three following words around the target one. If the 4-gram is not found, the process continues by backing off to trigrams, bigrams, and, if necessary, to unigrams. The whole diacritization process is iterative and sequential: after the diacritized equivalent for some targeting word is determined, the new target is set. A similar method is applied to the Igbo language [16]. The authors tested the bigram and trigram language models with the back-off strategy and various smoothing techniques, experimentally proving the trigram language model with the Add-1 smoothing to be the most accurate for their diacritization problems. However, the back-off strategy does not always appear to be the best. An experimentally investigated token bigram language model achieved the highest accuracy on the Spanish texts [17]. It outperformed not only the unigram model, but a bigram language model with the back-off strategy. The diacritics restoration problem for Spanish is also tackled in [18] and three different methods are investigated. Their first method relies on the Bayesian framework. The idea behind it is that words closer to the target would give more clues about its correct disambiguation and diacritization. The basis of the second method is the Hidden Markov Model (HMM) method, which is able to solve ambiguity problems by indicating different parts of speech.