# Contemporary Approaches in Evolving Language Models
## 2 Statistical Approaches to Language Modeling
### N-Gram  
The n-gram model is one of the statistical models used for language modeling, employed for predicting the next word in a sequence or for assessing the probability of a sentence. The core principle revolves around predicting a sequence of \(n\) words or characters. In an n-gram model, the probability of producing the next word is determined based on transition probabilities and observations. The general formula for calculating \(P(w)\) depending on the number of words in a sentence is presented as follows (1):  
\[P(w_{1},\ \ldots,\ w_{n})=P(w_{n}|w_{n-1},\ \ldots,\ w_{1})\cdot P(w_{n-1},\ \ldots,\ w_{1}), \tag{1}\]  
where \(w\) represents individual words, and \(n\) is the number of words. There are various types of n-gram models (Figure 1), such as those listed below. * Unigrams, which evaluate each word independently;
* Bigrams, which consider the probability of a word given the previous word;
* Trigrams, which consider the probability of a word given the two preceding words, and so on. Unigrams play a fundamental role in n-gram models, assessing the probability of each word occurring independently within a specific context. Known for their simplicity and efficiency, unigram models serve as a foundational reference point for more intricate language models. Despite assuming independence, unigrams make substantial contributions to language comprehension and are essential in diverse natural language processing tasks, such as information retrieval, text classification, and sentiment analysis. Integrating unigrams into an n-gram model enhances its adaptability and suitability for various linguistic applications. * Data sparsity problem: This issue arises with low-frequency words or the absence of word groups in training data. To alleviate this, a Laplace smoothing method or Neyman-Morgenstern smoothing was proposed in [5]. In the realm of language modeling, n-gram models inherently fail to account for the long-term contextual dependencies present within word sequences. * Model size problem: This leads to an exponential increase in possible combinations, demanding high memory and computational resources. * Ignoring full sentence context: N-gram models consider only the previous context, overlooking the interrelation and broader context between words. Recurrent neural networks and their variants, as well as transformer-based models, have been proposed [6,7]. * Incorrect modeling of incomplete sentences. : N-grams work well with complete word sequences, but errors occur with incomplete phrases or sentences, a common scenario in real life. Methods of stream data processing or incremental model updating were applied to mitigate this drawback [8,9]. Some Modifications and Improvements  
[PERSON] et al. [10] investigated the effective memory depth of RNN models by employing them for n-gram modeling. Experiments on a small UPenn Treebank corpus indicated that the LSTM cell with dropout is a superior model for encoding n-gram state data compared to a feedforward RNN. When maintaining the sentence independence assumption, the LSTM n-gram achieves performance comparable to the LSTM LM for \(n=9\) and slightly surpasses it when \(n=13\). The authors of [11] introduced the n-distant-max language model, which significantly mitigated the shortcomings of n-gram models with a smaller parameter for evaluation. However, a limitation of the n-distant-max model was identified: it only considers the word with the highest probability, even though there are instances in which a word is associated with more than one word. It was demonstrated in [12] that  
Figure 1: Types of n-gram models. by incorporating multi-granularity information about unseen and domain-specific words through word-based n-gram adaptation, the performance of the overall pre-trained model saw significant modeling improvements. This includes the Transformer-based Domain-aware N-gram Adaptor, which effectively learns and integrates semantic representations of various word combinations into a new domain. [PERSON] et al.