# Contemporary Approaches in Evolving Language Models
## 2 Statistical Approaches to Language Modeling
### Hidden Markov Model (HMM)  
The Hidden Markov Model (HMM) was one of the prevalent models used before the adoption of artificial neural networks and is applied in various NLP tasks, including language modeling (Figure 2). An HMM simulates the operation of a Markov process with hidden internal states. The Markov process itself represents a sequence of random events in which the probability of each event depends solely on the current state of the process and is independent of prior states ([PERSON] et al., 2013). In language modeling, an HMM is utilized to model word sequences while accounting for fundamental linguistic structures. An HMM consists of two primary elements: a hidden Markov chain and an observable sequence. The former comprises a series of hidden states with an unknown structure for observation. In many instances, these states can correspond to parts of speech. The latter element, the observable sequence, represents a series of characters or words observed during the modeling process. The HMM is characterized by the following parameters:  
* \(S=\{s_{1},s_{2},...,s_{n}\}\)--the set of hidden states;
* \(O=\{o_{1},o_{2},...,o_{m}\}\)--the set of observable symbols or words;
* \(A=\{a_{ij}\}\)--the transition probability matrix, where \(a_{ij}\) is the probability of transitioning from state \(s_{i}\) to state \(s_{j}\);
* \(B=\{b_{j}(k)\}\)--the observation probability matrix, where \(b_{j}(k)\)--is the probability of observing the symbol \(o_{k}\) in state \(s_{j}\);
* \(\pi=\{\pi_{i}\}\)--the initial probability vector, where \(\pi_{i}\)--is the probability of starting in state \(s_{i}\).  
In Figure 2, \(X\) represents the hidden state which depends on \(Y\). \(Y\) in turn is the observable element, that is, a word, whereas \(a\) signifies the transition probability from one state to another and \(b\) stands for the emission probabilities.  
In the domain of speech recognition, HMMs are typically employed in the construction of an acoustic model. They model the temporal sequences of phonemes and determine the most probable sequence of a model's states. This, in effect, accommodates both the transition probabilities between states and the observation probabilities. Moreover, HMMs  
Figure 2: Example of a Hidden Markov Model.  
are used in tasks like speaker identification and analysis and speech classification, among others [15].  
HMMs have certain limitations to consider in the context of language modeling:  
* Insufficient realism and model simplification: HMMs make independence assumptions that might be inconsistent in real-world conditions. In actual scenarios, observations might depend on hidden states with a more intricate structure [16; 17].
* The emergence of advanced models: To address this limitation, more sophisticated models have been introduced, such as the Conditional Random Fields (CRFs) and Deep Hidden Markov Models (DHMMs). CRFs cater to dependencies between observations and hidden states by considering contextual information. Meanwhile, DHMMs melt the principles of deep learning and HMMs, allowing for the modeling of intricate dependencies and providing a more precise description of linguistic structures.  
While HMMs were prevalent in the past, recent advancements in deep learning, such as recurrent neural networks (RNNs) and transformer-based models like BERT and GPT, have largely supplanted HMMs in many NLP tasks. These contemporary models better capture long-term dependencies and have achieved state-of-the-art results across a broad spectrum of NLP applications. Nonetheless, understanding HMMs and their principles can provide valuable insights into the fundamentals of sequence modeling and probabilistic modeling in language processing.