# Language Model Using Neural Turing Machine Based on Localized Content-Based Addressing
## 1 Introduction  
A language model (LM) estimates the probability of the current word based on the previous word sequence. For the word sequence \(W=(w_{1},w_{2},...,w_{N})\), the probability of the LM is denoted as \(P(W)\). \[P(W)=P(w_{1},w_{2},...,w_{N}) \tag{1}\]  
where \(N\) is the length of \(W\). When the number of words in the word history \((w_{1},w_{2},...,w_{i-1})\) increases, it becomes increasingly difficult to calculate the probability for the current word \(w_{i}\), because the word history will not appear in the text corpus. For this reason, the Markov assumption is applied to the LM to compute \(P(w_{i}|w_{1},...,w_{i-1})\). Here, the length of the word sequence which affects \(w_{i}\) is \((n-1)\). For modeling the LM, the conventional method is an \(n\)-gram but it has the following two problems: an unseen word sequence problem and a limitation in the length of the word history. To address these problems, an LM using a deep neural network (DNN) was proposed to model word sequences [1]. It could predict the probability of the unseen word sequence with a high-dimensional hyperplane of the DNN. However, the DNN-based LM cannot solve the limitation in the length of the word history. To solve the aforementioned limitation, a recurrent neural network (RNN) was used to model the longer word sequence [2]. In the RNN-based LM, a recurrent hidden layer performs the role ofmemory through recurrence. The recurrent hidden layer can learn from the first word to the \((n-1)\)-th word. Therefore, the RNN-based LM solves the aforementioned issue. However, the output of the recurrent hidden layer trained in the previous time step is gradually vanished and is replaced by the current input when the time step is increased. This is referred to as the vanish gradient problem. For the vanish gradient problem of the RNN-based LM, a long short-term memory (LSTM) [3] was used in the RNN-based LM [4]. The LSTM consists of an input gate, output gate, forget gate, and one or more memory cells [3]. The LSTM RNN-based LM stores the word sequence in the memory cells of the hidden nodes and deletes previous information of the word history depending on the context. However, LSTM language models can effectively use only 200 words or character sequences on average [5]. Therefore, although the LSTM RNN can potentially encode longer contextual information, the LSTN RNN cannot solve the limitation that exists in the length of the word history. Recently, attention mechanisms have been used in sequence-to-sequence models such as LMs. An advantage of using the attention mechanism is that it supports the interpretation of deep learning models by showing how a deep learning model attends to different blocks of the input sequence [6]. Typical attention-based deep learning models are a Transformer [7] and a neural Turing machine (NTM) architecture [8]. The Transformer replaces LSTM RNN models with multi-head self-attention and positional encoding. Multi-head self-attention computes a sequence of vector-space representations related to different positions in a sequence. Positional encoding makes the Transformer understand long-range dependency with relative word or character positions. The Transformer has been used in LMs, and has been reported to significantly outperform LSTM RNN-based LMs [9]. However, the Transformer can only be treated with a fixed-length input sequence [10]. The input sequence has to be divided into a number of sub-sequences before being input to the Transformer. Transformer-XL [10], GPT-2 [11], and bidirectional encoder representations from Transformers (BERT) [12] are proposed to solve the aforementioned problem of the Transformer. Another attention-based deep learning model is the NTM architecture. The NTM architecture comprises a controller and an external memory [8]. The controller is based on previous deep learning model topologies. The external memory comprises \(M\)-dimensional vectors. The controller writes its output to the external memory and reads information from the external memory.