# Language Model Using Neural Turing Machine Based on Localized Content-Based Addressing
## 5 Experiments and Discussion
### Experimental Results in Character-Level Penn Treebank LM Task
We measured the performance of the MDM-NTM architecture according to the number of read vectors. The number of read vectors was doubled to evaluate their effect. The MDM-NTM-based LM using a single read vector demonstrated a higher performance with a BPC of 1.5986 than the MDM-NTM-based LMs using two and four read vectors. Each BPC sample of experiment results on the PTB LM task is shown in Table 1. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**nWP**} & \multirow{2}{*}{**nRV**} & \multirow{2}{*}{**nVEM**} & \multirow{2}{*}{**WD**} & \multirow{2}{*}{**IT**} & \multicolumn{3}{c}{**BPC**} \\ \cline{5-8}  & & & & & & & **Validation** & & **Test** \\ \cline{5-8}  & & & & & & & \(\mu\) & \(\sigma\) & \(\mu\) & \(\sigma\) \\ \hline LSTM & 13.2 M & - & - & \(1\times 10^{-6}\) & 141 & 1.8949 & 0.0033 & 1.8254 & 0.0031 \\ Transformer & 13.2 M & - & - & \(1\times 10^{-6}\) & 22 & 1.6042 & 0.0006 & 1.5954 & 0.0004 \\ Trellis Network & 13.4 M & - & - & \(8\times 10^{-7}\) & 4915 & 1.3858 & 0.0013 & 1.3578 & 0.0019 \\ AWD-LSTM & 13.8 M & - & - & \(1.2\times 10^{-6}\) & 61 & 1.5224 & 0.0039 & 1.4720 & 0.0036 \\ MDM-NTM & 18.3 M & 1 & 65 & \(1\times 10^{-7}\) & 1348 & 1.7335 & 0.0011 & 1.6977 & 0.0010 \\  & 18.3 M & 1 & 129 & \(1\times 10^{-7}\) & 1481 & 1.6362 & 0.0008 & 1.6019 & 0.0009 \\  & 18.3 M & 1 & 257 & \(1\times 10^{-7}\) & 1710 & 1.6592 & 0.0011 & 1.6288 & 0.0009 \\  & 18.3 M & 1 & 513 & \(1\times 10^{-7}\) & 2147 & 1.6688 & 0.0006 & 1.6316 & 0.0007 \\  & 18.3 M & 1 & 769 & \(1\times 10^{-7}\) & 2625 & 1.6529 & 0.0006 & 1.6214 & 0.0008 \\  & 18.3 M & 1 & 1024 & \(1\times 10^{-5}\) & 2872 & 1.7354 & 0.0004 & 1.6998 & 0.0007 \\  & 18.3 M & 1 & 1024 & \(1\times 10^{-7}\) & 2710 & 1.6330 & 0.0006 & 1.5986 & 0.0007 \\  & 23.6 M & 2 & 1024 & \(1\times 10^{-5}\) & 2518 & 1.7716 & 0.0003 & 1.7341 & 0.0005 \\  & 23.6 M & 2 & 1024 & \(1\times 10^{-7}\) & 2590 & 1.7210 & 0.0009 & 1.6906 & 0.0002 \\  & 34.2 M & 4 & 1024 & \(1\times 10^{-7}\) & 3250 & 1.7428 & 0.0005 & 1.7044 & 0.0009 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Evaluation results of MDM-NTM-based LMs on the PTB LM task (nWP, number of weight parameters; nRV, number of read vectors; nVEM, number of vectors in the external memory; WD, weight decay; IT, inference time (ms/batch); \(\mu\), mean of BPC results; \(\sigma\), standard deviation of BPC results). The analysis for the performance of the MDM-NTM-based LM according to the number of read vectors led to three important findings: (1) the number of weight parameters increased with the number of read vectors. To generate the read vectors, the controller outputs the key vectors. The number of key vectors was the same as the number of read vectors. The key vectors were \(M\)-dimensional vectors, where \(M\) is a dimension of the vector in the external memory. Furthermore, the number of read vectors affected the dimension of the input layer in the controller, because the read vectors generated at time \((t-1)\) were used as the input to the controller. (2) BPC decreased depending on the number of read vectors. When the MDM-NTM-based LM used one to four read vectors, the performance decreased. This implies that the PTB LM task was insufficient for training all weight parameters of the MDM-NTM-based LM using two or more read vectors. (3) The inference time was disproportional to the number of read vectors. We assumed that a larger number of read vectors required a longer inference time. However, the MDM-NTM-based LM using two read vectors showed a faster inference time than that using one read vector. We evaluated the performance of the MDM-NTM architecture according to the weight decay. The weight decay reduced the model overfitting by imposing increasingly large penalties as the weight parameters increased [36]. To implement the weight decay, a set of assumed weight parameters such as \(W\), \(\frac{1}{2}\lambda WW^{T}\) was added to the loss function and a penalty is imposed.