# Language Model Using Neural Turing Machine Based on Localized Content-Based Addressing
## 5 Experiments and Discussion
### Experimental Results in enviks LM Task  
We compared the results achieved through the proposed LCA-NTM architecture with those of the Transformer [10]. For the baseline LM, we also used the previous experimental result of the LSTM RNN-based LM [37]. For training the MDM-NTM-based LM, we used the LSTM RNN as the controller. The number of dimensions of the hidden layers was 1024, and the number of hidden layers was 4. We used an external memory consisting of 128 vectors. The number of dimensions of each vector was 256. In addition, the batch size was 20. We could not train or evaluate the large-scale MDM-NTM-based \begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline  & & & & & & & & \multicolumn{3}{c}{**BPC**} \\ \cline{4-10} \multirow{-2}{*}{**Model**} & \multirow{-2}{*}{**nWP**} & \multirow{-2}{*}{**nRV**} & \multirow{-2}{*}{**nVEM**} & \multirow{-2}{*}{**nSVEM**} & \multirow{-2}{*}{**nSVEM**} & \multirow{-2}{*}{**WD**} & \multirow{-2}{*}{**IT**} & \multicolumn{2}{c}{**Validation**} & \multicolumn{2}{c}{**Test**} \\ \cline{5-10}  & & & & & & & & \(\mu\) & \(\sigma\) & \(\mu\) & \(\sigma\) \\ \hline LSTM & 13.2 M & - & - & - & - & \(1\times 10^{-6}\) & 141 & 1.8937 & 0.0032 & 1.8254 & 0.0029 \\ Transformer & 13.2 M & - & - & - & \(1\times 10^{-6}\) & 22 & 1.6042 & 0.0006 & 1.5954 & 0.0004 \\ Trellis & 13.4 M & - & - & - & \(8\times 10^{-7}\) & 4915 & 1.3858 & 0.0013 & 1.3578 & 0.0019 \\ AWD-LSTM & 13.8 M & - & - & - & \(1.2\times 10^{-6}\) & 61 & 1.5224 & 0.0039 & 1.4720 & 0.0036 \\ MDM-NTM & 18.3 M & 1 & 1024 & - & \(1\times 10^{-7}\) & 2710 & 1.6330 & 0.0006 & 1.5986 & 0.0007 \\ LCA-NTM & 18.3 M & 1 & 1024 & 65 & \(1\times 10^{-7}\) & 158 & 1.6580 & 0.0022 & 1.6249 & 0.0024 \\  & 18.3 M & 1 & 1024 & 129 & \(1\times 10^{-7}\) & 154 & 1.6432 & 0.0029 & 1.6103 & 0.0027 \\  & 18.3 M & 1 & 1024 & 257 & \(1\times 10^{-7}\) & 159 & 1.6172 & 0.0021 & 1.5648 & 0.0018 \\  & 18.3 M & 1 & 1024 & 513 & \(1\times 10^{-7}\) & 155 & 1.6230 & 0.0024 & 1.6071 & 0.0022 \\  & 18.3 M & 1 & 1024 & 769 & \(1\times 10^{-7}\) & 157 & 1.6419 & 0.0015 & 1.6297 & 0.0014 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation results of LCA-NTM-based LM according to number of selected memory addresses on the PTB LM task (nWP, number of weight parameters; nRV, number of read vectors; nVEM, number of vectors in the external memory; nSVEM, number of selected vectors in the external memory; WD, weight decay; IT, inference time (ms/batch); \(\mu\), mean of BPC results; \(\sigma\), standard deviation of BPC results). LM because the Nvidia GTX 1080 Ti GPU has a 11-GB memory capacity, and the batch size would have been considerably small if we trained a large-scale MDM-NTM-based LM. We evaluated the performance of the MDM-NTM architecture according to the number of read vectors. Table 3 shows the evaluation results. When we used four read vectors, the MDM-NTM-based LM demonstrated the highest BPC performance of 1.3922. The inference time increased when the number of read vectors was increased. Each BPC sample of experiment results on the PTB LM Task is shown in Table 1.  
The analysis of the performance of the MDM-NTM-based LM according to the number of read vectors led to two important findings: (1) BPC decreased depending on the number of read vectors. When the MDM-NTM-based LM used from one to four read vectors, the performance improved. (2) The inference time was proportional to the number of read vectors. We assumed that a larger number of read vectors required a longer inference time. The MDM-NTM-based LM using one read vector showed a faster inference time than that using two or more read vectors. Furthermore, we applied the proposed LCA mechanism to the pre-trained MDM-NTM architecture during the test stage. The pre-trained MDM-NTM architecture showed that the highest performance in Table 3 was used. In Table 4, when the selected memory address was 97, the performance of the LCA-NTM-based LM showed a BPC of 1.3887. The performance was higher than that of the MDM-NTM-based LM.