# Language Model Using Neural Turing Machine Based on Localized Content-Based Addressing
## 4 Neural Turing Machine Using Localized Content-Based Addressing  
Previous content-based addressing, as described in Section 3.1, is necessary for accessing all memory addresses for the calculation of the cosine similarities and using these similarities for generating the attention vector. Consequently, the attention vector can include weights to the memory address, which are not related to the input vector. The unnecessary memory area is reflected in the external memory, which results in performance deterioration.  
Hence, we introduce the NTM architecture using LCA-NTM. As shown in Figure 2, cosine similarities are calculated using a key vector \(k_{t}\) and each external memory address. Subsequently, the LCA selects a memory address that shows the maximum cosine similarity. This process is the main difference with vanilla content-based addressing, because vanilla content-based addressing does not select the maximum cosine similarity. If all the cosine similarities are negative, vanilla content-based addressing is used. After searching for the maximum value of cosine similarity, softmax normalization is performed on \(d\) memory addresses on both sides of the selected memory address to generate the content-based addressing vector \(CA_{t}\). \(CA_{t}[i]\), where softmax is not performed, is \(0\). Algorithm 1 describes the proposed LCA procedure in detail.  
The proposed NTM architecture attends to a specific memory area selected by LCA. The result of the LCA is the content-based addressing vector. Vanilla location-based addressing is applied to the content-based addressing vector generated from LCA. The final weighting vector is generated by location-based addressing. The proposed LCA method is applied to test the stage on pre-trained NTM-based LM.  
For the \(i\)-th read vector at time \(t\), we define,  
\[r_{t}^{ LCA,i}=EM_{t}^{\top}\omega_{t}^{ LCA,i} \tag{8}\]where \(\omega_{t}^{LCA_{r}i}\) is the \(i\)-th \(N\)-dimensional read weighting vector at time \(t\). The proposed NTM architecture stores the converted input vector to the selected memory addresses. Selected memory addresses are decided by the write weighting vector. We define the write operation in the proposed LCA-NTM architecture as,  
\[EM_{t}=EM_{t-1}\circ(OM-\omega_{t}^{LCA_{w}}e_{t}^{\top})+\omega_{t}^{LCA_{w}}v _{t}^{\top} \tag{9}\]  
where \(\omega_{t}^{LCA_{w}}\) is an \(N\)-dimensional write weighting vector at time \(t\). LCA searches for the maximum cosine similarity value. To search for this value, the cosine similarity values are not sorted because all values have to be considered in the deterministic algorithm for obtaining the maximum value. It takes \(\mathcal{O}(N)\), where \(N\) is the number of vectors in the external memory. The time complexity of softmax normalization used for LCA is \(\mathcal{O}(2d+1)\) because the denominator for softmax normalization is dependent on the selected memory area and comprises \((2d+1)\) memory addresses. Therefore, the time complexity of the LCA is \(\mathcal{O}(N+2d+1)\).  
```
Data:\(EM_{t}\), \(k_{t}\), \(ks_{t}\) Result:\(CA_{t}\)
1\(CA_{t}\leftarrow\vec{0}\);
2\(CS_{t}\leftarrow\vec{0}\);
3 for\(i=0\) to \((N-1)\)do
4\(CS_{t}[i]\leftarrow\frac{EM_{t}[i_{c}]\cdot k_{t}}{\|EM_{t}[i_{c}]\|\|k_{t}\|}\);
5
6 end for
7 if\(CS_{t}<\vec{0}\)then
8 for\(i=0\) to \((N-1)\)do
9\(CA_{t}[i]\leftarrow\frac{exp(CS_{t}[i])^{ks_{t}}}{\sum_{j=0}^{N-1}exp(CS_{t}[i ])^{ks_{t}}}\);
10
11 end for
12 else
13\(max\gets CS_{t}[0]\);
14\(idx\gets 0\);
15 for\(i=1\) to \((N-1)\)do
16 if\(max<CS_{t}[i]\)then
17\(max\gets CS_{t}[i]\);
18\(idx\gets i\)
19 end if
20
21 end for
22 for\(i=0\) to \((idx-d-1)\)do
23\(CA_{t}[i]\gets 0\)
24 end for
25 for\(i=(idx-d)\) to \((idx+d)\)do
26\(CA_{t}[i]\leftarrow\frac{exp(CS_{t}[i])^{ks_{t}}}{\sum_{j=idx-d}^{idx+d}exp(CS_{t }[i])^{ks_{t}}}\);
27
28 end for
29 for\(i=(idx+d+1)\) to \((N-1)\)do
30\(CA_{t}[i]\gets 0\)



50 end for
```  
**Algorithm 1**Proposed LCA procedure