# Capsule Network Improved Multi-Head Attention for Word Sense Disambiguation
## 3 Methodology
### Model Details
In this way, the length of the output vector \(v\) is a number between \(0\) and \(1\), so the length can be interpreted as the probability that \(v\) has a specific feature. \[v=\frac{||s||^{2}}{1+||s||^{2}}\frac{s}{||s||}. \tag{4}\]  
Taking advantage of the ability to recognize overlapping features of the capsule network, we incorporate the capsule routing into the multi-head attention to measure importance of information contained by various heads. Based on Equation (2), we see \(e\) computed by Equation (5) as low layer capsules input to high layer capsules in the capsule network. \[e=\left[ \begin{array}{c}e_{1}\\ \ldots\\ e_{H}\end{array} \right];h\in[1,H], \tag{5}\]  
\[e_{h}=\left[ \begin{array}{cc}e_{1h}&\ldots&e_{Lh}\end{array} \right]. \tag{6}\]  
To attain deeper contextualized information, we consider the attention weight from two aspects: the sentence itself and the multiple heads. We call them sequence routing (SR) and head routing (HR), respectively. In the sequence routing, the capsules are as much as the heads and each capsule has L vectors, where L is the length of the sequence. The process of sequence routing is shown in Figure 4. We view \(e\) as the \(H\times L\) input vectors. The input vectors generate output vectors \(u_{1l},u_{2l},\ldots,u_{HI}\) and weights \(b_{1l},b_{2l},\ldots,b_{HI},l\in[1,L]\) (indicated by the circle groups in Figure 4.) after the dynamic routing. The concatent in figures means the concatenate operation. Considering \(H\) heads have different effects on the output, softmax function is applied to the weights of each head for the sequence:  
\[u=softmax(\
abla)u_{L}, \tag{7}\]  
\[\
abla=\left[\sum_{l=1}^{L}b_{1l},\ldots,\sum_{l=1}^{L}b_{HI}\right], \tag{8}\]  
\[u_{L}=\left[ \begin{array}{c}u_{l1}\\ \ldots\\ u_{HI}\end{array} \right];l\in[1,L]. \tag{9}\]The head routing is shown in Figure 5. There are \(L\) capsules, and each capsule generates \(H\) vectors in the head routing. We view \(e\) as the \(L\times H\) input vectors. Taking measures to capture positional information among input tokens in the head routing is necessary because of its order dependence. Here, each capsule has a partial routing so that the sequential information is involved into the output capsules. Specifically speaking, the number of the head capsules is equal to the length of the sequence. For the \(l^{th}\) capsule, its routing output is \(v_{l}\) after the dynamic routing algorithm (Algorithm 1), which is computed by routing the top \(l\) capsules. Then, we have the last head routing result \(v\) computed as follows:  
\[v=[v_{1},v_{2},\dots,v_{L}]. \tag{10}\]  
At last, similar to a residual connection, we add \(e\) to the sum of \(u\) and \(v\). The sum of the three vectors is followed by a softmax function, which is multiplied by value \(V_{H}\) to get the final output representations as follows:  
\[r=softmax(e+u+v)V_{H}, \tag{11}\]  
\[V_{H}=[V_{1},V_{2},\dots,V_{h}],h\in[1,H]. \tag{12}\]  
Figure 4: The process of sequence routing. Figure 5: The process of head routing. In the context module, the encoder generates \(r_{w}=\left[r_{w_{1}},r_{w_{2}},\ldots,r_{w_{m}}\right]\) for input sequence, where \(l\) is the length of context sequence. Because target words may be segmented into word pieces, we assume that target word \(w_{l}\) corresponds to word list \(w_{m},\ldots,w_{m+k-1}\). We average representation to present the contextualized target word:  
\[r_{w_{l}}=\frac{1}{k}\sum_{i=0}^{k-1}r_{w_{i}}. \tag{13}\]  
For each sense of the target word, the sense glosses module generates \(r_{s}=\left[r_{g_{1}},r_{g_{2}},\ldots,r_{g_{p}}\right],\) where \(p\) is the length of the current sense gloss. To identify the correct meaning, we score each sense simply by the dot product of word and its sense:  
\[score(w_{t},s_{i})=r_{w_{t}},r_{s_{i}};i\in[0,n], \tag{14}\]  
where \(n\) denotes the sense inventory number of target word listed in WordNet. We choose the one who has the highest score as the most suitable sense of the polyseme.