## 3 Dual Pointer Network Model for Relation Extraction
### Context and Entity Encoder  
The context and entity encoder computes the degree of association between words and entities in a given sentence. For example, \(\{w_{1},w_{2},\ldots,w_{i}\}\) and \(\{e_{1},e_{2},\ldots,e_{m}\}\) refer to word and entity embedding vectors, respectively. Figure 3 illustrates the process of word and entity embedding.  
As shown in Figure 3, the word embedding vectors are concatenations of two types of embeddings: word-level GloVe [12] embeddings for representing the meaning of words and character-level CNN embeddings [13] for addressing out-of-vocabulary problems. The entity embedding vectors are concatenations of three types of embeddings: word-level CNN embedding for representing the meaning of entities composed of multiple words, character-level CNN embedding for addressing out-of-vocabulary problems, and entity type embedding for representing the categorical information of input entities. Word-level GloVe embeddings represent each word in the word-level CNN embedding.  
Figure 3: (a) Word embedding process, (b) Entity embedding process.  
Figure 2: The overall architecture of dual pointer networks for relation extraction.  
The word embedding vectors are used as input for a bidirectional LSTM network to obtain contextual information as follows:  
\[\begin{array}{l}\overset{\rightarrow}{c}_{i}=\text{LSTM}(w_{i},\overset{ \rightarrow}{c}_{i-1}),\\ \overset{\leftarrow}{c}_{i}=\text{LSTM}(w_{i},\overset{\rightarrow}{c}_{i-1}), \\ c_{i}=[\overset{\rightarrow}{c}_{i};\overset{\rightarrow}{c}_{i}],\end{array} \tag{1}\]  
where \(w_{i}\) is an embedding vector of the \(i\)-th word in a sentence, and \([\overset{\rightarrow}{c}_{i};\overset{\leftarrow}{c}_{i}]\) is a concatenation of \(\overset{\rightarrow}{c}_{i}\) and \(\overset{\leftarrow}{c}_{i}\) that represents the output vectors of a forward LSTM and a backward LSTM, respectively. The entity embedding vectors are used as input for a forward LSTM network because the entities are listed in the order that they appear in a sentence, as shown below.  
\[s_{t}=\text{LSTM}(e_{t},s_{t-1}), \tag{2}\]  
where \(e_{t}\) is an embedding vector of the \(t\)-th one among all entities occurring in a sentence, and \(s_{t}\) is an output vector encoded by a forward LSTM. The output vectors of the bidirectional LSTM network \(\{c_{1},c_{2},\dots,c_{i}\}\) and the forward LSTM network \(\{s_{1},s_{2},\dots,s_{t}\}\) are used as input for the context-to-entity attention layer (as shown in Figure 2), to compute the relative degrees of association between words and entities. This is similar to the well-known multi-head attention mechanism [14], as shown below.  
\[\begin{array}{l}q_{j}=w^{d}*splitsplitsplit(q,n)_{j},\\ a_{j}=softmax(\frac{q_{j}k_{j}}{q}),\\ head_{j}=a_{j}v_{j},\\ o_{t}=\text{relu}(w^{\rho}[head_{0};\ head_{1};\ head_{2};\dots;head_{n}] +b^{\rho}),\end{array} \tag{3}\]  
where the query \(q\) is set to \(s_{t}\), the key \(k\) and the value \(v\) are set to C's. The query \(q\) is split into \(n\) vectors, where \(n\) is the number of heads. The attention score \(a_{j}\) is calculated by a scaled-dot product, where \(d\) is a normalization factor. The context-to-entity layer output \(o_{t}\) is determined through a fully-connected neural network (FNN) using a concatenation of \(n\) heads as input.