## 3 Proposed Method
### Sentence Encoder
#### 3.2.2 Encoding Layer  
The piecewise convolutional neural network (PCNN) [6] has been utilized by a number of previous RE works and its powerful capability of capturing local features and positional information of sentences has been proven. However, PCNN usually performs badly when processing long sentences due to its limitation of not being able to obtain the global dependencies of sentences. Therefore, a multi-head self-attention mechanism is employed in our framework to obtain long-distance dependency information from multiple perspectives.  
As shown in Figure 3, given an input representation S, we firstly apply a convolution kernel with a sliding window over S, where the window's size is set to k. In order to keep  
Figure 2: An example of position embedding.  
the original size of the input, we add \(\frac{k-1}{2}\) padding tokens on both sides of the sentence boundaries. The hidden output of the convolution layer can be expressed as:  
\[\mathbf{H^{C}}=\{\mathbf{h^{c}_{1}},\mathbf{h^{c}_{2}},\dots,\mathbf{h^{c}_{m}}\},\mathbf{h^{c}_{i}}=\mathbf{W_{c}}\mathbf{x_{i_{-}\frac{k-1}{2}+\frac{k-1}{2}} +\mathbf{b_{c}}} \tag{1}\]  
where \(\mathbf{W_{c}}\in\mathbb{R}^{d^{c}\times(k\times d)}\), \(\mathbf{b_{c}}\in\mathbb{R}^{d^{c}}\), and \(\mathbf{H^{C}}\in\mathbb{R}^{m\times d^{c}}\).  
Next, to exploit the global dependency of a sentence, we adopt the multi-head self-attention mechanism of Transformer [21], which has successfully achieved promising results in most NLP tasks. The basic idea of a self-attention mechanism is to find the interaction between each word and the whole sentence. The importance or the weight of each word in the sentence can be calculated by the degree of its interaction and is utilized to adjust the representation of the sentence. In this way, the global information of the whole sentence will be contained in the new representation.  
Formally, a self-attention mechanism is defined as follows:  
\[Attention(\mathbf{a},\mathbf{a},\mathbf{a})=softmax(\frac{\mathbf{a}^{ \mathrm{T}}}{\sqrt{\mathrm{d}^{d}}})\mathbf{a} \tag{2}\]  
where the \(\mathit{Softmax}\) (\(\cdot\)) function is used to calculate the weights that are applied to the words, T is a transpose operation, \(\mathbf{a}^{\mathrm{T}}\) is the dot product and \(\mathrm{d}^{a}\) is the dimension of \(\mathbf{a}\), and the purpose of scaling through \(\sqrt{\mathrm{d}^{a}}\) is to avoid the dot product result being too large.  
Further, we adopt a multi-head operation on the self-attention mechanism to extract the features about the different levels of a sentence. We get the sentence representation Ha, as represented in the following equation:  
\[head_{j}=Attention(\mathbf{W^{Q}_{j}}\mathbf{H^{c}},\mathbf{W^{K}_{j}} \mathbf{H^{c}},\mathbf{W^{V}_{j}}\mathbf{H^{c}})(j=1,2,\dots,A) \tag{3}\]  
\[\mathbf{H^{a}}=\mathbf{W^{d}}[\mathbf{head_{1}};\mathbf{head_{2}};\dots; \mathbf{head_{A}}] \tag{4}\]  
where \(\mathbf{W^{Q}_{j}},\mathbf{W^{K}_{j}},\mathbf{W^{V}_{j}}\in\mathbb{R}^{d^{k} \times d^{c}}\) and \(\mathbf{W^{n}}\in\mathbb{R}^{d^{k}\times d^{c}}\) are learnable parameters; \(d^{h}\) is the dimension of the head; and \([\cdot]\) is a concatenate operation. Notice that A is the number of heads and \(d^{c}=d^{h}\times A\). The parameters of different heads are independent of each other.  
Then, piecewise max-pooling is applied over \(\mathbf{H^{a}}=\{\mathbf{H^{a}_{1}},\mathbf{H^{a}_{2}},\dots,\mathbf{H^{a}_{m }}\}\) to capture the sentence-level structural information, which is formulated as  
\[\mathbf{s}=relu([Pool(\mathbf{H^{a}_{1:h}});Pool(\mathbf{H^{a}_{h+1:t}});Pool (\mathbf{H^{a}_{t+1:m}})]) \tag{5}\]  
Figure 3: Infrastructure of our sentence encoder. Relu: Rectified Linear Unit.  
where \(\mathbf{H}^{a}\) is segmented according to the position of (h, t), Pool(\(\cdot\)) is a max-pooling operation and relu(\(\cdot\)) is a non-linear activation function. As a result, \(\mathbf{s}=\mathbb{R}^{m\times(3\times d^{2})}\) concatenated by three pooling results is the final sentence vector representation.