# KGGCN: Knowledge-Guided Graph Convolutional Networks for Distantly Supervised Relation Extraction
## 3 Methodology
### Sentence Representation
#### 3.1.2 Knowledge Attention over Words  
Each sentence \(S\) in the sentence bag consists of a sequence of words, i.e., \(S=\{w_{1},w_{2},...,w_{m}\}\), where \(m\) is the length of the sentence. We first project the discrete words to low-dimensional word vectors by looking up the pre-trained word embeddings. Thus, these words can be processed and modeled by the knowledge attention layer and graph convolutional layer. The same word embedding matrix used in the LUs embedding procedure is employed to embedding the words in sentences. The word embedding of the \(i\)-th word in sentence \(S\) is denoted by \(\mathbf{e}_{i}^{w}\in\mathbb{R}^{1\times d_{w}}\), where \(d_{w}\) is the size of the word embeddings. The sentence can be expressed as the concatenation of the word embeddings, \(\mathbf{S}=\{\mathbf{e}_{1}^{w},\mathbf{e}_{2}^{w},...,\mathbf{e}_{m}^{w}\}\in\mathbb{R}^{m \times d_{w}}\).  
Not all words in a sentence are equally important for relation extraction. In order to distinguish the importance of each word in a sentence, we adopt the recently-promoted self-attention mechanism [35; 36] to measure the contribution (importance) of each word to the expression of a relevant relation. It helps in highlighting important relation words with respect to each of the relation indicators present in the LUs set. The generation of the word-level knowledge attention is illustrated in Figure 2.  
Figure 1: Semantic frame of the relation _founder_. The relation indicators are extracted from the Lexical units.  
Formally, the query(\(\mathbf{Q}\)) is the word embeddings of a sentence, and the key(\(\mathbf{K}\))-value(\(\mathbf{V}\)) pairs are both the relation indicator embeddings, i.e., \(\mathbf{Q}=\mathbf{S}\in\mathbb{R}^{m\times d_{w}}\mathbf{K}=\mathbf{V}=\mathbf{U}\in\mathbb{R}^{n \times d_{w}}\). Thus, the hidden representation of the input sentence can be obtained from  
\[\mathbf{H}=Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=softmax(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d _{w}}})\mathbf{V}. \tag{1}\]  
\(d_{w}\) is a scaling factor, which is the dimension of word embeddings. Specifically, for each word \(e_{i}^{w}\) of the input sentence, the attention probability \(\mathbf{p}^{i}\) is expressed as  
\[\mathbf{p}^{i}=softmax(\frac{e_{i}^{w}\mathbf{K}^{T}}{\sqrt{d_{w}}}). \tag{2}\]  
Then, the hidden representation of each word can be calculated as a weighted sum of the values  
\[\mathbf{h}_{i}=\sum_{j=1}^{n}\mathbf{p}_{j}^{i}\odot\mathbf{V}_{j}, \tag{3}\]  
where \(\mathbf{h}_{i}\in\mathbb{R}^{1\times d_{w}}\) is the hidden representation of the \(i\)-th word in the sentence, \(\odot\) means the element-wise multiplication, and \(\sum\) performs along sequential dimension.  
Eventually, the result of the knowledge attention can be calculated as  
\[\alpha_{i}=\frac{\exp(\mu_{i})}{\sum_{j=1}^{m}\exp(\mu_{j})},\mu_{i}=\mathbf{h}_{ i}\mathbf{W}_{1}\mathbf{r}, \tag{4}\]  
where \(\mathbf{W}_{1}\in\mathbb{R}^{d_{w}\times d_{w}}\) is a square matrix, and \(\mathbf{r}\in\mathbb{R}^{d_{w}\times 1}\) is a random query vector. \(\alpha_{i}\) is the knowledge attention score of the \(i\)-th word in the sentence, which is calculated by attending to the relation indicators. This attention score represents the importance of the word for relation extraction.