# GREG: A Global Level Relation Extraction with Knowledge Graph Embedding
## 3 Model
### Context-Aware Relation Extraction  
The purpose of the first module is aimed to extract and classify local relations within a document. The overall module can be seen in Figure 3. In addition, to perform local relation extractions without suffering from a bottleneck issue [8], the input document is to be analyzed sentence by sentence. Once the document input is preprocessed into smaller sentences, the sentences are then given to a named entity recognition module to identify any existing entities within the sentence. In this research, we used two separate methods to extract named entities from the text for this experiment. The first module is the Stanford CoreNLP toolkit [18] developed by [PERSON] et al. and the second named entity recognition module is a combined Bidirectional Long Short Term Memory and Convolutional Neural Network with a Conditional Random Field (Bi-LSTM-CNN-CRF) Korean named entity recognition module [17] developed by [PERSON] et al. The reason behind using a Korean Named Entity recognition module is because the document-knowledge graph dataset used for the synchronization process of the two modules are made in Korean. Thus, to exclusively evaluate the model's viability on global relation extraction, the module has been trained to process both languages separately.  
Given sentence \(x\) and its token as \(x=\{x_{1},x_{2},x_{3},...,x_{n}\}\), each token within the sentence is to be embedded into a vector of \(k\) dimensions by using matrix \(W\in R^{|V|\times k}\). Here, \(|V|\) Indicates the size of the vocabulary from the word embedding. For this experiment, we used the 50-dimensional GloVe embeddings pre-trained on a 6 billion corpus by [PERSON] et al. [19] for our English module and a 200-dimensional Korean word embedding constructed from the Korean document dataset of history for our Korean module [6]. Additionally, the LSTM context-aware module has a layer size of 256 and the entity marker embeddings as \(d=3\). Once all the entities are recognized, every two entities from the same sentence are paired as candidate entities for a possible local relation. As the purpose of this module is to extract every inch of local relation information, every possible combination of two entities within a sentence is to be given input to the local relation extraction module.  
For every pair of candidate entities given as inputs, along with the sentence it belongs to, the other entities within the sentence are temporarily considered as tokens. All tokens are then marked as to whether they belong to one of the candidate entities, or are independent based on their grammatical distances. These distances are measured through a part of speech tagging method [20]. A marker embedding matrix of \(P\in R^{3\times d}\) is randomly initialized of the three marker types: entity 1, entity 2, and none for independent tokens. Each token's marker embedding will then be concatenated to the word embedding with \(d\) as the dimension for the position embedding: \((W_{n},P_{n})\).  
By applying the sequence of \(n\) token embeddings to a recurrent neural network, the LSTM encoder converts it to a fixed-sized output vector \(\mathbf{o}_{\mathbf{v}}\). \(\mathbf{o}_{\mathbf{v}}\) acts as a representation of the relation between the target entities. The vector \(\mathbf{o}_{\mathbf{v}}\) is then passed to a softmax layer which will calculate which relation class it most likely belongs to. This can be shown as in Equation (1) with \(P\left(r\right)\) as the classifier and the entities marked as \(e_{1}\) and \(e_{2}\), along with the softmax layer \(f_{i}\) defined in Equation (2):  
Figure 3: Long Short Term Memory(LSTM) based Mention-level ‘Local’ Relation Extraction model.  
\[P(r|\left<e_{1},e_{2}\right>,x;\theta)=\frac{\exp(f_{r})}{\sum_{i=1}^{n_{r}}\exp(f _{i})} \tag{1}\]  
\[f_{i}=w_{i}\cdot o_{v}\oplus b_{i} \tag{2}\]