# An Open Relation Extraction System for Web Text Information
## 3 Methodology
### Encoder
#### 3.3.2 K-BERT Encoder  
BERT does well in common named-entity representation, but not in the professional field of named entities. It is related to the pre-train mechanism that BERT captures entity representation on a large-scale common corpus but lacks domain-specific knowledge. The underlying reason is that domain knowledge is rarely present in the pre-training stage. Although expertise can be absorbed during the fine-tuning stage, the actual effect is not good. If it is directly trained on the corpus in the professional field, it will need to invest a significant cost and have low transferability. Therefore, [PERSON] proposes directly injecting domain knowledge annotation that would only appear in the fine-tuning stage into the pre-training stage. The knowledge graph is injected into the instance as external domain knowledge. This will not only improve performance but also effectively reduce costs. There are two reasons for choosing a knowledge graph [32] as an external knowledge source. The first is that a knowledge graph is marked and verified structured knowledge, and thus is rich, reliable, and easy to query. The other is that the construction of the knowledge graph is controllable, and the model has good interpretability. Knowledge injection needs to solve two problems. The first one is physical alignment. Generally speaking, the entity's word embedding vector space in text and external knowledge are different. Therefore, it is necessary to align the entities so that they have the same word vector representation. The second is the problem of knowledge noise. Improperly injecting external knowledge can adversely affect the original semantics. Correct and reasonable injection methods can effectively reduce or even avoid noise generation. [PERSON] proposed K-BERT [33], which is compatible with any pre-trained BERT model and does well in embedding space. For this paper, the authors designed sample selector 2 based on K-BERT. As shown in Figure 6, the K-BERT working process can be divided into three stages:  
Stage 1: The knowledge layer. This stage injects domain knowledge into the original sentence sequence and outputs an abundant graph-structure sentence tree. It can be further subdivided into two processes: knowledge injection and sentence tree transformation. The external professional knowledge base KG equips the model with knowledge triple-tuple and then injects domain knowledge. The knowledge layer transforms primary sentences into abundant sentence trees. First, perform a knowledge query (\(K\_Query\)). Take the entities in the original sentence as the query, and traverse the KG. \(K\_Query\) is a knowledge query function, \(S\) represents a query sentence, and \(K\) represents an external knowledge graph. External knowledge \(E\) can be expressed as:  
\[E=K\_Query(S,K) \tag{16}\]Its specific form is:  
\[E=\{(w_{i},r_{i0},w_{i0}),\ldots,(w_{i},r_{ik},w_{ik})\} \tag{17}\]  
where is a collection of the corresponding triples. The specific form of the sentence \(S\) is:  
\[S=\{w_{0},w_{1},w_{2},\ldots,w_{n}\} \tag{18}\]  
The specific form of the sentence tree \(St\) is:  
\[St=\{w_{0},w_{1},\ldots,w_{i}\{(r_{i0},w_{i0}),\ldots(r_{ik},w_{ik})\},\ldots,w _{n}\} \tag{19}\]  
Injecting external knowledge according to the knowledge graph can be formulated as:  
\[S_{i}=K\_Inject(S,K\_Query(S,K)) \tag{20}\]  
where \(K\_Inject\) is a function that injects knowledge into the sentence tree and keeps structure index information. The queried knowledge is placed directly after the entity, and the retrieval process is based on the visible matrix and soft position. In this paper, there are many branches in a sentence tree. Furthermore, its depth is limited to 1 for subsequent processing. As shown in Figure 6, the input sentence \(\chi\) is: \"[PERSON] created the theory of relativity\". \"[PERSON]\" is associated with the tuple information \"is born in Germany\" in the knowledge graph. The relativity theory is related to \"a non-inertial-system theory\".