# An Open Relation Extraction System for Web Text Information
## 3 Methodology
### Encoder
#### 3.3.2 K-BERT Encoder
After tuple injection, a sentence tree with a rich knowledge background is generated. Figure 6: An illustration of K-BERT structure and the processing of a sentence sample. Black numbers represent hard position indices. Red numbers represent soft position indices. Stage 2: The embedding layer and seeing layer. BERT cannot handle graph-structured sentences. The role of the embedding layer is to solve this problem. The role of the seeing layer is to control the reference domain of external knowledge and avoid knowledge noise. * Process 1: BERT can only process sentence input with sequence structure. The structural information will be lost if the sentence tree is tiled into a sequence. The visual matrix can convert the sentence tree into a sequence and save the structural information, and its principle is shown in Figure 7. The embedding representation includes three parts: token embedding, position embedding, and segment embedding. In the token embedding section, K-BERT flattens the tokens in the sentence tree into embedding sequences according to the hard position index. \"[CLS](0) [PERSON](1) is_born_in(2) German(3) founded(4) the_theory_of relativity(5) is_proposed_by(6) [PERSON](7)\". Obviously, tiling causes losses in sentence structure and readability. For example, the positional index of \"is born in(2)\" and \"founded(2)\" are both (2), and they all follow \"[PERSON](1)\". This information can be recovered by soft-position coding. As shown in Figure 7, \"[CLS](0) [PERSON](1) is born in(2) German(3) founded(2) the(3) theory of relativity(4) a(5) non-inertial-system(6) theory (7)\". In the soft-position embedding section, K-BERT restores the graph structure and readability of the sentence tree through soft-position encoding. K-BERT uses soft-position embedding to map the readable information of sentence trees. Nevertheless, only using soft positions is not enough because it will make the model misunderstand that found(2) follows German(3), which will also lead to knowledge noise. In order to solve this problem, we can convert the soft-position coding into a visible matrix through the process 2 seeing layer. The role of this last section segment embedding is to identify sentence 1 and sentence 2. For the sentence pair, it is marked with a sequence of segment tags, \(S_{1},S_{1},\ldots,S_{1},S_{2},S_{2},\ldots,S_{2}\). * Process 2: The seeing layer's role is to limit the field of view of the knowledge referenced by the entities. It can avoid false references to knowledge and the noise caused by mutual interference between knowledge. The visible matrix \(M\) is defined as: \[M_{ij}=\left\{ \begin{aligned} 0,& W_{i}\odot W_{j}\\ -\infty,& W_{i}\odot W_{j}\end{aligned} \right.\] (21) where \(i\) and \(j\) are the hard-position index, \(W_{i}\) and \(W_{j}\) are co-branched, meaning that they are the same entity, and its distance is 0. On the contrary, its distance is negative infinity. The seeing layer's core idea is to ensure that the original sentence entities and the injected background knowledge do not interfere. As shown in Figure 7, the sentence tree has nine tokens, which are mapped into a 9 * 9 visible matrix. White means that the two tokens in the corresponding position are visible to each other, and black means that they are invisible to each other. The black cell in column 3, row 9, means German(3) cannot see \"theory(9)\". Similarly, [German] is invisible to [CLS], but through [[PERSON]] it indirectly acts on [CLS], thereby reducing knowledge noise. Stage 3: The mask-transformer encoder processing. Traditional transformer encoders cannot accept a visible matrix as input. The role of the mask-transformer encoder is to introduce the structural information in the sentence tree into the model. It is a stack of multiple mask self-attention blocks.