# An Entity-Relation Extraction Method Based on the Mixture-of-Experts Model and Dependency Parsing
## 3 Methodology
### Encoding Layer  
3.1.1. BERT Encoding  
The proposed model employs BERT as the base encoder. The input to BERT consists of three components: word embeddings, sentence embeddings, and position embeddings. Word embeddings are used to represent each word or subword in a sentence, while sentence embeddings distinguish between different sentences, and position embeddings capture the relative positions of words within a sentence. For a sentence \(S=\{t_{1},t_{2},\ldots,t_{n}\}\), composed of n characters, it is processed by the pre-trained BERT model to obtain the corresponding representations for each character.  
\[\{b_{1},b_{2},\ldots,b_{n}\}=BERT(\{t_{1},t_{2},\ldots,t_{n}\})\]  
In this context, \(b_{i}\) denotes the semantic representation of character \(t_{i}\), produced by BERT.  
3.1.2. Syntax-Information Encoding  
Since the output of BERT is based on character-level granularity, it cannot fully capture the syntactic structure of entities and their relationships. To address this, we introduce an additional syntax-information encoding module, which enhances the model's ability to capture the syntactic relationships between entities by effectively modeling word-level syntactic features. Initially, the input text is tokenized into words, yielding a set of word-level representations, \(V=\{v_{1},v_{2},\ldots,v_{n}\}\), where n represents the total number of words. These words are then mapped onto a higher-dimensional vector space via an embedding layer to obtain word-level embeddings:  
\[\{e_{1},e_{2},\ldots,e_{n}\}=Embedding(\{v_{1},v_{2},\ldots,v_{n}\})\]  
where \(e_{i}\) represents the initial embedding vector of word \(v_{i}\). To capture word-level contextual information, BiLSTM is used to learn the contextual relationships at the word level. BiLSTM effectively models the sequential order between words and bidirectional dependencies, outputting the word's contextual representations:  
\[\{m_{1},m_{2},\ldots,m_{n}\}=BiLSTM(\{e_{1},e_{2},\ldots,e_{n}\})\]  
where \(m_{i}\) represents the word \(v_{i}\)'s contextual semantic feature, incorporating both its preceding and succeeding contextual information. For the given input dependency parsing, we construct connections between the source and target nodes for each word pair with a dependency relationship, resulting in a dependency syntax graph. We then model the word-to-word relationships in the dependency syntax graph using GNN and use GAT (Graph Attention Network) to learn the propagation and updating of dependency relationships in the sentence layer by layer. Each word is represented as a node in the dependency syntax graph, with edges representing the syntactic relationships between words. GNN captures syntactic information based on the graph structure to obtain a deep representation of each word:  
\[\{g_{1},g_{2},\ldots,g_{n}\}=GNN(\{m_{1},m_{2},\ldots,m_{n}\})\]  
where \(g_{i}\) denotes the syntax-enhanced feature after the GNN processes the dependency syntax graph. During the process of updating the information between nodes in the graph, each node aggregates information from its neighboring nodes while integrating its features for updates. This enables the model to capture long-range dependencies between words and effectively incorporate this information into the representation of each word. Specifically, when GNN propagates information between nodes, it not only considers the contextual semantics of the words but also incorporates structural information derived from dependency syntax, thus more accurately capturing the structured dependency relationships between words.